2024-09-13T08:25:50.634 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T08:25:50.657 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T08:25:50.750 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322 branch: wip-athakkar-testing-2024-09-12-1849 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} email: null first_in_suite: false job_id: '7903322' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-athakkar-testing-2024-09-12-1849 ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: client mount timeout: 30 debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-athakkar-testing-2024-09-12-1849 sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e owner: scheduled_khiremat@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6477 sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e sleep_before_teardown: 0 subset: 3/4 suite: fs:volumes suite_branch: wip-athakkar-testing-2024-09-12-1849 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d454760ef8a93a9dd7f4e4dedfb085faac216e7e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcbKOtiJPLoQxqG6RdpIVoSZReEDe5VYIb/hmIkntx3ONKKtHb1dPLEuF9fCyl9Qje4FvG1DQ9BVtu+2fdOAoQ= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS+xAWN3T0vHWYn3BI/lHTunEw9YoQf0AKEdGArLvZOJ4u0GAjwvd/wbp+/kOMNlnlnZvTaUy8ZT6+0O6+zwAw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_08:07:46 tube: smithi user: khiremat verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T08:25:50.750 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d454760ef8a93a9dd7f4e4dedfb085faac216e7e/qa; will attempt to use it 2024-09-13T08:25:50.751 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d454760ef8a93a9dd7f4e4dedfb085faac216e7e/qa/tasks 2024-09-13T08:25:50.751 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T08:25:50.753 INFO:teuthology.task.internal:Checking packages... 2024-09-13T08:25:50.775 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e' 2024-09-13T08:25:50.775 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T08:25:50.775 INFO:teuthology.packaging:ref: None 2024-09-13T08:25:50.775 INFO:teuthology.packaging:tag: None 2024-09-13T08:25:50.775 INFO:teuthology.packaging:branch: wip-athakkar-testing-2024-09-12-1849 2024-09-13T08:25:50.775 INFO:teuthology.packaging:sha1: d454760ef8a93a9dd7f4e4dedfb085faac216e7e 2024-09-13T08:25:50.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-athakkar-testing-2024-09-12-1849 2024-09-13T08:25:50.939 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4881.gd454760e 2024-09-13T08:25:50.941 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T08:25:50.947 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T08:25:50.947 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T08:25:50.953 INFO:teuthology.task.internal:Saving configuration 2024-09-13T08:25:50.969 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T08:25:50.975 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T08:25:51.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 08:21:04.234428', 'locked_by': 'scheduled_khiremat@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcbKOtiJPLoQxqG6RdpIVoSZReEDe5VYIb/hmIkntx3ONKKtHb1dPLEuF9fCyl9Qje4FvG1DQ9BVtu+2fdOAoQ='} 2024-09-13T08:25:51.038 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 08:21:04.236226', 'locked_by': 'scheduled_khiremat@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS+xAWN3T0vHWYn3BI/lHTunEw9YoQf0AKEdGArLvZOJ4u0GAjwvd/wbp+/kOMNlnlnZvTaUy8ZT6+0O6+zwAw='} 2024-09-13T08:25:51.038 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T08:25:51.045 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-13T08:25:51.045 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-13T08:25:51.045 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T08:25:51.119 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc2321ffac0>, signals=[15]) 2024-09-13T08:25:51.119 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T08:25:51.127 INFO:teuthology.task.internal:Opening connections... 2024-09-13T08:25:51.127 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:25:51.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:25:51.212 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:25:51.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:25:51.294 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T08:25:51.300 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-09-13T08:25:51.318 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-09-13T08:25:51.319 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-09-13T08:25:51.376 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-09-13T08:25:51.376 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-09-13T08:25:51.376 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T08:25:51.377 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T08:25:51.378 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-09-13T08:25:51.407 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-09-13T08:25:51.425 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-09-13T08:25:51.425 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-09-13T08:25:51.482 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T08:25:51.483 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T08:25:51.483 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-09-13T08:25:51.515 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T08:25:51.524 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T08:25:51.529 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T08:25:51.529 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-09-13T08:25:51.532 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-09-13T08:25:51.548 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T08:25:51.554 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T08:25:51.554 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-09-13T08:25:51.589 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-09-13T08:25:51.615 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T08:25:51.688 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T08:25:51.703 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-13T08:25:51.703 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-09-13T08:25:51.704 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-09-13T08:25:51.704 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T08:25:51.704 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T08:25:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:25:51.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-13T08:25:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:25:51.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-13T08:25:51.894 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:25:51.894 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-13T08:25:51.895 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-13T08:25:51.913 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-503.el9.x86_64 2024-09-13T08:25:51.913 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:25:51.913 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-13T08:25:51.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum upgrade -y linux-firmware 2024-09-13T08:25:51.916 DEBUG:teuthology.task.kernel:sha1 for client.1 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:25:51.916 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-13T08:25:51.916 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-13T08:25:51.934 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-503.el9.x86_64 2024-09-13T08:25:51.934 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:25:51.934 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-13T08:25:51.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum upgrade -y linux-firmware 2024-09-13T08:25:54.333 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-13T08:25:54.567 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-13T08:25:56.934 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-13T08:26:01.109 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 20 MB 00:05 2024-09-13T08:26:02.776 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-13T08:26:04.477 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-13T08:26:05.348 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-13T08:26:06.717 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T08:26:08.160 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 19 kB 00:00 2024-09-13T08:26:09.088 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-13T08:26:10.945 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-13T08:26:13.086 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-13T08:26:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-13T08:26:13.087 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-13T08:26:13.209 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-13T08:26:13.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:26:13.360 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-13T08:26:13.360 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-13T08:26:13.363 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi045.front.sepia.ceph.com to complete... 2024-09-13T08:26:14.735 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T08:26:16.914 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-13T08:26:16.915 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-13T08:26:16.915 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-13T08:26:17.002 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T08:26:17.003 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T08:26:17.152 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-13T08:26:17.153 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-13T08:26:17.155 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi031.front.sepia.ceph.com to complete... 2024-09-13T08:26:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13 08:26:25 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-13T08:26:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:FINISHED --2024-09-13 08:26:25-- 2024-09-13T08:26:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:Total wall clock time: 12s 2024-09-13T08:26:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:Downloaded: 1 files, 343M in 12s (29.3 MB/s) 2024-09-13T08:26:25.440 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-13T08:26:25.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-13T08:26:27.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13 08:26:27 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-13T08:26:27.274 INFO:teuthology.orchestra.run.smithi031.stderr:FINISHED --2024-09-13 08:26:27-- 2024-09-13T08:26:27.274 INFO:teuthology.orchestra.run.smithi031.stderr:Total wall clock time: 10s 2024-09-13T08:26:27.274 INFO:teuthology.orchestra.run.smithi031.stderr:Downloaded: 1 files, 343M in 9.8s (35.0 MB/s) 2024-09-13T08:26:27.276 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T08:26:27.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-13T08:26:27.648 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying... ######################################## 2024-09-13T08:26:27.789 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing... ######################################## 2024-09-13T08:26:27.905 INFO:teuthology.orchestra.run.smithi045.stdout:Updating / installing... 2024-09-13T08:26:29.471 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying... ######################################## 2024-09-13T08:26:29.620 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing... ######################################## 2024-09-13T08:26:29.728 INFO:teuthology.orchestra.run.smithi031.stdout:Updating / installing... 2024-09-13T08:26:36.287 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-13T08:26:38.204 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-13T08:27:16.604 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-13T08:27:16.605 DEBUG:teuthology.orchestra.run.smithi045:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-13T08:27:16.701 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.701 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.701 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.701 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-13T08:27:16.702 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-13T08:27:16.703 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-13T08:27:16.704 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-13T08:27:16.705 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-13T08:27:16.706 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-13T08:27:16.707 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-13T08:27:16.708 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-13T08:27:16.709 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-13T08:27:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-13T08:27:16.711 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-13T08:27:16.712 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-13T08:27:16.713 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-13T08:27:16.714 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-13T08:27:16.715 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-13T08:27:16.716 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-13T08:27:16.717 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-13T08:27:16.718 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-13T08:27:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-13T08:27:16.720 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-13T08:27:16.721 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-13T08:27:16.722 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-13T08:27:16.723 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-13T08:27:16.724 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-13T08:27:16.725 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-13T08:27:16.726 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-13T08:27:16.727 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-13T08:27:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-13T08:27:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-13T08:27:16.730 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-13T08:27:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-13T08:27:16.732 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-13T08:27:16.733 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-13T08:27:16.734 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-13T08:27:16.735 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-13T08:27:16.736 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-13T08:27:16.737 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-13T08:27:16.738 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-13T08:27:16.738 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-13T08:27:16.738 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-13T08:27:16.738 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-13T08:27:16.738 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-13T08:27:16.740 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-13T08:27:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-13T08:27:16.742 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-13T08:27:16.743 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-13T08:27:16.744 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-13T08:27:16.745 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-13T08:27:16.746 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-13T08:27:16.747 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-13T08:27:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-13T08:27:16.749 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-13T08:27:16.750 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-13T08:27:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-13T08:27:16.752 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-13T08:27:16.753 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-13T08:27:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-13T08:27:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-13T08:27:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-13T08:27:16.757 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-13T08:27:16.758 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-13T08:27:16.759 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-13T08:27:16.760 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-13T08:27:16.760 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-13T08:27:16.760 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-13T08:27:16.760 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-13T08:27:16.760 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-13T08:27:16.761 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-13T08:27:16.762 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-13T08:27:16.762 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-13T08:27:16.762 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-13T08:27:16.762 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-13T08:27:16.762 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-13T08:27:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-13T08:27:16.764 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-13T08:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-13T08:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-13T08:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-13T08:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-13T08:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-13T08:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-13T08:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-13T08:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-13T08:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-13T08:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-13T08:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-13T08:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-13T08:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-13T08:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-13T08:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-13T08:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-13T08:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-13T08:27:16.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-13T08:27:16.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-13T08:27:16.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-13T08:27:16.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-13T08:27:16.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-13T08:27:16.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-13T08:27:16.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-13T08:27:16.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-13T08:27:16.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-13T08:27:16.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-13T08:27:16.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-13T08:27:16.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-13T08:27:16.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-13T08:27:16.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-13T08:27:16.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-13T08:27:16.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-13T08:27:16.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-13T08:27:16.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-13T08:27:16.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-13T08:27:16.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-13T08:27:16.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-13T08:27:16.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-13T08:27:16.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-13T08:27:16.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-13T08:27:16.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-13T08:27:16.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-13T08:27:16.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-13T08:27:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-13T08:27:16.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-13T08:27:16.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-13T08:27:16.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-13T08:27:16.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-13T08:27:16.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-13T08:27:16.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-13T08:27:16.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-13T08:27:16.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-13T08:27:16.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-13T08:27:16.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-13T08:27:16.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-13T08:27:16.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-13T08:27:16.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-13T08:27:16.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-13T08:27:16.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-13T08:27:16.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-13T08:27:16.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-13T08:27:16.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-13T08:27:16.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-13T08:27:16.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-13T08:27:16.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-13T08:27:16.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-13T08:27:16.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-13T08:27:16.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-13T08:27:16.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-13T08:27:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-13T08:27:16.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-13T08:27:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-13T08:27:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-13T08:27:16.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-13T08:27:16.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-13T08:27:16.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-13T08:27:16.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-13T08:27:16.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-13T08:27:16.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-13T08:27:16.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-13T08:27:16.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-13T08:27:16.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-13T08:27:16.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-13T08:27:16.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-13T08:27:16.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-13T08:27:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-13T08:27:16.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-13T08:27:16.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-13T08:27:16.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-13T08:27:16.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-13T08:27:16.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-13T08:27:16.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-13T08:27:16.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-13T08:27:16.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-13T08:27:16.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-13T08:27:16.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-13T08:27:16.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-13T08:27:16.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T08:27:16.864 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.864 DEBUG:teuthology.orchestra.run.smithi045:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-13T08:27:16.888 INFO:teuthology.orchestra.run.smithi045.stdout:postinstall scriptlet (using /bin/sh): 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout:for file in vmlinuz System.map config; do 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: fi 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-13T08:27:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-09-13T08:27:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-13T08:27:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T08:27:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-09-13T08:27:16.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 82.el9 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T08:27:16.977 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8274670 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-09-13T08:27:16.978 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:27:16.979 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T08:27:16.980 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T08:27:16.980 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:16.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T08:27:17.547 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-09-13T08:27:17.976 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-13T08:27:17.977 DEBUG:teuthology.orchestra.run.smithi031:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-13T08:27:18.065 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.065 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-13T08:27:18.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-13T08:27:18.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-13T08:27:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-13T08:27:18.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-13T08:27:18.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-13T08:27:18.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-13T08:27:18.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-13T08:27:18.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-13T08:27:18.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-13T08:27:18.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-13T08:27:18.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-13T08:27:18.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-13T08:27:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-13T08:27:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-13T08:27:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-13T08:27:18.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-13T08:27:18.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-13T08:27:18.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-13T08:27:18.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-13T08:27:18.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-13T08:27:18.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-13T08:27:18.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-13T08:27:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-13T08:27:18.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-13T08:27:18.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-13T08:27:18.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-13T08:27:18.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-13T08:27:18.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-13T08:27:18.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-13T08:27:18.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-13T08:27:18.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-13T08:27:18.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-13T08:27:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-13T08:27:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-13T08:27:18.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-13T08:27:18.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-13T08:27:18.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-13T08:27:18.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-13T08:27:18.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-13T08:27:18.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-13T08:27:18.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-13T08:27:18.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-13T08:27:18.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-13T08:27:18.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-13T08:27:18.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-13T08:27:18.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-13T08:27:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-13T08:27:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-13T08:27:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-13T08:27:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-13T08:27:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-13T08:27:18.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-13T08:27:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-13T08:27:18.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-13T08:27:18.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-13T08:27:18.121 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-13T08:27:18.122 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-13T08:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-13T08:27:18.124 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-13T08:27:18.125 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-13T08:27:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-13T08:27:18.127 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-13T08:27:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-13T08:27:18.129 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-13T08:27:18.130 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-13T08:27:18.131 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-13T08:27:18.132 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-13T08:27:18.133 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-13T08:27:18.134 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-13T08:27:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-13T08:27:18.136 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-13T08:27:18.137 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-13T08:27:18.138 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-13T08:27:18.139 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-13T08:27:18.140 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-13T08:27:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-13T08:27:18.142 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-13T08:27:18.143 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-13T08:27:18.144 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-13T08:27:18.145 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-13T08:27:18.146 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-13T08:27:18.147 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-13T08:27:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-13T08:27:18.149 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-13T08:27:18.150 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-13T08:27:18.151 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-13T08:27:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-13T08:27:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-13T08:27:18.154 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-13T08:27:18.155 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-13T08:27:18.156 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-13T08:27:18.157 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-13T08:27:18.158 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-13T08:27:18.159 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-13T08:27:18.160 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-13T08:27:18.161 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-13T08:27:18.162 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-13T08:27:18.163 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-13T08:27:18.164 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-13T08:27:18.165 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-13T08:27:18.166 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-13T08:27:18.167 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-13T08:27:18.168 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-13T08:27:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-13T08:27:18.170 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-13T08:27:18.171 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-13T08:27:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-13T08:27:18.173 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-13T08:27:18.174 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-13T08:27:18.175 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-13T08:27:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-13T08:27:18.177 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-13T08:27:18.178 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-13T08:27:18.179 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-13T08:27:18.180 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-13T08:27:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-13T08:27:18.182 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-13T08:27:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-13T08:27:18.184 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-13T08:27:18.185 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-13T08:27:18.186 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-13T08:27:18.187 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-13T08:27:18.188 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-13T08:27:18.189 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-13T08:27:18.190 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-13T08:27:18.191 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-13T08:27:18.192 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-13T08:27:18.193 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-13T08:27:18.194 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-13T08:27:18.195 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-13T08:27:18.196 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-13T08:27:18.197 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-13T08:27:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-13T08:27:18.199 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-13T08:27:18.200 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-13T08:27:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-13T08:27:18.202 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-13T08:27:18.203 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-13T08:27:18.204 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-13T08:27:18.205 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-13T08:27:18.206 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-13T08:27:18.207 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-13T08:27:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-13T08:27:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-13T08:27:18.210 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-13T08:27:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-13T08:27:18.212 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-13T08:27:18.213 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-13T08:27:18.214 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T08:27:18.218 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.218 DEBUG:teuthology.orchestra.run.smithi031:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout:postinstall scriptlet (using /bin/sh): 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout:for file in vmlinuz System.map config; do 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-13T08:27:18.245 INFO:teuthology.orchestra.run.smithi031.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout: fi 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:done 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.246 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-13T08:27:18.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-09-13T08:27:18.324 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-09-13T08:27:18.324 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 82.el9 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8274670 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-09-13T08:27:18.325 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:27:18.326 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T08:27:18.327 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T08:27:18.327 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:18.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T08:27:18.929 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-09-13T08:27:18.999 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T08:27:19.030 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-09-13T08:27:19.032 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-13T08:27:19.047 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.PIFnEFV8hA 2024-09-13T08:27:19.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PIFnEFV8hA 2024-09-13T08:27:19.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.PIFnEFV8hA 2024-09-13T08:27:19.254 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.PIFnEFV8hA is 6KB 2024-09-13T08:27:19.303 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.PIFnEFV8hA 2024-09-13T08:27:19.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T08:27:19.380 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T08:27:19.380 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T08:27:19.380 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T08:27:19.380 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-13T08:27:19.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:19.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-09-13T08:27:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T08:27:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-09-13T08:27:20.328 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-13T08:27:20.343 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.lgN2LZIQJk 2024-09-13T08:27:20.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lgN2LZIQJk 2024-09-13T08:27:20.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.lgN2LZIQJk 2024-09-13T08:27:20.546 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.lgN2LZIQJk is 6KB 2024-09-13T08:27:20.594 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.lgN2LZIQJk 2024-09-13T08:27:20.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T08:27:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T08:27:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T08:27:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T08:27:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-13T08:27:20.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:27:20.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-09-13T08:27:49.535 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-09-13T08:27:49.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:27:49.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:27:50.830 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-13T08:27:50.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:27:50.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:28:07.869 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-09-13T08:28:09.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-13T08:28:16.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:28:16.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:28:18.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:28:18.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:28:35.265 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-09-13T08:28:36.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-13T08:28:47.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:28:47.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:28:48.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:28:48.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:28:50.369 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-09-13T08:28:52.323 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-13T08:28:52.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:28:52.898 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-13T08:28:52.898 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-13T08:28:52.932 INFO:teuthology.orchestra.run.smithi031.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:28:52.933 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:28:52.933 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T08:28:52.933 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T08:28:53.934 DEBUG:teuthology.parallel:result is None 2024-09-13T08:29:05.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:29:05.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:29:05.704 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-13T08:29:06.164 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:29:06.164 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-13T08:29:06.165 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-13T08:29:06.200 INFO:teuthology.orchestra.run.smithi045.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:29:06.200 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T08:29:06.200 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T08:29:06.201 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T08:29:07.203 DEBUG:teuthology.parallel:result is None 2024-09-13T08:29:07.203 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T08:29:07.210 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T08:29:07.211 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T08:29:07.215 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T08:29:07.251 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T08:29:07.257 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T08:29:07.272 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T08:29:07.273 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T08:29:07.293 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T08:29:07.341 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T08:29:07.347 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T08:29:07.348 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T08:29:07.377 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T08:29:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T08:29:07.518 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T08:29:07.545 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T08:29:07.568 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T08:29:07.571 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T08:29:07.582 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T08:29:07.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T08:29:07.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T08:29:07.642 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T08:29:07.651 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T08:29:07.651 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T08:29:07.655 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T08:29:07.689 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T08:29:07.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T08:29:07.964 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T08:29:08.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T08:29:08.098 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:29:08.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T08:29:08.153 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T08:29:08.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T08:29:08.302 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T08:29:08.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T08:29:08.434 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:29:08.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T08:29:08.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-09-13T08:29:08.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-09-13T08:29:08.567 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T08:29:08.618 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T08:29:09.129 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T08:29:09.137 INFO:teuthology.task.internal:Starting timer... 2024-09-13T08:29:09.137 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T08:29:09.148 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T08:29:09.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-09-13T08:29:09.232 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-09-13T08:29:09.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-09-13T08:29:09.371 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-09-13T08:29:09.374 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T08:29:09.374 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-09-13T08:29:09.416 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-09-13T08:29:09.416 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-09-13T08:29:09.459 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-09-13T08:29:09.460 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive'} 2024-09-13T08:29:09.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T08:29:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:29:09.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T08:29:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:29:09.577 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T08:29:09.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-09-13T08:29:09.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-09-13T08:29:09.951 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T08:29:09.960 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T08:29:09.963 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T08:29:10.087 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T08:29:10.098 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T08:29:10.099 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T08:33:55.450 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2024-09-13T08:33:55.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:33:55.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:33:55.556 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-13T08:33:55.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-13T08:33:55.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:33:55.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T08:33:55.783 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-13T08:33:55.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-13T08:33:55.891 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T08:33:55.907 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T08:33:55.908 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T08:33:55.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T08:33:55.913 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T08:33:55.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T08:33:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T08:33:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T08:33:56.001 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T08:33:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T08:33:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-09-13T08:33:56.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-09-13T08:33:56.077 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-09-13T08:33:56.097 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-09-13T08:33:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T08:33:56.126 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T08:33:56.137 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T08:33:56.156 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T08:33:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-13T08:33:56.239 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T08:33:56.239 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-13T08:33:56.240 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.240 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.240 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.240 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.264 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:33:56.265 INFO:teuthology.run_tasks:Running task install... 2024-09-13T08:33:56.275 DEBUG:teuthology.task.install:project ceph 2024-09-13T08:33:56.275 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'}} 2024-09-13T08:33:56.275 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'} 2024-09-13T08:33:56.275 INFO:teuthology.task.install:Using flavor: default 2024-09-13T08:33:56.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T08:33:56.281 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-13T08:33:56.281 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e', 'tag': None, 'wait_for_package': False} 2024-09-13T08:33:56.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d454760ef8a93a9dd7f4e4dedfb085faac216e7e 2024-09-13T08:33:56.284 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e', 'tag': None, 'wait_for_package': False} 2024-09-13T08:33:56.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d454760ef8a93a9dd7f4e4dedfb085faac216e7e 2024-09-13T08:33:56.437 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/ 2024-09-13T08:33:56.438 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4881.gd454760e 2024-09-13T08:33:56.449 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/ 2024-09-13T08:33:56.449 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4881.gd454760e 2024-09-13T08:33:56.560 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T08:33:56.560 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:33:56.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T08:33:56.570 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1849/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T08:33:56.571 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:33:56.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T08:33:56.618 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T08:33:56.618 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T08:33:56.628 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T08:33:56.629 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d454760ef8a93a9dd7f4e4dedfb085faac216e7e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T08:33:56.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T08:33:56.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T08:33:56.797 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T08:33:56.805 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T08:33:56.852 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-09-13T08:33:56.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-09-13T08:33:56.860 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-09-13T08:33:56.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-09-13T08:33:57.134 INFO:teuthology.orchestra.run.smithi031.stdout:45 files removed 2024-09-13T08:33:57.145 INFO:teuthology.orchestra.run.smithi045.stdout:45 files removed 2024-09-13T08:33:57.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T08:33:57.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T08:33:57.865 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 291 kB/s | 84 kB 00:00 2024-09-13T08:33:57.885 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 284 kB/s | 84 kB 00:00 2024-09-13T08:33:58.174 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 92 kB/s | 24 kB 00:00 2024-09-13T08:33:58.310 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-09-13T08:33:58.466 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 7.5 kB/s | 1.9 kB 00:00 2024-09-13T08:33:58.603 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 7.5 kB/s | 1.9 kB 00:00 2024-09-13T08:33:59.244 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-13T08:34:01.158 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-13T08:34:02.563 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-13T08:34:08.492 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T08:34:10.147 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 19 kB 00:00 2024-09-13T08:34:11.302 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-13T08:34:17.841 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-13T08:34:18.623 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:16 2024-09-13T08:34:20.487 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T08:34:20.488 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T08:34:20.500 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T08:34:20.501 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T08:34:20.502 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T08:34:20.503 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T08:34:20.505 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T08:34:20.507 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T08:34:20.508 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T08:34:20.577 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.3.0-4881.gd454760e.el9 ceph 6.6 k 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.3.0-4881.gd454760e.el9 ceph 5.5 M 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 899 k 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4881.gd454760e.el9 ceph 145 k 2024-09-13T08:34:20.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.3.0-4881.gd454760e.el9 ceph 1.5 M 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 154 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 14 M 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 7.4 M 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 49 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.3.0-4881.gd454760e.el9 ceph 11 M 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.3.0-4881.gd454760e.el9 ceph 50 M 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 285 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 784 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:19.3.0-4881.gd454760e.el9 ceph 225 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell x86_64 2:19.3.0-4881.gd454760e.el9 ceph 27 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 18 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T08:34:20.595 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.3.0-4881.gd454760e.el9 ceph 32 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 756 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.3.0-4881.gd454760e.el9 ceph 126 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.3.0-4881.gd454760e.el9 ceph 161 k 2024-09-13T08:34:20.596 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.3.0-4881.gd454760e.el9 ceph 320 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 305 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.3.0-4881.gd454760e.el9 ceph 100 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 85 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.2 M 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 170 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.5 M 2024-09-13T08:34:20.597 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.2 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.3.0-4881.gd454760e.el9 ceph 22 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 34 k 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.3.0-4881.gd454760e.el9 ceph 2.4 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 291 k 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.3.0-4881.gd454760e.el9 ceph 4.8 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 17 M 2024-09-13T08:34:20.598 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 16 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.3.0-4881.gd454760e.el9 ceph 25 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.3.0-4881.gd454760e.el9 ceph 161 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-13T08:34:20.599 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 508 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 5.5 M 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T08:34:20.600 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 45 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.3.0-4881.gd454760e.el9 ceph 148 k 2024-09-13T08:34:20.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T08:34:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T08:34:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T08:34:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T08:34:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-13T08:34:20.606 INFO:teuthology.orchestra.run.smithi045.stdout:Install 149 Packages 2024-09-13T08:34:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 8 Packages 2024-09-13T08:34:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:34:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 240 M 2024-09-13T08:34:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-13T08:34:21.217 INFO:teuthology.orchestra.run.smithi045.stdout:(1/157): ceph-19.3.0-4881.gd454760e.el9.x86_64. 52 kB/s | 6.6 kB 00:00 2024-09-13T08:34:21.393 INFO:teuthology.orchestra.run.smithi045.stdout:(2/157): ceph-fuse-19.3.0-4881.gd454760e.el9.x8 5.0 MB/s | 899 kB 00:00 2024-09-13T08:34:21.476 INFO:teuthology.orchestra.run.smithi045.stdout:(3/157): ceph-immutable-object-cache-19.3.0-488 1.7 MB/s | 145 kB 00:00 2024-09-13T08:34:21.610 INFO:teuthology.orchestra.run.smithi045.stdout:(4/157): ceph-base-19.3.0-4881.gd454760e.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-13T08:34:21.685 INFO:teuthology.orchestra.run.smithi045.stdout:(5/157): ceph-mds-19.3.0-4881.gd454760e.el9.x86 11 MB/s | 2.4 MB 00:00 2024-09-13T08:34:21.819 INFO:teuthology.orchestra.run.smithi045.stdout:(6/157): ceph-mgr-19.3.0-4881.gd454760e.el9.x86 7.3 MB/s | 1.5 MB 00:00 2024-09-13T08:34:22.053 INFO:teuthology.orchestra.run.smithi045.stdout:(7/157): ceph-mon-19.3.0-4881.gd454760e.el9.x86 13 MB/s | 4.8 MB 00:00 2024-09-13T08:34:22.378 INFO:teuthology.orchestra.run.smithi045.stdout:(8/157): ceph-common-19.3.0-4881.gd454760e.el9. 17 MB/s | 22 MB 00:01 2024-09-13T08:34:22.428 INFO:teuthology.orchestra.run.smithi045.stdout:(9/157): ceph-selinux-19.3.0-4881.gd454760e.el9 509 kB/s | 25 kB 00:00 2024-09-13T08:34:22.670 INFO:teuthology.orchestra.run.smithi045.stdout:(10/157): ceph-radosgw-19.3.0-4881.gd454760e.el 18 MB/s | 11 MB 00:00 2024-09-13T08:34:22.729 INFO:teuthology.orchestra.run.smithi045.stdout:(11/157): cephfs-mirror-19.3.0-4881.gd454760e.e 3.8 MB/s | 225 kB 00:00 2024-09-13T08:34:22.904 INFO:teuthology.orchestra.run.smithi045.stdout:(12/157): ceph-osd-19.3.0-4881.gd454760e.el9.x8 16 MB/s | 17 MB 00:01 2024-09-13T08:34:22.929 INFO:teuthology.orchestra.run.smithi045.stdout:(13/157): cephfs-shell-19.3.0-4881.gd454760e.el 135 kB/s | 27 kB 00:00 2024-09-13T08:34:22.954 INFO:teuthology.orchestra.run.smithi045.stdout:(14/157): libcephfs-devel-19.3.0-4881.gd454760e 632 kB/s | 32 kB 00:00 2024-09-13T08:34:22.988 INFO:teuthology.orchestra.run.smithi045.stdout:(15/157): libcephfs2-19.3.0-4881.gd454760e.el9. 13 MB/s | 756 kB 00:00 2024-09-13T08:34:23.038 INFO:teuthology.orchestra.run.smithi045.stdout:(16/157): libcephsqlite-19.3.0-4881.gd454760e.e 1.9 MB/s | 161 kB 00:00 2024-09-13T08:34:23.072 INFO:teuthology.orchestra.run.smithi045.stdout:(17/157): librados-devel-19.3.0-4881.gd454760e. 1.5 MB/s | 126 kB 00:00 2024-09-13T08:34:23.114 INFO:teuthology.orchestra.run.smithi045.stdout:(18/157): libradosstriper1-19.3.0-4881.gd454760 6.6 MB/s | 508 kB 00:00 2024-09-13T08:34:23.181 INFO:teuthology.orchestra.run.smithi045.stdout:(19/157): python3-ceph-argparse-19.3.0-4881.gd4 678 kB/s | 45 kB 00:00 2024-09-13T08:34:23.307 INFO:teuthology.orchestra.run.smithi045.stdout:(20/157): librgw2-19.3.0-4881.gd454760e.el9.x86 24 MB/s | 5.5 MB 00:00 2024-09-13T08:34:23.332 INFO:teuthology.orchestra.run.smithi045.stdout:(21/157): python3-ceph-common-19.3.0-4881.gd454 987 kB/s | 148 kB 00:00 2024-09-13T08:34:23.365 INFO:teuthology.orchestra.run.smithi045.stdout:(22/157): python3-cephfs-19.3.0-4881.gd454760e. 2.7 MB/s | 161 kB 00:00 2024-09-13T08:34:23.390 INFO:teuthology.orchestra.run.smithi045.stdout:(23/157): python3-rados-19.3.0-4881.gd454760e.e 5.4 MB/s | 320 kB 00:00 2024-09-13T08:34:23.416 INFO:teuthology.orchestra.run.smithi045.stdout:(24/157): python3-rbd-19.3.0-4881.gd454760e.el9 6.0 MB/s | 305 kB 00:00 2024-09-13T08:34:23.475 INFO:teuthology.orchestra.run.smithi045.stdout:(25/157): python3-rgw-19.3.0-4881.gd454760e.el9 1.2 MB/s | 100 kB 00:00 2024-09-13T08:34:23.500 INFO:teuthology.orchestra.run.smithi045.stdout:(26/157): rbd-fuse-19.3.0-4881.gd454760e.el9.x8 1.0 MB/s | 85 kB 00:00 2024-09-13T08:34:23.507 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.5 MB 00:00 2024-09-13T08:34:23.559 INFO:teuthology.orchestra.run.smithi045.stdout:(27/157): rbd-nbd-19.3.0-4881.gd454760e.el9.x86 2.9 MB/s | 170 kB 00:00 2024-09-13T08:34:23.651 INFO:teuthology.orchestra.run.smithi045.stdout:(28/157): rbd-mirror-19.3.0-4881.gd454760e.el9. 18 MB/s | 3.2 MB 00:00 2024-09-13T08:34:23.685 INFO:teuthology.orchestra.run.smithi045.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4881.g 273 kB/s | 34 kB 00:00 2024-09-13T08:34:23.701 INFO:teuthology.orchestra.run.smithi045.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4881.gd454760 3.0 MB/s | 154 kB 00:00 2024-09-13T08:34:24.385 INFO:teuthology.orchestra.run.smithi045.stdout:(31/157): ceph-mgr-dashboard-19.3.0-4881.gd4547 20 MB/s | 14 MB 00:00 2024-09-13T08:34:24.477 INFO:teuthology.orchestra.run.smithi045.stdout:(32/157): ceph-mgr-diskprediction-local-19.3.0- 9.5 MB/s | 7.4 MB 00:00 2024-09-13T08:34:24.502 INFO:teuthology.orchestra.run.smithi045.stdout:(33/157): ceph-mgr-modules-core-19.3.0-4881.gd4 2.4 MB/s | 291 kB 00:00 2024-09-13T08:34:24.528 INFO:teuthology.orchestra.run.smithi045.stdout:(34/157): ceph-mgr-rook-19.3.0-4881.gd454760e.e 984 kB/s | 49 kB 00:00 2024-09-13T08:34:24.553 INFO:teuthology.orchestra.run.smithi045.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4881.gd 329 kB/s | 16 kB 00:00 2024-09-13T08:34:24.586 INFO:teuthology.orchestra.run.smithi045.stdout:(36/157): ceph-volume-19.3.0-4881.gd454760e.el9 4.8 MB/s | 285 kB 00:00 2024-09-13T08:34:24.620 INFO:teuthology.orchestra.run.smithi045.stdout:(37/157): cephadm-19.3.0-4881.gd454760e.el9.noa 11 MB/s | 784 kB 00:00 2024-09-13T08:34:24.645 INFO:teuthology.orchestra.run.smithi045.stdout:(38/157): cephfs-top-19.3.0-4881.gd454760e.el9. 312 kB/s | 18 kB 00:00 2024-09-13T08:34:24.754 INFO:teuthology.orchestra.run.smithi045.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 598 kB/s | 80 kB 00:00 2024-09-13T08:34:24.813 INFO:teuthology.orchestra.run.smithi045.stdout:(40/157): libquadmath-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-13T08:34:24.847 INFO:teuthology.orchestra.run.smithi045.stdout:(41/157): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-13T08:34:24.872 INFO:teuthology.orchestra.run.smithi045.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-09-13T08:34:24.914 INFO:teuthology.orchestra.run.smithi045.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T08:34:24.997 INFO:teuthology.orchestra.run.smithi045.stdout:(44/157): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-09-13T08:34:25.056 INFO:teuthology.orchestra.run.smithi045.stdout:(45/157): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-13T08:34:25.200 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-13T08:34:25.532 INFO:teuthology.orchestra.run.smithi045.stdout:(46/157): ceph-test-19.3.0-4881.gd454760e.el9.x 16 MB/s | 50 MB 00:03 2024-09-13T08:34:25.578 INFO:teuthology.orchestra.run.smithi045.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 412 kB/s | 215 kB 00:00 2024-09-13T08:34:25.615 INFO:teuthology.orchestra.run.smithi045.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 1.6 MB/s | 1.2 MB 00:00 2024-09-13T08:34:25.791 INFO:teuthology.orchestra.run.smithi045.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 611 kB/s | 107 kB 00:00 2024-09-13T08:34:25.850 INFO:teuthology.orchestra.run.smithi045.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-13T08:34:25.875 INFO:teuthology.orchestra.run.smithi045.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 260 kB/s | 22 kB 00:00 2024-09-13T08:34:25.908 INFO:teuthology.orchestra.run.smithi045.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-09-13T08:34:25.933 INFO:teuthology.orchestra.run.smithi045.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-13T08:34:25.959 INFO:teuthology.orchestra.run.smithi045.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-13T08:34:26.015 INFO:teuthology.orchestra.run.smithi045.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 224 kB/s | 18 kB 00:00 2024-09-13T08:34:26.043 INFO:teuthology.orchestra.run.smithi045.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-09-13T08:34:26.045 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-13T08:34:26.068 INFO:teuthology.orchestra.run.smithi045.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.2 MB/s | 171 kB 00:00 2024-09-13T08:34:26.118 INFO:teuthology.orchestra.run.smithi045.stdout:(58/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-09-13T08:34:26.144 INFO:teuthology.orchestra.run.smithi045.stdout:(59/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-09-13T08:34:26.211 INFO:teuthology.orchestra.run.smithi045.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-13T08:34:26.261 INFO:teuthology.orchestra.run.smithi045.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 4.9 MB/s | 581 kB 00:00 2024-09-13T08:34:26.286 INFO:teuthology.orchestra.run.smithi045.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 326 kB/s | 24 kB 00:00 2024-09-13T08:34:26.312 INFO:teuthology.orchestra.run.smithi045.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-13T08:34:26.370 INFO:teuthology.orchestra.run.smithi045.stdout:(64/157): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-09-13T08:34:26.442 INFO:teuthology.orchestra.run.smithi045.stdout:(65/157): openblas-0.3.26-2.el9.x86_64.rpm 537 kB/s | 38 kB 00:00 2024-09-13T08:34:26.471 INFO:teuthology.orchestra.run.smithi045.stdout:(66/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-09-13T08:34:27.121 INFO:teuthology.orchestra.run.smithi045.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-09-13T08:34:27.905 INFO:teuthology.orchestra.run.smithi045.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-09-13T08:34:28.263 INFO:teuthology.orchestra.run.smithi045.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 2.5 MB/s | 910 kB 00:00 2024-09-13T08:34:28.339 INFO:teuthology.orchestra.run.smithi045.stdout:(70/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.6 MB/s | 4.9 MB 00:01 2024-09-13T08:34:28.381 INFO:teuthology.orchestra.run.smithi045.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-13T08:34:28.448 INFO:teuthology.orchestra.run.smithi045.stdout:(72/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:01 2024-09-13T08:34:28.490 INFO:teuthology.orchestra.run.smithi045.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-09-13T08:34:28.606 INFO:teuthology.orchestra.run.smithi045.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 552 kB/s | 87 kB 00:00 2024-09-13T08:34:28.674 INFO:teuthology.orchestra.run.smithi045.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 916 kB/s | 61 kB 00:00 2024-09-13T08:34:28.915 INFO:teuthology.orchestra.run.smithi045.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-13T08:34:29.066 INFO:teuthology.orchestra.run.smithi045.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-09-13T08:34:29.116 INFO:teuthology.orchestra.run.smithi045.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 515 kB/s | 26 kB 00:00 2024-09-13T08:34:29.166 INFO:teuthology.orchestra.run.smithi045.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-13T08:34:29.283 INFO:teuthology.orchestra.run.smithi045.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-13T08:34:29.334 INFO:teuthology.orchestra.run.smithi045.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-09-13T08:34:30.751 INFO:teuthology.orchestra.run.smithi045.stdout:(82/157): python3-babel-2.9.1-2.el9.noarch.rpm 2.6 MB/s | 6.0 MB 00:02 2024-09-13T08:34:30.893 INFO:teuthology.orchestra.run.smithi045.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-09-13T08:34:30.968 INFO:teuthology.orchestra.run.smithi045.stdout:(84/157): postgresql-server-13.16-1.el9.x86_64. 2.2 MB/s | 5.8 MB 00:02 2024-09-13T08:34:31.051 INFO:teuthology.orchestra.run.smithi045.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 931 kB/s | 77 kB 00:00 2024-09-13T08:34:31.077 INFO:teuthology.orchestra.run.smithi045.stdout:(86/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-09-13T08:34:31.118 INFO:teuthology.orchestra.run.smithi045.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 638 kB/s | 42 kB 00:00 2024-09-13T08:34:31.244 INFO:teuthology.orchestra.run.smithi045.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T08:34:31.269 INFO:teuthology.orchestra.run.smithi045.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-09-13T08:34:31.320 INFO:teuthology.orchestra.run.smithi045.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T08:34:31.378 INFO:teuthology.orchestra.run.smithi045.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 925 kB/s | 54 kB 00:00 2024-09-13T08:34:31.437 INFO:teuthology.orchestra.run.smithi045.stdout:(92/157): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-09-13T08:34:31.487 INFO:teuthology.orchestra.run.smithi045.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-09-13T08:34:31.554 INFO:teuthology.orchestra.run.smithi045.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 665 kB/s | 44 kB 00:00 2024-09-13T08:34:31.705 INFO:teuthology.orchestra.run.smithi045.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-13T08:34:31.763 INFO:teuthology.orchestra.run.smithi045.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 620 kB/s | 36 kB 00:00 2024-09-13T08:34:31.813 INFO:teuthology.orchestra.run.smithi045.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-09-13T08:34:32.022 INFO:teuthology.orchestra.run.smithi045.stdout:(98/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-09-13T08:34:32.047 INFO:teuthology.orchestra.run.smithi045.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-13T08:34:32.073 INFO:teuthology.orchestra.run.smithi045.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T08:34:32.131 INFO:teuthology.orchestra.run.smithi045.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.1 MB/s | 96 kB 00:00 2024-09-13T08:34:32.157 INFO:teuthology.orchestra.run.smithi045.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 762 kB/s | 64 kB 00:00 2024-09-13T08:34:32.182 INFO:teuthology.orchestra.run.smithi045.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 896 kB/s | 45 kB 00:00 2024-09-13T08:34:32.266 INFO:teuthology.orchestra.run.smithi045.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-09-13T08:34:32.324 INFO:teuthology.orchestra.run.smithi045.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-13T08:34:32.358 INFO:teuthology.orchestra.run.smithi045.stdout:(106/157): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-09-13T08:34:32.400 INFO:teuthology.orchestra.run.smithi045.stdout:(107/157): gperftools-libs-2.9.1-3.el9.x86_64.r 4.0 MB/s | 308 kB 00:00 2024-09-13T08:34:32.472 INFO:teuthology.orchestra.run.smithi045.stdout:(108/157): grpc-data-1.46.7-10.el9.noarch.rpm 171 kB/s | 19 kB 00:00 2024-09-13T08:34:32.501 INFO:teuthology.orchestra.run.smithi045.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 921 kB/s | 26 kB 00:00 2024-09-13T08:34:32.535 INFO:teuthology.orchestra.run.smithi045.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-13T08:34:32.560 INFO:teuthology.orchestra.run.smithi045.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-13T08:34:32.594 INFO:teuthology.orchestra.run.smithi045.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-13T08:34:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 17 MB/s | 839 kB 00:00 2024-09-13T08:34:32.691 INFO:teuthology.orchestra.run.smithi045.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-09-13T08:34:32.716 INFO:teuthology.orchestra.run.smithi045.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-13T08:34:32.734 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-13T08:34:32.742 INFO:teuthology.orchestra.run.smithi045.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-13T08:34:32.775 INFO:teuthology.orchestra.run.smithi045.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 426 kB/s | 14 kB 00:00 2024-09-13T08:34:32.809 INFO:teuthology.orchestra.run.smithi045.stdout:(118/157): python3-cheroot-8.6.0-4.el9.noarch.r 5.1 MB/s | 172 kB 00:00 2024-09-13T08:34:32.843 INFO:teuthology.orchestra.run.smithi045.stdout:(119/157): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-09-13T08:34:32.893 INFO:teuthology.orchestra.run.smithi045.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 6.0 MB/s | 303 kB 00:00 2024-09-13T08:34:32.927 INFO:teuthology.orchestra.run.smithi045.stdout:(121/157): python3-google-auth-2.34.0-1.el9.noa 6.4 MB/s | 220 kB 00:00 2024-09-13T08:34:32.994 INFO:teuthology.orchestra.run.smithi045.stdout:(122/157): libarrow-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-09-13T08:34:33.019 INFO:teuthology.orchestra.run.smithi045.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-13T08:34:33.044 INFO:teuthology.orchestra.run.smithi045.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-13T08:34:33.070 INFO:teuthology.orchestra.run.smithi045.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-09-13T08:34:33.095 INFO:teuthology.orchestra.run.smithi045.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 937 kB/s | 23 kB 00:00 2024-09-13T08:34:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 778 kB/s | 19 kB 00:00 2024-09-13T08:34:33.146 INFO:teuthology.orchestra.run.smithi045.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 791 kB/s | 20 kB 00:00 2024-09-13T08:34:33.205 INFO:teuthology.orchestra.run.smithi045.stdout:(129/157): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-09-13T08:34:33.230 INFO:teuthology.orchestra.run.smithi045.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-13T08:34:33.304 INFO:teuthology.orchestra.run.smithi045.stdout:(131/157): python3-grpcio-1.46.7-10.el9.x86_64. 5.4 MB/s | 2.0 MB 00:00 2024-09-13T08:34:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 724 kB/s | 79 kB 00:00 2024-09-13T08:34:33.365 INFO:teuthology.orchestra.run.smithi045.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 961 kB/s | 58 kB 00:00 2024-09-13T08:34:33.390 INFO:teuthology.orchestra.run.smithi045.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 661 kB/s | 16 kB 00:00 2024-09-13T08:34:33.415 INFO:teuthology.orchestra.run.smithi045.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.6 MB/s | 90 kB 00:00 2024-09-13T08:34:33.440 INFO:teuthology.orchestra.run.smithi045.stdout:(136/157): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-13T08:34:33.466 INFO:teuthology.orchestra.run.smithi045.stdout:(137/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-13T08:34:33.566 INFO:teuthology.orchestra.run.smithi045.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-09-13T08:34:33.591 INFO:teuthology.orchestra.run.smithi045.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T08:34:33.617 INFO:teuthology.orchestra.run.smithi045.stdout:(140/157): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-13T08:34:33.642 INFO:teuthology.orchestra.run.smithi045.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-09-13T08:34:33.667 INFO:teuthology.orchestra.run.smithi045.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-13T08:34:33.693 INFO:teuthology.orchestra.run.smithi045.stdout:(143/157): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-13T08:34:33.740 INFO:teuthology.orchestra.run.smithi045.stdout:(144/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2024-09-13T08:34:33.768 INFO:teuthology.orchestra.run.smithi045.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 706 kB/s | 20 kB 00:00 2024-09-13T08:34:33.802 INFO:teuthology.orchestra.run.smithi045.stdout:(146/157): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T08:34:33.828 INFO:teuthology.orchestra.run.smithi045.stdout:(147/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-13T08:34:33.886 INFO:teuthology.orchestra.run.smithi045.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-13T08:34:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:(149/157): librados2-19.3.0-4881.gd454760e.el9. 12 MB/s | 3.5 MB 00:00 2024-09-13T08:34:34.212 INFO:teuthology.orchestra.run.smithi045.stdout:(150/157): librbd1-19.3.0-4881.gd454760e.el9.x8 10 MB/s | 3.2 MB 00:00 2024-09-13T08:34:34.687 INFO:teuthology.orchestra.run.smithi045.stdout:(151/157): NetworkManager-libnm-1.49.90-1.el9.x 3.8 MB/s | 1.8 MB 00:00 2024-09-13T08:34:34.737 INFO:teuthology.orchestra.run.smithi045.stdout:(152/157): NetworkManager-1.49.90-1.el9.x86_64. 3.8 MB/s | 2.3 MB 00:00 2024-09-13T08:34:34.763 INFO:teuthology.orchestra.run.smithi045.stdout:(153/157): NetworkManager-team-1.49.90-1.el9.x8 544 kB/s | 41 kB 00:00 2024-09-13T08:34:34.838 INFO:teuthology.orchestra.run.smithi045.stdout:(154/157): NetworkManager-tui-1.49.90-1.el9.x86 2.4 MB/s | 249 kB 00:00 2024-09-13T08:34:35.155 INFO:teuthology.orchestra.run.smithi045.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 3.5 MB/s | 1.4 MB 00:00 2024-09-13T08:34:35.382 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T08:34:35.383 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T08:34:35.396 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T08:34:35.397 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T08:34:35.398 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T08:34:35.399 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T08:34:35.401 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T08:34:35.404 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T08:34:35.405 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T08:34:35.478 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-13T08:34:35.498 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-13T08:34:35.498 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-13T08:34:35.498 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-13T08:34:35.498 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.3.0-4881.gd454760e.el9 ceph 6.6 k 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.3.0-4881.gd454760e.el9 ceph 5.5 M 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 899 k 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4881.gd454760e.el9 ceph 145 k 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.3.0-4881.gd454760e.el9 ceph 1.5 M 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 154 k 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 14 M 2024-09-13T08:34:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 7.4 M 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 49 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.3.0-4881.gd454760e.el9 ceph 11 M 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.3.0-4881.gd454760e.el9 ceph 50 M 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 285 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 784 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.3.0-4881.gd454760e.el9 ceph 225 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.3.0-4881.gd454760e.el9 ceph 27 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 18 k 2024-09-13T08:34:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.3.0-4881.gd454760e.el9 ceph 32 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 756 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.3.0-4881.gd454760e.el9 ceph 126 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T08:34:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.3.0-4881.gd454760e.el9 ceph 161 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.3.0-4881.gd454760e.el9 ceph 320 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 305 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.3.0-4881.gd454760e.el9 ceph 100 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 85 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.2 M 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 170 k 2024-09-13T08:34:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.5 M 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 3.2 M 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T08:34:35.503 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.3.0-4881.gd454760e.el9 ceph 22 M 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 34 k 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.3.0-4881.gd454760e.el9 ceph 2.4 M 2024-09-13T08:34:35.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 291 k 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.3.0-4881.gd454760e.el9 ceph 4.8 M 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.3.0-4881.gd454760e.el9 ceph 17 M 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4881.gd454760e.el9 ceph-noarch 16 k 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.3.0-4881.gd454760e.el9 ceph 25 k 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T08:34:35.505 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.3.0-4881.gd454760e.el9 ceph 161 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T08:34:35.506 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 508 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.3.0-4881.gd454760e.el9 ceph 5.5 M 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T08:34:35.507 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T08:34:35.508 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T08:34:35.508 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T08:34:35.508 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T08:34:35.508 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T08:34:35.508 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T08:34:35.509 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T08:34:35.509 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T08:34:35.510 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T08:34:35.510 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T08:34:35.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.3.0-4881.gd454760e.el9 ceph 45 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.3.0-4881.gd454760e.el9 ceph 148 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T08:34:35.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T08:34:35.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T08:34:35.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T08:34:35.514 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T08:34:35.515 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T08:34:35.516 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T08:34:35.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:34:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-13T08:34:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-13T08:34:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:Install 149 Packages 2024-09-13T08:34:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 8 Packages 2024-09-13T08:34:35.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:34:35.519 INFO:teuthology.orchestra.run.smithi045.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.5 MB/s | 2.4 MB 00:00 2024-09-13T08:34:35.519 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 240 M 2024-09-13T08:34:35.519 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-13T08:34:36.044 INFO:teuthology.orchestra.run.smithi031.stdout:(1/157): ceph-19.3.0-4881.gd454760e.el9.x86_64. 44 kB/s | 6.6 kB 00:00 2024-09-13T08:34:36.228 INFO:teuthology.orchestra.run.smithi031.stdout:(2/157): ceph-fuse-19.3.0-4881.gd454760e.el9.x8 4.8 MB/s | 899 kB 00:00 2024-09-13T08:34:36.312 INFO:teuthology.orchestra.run.smithi031.stdout:(3/157): ceph-immutable-object-cache-19.3.0-488 1.7 MB/s | 145 kB 00:00 2024-09-13T08:34:36.387 INFO:teuthology.orchestra.run.smithi031.stdout:(4/157): ceph-base-19.3.0-4881.gd454760e.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-13T08:34:36.422 INFO:teuthology.orchestra.run.smithi045.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.2 MB/s | 16 MB 00:05 2024-09-13T08:34:36.427 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-13T08:34:36.427 INFO:teuthology.orchestra.run.smithi045.stdout:Total 15 MB/s | 240 MB 00:15 2024-09-13T08:34:36.487 INFO:teuthology.orchestra.run.smithi031.stdout:(5/157): ceph-mgr-19.3.0-4881.gd454760e.el9.x86 15 MB/s | 1.5 MB 00:00 2024-09-13T08:34:36.537 INFO:teuthology.orchestra.run.smithi031.stdout:(6/157): ceph-mds-19.3.0-4881.gd454760e.el9.x86 11 MB/s | 2.4 MB 00:00 2024-09-13T08:34:37.088 INFO:teuthology.orchestra.run.smithi031.stdout:(7/157): ceph-common-19.3.0-4881.gd454760e.el9. 18 MB/s | 22 MB 00:01 2024-09-13T08:34:37.163 INFO:teuthology.orchestra.run.smithi031.stdout:(8/157): ceph-mon-19.3.0-4881.gd454760e.el9.x86 7.1 MB/s | 4.8 MB 00:00 2024-09-13T08:34:37.205 INFO:teuthology.orchestra.run.smithi031.stdout:(9/157): ceph-selinux-19.3.0-4881.gd454760e.el9 608 kB/s | 25 kB 00:00 2024-09-13T08:34:37.855 INFO:teuthology.orchestra.run.smithi031.stdout:(10/157): ceph-osd-19.3.0-4881.gd454760e.el9.x8 13 MB/s | 17 MB 00:01 2024-09-13T08:34:37.989 INFO:teuthology.orchestra.run.smithi031.stdout:(11/157): ceph-radosgw-19.3.0-4881.gd454760e.el 12 MB/s | 11 MB 00:00 2024-09-13T08:34:38.014 INFO:teuthology.orchestra.run.smithi031.stdout:(12/157): cephfs-mirror-19.3.0-4881.gd454760e.e 1.4 MB/s | 225 kB 00:00 2024-09-13T08:34:38.039 INFO:teuthology.orchestra.run.smithi031.stdout:(13/157): cephfs-shell-19.3.0-4881.gd454760e.el 543 kB/s | 27 kB 00:00 2024-09-13T08:34:38.064 INFO:teuthology.orchestra.run.smithi031.stdout:(14/157): libcephfs-devel-19.3.0-4881.gd454760e 636 kB/s | 32 kB 00:00 2024-09-13T08:34:38.115 INFO:teuthology.orchestra.run.smithi031.stdout:(15/157): libcephsqlite-19.3.0-4881.gd454760e.e 3.2 MB/s | 161 kB 00:00 2024-09-13T08:34:38.140 INFO:teuthology.orchestra.run.smithi031.stdout:(16/157): libcephfs2-19.3.0-4881.gd454760e.el9. 7.3 MB/s | 756 kB 00:00 2024-09-13T08:34:38.173 INFO:teuthology.orchestra.run.smithi031.stdout:(17/157): librados-devel-19.3.0-4881.gd454760e. 2.1 MB/s | 126 kB 00:00 2024-09-13T08:34:38.207 INFO:teuthology.orchestra.run.smithi031.stdout:(18/157): libradosstriper1-19.3.0-4881.gd454760 7.5 MB/s | 508 kB 00:00 2024-09-13T08:34:38.249 INFO:teuthology.orchestra.run.smithi031.stdout:(19/157): python3-ceph-argparse-19.3.0-4881.gd4 1.1 MB/s | 45 kB 00:00 2024-09-13T08:34:38.316 INFO:teuthology.orchestra.run.smithi031.stdout:(20/157): python3-ceph-common-19.3.0-4881.gd454 2.2 MB/s | 148 kB 00:00 2024-09-13T08:34:38.383 INFO:teuthology.orchestra.run.smithi031.stdout:(21/157): python3-cephfs-19.3.0-4881.gd454760e. 2.4 MB/s | 161 kB 00:00 2024-09-13T08:34:38.466 INFO:teuthology.orchestra.run.smithi031.stdout:(22/157): python3-rados-19.3.0-4881.gd454760e.e 3.8 MB/s | 320 kB 00:00 2024-09-13T08:34:38.549 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-13T08:34:38.583 INFO:teuthology.orchestra.run.smithi031.stdout:(23/157): librgw2-19.3.0-4881.gd454760e.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-13T08:34:38.608 INFO:teuthology.orchestra.run.smithi031.stdout:(24/157): python3-rbd-19.3.0-4881.gd454760e.el9 2.1 MB/s | 305 kB 00:00 2024-09-13T08:34:38.633 INFO:teuthology.orchestra.run.smithi031.stdout:(25/157): python3-rgw-19.3.0-4881.gd454760e.el9 2.0 MB/s | 100 kB 00:00 2024-09-13T08:34:38.659 INFO:teuthology.orchestra.run.smithi031.stdout:(26/157): rbd-fuse-19.3.0-4881.gd454760e.el9.x8 1.7 MB/s | 85 kB 00:00 2024-09-13T08:34:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:(27/157): rbd-nbd-19.3.0-4881.gd454760e.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-13T08:34:38.748 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-13T08:34:38.748 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-13T08:34:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:(28/157): ceph-grafana-dashboards-19.3.0-4881.g 413 kB/s | 34 kB 00:00 2024-09-13T08:34:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:(29/157): rbd-mirror-19.3.0-4881.gd454760e.el9. 13 MB/s | 3.2 MB 00:00 2024-09-13T08:34:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4881.gd454760 1.4 MB/s | 154 kB 00:00 2024-09-13T08:34:39.535 INFO:teuthology.orchestra.run.smithi031.stdout:(31/157): ceph-test-19.3.0-4881.gd454760e.el9.x 22 MB/s | 50 MB 00:02 2024-09-13T08:34:39.586 INFO:teuthology.orchestra.run.smithi031.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4881.gd4 5.7 MB/s | 291 kB 00:00 2024-09-13T08:34:39.636 INFO:teuthology.orchestra.run.smithi031.stdout:(33/157): ceph-mgr-rook-19.3.0-4881.gd454760e.e 997 kB/s | 49 kB 00:00 2024-09-13T08:34:39.761 INFO:teuthology.orchestra.run.smithi031.stdout:(34/157): ceph-mgr-diskprediction-local-19.3.0- 8.6 MB/s | 7.4 MB 00:00 2024-09-13T08:34:39.787 INFO:teuthology.orchestra.run.smithi031.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4881.gd 109 kB/s | 16 kB 00:00 2024-09-13T08:34:39.812 INFO:teuthology.orchestra.run.smithi031.stdout:(36/157): ceph-volume-19.3.0-4881.gd454760e.el9 5.5 MB/s | 285 kB 00:00 2024-09-13T08:34:39.862 INFO:teuthology.orchestra.run.smithi031.stdout:(37/157): cephadm-19.3.0-4881.gd454760e.el9.noa 10 MB/s | 784 kB 00:00 2024-09-13T08:34:39.887 INFO:teuthology.orchestra.run.smithi031.stdout:(38/157): cephfs-top-19.3.0-4881.gd454760e.el9. 244 kB/s | 18 kB 00:00 2024-09-13T08:34:40.104 INFO:teuthology.orchestra.run.smithi031.stdout:(39/157): ceph-mgr-dashboard-19.3.0-4881.gd4547 12 MB/s | 14 MB 00:01 2024-09-13T08:34:40.129 INFO:teuthology.orchestra.run.smithi031.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 299 kB/s | 80 kB 00:00 2024-09-13T08:34:40.163 INFO:teuthology.orchestra.run.smithi031.stdout:(41/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-13T08:34:40.188 INFO:teuthology.orchestra.run.smithi031.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-13T08:34:40.230 INFO:teuthology.orchestra.run.smithi031.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T08:34:40.264 INFO:teuthology.orchestra.run.smithi031.stdout:(44/157): python3-pycparser-2.20-6.el9.noarch.r 4.0 MB/s | 135 kB 00:00 2024-09-13T08:34:40.289 INFO:teuthology.orchestra.run.smithi031.stdout:(45/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-13T08:34:40.314 INFO:teuthology.orchestra.run.smithi031.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-13T08:34:40.348 INFO:teuthology.orchestra.run.smithi031.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-13T08:34:40.407 INFO:teuthology.orchestra.run.smithi031.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 5.0 MB/s | 1.2 MB 00:00 2024-09-13T08:34:40.590 INFO:teuthology.orchestra.run.smithi031.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 584 kB/s | 107 kB 00:00 2024-09-13T08:34:40.624 INFO:teuthology.orchestra.run.smithi031.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-09-13T08:34:40.649 INFO:teuthology.orchestra.run.smithi031.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 371 kB/s | 22 kB 00:00 2024-09-13T08:34:40.716 INFO:teuthology.orchestra.run.smithi031.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 943 kB 00:00 2024-09-13T08:34:40.741 INFO:teuthology.orchestra.run.smithi031.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-13T08:34:40.766 INFO:teuthology.orchestra.run.smithi031.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-09-13T08:34:40.791 INFO:teuthology.orchestra.run.smithi031.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-09-13T08:34:40.825 INFO:teuthology.orchestra.run.smithi031.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-09-13T08:34:40.875 INFO:teuthology.orchestra.run.smithi031.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.0 MB/s | 171 kB 00:00 2024-09-13T08:34:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-09-13T08:34:40.918 INFO:teuthology.orchestra.run.smithi031.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-13T08:34:40.951 INFO:teuthology.orchestra.run.smithi031.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-09-13T08:34:41.002 INFO:teuthology.orchestra.run.smithi031.stdout:(61/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 491 kB/s | 24 kB 00:00 2024-09-13T08:34:41.035 INFO:teuthology.orchestra.run.smithi031.stdout:(62/157): libtool-2.4.6-46.el9.x86_64.rpm 4.9 MB/s | 581 kB 00:00 2024-09-13T08:34:41.060 INFO:teuthology.orchestra.run.smithi031.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-09-13T08:34:41.094 INFO:teuthology.orchestra.run.smithi031.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-13T08:34:41.144 INFO:teuthology.orchestra.run.smithi031.stdout:(65/157): openblas-0.3.26-2.el9.x86_64.rpm 775 kB/s | 38 kB 00:00 2024-09-13T08:34:41.169 INFO:teuthology.orchestra.run.smithi031.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-13T08:34:41.362 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-13T08:34:41.363 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-13T08:34:41.987 INFO:teuthology.orchestra.run.smithi031.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.4 MB/s | 3.0 MB 00:01 2024-09-13T08:34:42.779 INFO:teuthology.orchestra.run.smithi031.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-09-13T08:34:43.237 INFO:teuthology.orchestra.run.smithi031.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 1.9 MB/s | 910 kB 00:00 2024-09-13T08:34:43.329 INFO:teuthology.orchestra.run.smithi031.stdout:(70/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.3 MB/s | 4.9 MB 00:02 2024-09-13T08:34:43.421 INFO:teuthology.orchestra.run.smithi031.stdout:(71/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-09-13T08:34:43.446 INFO:teuthology.orchestra.run.smithi031.stdout:(72/157): postgresql-private-libs-13.16-1.el9.x 658 kB/s | 137 kB 00:00 2024-09-13T08:34:43.530 INFO:teuthology.orchestra.run.smithi031.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-09-13T08:34:43.963 INFO:teuthology.orchestra.run.smithi031.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 1.9 MB/s | 1.0 MB 00:00 2024-09-13T08:34:44.022 INFO:teuthology.orchestra.run.smithi031.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-09-13T08:34:44.172 INFO:teuthology.orchestra.run.smithi031.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-13T08:34:44.322 INFO:teuthology.orchestra.run.smithi031.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-09-13T08:34:44.373 INFO:teuthology.orchestra.run.smithi031.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-09-13T08:34:44.423 INFO:teuthology.orchestra.run.smithi031.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 385 kB/s | 19 kB 00:00 2024-09-13T08:34:44.556 INFO:teuthology.orchestra.run.smithi031.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T08:34:44.607 INFO:teuthology.orchestra.run.smithi031.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-09-13T08:34:44.915 INFO:teuthology.orchestra.run.smithi031.stdout:(82/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-13T08:34:44.974 INFO:teuthology.orchestra.run.smithi031.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-13T08:34:45.032 INFO:teuthology.orchestra.run.smithi031.stdout:(84/157): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-09-13T08:34:45.091 INFO:teuthology.orchestra.run.smithi031.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-13T08:34:45.141 INFO:teuthology.orchestra.run.smithi031.stdout:(86/157): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-09-13T08:34:45.200 INFO:teuthology.orchestra.run.smithi031.stdout:(87/157): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-13T08:34:45.251 INFO:teuthology.orchestra.run.smithi031.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 159 kB 00:00 2024-09-13T08:34:45.308 INFO:teuthology.orchestra.run.smithi031.stdout:(89/157): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-13T08:34:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T08:34:45.409 INFO:teuthology.orchestra.run.smithi031.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-13T08:34:45.784 INFO:teuthology.orchestra.run.smithi031.stdout:(92/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-13T08:34:45.843 INFO:teuthology.orchestra.run.smithi031.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 717 kB/s | 42 kB 00:00 2024-09-13T08:34:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-13T08:34:45.943 INFO:teuthology.orchestra.run.smithi031.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-13T08:34:46.069 INFO:teuthology.orchestra.run.smithi031.stdout:(96/157): postgresql-server-13.16-1.el9.x86_64. 2.1 MB/s | 5.8 MB 00:02 2024-09-13T08:34:46.086 INFO:teuthology.orchestra.run.smithi031.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 253 kB/s | 36 kB 00:00 2024-09-13T08:34:46.111 INFO:teuthology.orchestra.run.smithi031.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-13T08:34:46.195 INFO:teuthology.orchestra.run.smithi031.stdout:(99/157): python3-babel-2.9.1-2.el9.noarch.rpm 2.2 MB/s | 6.0 MB 00:02 2024-09-13T08:34:46.211 INFO:teuthology.orchestra.run.smithi031.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 577 kB/s | 58 kB 00:00 2024-09-13T08:34:46.237 INFO:teuthology.orchestra.run.smithi031.stdout:(101/157): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-13T08:34:46.279 INFO:teuthology.orchestra.run.smithi031.stdout:(102/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.1 MB/s | 96 kB 00:00 2024-09-13T08:34:46.294 INFO:teuthology.orchestra.run.smithi031.stdout:(103/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 772 kB/s | 64 kB 00:00 2024-09-13T08:34:46.311 INFO:teuthology.orchestra.run.smithi031.stdout:(104/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-13T08:34:46.362 INFO:teuthology.orchestra.run.smithi031.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-13T08:34:46.396 INFO:teuthology.orchestra.run.smithi031.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.0 MB/s | 308 kB 00:00 2024-09-13T08:34:46.413 INFO:teuthology.orchestra.run.smithi031.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-13T08:34:46.529 INFO:teuthology.orchestra.run.smithi031.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-09-13T08:34:46.563 INFO:teuthology.orchestra.run.smithi031.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.8 MB/s | 26 kB 00:00 2024-09-13T08:34:46.580 INFO:teuthology.orchestra.run.smithi031.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-09-13T08:34:46.605 INFO:teuthology.orchestra.run.smithi031.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-13T08:34:46.630 INFO:teuthology.orchestra.run.smithi031.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-13T08:34:46.673 INFO:teuthology.orchestra.run.smithi031.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-13T08:34:46.706 INFO:teuthology.orchestra.run.smithi031.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-13T08:34:46.731 INFO:teuthology.orchestra.run.smithi031.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-13T08:34:46.756 INFO:teuthology.orchestra.run.smithi031.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-13T08:34:46.782 INFO:teuthology.orchestra.run.smithi031.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 560 kB/s | 14 kB 00:00 2024-09-13T08:34:46.807 INFO:teuthology.orchestra.run.smithi031.stdout:(118/157): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-09-13T08:34:46.841 INFO:teuthology.orchestra.run.smithi031.stdout:(119/157): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-09-13T08:34:46.874 INFO:teuthology.orchestra.run.smithi031.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 8.9 MB/s | 303 kB 00:00 2024-09-13T08:34:46.899 INFO:teuthology.orchestra.run.smithi031.stdout:(121/157): python3-google-auth-2.34.0-1.el9.noa 8.6 MB/s | 220 kB 00:00 2024-09-13T08:34:46.925 INFO:teuthology.orchestra.run.smithi031.stdout:(122/157): lua-devel-5.4.4-4.el9.x86_64.rpm 34 kB/s | 22 kB 00:00 2024-09-13T08:34:46.975 INFO:teuthology.orchestra.run.smithi031.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-13T08:34:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-13T08:34:47.025 INFO:teuthology.orchestra.run.smithi031.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 713 kB/s | 18 kB 00:00 2024-09-13T08:34:47.053 INFO:teuthology.orchestra.run.smithi031.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 930 kB/s | 23 kB 00:00 2024-09-13T08:34:47.076 INFO:teuthology.orchestra.run.smithi031.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 781 kB/s | 19 kB 00:00 2024-09-13T08:34:47.101 INFO:teuthology.orchestra.run.smithi031.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 787 kB/s | 20 kB 00:00 2024-09-13T08:34:47.160 INFO:teuthology.orchestra.run.smithi031.stdout:(129/157): python3-grpcio-1.46.7-10.el9.x86_64. 7.9 MB/s | 2.0 MB 00:00 2024-09-13T08:34:47.185 INFO:teuthology.orchestra.run.smithi031.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-13T08:34:47.210 INFO:teuthology.orchestra.run.smithi031.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-13T08:34:47.236 INFO:teuthology.orchestra.run.smithi031.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-13T08:34:47.278 INFO:teuthology.orchestra.run.smithi031.stdout:(133/157): protobuf-compiler-3.14.0-13.el9.x86_ 878 kB/s | 863 kB 00:00 2024-09-13T08:34:47.303 INFO:teuthology.orchestra.run.smithi031.stdout:(134/157): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-13T08:34:47.337 INFO:teuthology.orchestra.run.smithi031.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.8 MB/s | 90 kB 00:00 2024-09-13T08:34:47.370 INFO:teuthology.orchestra.run.smithi031.stdout:(136/157): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-09-13T08:34:47.418 INFO:teuthology.orchestra.run.smithi031.stdout:(137/157): python3-portend-3.1.0-2.el9.noarch.r 117 kB/s | 16 kB 00:00 2024-09-13T08:34:47.446 INFO:teuthology.orchestra.run.smithi031.stdout:(138/157): python3-pyperclip-1.8.2-2.el9.noarch 264 kB/s | 28 kB 00:00 2024-09-13T08:34:47.463 INFO:teuthology.orchestra.run.smithi031.stdout:(139/157): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-09-13T08:34:47.480 INFO:teuthology.orchestra.run.smithi031.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-13T08:34:47.497 INFO:teuthology.orchestra.run.smithi031.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-09-13T08:34:47.522 INFO:teuthology.orchestra.run.smithi031.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-13T08:34:47.547 INFO:teuthology.orchestra.run.smithi031.stdout:(143/157): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-09-13T08:34:47.573 INFO:teuthology.orchestra.run.smithi031.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-13T08:34:47.597 INFO:teuthology.orchestra.run.smithi031.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T08:34:47.614 INFO:teuthology.orchestra.run.smithi031.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 481 kB/s | 20 kB 00:00 2024-09-13T08:34:47.631 INFO:teuthology.orchestra.run.smithi031.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T08:34:47.656 INFO:teuthology.orchestra.run.smithi031.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-09-13T08:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-13T08:34:47.924 INFO:teuthology.orchestra.run.smithi031.stdout:(150/157): librados2-19.3.0-4881.gd454760e.el9. 12 MB/s | 3.5 MB 00:00 2024-09-13T08:34:47.974 INFO:teuthology.orchestra.run.smithi031.stdout:(151/157): librbd1-19.3.0-4881.gd454760e.el9.x8 10 MB/s | 3.2 MB 00:00 2024-09-13T08:34:48.033 INFO:teuthology.orchestra.run.smithi031.stdout:(152/157): NetworkManager-team-1.49.90-1.el9.x8 700 kB/s | 41 kB 00:00 2024-09-13T08:34:48.133 INFO:teuthology.orchestra.run.smithi031.stdout:(153/157): NetworkManager-1.49.90-1.el9.x86_64. 5.6 MB/s | 2.3 MB 00:00 2024-09-13T08:34:48.158 INFO:teuthology.orchestra.run.smithi031.stdout:(154/157): NetworkManager-tui-1.49.90-1.el9.x86 1.9 MB/s | 249 kB 00:00 2024-09-13T08:34:48.200 INFO:teuthology.orchestra.run.smithi031.stdout:(155/157): NetworkManager-libnm-1.49.90-1.el9.x 6.5 MB/s | 1.8 MB 00:00 2024-09-13T08:34:48.242 INFO:teuthology.orchestra.run.smithi031.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 12 MB/s | 1.4 MB 00:00 2024-09-13T08:34:48.334 INFO:teuthology.orchestra.run.smithi031.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-09-13T08:34:48.339 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-13T08:34:48.475 INFO:teuthology.orchestra.run.smithi031.stdout:Total 19 MB/s | 240 MB 00:12 2024-09-13T08:34:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-13T08:34:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-13T08:34:49.232 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-13T08:34:49.340 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-13T08:34:49.389 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-13T08:34:49.420 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-13T08:34:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-13T08:34:49.875 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-13T08:34:50.070 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:34:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:34:50.371 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:34:50.462 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-13T08:34:50.478 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:34:50.517 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 10/165 2024-09-13T08:34:50.550 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-13T08:34:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-13T08:34:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-13T08:34:50.802 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-13T08:34:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-13T08:34:50.893 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:34:51.024 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:34:51.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 15/165 2024-09-13T08:34:51.143 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 15/165 2024-09-13T08:34:51.170 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-13T08:34:51.216 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-13T08:34:51.415 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-13T08:34:51.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-13T08:34:51.804 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-13T08:34:51.933 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:34:52.038 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:34:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:34:52.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-13T08:34:52.519 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-13T08:34:52.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-13T08:34:52.650 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-13T08:34:52.838 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-13T08:34:52.898 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-13T08:34:52.961 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-13T08:34:53.008 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-13T08:34:53.041 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-13T08:34:53.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-13T08:34:53.195 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-13T08:34:53.255 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-13T08:34:53.308 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-13T08:34:53.339 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-13T08:34:53.340 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-13T08:34:53.406 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-13T08:34:53.440 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.3.0-4881.gd454760e.el9. 34/165 2024-09-13T08:34:53.486 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.3.0-4881.gd454760e.el 35/165 2024-09-13T08:34:53.512 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_6 36/165 2024-09-13T08:34:53.605 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-13T08:34:53.658 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-13T08:34:53.752 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-13T08:34:53.924 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-13T08:34:53.987 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-13T08:34:54.660 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-13T08:34:54.764 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-13T08:34:54.796 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-13T08:34:54.841 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-13T08:34:54.880 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.3.0-4881.gd454760e.el9.x86_6 46/165 2024-09-13T08:34:54.929 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-13T08:34:55.142 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-13T08:34:55.216 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-13T08:34:55.336 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-13T08:34:55.433 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-13T08:34:55.527 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-13T08:34:55.563 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-13T08:34:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-13T08:34:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-13T08:34:55.699 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-13T08:34:55.745 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-13T08:34:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-13T08:34:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-13T08:34:56.079 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-13T08:34:56.122 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-13T08:34:56.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-13T08:34:56.296 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-13T08:34:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-13T08:34:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-13T08:34:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-13T08:34:56.508 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-13T08:34:56.676 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-13T08:34:56.717 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-13T08:34:56.762 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-13T08:34:56.803 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-13T08:34:56.846 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-13T08:34:56.885 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-13T08:34:56.921 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-13T08:34:56.956 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-13T08:34:56.990 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-13T08:34:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-13T08:34:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-13T08:34:57.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-13T08:34:57.258 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-13T08:34:57.295 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-13T08:34:57.423 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-13T08:34:57.492 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-13T08:34:58.452 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-13T08:34:58.519 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-13T08:34:58.553 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-13T08:34:58.672 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-13T08:34:58.745 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-13T08:34:58.798 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-13T08:34:58.907 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-13T08:34:58.947 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-13T08:34:59.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-13T08:34:59.251 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-13T08:34:59.350 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-13T08:35:00.067 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-13T08:35:00.210 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-13T08:35:00.213 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-13T08:35:00.267 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-13T08:35:00.341 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-13T08:35:00.363 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-13T08:35:00.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-13T08:35:00.414 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-13T08:35:00.469 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-13T08:35:00.863 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-13T08:35:00.964 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:35:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:35:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:35:01.259 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:35:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 10/165 2024-09-13T08:35:01.336 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-13T08:35:01.552 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-13T08:35:01.612 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-13T08:35:01.637 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:35:01.771 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:35:01.826 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 15/165 2024-09-13T08:35:01.912 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 15/165 2024-09-13T08:35:01.939 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-13T08:35:01.986 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-13T08:35:02.009 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-13T08:35:02.101 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-13T08:35:02.143 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-13T08:35:02.183 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-13T08:35:02.367 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-13T08:35:02.529 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-13T08:35:02.545 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-13T08:35:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:35:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:35:02.915 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-13T08:35:02.972 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 102/165 2024-09-13T08:35:03.094 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 102/165 2024-09-13T08:35:03.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 103/165 2024-09-13T08:35:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-13T08:35:03.187 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-13T08:35:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-13T08:35:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-13T08:35:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-13T08:35:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-13T08:35:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-13T08:35:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-13T08:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-13T08:35:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-13T08:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 106/165 2024-09-13T08:35:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-13T08:35:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-13T08:35:03.884 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 106/165 2024-09-13T08:35:03.911 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 107/165 2024-09-13T08:35:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-13T08:35:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-13T08:35:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-13T08:35:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-13T08:35:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-13T08:35:04.230 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.3.0-4881.gd454760e.el9. 34/165 2024-09-13T08:35:04.275 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.3.0-4881.gd454760e.el 35/165 2024-09-13T08:35:04.302 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_6 36/165 2024-09-13T08:35:04.394 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-13T08:35:04.449 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-13T08:35:04.541 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-13T08:35:04.712 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-13T08:35:04.775 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-13T08:35:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-13T08:35:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-13T08:35:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-13T08:35:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-13T08:35:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.3.0-4881.gd454760e.el9.x86_6 46/165 2024-09-13T08:35:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-13T08:35:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-13T08:35:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-13T08:35:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-13T08:35:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-13T08:35:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-13T08:35:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-13T08:35:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-13T08:35:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-13T08:35:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-13T08:35:06.562 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-13T08:35:06.639 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-13T08:35:06.829 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-13T08:35:06.872 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-13T08:35:06.916 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-13T08:35:07.045 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-13T08:35:07.081 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-13T08:35:07.123 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-13T08:35:07.152 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-13T08:35:07.177 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-13T08:35:07.291 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-13T08:35:07.311 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:07.353 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 109/165 2024-09-13T08:35:07.460 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-13T08:35:07.485 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 109/165 2024-09-13T08:35:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T08:35:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T08:35:07.485 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:07.501 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-13T08:35:07.530 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 110/165 2024-09-13T08:35:07.538 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-13T08:35:07.628 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-13T08:35:07.672 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-13T08:35:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-13T08:35:07.757 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-13T08:35:07.792 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-13T08:35:07.825 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-13T08:35:07.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-13T08:35:07.918 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-13T08:35:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-13T08:35:08.077 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-13T08:35:08.112 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-13T08:35:08.240 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-13T08:35:08.310 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-13T08:35:09.243 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-13T08:35:09.294 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-13T08:35:09.327 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-13T08:35:09.445 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-13T08:35:09.532 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-13T08:35:09.589 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-13T08:35:09.698 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-13T08:35:09.738 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-13T08:35:09.984 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-13T08:35:10.017 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-13T08:35:10.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-13T08:35:10.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-13T08:35:11.126 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-13T08:35:12.787 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-13T08:35:12.888 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-13T08:35:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-13T08:35:12.957 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-13T08:35:13.325 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-13T08:35:13.368 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 102/165 2024-09-13T08:35:13.481 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 102/165 2024-09-13T08:35:13.516 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 103/165 2024-09-13T08:35:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-13T08:35:14.132 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-13T08:35:14.162 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 106/165 2024-09-13T08:35:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 106/165 2024-09-13T08:35:14.331 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 107/165 2024-09-13T08:35:17.003 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 108/165 2024-09-13T08:35:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 109/165 2024-09-13T08:35:17.945 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 109/165 2024-09-13T08:35:17.945 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T08:35:17.945 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T08:35:17.945 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:17.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 110/165 2024-09-13T08:35:21.368 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 110/165 2024-09-13T08:35:21.368 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-09-13T08:35:21.368 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-09-13T08:35:21.368 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-09-13T08:35:21.368 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-09-13T08:35:21.369 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-09-13T08:35:21.369 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-09-13T08:35:21.369 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-09-13T08:35:21.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 111/165 2024-09-13T08:35:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 111/165 2024-09-13T08:35:21.626 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:21.626 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T08:35:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:23.338 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 112/165 2024-09-13T08:35:23.397 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 112/165 2024-09-13T08:35:23.397 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:23.397 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T08:35:23.397 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T08:35:23.397 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T08:35:23.398 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:23.658 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 113/165 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 113/165 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T08:35:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:24.225 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 114/165 2024-09-13T08:35:24.277 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 114/165 2024-09-13T08:35:24.277 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:24.277 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T08:35:24.277 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T08:35:24.278 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T08:35:24.278 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:24.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-13T08:35:24.480 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-13T08:35:24.553 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-13T08:35:24.806 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-13T08:35:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-13T08:35:24.962 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-13T08:35:25.087 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-13T08:35:25.268 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-13T08:35:25.332 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-13T08:35:25.363 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-13T08:35:25.396 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-13T08:35:26.002 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.3.0-4881.gd454760e.el9.noarch 126/165 2024-09-13T08:35:26.038 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.3.0-4881.gd454760e.el9.noarch 126/165 2024-09-13T08:35:26.080 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.e 127/165 2024-09-13T08:35:26.164 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4881.gd454760e. 128/165 2024-09-13T08:35:26.193 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 129/165 2024-09-13T08:35:30.035 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 129/165 2024-09-13T08:35:30.100 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 130/165 2024-09-13T08:35:31.252 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 130/165 2024-09-13T08:35:31.283 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 131/165 2024-09-13T08:35:31.448 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 131/165 2024-09-13T08:35:31.649 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el 132/165 2024-09-13T08:35:31.678 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 133/165 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 133/165 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T08:35:31.731 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:31.787 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 134/165 2024-09-13T08:35:31.814 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 134/165 2024-09-13T08:35:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 110/165 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-09-13T08:35:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:31.926 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.3.0-4881.gd454760e.el9.x86_64 135/165 2024-09-13T08:35:32.000 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 136/165 2024-09-13T08:35:32.125 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 111/165 2024-09-13T08:35:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 111/165 2024-09-13T08:35:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T08:35:32.163 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:33.098 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-13T08:35:33.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 138/165 2024-09-13T08:35:33.184 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 138/165 2024-09-13T08:35:33.185 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:33.185 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T08:35:33.185 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T08:35:33.185 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T08:35:33.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:33.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4881.gd4547 139/165 2024-09-13T08:35:33.283 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4881.gd4547 139/165 2024-09-13T08:35:33.283 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:33.283 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T08:35:33.283 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:33.333 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 140/165 2024-09-13T08:35:33.384 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 140/165 2024-09-13T08:35:33.385 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:33.385 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T08:35:33.385 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T08:35:33.385 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T08:35:33.385 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:33.759 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 141/165 2024-09-13T08:35:33.810 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 141/165 2024-09-13T08:35:33.811 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:33.811 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T08:35:33.811 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T08:35:33.811 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T08:35:33.811 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 112/165 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 112/165 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T08:35:33.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:34.170 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 113/165 2024-09-13T08:35:34.221 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 113/165 2024-09-13T08:35:34.221 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:34.221 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T08:35:34.221 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T08:35:34.221 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T08:35:34.222 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 114/165 2024-09-13T08:35:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 114/165 2024-09-13T08:35:34.789 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:34.790 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T08:35:34.790 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T08:35:34.790 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T08:35:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:34.922 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-13T08:35:34.983 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-13T08:35:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-13T08:35:35.292 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-13T08:35:35.350 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-13T08:35:35.441 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-13T08:35:35.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-13T08:35:35.736 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-13T08:35:35.783 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-13T08:35:35.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-13T08:35:35.856 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-13T08:35:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.3.0-4881.gd454760e.el9.noarch 126/165 2024-09-13T08:35:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.3.0-4881.gd454760e.el9.noarch 126/165 2024-09-13T08:35:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.e 127/165 2024-09-13T08:35:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4881.gd454760e. 128/165 2024-09-13T08:35:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 129/165 2024-09-13T08:35:39.224 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 142/165 2024-09-13T08:35:39.266 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 143/165 2024-09-13T08:35:39.310 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 144/165 2024-09-13T08:35:39.346 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-13T08:35:39.382 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-13T08:35:39.628 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 147/165 2024-09-13T08:35:39.679 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-13T08:35:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_ 149/165 2024-09-13T08:35:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-13T08:35:39.784 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-13T08:35:40.671 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 129/165 2024-09-13T08:35:40.733 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 130/165 2024-09-13T08:35:40.737 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:40.825 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:41.886 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 130/165 2024-09-13T08:35:41.925 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 131/165 2024-09-13T08:35:42.085 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 131/165 2024-09-13T08:35:42.210 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 153/165 2024-09-13T08:35:42.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el 132/165 2024-09-13T08:35:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 133/165 2024-09-13T08:35:42.323 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 133/165 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T08:35:42.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:42.383 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-13T08:35:42.411 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 134/165 2024-09-13T08:35:42.437 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 134/165 2024-09-13T08:35:42.500 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-13T08:35:42.549 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.3.0-4881.gd454760e.el9.x86_64 135/165 2024-09-13T08:35:42.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-13T08:35:42.610 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-13T08:35:42.624 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 136/165 2024-09-13T08:35:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-13T08:35:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-13T08:35:42.745 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-13T08:35:42.745 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-13T08:35:42.838 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-13T08:35:42.862 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-13T08:35:42.878 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:42.878 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:43.131 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:43.132 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-13T08:35:43.233 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-13T08:35:43.317 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:35:43.520 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:35:43.521 INFO:teuthology.orchestra.run.smithi045.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-13T08:35:43.521 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:43.727 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-13T08:35:43.755 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 138/165 2024-09-13T08:35:43.806 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 138/165 2024-09-13T08:35:43.807 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:43.807 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T08:35:43.807 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T08:35:43.807 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T08:35:43.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:43.855 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4881.gd4547 139/165 2024-09-13T08:35:43.906 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4881.gd4547 139/165 2024-09-13T08:35:43.907 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:43.907 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T08:35:43.907 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:43.956 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 140/165 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 140/165 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T08:35:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 141/165 2024-09-13T08:35:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 141/165 2024-09-13T08:35:44.426 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T08:35:44.426 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T08:35:44.426 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T08:35:44.426 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T08:35:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:35:49.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.3.0-4881.gd454760e.el9.x86_64 1/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 2/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 3/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 4/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4881.gd4547 5/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 6/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 7/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 10/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 11/165 2024-09-13T08:35:49.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 12/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 13/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_ 15/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 16/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 17/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.3.0-4881.gd454760e.el9.x86_6 18/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 19/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 20/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4881.gd454760e.el 21/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.3.0-4881.gd454760e.el9. 22/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_6 23/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 24/165 2024-09-13T08:35:49.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 25/165 2024-09-13T08:35:49.786 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 26/165 2024-09-13T08:35:49.786 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 27/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 28/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 29/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4881.gd454760e. 30/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 31/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 32/165 2024-09-13T08:35:49.798 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 33/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el 34/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 35/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.e 36/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 37/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.3.0-4881.gd454760e.el9.noarch 38/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 39/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-13T08:35:49.799 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-13T08:35:49.800 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-13T08:35:49.801 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-13T08:35:49.802 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-13T08:35:49.803 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-13T08:35:49.804 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-13T08:35:49.805 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-13T08:35:49.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-13T08:35:49.807 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.3.0-4881.gd454760e.el9.x86_64 150/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 152/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-13T08:35:49.808 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-13T08:35:49.809 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-13T08:35:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 142/165 2024-09-13T08:35:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 143/165 2024-09-13T08:35:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 144/165 2024-09-13T08:35:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-13T08:35:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-13T08:35:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 147/165 2024-09-13T08:35:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-13T08:35:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_ 149/165 2024-09-13T08:35:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-13T08:35:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-13T08:35:51.456 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:51.511 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:51.553 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-13T08:35:51.989 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T08:35:51.990 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.991 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T08:35:51.992 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.993 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T08:35:51.994 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T08:35:51.995 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T08:35:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T08:35:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T08:35:51.998 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T08:35:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T08:35:52.000 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.001 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:35:52.002 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-13T08:35:52.664 DEBUG:teuthology.parallel:result is None 2024-09-13T08:35:52.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 153/165 2024-09-13T08:35:53.066 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-13T08:35:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-13T08:35:53.233 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-13T08:35:53.268 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-13T08:35:53.302 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-13T08:35:53.338 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-13T08:35:53.339 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-13T08:35:53.454 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-13T08:35:53.454 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-13T08:35:53.638 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-13T08:35:53.663 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-13T08:35:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-13T08:35:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-13T08:35:54.066 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-13T08:35:54.143 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:35:54.354 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:35:54.354 INFO:teuthology.orchestra.run.smithi031.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-13T08:35:54.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:00.327 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.3.0-4881.gd454760e.el9.x86_64 1/165 2024-09-13T08:36:00.327 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 2/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 3/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 4/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4881.gd4547 5/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 6/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 7/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 8/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 9/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 10/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 11/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 12/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 13/165 2024-09-13T08:36:00.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 14/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_ 15/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 16/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 17/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.3.0-4881.gd454760e.el9.x86_6 18/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86 19/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 20/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4881.gd454760e.el 21/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.3.0-4881.gd454760e.el9. 22/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_6 23/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 24/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 25/165 2024-09-13T08:36:00.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 26/165 2024-09-13T08:36:00.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 27/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 28/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 29/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4881.gd454760e. 30/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noa 31/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.n 32/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4881.gd45 33/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el 34/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 35/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.e 36/165 2024-09-13T08:36:00.342 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 37/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.3.0-4881.gd454760e.el9.noarch 38/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 39/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-13T08:36:00.343 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-13T08:36:00.344 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-13T08:36:00.345 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-13T08:36:00.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-13T08:36:00.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-13T08:36:00.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-13T08:36:00.349 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-13T08:36:00.350 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.3.0-4881.gd454760e.el9.x86_64 150/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 152/165 2024-09-13T08:36:00.351 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-13T08:36:00.352 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-13T08:36:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-13T08:36:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.3.0-4881.gd454760e.el9.noarch 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T08:36:02.489 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.490 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T08:36:02.491 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T08:36:02.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T08:36:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T08:36:02.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T08:36:02.496 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.3.0-4881.gd454760e.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:02.498 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-13T08:36:03.158 DEBUG:teuthology.parallel:result is None 2024-09-13T08:36:03.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d454760ef8a93a9dd7f4e4dedfb085faac216e7e 2024-09-13T08:36:03.303 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T08:36:03.349 INFO:teuthology.orchestra.run.smithi031.stdout:19.3.0-4881.gd454760e.el9 2024-09-13T08:36:03.350 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4881.gd454760e.el9 2024-09-13T08:36:03.350 INFO:teuthology.task.install:The correct ceph version 19.3.0-4881.gd454760e is installed. 2024-09-13T08:36:03.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d454760ef8a93a9dd7f4e4dedfb085faac216e7e 2024-09-13T08:36:03.490 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T08:36:03.541 INFO:teuthology.orchestra.run.smithi045.stdout:19.3.0-4881.gd454760e.el9 2024-09-13T08:36:03.542 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4881.gd454760e.el9 2024-09-13T08:36:03.542 INFO:teuthology.task.install:The correct ceph version 19.3.0-4881.gd454760e is installed. 2024-09-13T08:36:03.544 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T08:36:03.544 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:03.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T08:36:03.606 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:03.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T08:36:03.669 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T08:36:03.669 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:03.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T08:36:03.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T08:36:03.823 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:03.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T08:36:03.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T08:36:03.980 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T08:36:03.980 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:03.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T08:36:04.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T08:36:04.134 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:04.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T08:36:04.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T08:36:04.286 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T08:36:04.286 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:04.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T08:36:04.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T08:36:04.439 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:04.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T08:36:04.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T08:36:04.590 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T08:36:04.704 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T08:36:04.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-09-13T08:36:04.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-09-13T08:36:04.764 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T08:36:04.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T08:36:04.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T08:36:04.823 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T08:36:04.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T08:36:04.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T08:36:04.888 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-13T08:36:04.888 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-13T08:36:04.889 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-13T08:36:04.889 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:04.889 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T08:36:04.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:05.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:05.148 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:05.149 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T08:36:05.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:05.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:05.416 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T08:36:05.416 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T08:36:05.416 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322', 'branch': 'wip-athakkar-testing-2024-09-12-1849', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}}', 'email': None, 'first_in_suite': False, 'job_id': '7903322', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-athakkar-testing-2024-09-12-1849'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-athakkar-testing-2024-09-12-1849', 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'}}, 'owner': 'scheduled_khiremat@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6477, 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e', 'sleep_before_teardown': 0, 'subset': '3/4', 'suite': 'fs:volumes', 'suite_branch': 'wip-athakkar-testing-2024-09-12-1849', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d454760ef8a93a9dd7f4e4dedfb085faac216e7e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcbKOtiJPLoQxqG6RdpIVoSZReEDe5VYIb/hmIkntx3ONKKtHb1dPLEuF9fCyl9Qje4FvG1DQ9BVtu+2fdOAoQ=', 'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS+xAWN3T0vHWYn3BI/lHTunEw9YoQf0AKEdGArLvZOJ4u0GAjwvd/wbp+/kOMNlnlnZvTaUy8ZT6+0O6+zwAw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd454760ef8a93a9dd7f4e4dedfb085faac216e7e', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_08:07:46', 'tube': 'smithi', 'user': 'khiremat', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T08:36:05.417 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T08:36:05.421 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T08:36:05.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T08:36:05.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T08:36:05.553 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:05.553 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T08:36:05.588 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T08:36:05.588 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-09-13T08:36:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T08:36:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-13 08:35:54.708947360 +0000 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-13 08:33:30.986718251 +0000 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-13 08:33:30.986718251 +0000 2024-09-13T08:36:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-13 08:33:30.986718251 +0000 2024-09-13T08:36:05.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T08:36:05.757 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:36:05.757 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:36:05.757 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000315114 s, 1.6 MB/s 2024-09-13T08:36:05.759 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T08:36:05.796 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-09-13T08:36:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T08:36:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:05.871 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-09-13T08:36:05.871 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:05.871 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:05.872 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-13 08:35:54.708947360 +0000 2024-09-13T08:36:05.872 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-13 08:33:31.461710370 +0000 2024-09-13T08:36:05.872 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-13 08:33:31.461710370 +0000 2024-09-13T08:36:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-13 08:33:31.461710370 +0000 2024-09-13T08:36:05.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T08:36:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:36:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:36:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000264088 s, 1.9 MB/s 2024-09-13T08:36:05.967 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T08:36:06.004 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-09-13T08:36:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-13 08:35:54.709947343 +0000 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-13 08:33:31.923702705 +0000 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-13 08:33:31.923702705 +0000 2024-09-13T08:36:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-13 08:33:31.923702705 +0000 2024-09-13T08:36:06.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T08:36:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:36:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:36:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000246335 s, 2.1 MB/s 2024-09-13T08:36:06.175 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T08:36:06.211 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-09-13T08:36:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-13 08:35:54.710947327 +0000 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-13 08:33:32.375695205 +0000 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-13 08:33:32.376695189 +0000 2024-09-13T08:36:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-13 08:33:32.375695205 +0000 2024-09-13T08:36:06.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T08:36:06.380 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:36:06.380 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:36:06.380 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000307316 s, 1.7 MB/s 2024-09-13T08:36:06.382 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T08:36:06.419 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:06.419 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:06.419 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T08:36:06.458 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T08:36:06.458 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 639 Links: 1 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-13 08:35:43.856837703 +0000 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-13 08:33:31.002523782 +0000 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-13 08:33:31.002523782 +0000 2024-09-13T08:36:06.538 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-13 08:33:31.002523782 +0000 2024-09-13T08:36:06.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T08:36:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-13T08:36:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-13T08:36:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.00017587 s, 2.9 MB/s 2024-09-13T08:36:06.637 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T08:36:06.674 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 658 Links: 1 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-13 08:35:43.857837687 +0000 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-13 08:33:31.467516614 +0000 2024-09-13T08:36:06.749 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-13 08:33:31.467516614 +0000 2024-09-13T08:36:06.750 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-13 08:33:31.467516614 +0000 2024-09-13T08:36:06.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T08:36:06.841 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-13T08:36:06.841 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-13T08:36:06.841 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000175587 s, 2.9 MB/s 2024-09-13T08:36:06.844 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T08:36:06.881 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 678 Links: 1 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-13 08:35:43.858837671 +0000 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-13 08:33:31.942509291 +0000 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-13 08:33:31.942509291 +0000 2024-09-13T08:36:06.956 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-13 08:33:31.942509291 +0000 2024-09-13T08:36:06.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T08:36:07.049 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-13T08:36:07.050 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-13T08:36:07.050 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000170699 s, 3.0 MB/s 2024-09-13T08:36:07.052 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T08:36:07.089 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 697 Links: 1 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-13 08:35:43.858837671 +0000 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-13 08:33:32.425501845 +0000 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-13 08:33:32.425501845 +0000 2024-09-13T08:36:07.164 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-13 08:33:32.425501845 +0000 2024-09-13T08:36:07.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T08:36:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-13T08:36:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-13T08:36:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000182382 s, 2.8 MB/s 2024-09-13T08:36:07.259 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T08:36:07.295 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:07.295 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T08:36:07.295 INFO:tasks.ceph:Generating config... 2024-09-13T08:36:07.297 INFO:tasks.ceph:[client] pid file = 2024-09-13T08:36:07.297 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mgr] client mount timeout = 30 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T08:36:07.297 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T08:36:07.298 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T08:36:07.298 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T08:36:07.298 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T08:36:07.298 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T08:36:07.299 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T08:36:07.299 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T08:36:07.299 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T08:36:07.299 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T08:36:07.299 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T08:36:07.299 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T08:36:07.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T08:36:07.403 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T08:36:07.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T08:36:07.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T08:36:07.553 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.b', '172.21.15.45'), ('mon.c', '[v2:172.21.15.45:3301,v1:172.21.15.45:6790]')] 2024-09-13T08:36:07.553 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,172.21.15.45,[v2:172.21.15.45:3301,v1:172.21.15.45:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-09-13T08:36:07.554 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:07.554 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T08:36:07.589 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --add b 172.21.15.45 --addv c '[v2:172.21.15.45:3301,v1:172.21.15.45:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 0251b593-a02a-47fa-b938-c065b7284822 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 0251b593-a02a-47fa-b938-c065b7284822 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-09-13T08:36:07.706980+0000 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-13T08:36:07.706980+0000 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-09-13T08:36:07.707 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-09-13T08:36:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-09-13T08:36:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-09-13T08:36:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] mon.c 2024-09-13T08:36:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T08:36:07.710 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T08:36:07.744 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0251b593-a02a-47fa-b938-c065b7284822... 2024-09-13T08:36:07.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T08:36:07.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T08:36:07.882 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-13T08:36:07.882 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-13T08:36:07.882 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T08:36:07.882 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-13T08:36:07.882 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T08:36:07.883 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-13T08:36:07.884 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,172.21.15.45,[v2:172.21.15.45:3301,v1:172.21.15.45:6790]" 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on skipped message = False 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 0251b593-a02a-47fa-b938-c065b7284822 2024-09-13T08:36:07.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-13T08:36:07.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-13T08:36:07.887 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = bitmap 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-13T08:36:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: client mount timeout = 30 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 20 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: debug finisher = 20 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.889 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.890 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: mon down mkfs grace = 300 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T08:36:07.891 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = "" 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-09-13T08:36:07.892 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T08:36:07.893 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T08:36:07.894 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "172.21.15.31,172.21.15.45,[v2:172.21.15.45:3301,v1:172.21.15.45:6790]" 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on skipped message = False 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 0251b593-a02a-47fa-b938-c065b7284822 2024-09-13T08:36:07.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-09-13T08:36:07.896 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-09-13T08:36:07.897 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bdev async discard = True 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bdev enable discard = True 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore allocator = bitmap 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 1/20 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 1/20 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-09-13T08:36:07.898 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 4/10 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-09-13T08:36:07.899 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: client mount timeout = 30 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug client = 20 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug finisher = 20 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-09-13T08:36:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-09-13T08:36:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-09-13T08:36:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T08:36:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T08:36:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: mon down mkfs grace = 300 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-09-13T08:36:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = "" 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2024-09-13T08:36:07.903 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2024-09-13T08:36:07.913 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T08:36:07.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T08:36:08.011 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T08:36:08.012 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:08.012 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T08:36:08.046 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:08.047 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T08:36:08.121 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:36:08.121 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:08.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T08:36:08.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T08:36:08.238 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:08.238 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:08.312 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:36:08.312 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:08.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T08:36:08.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T08:36:08.391 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:08.391 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:08.465 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T08:36:08.466 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T08:36:08.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T08:36:08.570 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T08:36:08.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T08:36:08.689 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T08:36:08.696 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T08:36:08.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T08:36:08.802 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T08:36:08.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-13T08:36:08.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T08:36:08.967 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T08:36:08.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-13T08:36:09.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T08:36:09.133 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T08:36:09.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-13T08:36:09.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T08:36:09.298 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T08:36:09.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-13T08:36:09.359 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T08:36:09.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T08:36:09.446 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T08:36:09.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-13T08:36:09.557 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-13T08:36:09.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T08:36:09.667 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-13T08:36:09.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-13T08:36:09.776 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-13T08:36:09.803 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T08:36:09.803 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T08:36:09.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T08:36:09.860 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:09.860 INFO:tasks.ceph:role: osd.0 2024-09-13T08:36:09.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:36:09.860 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:09.923 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:09.924 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:09.924 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:09.924 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:10.324 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-13T08:36:10.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-13T08:36:10.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T08:36:10.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T08:36:10.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T08:36:10.590 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:10.590 INFO:tasks.ceph:role: osd.1 2024-09-13T08:36:10.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:36:10.590 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T08:36:10.653 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:10.654 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:11.009 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-13T08:36:11.013 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-13T08:36:11.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T08:36:11.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T08:36:11.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T08:36:11.225 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:11.225 INFO:tasks.ceph:role: osd.2 2024-09-13T08:36:11.225 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:36:11.226 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:11.290 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:11.673 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-13T08:36:11.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-13T08:36:11.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T08:36:11.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T08:36:11.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T08:36:11.896 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:11.896 INFO:tasks.ceph:role: osd.3 2024-09-13T08:36:11.896 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:36:11.896 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T08:36:11.958 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:11.959 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:12.383 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-13T08:36:12.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-13T08:36:12.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-13T08:36:12.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T08:36:12.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:12.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:12.690+0000 7f6c5dc11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T08:36:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:12.693+0000 7f6c5dc11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T08:36:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:12.693+0000 7f6c5dc11640 -1 bdev(0x5613b0653800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:12.693+0000 7f6c5dc11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T08:36:15.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T08:36:15.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:15.828+0000 7f2cf08e2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T08:36:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:15.829+0000 7f2cf08e2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T08:36:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:15.829+0000 7f2cf08e2640 -1 bdev(0x557433d0f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:15.829+0000 7f2cf08e2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T08:36:19.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T08:36:19.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:19.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:19.224+0000 7f25c261e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T08:36:19.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:19.224+0000 7f25c261e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T08:36:19.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:19.224+0000 7f25c261e640 -1 bdev(0x556f4765d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:19.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:19.224+0000 7f25c261e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T08:36:22.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T08:36:22.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:22.417+0000 7f4914c14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T08:36:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:22.418+0000 7f4914c14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T08:36:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:22.418+0000 7f4914c14640 -1 bdev(0x558df2bcd800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:22.418+0000 7f4914c14640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T08:36:25.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T08:36:25.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T08:36:25.578 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:25.578 INFO:tasks.ceph:role: osd.4 2024-09-13T08:36:25.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:36:25.579 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:25.636 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:25.637 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:25.637 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:25.637 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:25.670 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-13T08:36:25.674 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-13T08:36:25.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-13T08:36:25.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T08:36:25.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T08:36:25.874 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:25.874 INFO:tasks.ceph:role: osd.5 2024-09-13T08:36:25.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:36:25.874 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:25.930 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-13T08:36:25.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-13T08:36:25.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T08:36:26.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T08:36:26.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T08:36:26.164 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:26.164 INFO:tasks.ceph:role: osd.6 2024-09-13T08:36:26.164 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:36:26.164 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:26.219 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:26.252 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-13T08:36:26.255 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-13T08:36:26.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-13T08:36:26.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T08:36:26.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T08:36:26.452 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T08:36:26.452 INFO:tasks.ceph:role: osd.7 2024-09-13T08:36:26.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:36:26.452 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T08:36:26.507 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T08:36:26.507 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T08:36:26.508 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T08:36:26.540 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-09-13T08:36:26.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-09-13T08:36:26.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-13T08:36:26.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T08:36:26.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:26.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:26.782+0000 7f207ee15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T08:36:26.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:26.782+0000 7f207ee15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T08:36:26.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:26.783+0000 7f207ee15640 -1 bdev(0x562e787ef800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:26.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:26.783+0000 7f207ee15640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T08:36:29.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T08:36:29.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:29.956+0000 7fafe1bdd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T08:36:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:29.956+0000 7fafe1bdd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T08:36:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:29.956+0000 7fafe1bdd640 -1 bdev(0x556a44c87800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:29.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:29.956+0000 7fafe1bdd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T08:36:32.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T08:36:33.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:33.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:33.121+0000 7fbe7b354640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T08:36:33.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:33.121+0000 7fbe7b354640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T08:36:33.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:33.122+0000 7fbe7b354640 -1 bdev(0x562789331800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:33.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:33.122+0000 7fbe7b354640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T08:36:35.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:35.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:36:36.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T08:36:36.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:36.263+0000 7f74a5262640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T08:36:36.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:36.263+0000 7f74a5262640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T08:36:36.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:36.263+0000 7f74a5262640 -1 bdev(0x5583773a9800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-13T08:36:36.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:36.263+0000 7f74a5262640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T08:36:39.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T08:36:39.383 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T08:36:39.383 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T08:36:39.446 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-13T08:36:39.544 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-13T08:36:39.642 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T08:36:39.737 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T08:36:39.833 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T08:36:39.928 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:39.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T08:36:40.023 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T08:36:40.085 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-13T08:36:40.184 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-13T08:36:40.281 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T08:36:40.377 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T08:36:40.474 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T08:36:40.569 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T08:36:40.665 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:40.665 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T08:36:40.702 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:40.702 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-13T08:36:40.777 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.777 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-13T08:36:40.812 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:40.813 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-13T08:36:40.887 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T08:36:40.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T08:36:40.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T08:36:40.942 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.y] 2024-09-13T08:36:40.942 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A== 2024-09-13T08:36:40.942 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q== 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw== 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ== 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA== 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ== 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-09-13T08:36:40.943 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.4] 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.5] 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ== 2024-09-13T08:36:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.6] 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw== 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.7] 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA== 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout:[client.2] 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw== 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw== 2024-09-13T08:36:40.945 INFO:teuthology.orchestra.run.smithi031.stdout:[client.3] 2024-09-13T08:36:40.946 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA== 2024-09-13T08:36:40.980 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ== 2024-09-13T08:36:40.981 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ== 2024-09-13T08:36:40.982 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw== 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA== 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw== 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw== 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout:[client.3] 2024-09-13T08:36:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA== 2024-09-13T08:36:40.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T08:36:40.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T08:36:41.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:41.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T08:36:41.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T08:36:41.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:41.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T08:36:42.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T08:36:42.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T08:36:42.812 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T08:36:42.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T08:36:42.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T08:36:42.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T08:36:43.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T08:36:43.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T08:36:43.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T08:36:43.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T08:36:43.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T08:36:43.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T08:36:43.472 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:43.476 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T08:36:43.513 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T08:36:43.513 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T08:36:43.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T08:36:43.554 INFO:tasks.ceph.mon.a:Started 2024-09-13T08:36:43.554 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T08:36:43.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T08:36:43.558 INFO:tasks.ceph.mon.b:Started 2024-09-13T08:36:43.558 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T08:36:43.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T08:36:43.561 INFO:tasks.ceph.mon.c:Started 2024-09-13T08:36:43.562 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T08:36:43.562 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T08:36:43.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T08:36:43.565 INFO:tasks.ceph.mgr.y:Started 2024-09-13T08:36:43.566 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T08:36:43.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T08:36:43.570 INFO:tasks.ceph.mgr.x:Started 2024-09-13T08:36:43.570 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-13T08:36:43.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-13T08:36:43.574 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-13T08:36:43.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-13T08:36:43.620 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-13T08:36:43.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-13T08:36:43.667 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-13T08:36:43.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-13T08:36:43.714 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-13T08:36:43.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-13T08:36:43.761 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-13T08:36:43.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-13T08:36:43.808 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-13T08:36:43.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-13T08:36:43.858 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-13T08:36:43.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-13T08:36:43.872 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-13T08:36:43.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-13T08:36:43.927 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-13T08:36:43.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-13T08:36:43.975 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-13T08:36:43.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-13T08:36:43.995 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-13T08:36:43.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-13T08:36:44.046 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-13T08:36:44.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-13T08:36:44.103 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-13T08:36:44.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-13T08:36:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.125+0000 7f2c69600640 1 -- 172.21.15.31:0/3737034536 <== mon.1 v2:172.21.15.45:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v4)=0 v4) ==== 109+0+0 (secure 0 0 0) 0x7f2c58024430 con 0x7f2c6c0c9ff0 2024-09-13T08:36:44.140 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-13T08:36:44.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-13T08:36:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 -- 172.21.15.31:0/3737034536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6c0c9ff0 msgr2=0x7f2c6c1f7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 --2- 172.21.15.31:0/3737034536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6c0c9ff0 0x7f2c6c1f7af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2c58001e50 tx=0x7f2c5800c6f0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 -- 172.21.15.31:0/3737034536 shutdown_connections 2024-09-13T08:36:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 --2- 172.21.15.31:0/3737034536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c6c0c9ff0 0x7f2c6c1f7af0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 --2- 172.21.15.31:0/3737034536 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c6c06fcd0 0x7f2c6c158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 --2- 172.21.15.31:0/3737034536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c6c06cb00 0x7f2c6c0b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.129+0000 7f2c5ea00640 1 -- 172.21.15.31:0/3737034536 >> 172.21.15.31:0/3737034536 conn(0x7f2c6c0cf9a0 msgr2=0x7f2c6c0af340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.130+0000 7f2c5ea00640 1 -- 172.21.15.31:0/3737034536 shutdown_connections 2024-09-13T08:36:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.130+0000 7f2c5ea00640 1 -- 172.21.15.31:0/3737034536 wait complete. 2024-09-13T08:36:44.190 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-13T08:36:44.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-13T08:36:44.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.177+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fcbc4000e60 con 0x7fcbec0d2170 2024-09-13T08:36:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.189+0000 7fe274800640 1 -- 172.21.15.31:0/4265746933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe254004800 msgr2=0x7fe254004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.189+0000 7fe274800640 1 --2- 172.21.15.31:0/4265746933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe254004800 0x7fe254004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b7c0 tx=0x7fe254005830 comp rx=0 tx=0).stop 2024-09-13T08:36:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.189+0000 7fe272800640 1 -- 172.21.15.31:0/4265746933 reap_dead start 2024-09-13T08:36:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.190+0000 7fe274800640 1 -- 172.21.15.31:0/4265746933 shutdown_connections 2024-09-13T08:36:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.193+0000 7fcbe2a00640 1 -- 172.21.15.31:0/978868907 <== mon.1 v2:172.21.15.45:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fcbdc0107d0 con 0x7fcbec0d2170 2024-09-13T08:36:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.193+0000 7fcbe2a00640 1 -- 172.21.15.31:0/978868907 <== mon.1 v2:172.21.15.45:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fcbdc010a60 con 0x7fcbec0d2170 2024-09-13T08:36:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.193+0000 7fd7dd600640 1 -- 172.21.15.31:0/53235567 <== mon.2 v2:172.21.15.45:3301/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fd7d403a640 con 0x7fd7e015d5e0 2024-09-13T08:36:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.193+0000 7fd471600640 1 -- 172.21.15.31:0/3276911337 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fd474093d80 con 0x7fd46c04b5a0 2024-09-13T08:36:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.191+0000 7fe274800640 1 -- 172.21.15.31:0/4265746933 >> 172.21.15.31:0/4265746933 conn(0x7fe26c0cfbb0 msgr2=0x7fe26c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.193+0000 7fe274800640 1 -- 172.21.15.31:0/4265746933 shutdown_connections 2024-09-13T08:36:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.194+0000 7fe274800640 1 -- 172.21.15.31:0/4265746933 wait complete. 2024-09-13T08:36:44.209 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-13T08:36:44.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-09-13T08:36:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.201+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbec0d2170 msgr2=0x7fcbec0c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.201+0000 7fcbf1c00640 1 --2- 172.21.15.31:0/978868907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbec0d2170 0x7fcbec0c9680 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c540 tx=0x7fcbdc00ca00 comp rx=0 tx=0).stop 2024-09-13T08:36:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 shutdown_connections 2024-09-13T08:36:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fcbf1c00640 1 --2- 172.21.15.31:0/978868907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbec0d2170 0x7fcbec0c9680 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fcbf1c00640 1 --2- 172.21.15.31:0/978868907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcbec0d1d90 0x7fcbec0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fcbf1c00640 1 --2- 172.21.15.31:0/978868907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbec0d1700 0x7fcbec14c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 >> 172.21.15.31:0/978868907 conn(0x7fcbec0d3f50 msgr2=0x7fcbec06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.205+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 shutdown_connections 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.205+0000 7fcbf1c00640 1 -- 172.21.15.31:0/978868907 wait complete. 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fe274800640 1 Processor -- start 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.202+0000 7fe274800640 1 -- start start 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.208+0000 7fe274800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.208+0000 7fe274800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26c14bc60 0x7fe26c150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.209+0000 7fe274800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 0x7fe26c0af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.209+0000 7f6e4b200640 1 -- 172.21.15.31:0/748576020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0d2000 msgr2=0x7f6e4c0d23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.209+0000 7f6e4b200640 1 --2- 172.21.15.31:0/748576020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0d2000 0x7f6e4c0d23e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0a1d80 tx=0x7f6e3c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.212+0000 7fe271e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.212+0000 7fe271e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60292/0 (socket says 172.21.15.31:60292) 2024-09-13T08:36:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.210+0000 7f6e4b200640 1 -- 172.21.15.31:0/748576020 shutdown_connections 2024-09-13T08:36:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.210+0000 7f6e4b200640 1 --2- 172.21.15.31:0/748576020 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.210+0000 7f6e4b200640 1 --2- 172.21.15.31:0/748576020 >> v2:172.21.15.45:3300/0 conn(0x7f6e4c0b4ee0 0x7f6e4c0b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.210+0000 7f6e4b200640 1 --2- 172.21.15.31:0/748576020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0d2000 0x7f6e4c0d23e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.210+0000 7f6e4b200640 1 -- 172.21.15.31:0/748576020 >> 172.21.15.31:0/748576020 conn(0x7f6e4c0d3e90 msgr2=0x7f6e4c0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.223 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-13T08:36:44.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7f6e4b200640 1 -- 172.21.15.31:0/748576020 shutdown_connections 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7fe271e00640 1 -- 172.21.15.31:0/3007081141 learned_addr learned my addr 172.21.15.31:0/3007081141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c072560 con 0x7fe26c1506c0 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c160890 con 0x7fe254004800 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe26c0726e0 con 0x7fe26c14bc60 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.213+0000 7fe272800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 0x7fe26c0af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.220+0000 7fe271400640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26c14bc60 0x7fe26c150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe271e00640 1 -- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26c14bc60 msgr2=0x7fe26c150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe271e00640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26c14bc60 0x7fe26c150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe271e00640 1 -- 172.21.15.31:0/3007081141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 msgr2=0x7fe26c0af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe271e00640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 0x7fe26c0af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe271e00640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26c153dc0 con 0x7fe254004800 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.222+0000 7fe272800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 0x7fe26c0af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.223+0000 7fe271e00640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe260006680 tx=0x7fe2600066b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.227+0000 7f6e4b200640 1 -- 172.21.15.31:0/748576020 wait complete. 2024-09-13T08:36:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.229+0000 7f6e4b200640 1 Processor -- start 2024-09-13T08:36:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.229+0000 7f6e4b200640 1 -- start start 2024-09-13T08:36:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.231+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.231+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0b4ee0 0x7f6e4c208fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.231+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.244 DEBUG:tasks.ceph:set 18 configs 2024-09-13T08:36:44.245 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.245 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.245 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.245 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e49e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c0b10f0 con 0x7f6e4c0b4ee0 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.234+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260015070 con 0x7fe254004800 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.234+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fe26000fab0 con 0x7fe254004800 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.234+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260011430 con 0x7fe254004800 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.234+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26c1540a0 con 0x7fe254004800 2024-09-13T08:36:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.234+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe26c14f980 con 0x7fe254004800 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.238+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe26c0d0490 con 0x7fe254004800 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c0b0f70 con 0x7f6e4c0b4a30 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4c0b1270 con 0x7f6e4c0d2000 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4a800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4a800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46862/0 (socket says 172.21.15.31:46862) 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e4a800640 1 -- 172.21.15.31:0/2198959517 learned_addr learned my addr 172.21.15.31:0/2198959517 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e49400640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0b4ee0 0x7f6e4c208fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.235+0000 7f6e49e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60304/0 (socket says 172.21.15.31:60304) 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.236+0000 7f6e4a800640 1 -- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 msgr2=0x7f6e4c0b8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.236+0000 7f6e4a800640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.236+0000 7f6e4a800640 1 -- 172.21.15.31:0/2198959517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0b4ee0 msgr2=0x7f6e4c208fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.236+0000 7f6e4a800640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0b4ee0 0x7f6e4c208fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.236+0000 7f6e4a800640 1 -- 172.21.15.31:0/2198959517 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c0b9500 con 0x7f6e4c0d2000 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.242+0000 7f6e4a800640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400b9c0 tx=0x7f6e4400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.244+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe260011960 con 0x7fe254004800 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.244+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe260024720 con 0x7fe254004800 2024-09-13T08:36:44.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.244+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe260025d20 con 0x7fe254004800 2024-09-13T08:36:44.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.252+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fd7b4005310 con 0x7fd7e015d5e0 2024-09-13T08:36:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.252+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44019070 con 0x7f6e4c0d2000 2024-09-13T08:36:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.252+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f6e4400cc90 con 0x7f6e4c0d2000 2024-09-13T08:36:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.252+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400ce60 con 0x7f6e4c0d2000 2024-09-13T08:36:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.252+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4c0b97e0 con 0x7f6e4c0d2000 2024-09-13T08:36:44.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.254+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4c0b9210 con 0x7f6e4c0d2000 2024-09-13T08:36:44.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.255+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6e4401c070 con 0x7f6e4c0d2000 2024-09-13T08:36:44.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.255+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6e4401ed40 con 0x7f6e4c0d2000 2024-09-13T08:36:44.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.258+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e4c0b44f0 con 0x7f6e4c0d2000 2024-09-13T08:36:44.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.267+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6e44023020 con 0x7f6e4c0d2000 2024-09-13T08:36:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.269+0000 7fd7dd600640 1 -- 172.21.15.31:0/53235567 <== mon.2 v2:172.21.15.45:3301/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7d403a640 con 0x7fd7e015d5e0 2024-09-13T08:36:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.269+0000 7fd7dd600640 1 -- 172.21.15.31:0/53235567 <== mon.2 v2:172.21.15.45:3301/0 10 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fd7d4043980 con 0x7fd7e015d5e0 2024-09-13T08:36:44.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.271+0000 7fd471600640 1 -- 172.21.15.31:0/3276911337 <== mon.0 v2:172.21.15.31:3300/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd47409e3f0 con 0x7fd46c04b5a0 2024-09-13T08:36:44.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.271+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e4401d3b0 con 0x7f6e4c0d2000 2024-09-13T08:36:44.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.272+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe260011c50 con 0x7fe254004800 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd7e015d5e0 msgr2=0x7fd7e00c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 --2- 172.21.15.31:0/53235567 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd7e015d5e0 0x7fd7e00c8ad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4006010 tx=0x7fd7d40046c0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 shutdown_connections 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 --2- 172.21.15.31:0/53235567 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd7e015d5e0 0x7fd7e00c8ad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 --2- 172.21.15.31:0/53235567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e00cd780 0x7fd7e0148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 --2- 172.21.15.31:0/53235567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e00ca200 0x7fd7e01555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.287+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 >> 172.21.15.31:0/53235567 conn(0x7fd7e00cfbb0 msgr2=0x7fd7e015ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.291+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 shutdown_connections 2024-09-13T08:36:44.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.291+0000 7fd7e8000640 1 -- 172.21.15.31:0/53235567 wait complete. 2024-09-13T08:36:44.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.296+0000 7f5d09800640 1 -- 172.21.15.31:0/3105476237 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 msgr2=0x7f5d040cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.299+0000 7f5d01600640 1 -- 172.21.15.31:0/3105476237 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc02c840 con 0x7f5d040cbb40 2024-09-13T08:36:44.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.299+0000 7f5d09800640 1 --2- 172.21.15.31:0/3105476237 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040cd940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d040749f0 tx=0x7f5cfc02aae0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 -- 172.21.15.31:0/3105476237 shutdown_connections 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 --2- 172.21.15.31:0/3105476237 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040cd940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 --2- 172.21.15.31:0/3105476237 >> v2:172.21.15.45:3300/0 conn(0x7f5d040cde80 0x7f5d04161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 --2- 172.21.15.31:0/3105476237 >> v2:172.21.15.31:3300/0 conn(0x7f5d0415df70 0x7f5d040cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 -- 172.21.15.31:0/3105476237 >> 172.21.15.31:0/3105476237 conn(0x7f5d040cfbb0 msgr2=0x7f5d040cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 -- 172.21.15.31:0/3105476237 shutdown_connections 2024-09-13T08:36:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.306+0000 7f5d09800640 1 -- 172.21.15.31:0/3105476237 wait complete. 2024-09-13T08:36:44.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.309+0000 7f5d09800640 1 Processor -- start 2024-09-13T08:36:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.316+0000 7f5d09800640 1 -- start start 2024-09-13T08:36:44.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.325+0000 7f5d09800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.325+0000 7f5d09800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.325+0000 7f5d02a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.326+0000 7f5d02a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46872/0 (socket says 172.21.15.31:46872) 2024-09-13T08:36:44.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.328+0000 7f5d02000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.328+0000 7f5d02000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60320/0 (socket says 172.21.15.31:60320) 2024-09-13T08:36:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.328+0000 7f5d02a00640 1 -- 172.21.15.31:0/629909646 learned_addr learned my addr 172.21.15.31:0/629909646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.328+0000 7f5d09800640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 0x7f5d0415e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.329+0000 7f5d03400640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 0x7f5d0415e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d04162210 con 0x7f5d0415df70 2024-09-13T08:36:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d04162090 con 0x7f5d040cde80 2024-09-13T08:36:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5d04162390 con 0x7f5d040cbb40 2024-09-13T08:36:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d02000640 1 -- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 msgr2=0x7f5d040c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d02000640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.331+0000 7f5d02000640 1 -- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 msgr2=0x7f5d0415e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.335 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.334+0000 7f5d02000640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 0x7f5d0415e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.334+0000 7f5d02000640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d040c9170 con 0x7f5d040cde80 2024-09-13T08:36:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.334+0000 7f5d03400640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 0x7f5d0415e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.336+0000 7f5d02000640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc032000 tx=0x7f5cfc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.339+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc044070 con 0x7f5d040cde80 2024-09-13T08:36:44.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.339+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cfc003e20 con 0x7f5d040cde80 2024-09-13T08:36:44.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.339+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc004850 con 0x7f5d040cde80 2024-09-13T08:36:44.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.339+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d040c9470 con 0x7f5d040cde80 2024-09-13T08:36:44.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.339+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d040c5890 con 0x7f5d040cde80 2024-09-13T08:36:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.346+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5cfc03f410 con 0x7f5d040cde80 2024-09-13T08:36:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.346+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5cfc02d410 con 0x7f5d040cde80 2024-09-13T08:36:44.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.349+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cdc005290 con 0x7f5d040cde80 2024-09-13T08:36:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.357+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5d040c9470 con 0x7f5d040cde80 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.364+0000 7ff81ce00640 1 -- 172.21.15.31:0/152065285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 msgr2=0x7ff8180d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.364+0000 7ff814a00640 1 -- 172.21.15.31:0/152065285 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808031600 con 0x7ff8180d1d90 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.364+0000 7ff81ce00640 1 --2- 172.21.15.31:0/152065285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180d1550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff81806fc10 tx=0x7ff80802a480 comp rx=0 tx=0).stop 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.365+0000 7ff81ce00640 1 -- 172.21.15.31:0/152065285 shutdown_connections 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.365+0000 7ff81ce00640 1 --2- 172.21.15.31:0/152065285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180d1550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.365+0000 7ff81ce00640 1 -- 172.21.15.31:0/152065285 >> 172.21.15.31:0/152065285 conn(0x7ff8180d3f50 msgr2=0x7ff8180d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.365+0000 7ff81ce00640 1 -- 172.21.15.31:0/152065285 shutdown_connections 2024-09-13T08:36:44.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.365+0000 7ff81ce00640 1 -- 172.21.15.31:0/152065285 wait complete. 2024-09-13T08:36:44.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.370+0000 7ff81ce00640 1 Processor -- start 2024-09-13T08:36:44.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.371+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fd4480052f0 con 0x7fd46c04b5a0 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.372+0000 7ff81ce00640 1 -- start start 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.373+0000 7ff81ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.373+0000 7ff81ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.373+0000 7ff81ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.376+0000 7ff815e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.376+0000 7ff815e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60328/0 (socket says 172.21.15.31:60328) 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.376+0000 7ff815400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.376+0000 7ff815400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46892/0 (socket says 172.21.15.31:46892) 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff816800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff816800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35990/0 (socket says 172.21.15.31:35990) 2024-09-13T08:36:44.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff815e00640 1 -- 172.21.15.31:0/238480923 learned_addr learned my addr 172.21.15.31:0/238480923 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff81ce00640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff818169560 con 0x7ff8180d2170 2024-09-13T08:36:44.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff81ce00640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8181693e0 con 0x7ff8180b4ae0 2024-09-13T08:36:44.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.377+0000 7ff81ce00640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff8181696e0 con 0x7ff8180d1d90 2024-09-13T08:36:44.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815400640 1 -- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 msgr2=0x7ff818201760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815400640 1 -- 172.21.15.31:0/238480923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 msgr2=0x7ff8180b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815400640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8180b29b0 con 0x7ff8180d1d90 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff815e00640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.378+0000 7ff816800640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.380+0000 7ff815400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff80400c390 tx=0x7ff80400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.383+0000 7fd471600640 1 -- 172.21.15.31:0/3276911337 <== mon.0 v2:172.21.15.31:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) ==== 108+0+0 (secure 0 0 0) 0x7fd46c031ca0 con 0x7fd46c04b5a0 2024-09-13T08:36:44.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.384+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd46c04b5a0 msgr2=0x7fd46c04ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.385+0000 7fd47ae00640 1 --2- 172.21.15.31:0/3276911337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd46c04b5a0 0x7fd46c04ba30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd474051c90 tx=0x7fd474069c90 comp rx=0 tx=0).stop 2024-09-13T08:36:44.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.385+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 shutdown_connections 2024-09-13T08:36:44.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.385+0000 7fd47ae00640 1 --2- 172.21.15.31:0/3276911337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd46c04b5a0 0x7fd46c04ba30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.387+0000 7fd47ae00640 1 --2- 172.21.15.31:0/3276911337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd46c04ad90 0x7fd46c032720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.387+0000 7fd47ae00640 1 --2- 172.21.15.31:0/3276911337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd45c003a00 0x7fd46c0d4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.387+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 >> 172.21.15.31:0/3276911337 conn(0x7fd46c01d770 msgr2=0x7fd46c0dfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.387+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 shutdown_connections 2024-09-13T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.387+0000 7fd47ae00640 1 -- 172.21.15.31:0/3276911337 wait complete. 2024-09-13T08:36:44.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.396+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804017070 con 0x7ff8180d1d90 2024-09-13T08:36:44.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.396+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8040035f0 con 0x7ff8180d1d90 2024-09-13T08:36:44.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.396+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804003a40 con 0x7ff8180d1d90 2024-09-13T08:36:44.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.401+0000 7ff81ce00640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8180b2c90 con 0x7ff8180d1d90 2024-09-13T08:36:44.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.402+0000 7ff81ce00640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8180c8ad0 con 0x7ff8180d1d90 2024-09-13T08:36:44.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.407+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec005290 con 0x7ff8180d1d90 2024-09-13T08:36:44.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.408+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff80401a070 con 0x7ff8180d1d90 2024-09-13T08:36:44.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.408+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff804012400 con 0x7ff8180d1d90 2024-09-13T08:36:44.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.410+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff804021020 con 0x7ff8180d1d90 2024-09-13T08:36:44.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.445+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fe26c14fd60 con 0x7fe254004800 2024-09-13T08:36:44.456 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.458+0000 7f2940400640 1 -- 172.21.15.31:0/3936199628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 msgr2=0x7f29380d3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.458+0000 7f2940400640 1 --2- 172.21.15.31:0/3936199628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29380d3c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2924005fd0 tx=0x7f292402a560 comp rx=0 tx=0).stop 2024-09-13T08:36:44.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 -- 172.21.15.31:0/3936199628 shutdown_connections 2024-09-13T08:36:44.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 --2- 172.21.15.31:0/3936199628 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f29380b70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 --2- 172.21.15.31:0/3936199628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29380d3c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 -- 172.21.15.31:0/3936199628 >> 172.21.15.31:0/3936199628 conn(0x7f29380d20e0 msgr2=0x7f29380d24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 -- 172.21.15.31:0/3936199628 shutdown_connections 2024-09-13T08:36:44.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.465+0000 7fe266000640 1 -- 172.21.15.31:0/3007081141 <== mon.1 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7fe260011e00 con 0x7fe254004800 2024-09-13T08:36:44.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.462+0000 7f2940400640 1 -- 172.21.15.31:0/3936199628 wait complete. 2024-09-13T08:36:44.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.469+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 msgr2=0x7fe26c14b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.469+0000 7fe274800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe260006680 tx=0x7fe2600066b0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 shutdown_connections 2024-09-13T08:36:44.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c1506c0 0x7fe26c0af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26c14bc60 0x7fe26c150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.473+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f6e4c0c9430 con 0x7f6e4c0d2000 2024-09-13T08:36:44.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.469+0000 7f2940400640 1 Processor -- start 2024-09-13T08:36:44.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 --2- 172.21.15.31:0/3007081141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe254004800 0x7fe26c14b720 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 >> 172.21.15.31:0/3007081141 conn(0x7fe26c0cfbb0 msgr2=0x7fe26c15d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.474+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 shutdown_connections 2024-09-13T08:36:44.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.475+0000 7fe274800640 1 -- 172.21.15.31:0/3007081141 wait complete. 2024-09-13T08:36:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.473+0000 7f2940400640 1 -- start start 2024-09-13T08:36:44.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.482+0000 7f2940400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f293814ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.486+0000 7f2940400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.486+0000 7f293da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f293814ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.486+0000 7f293da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f293814ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46904/0 (socket says 172.21.15.31:46904) 2024-09-13T08:36:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.486+0000 7f293d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.487+0000 7f293d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60342/0 (socket says 172.21.15.31:60342) 2024-09-13T08:36:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.487+0000 7f293d000640 1 -- 172.21.15.31:0/1038414449 learned_addr learned my addr 172.21.15.31:0/1038414449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.487+0000 7f2940400640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29381f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.488+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29380b79f0 con 0x7f29380d4b30 2024-09-13T08:36:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.491+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29380b7870 con 0x7f29380700a0 2024-09-13T08:36:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.491+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f29380b9160 con 0x7f293806fc60 2024-09-13T08:36:44.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.491+0000 7f293ee00640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29381f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293ee00640 1 -- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 msgr2=0x7f293814ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293ee00640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f293814ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293ee00640 1 -- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 msgr2=0x7f2938160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293ee00640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293d000640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:36:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.496+0000 7f293ee00640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29381504a0 con 0x7f29380d4b30 2024-09-13T08:36:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.501+0000 7f6e42000640 1 -- 172.21.15.31:0/2198959517 <== mon.2 v2:172.21.15.45:3301/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7f6e44014830 con 0x7f6e4c0d2000 2024-09-13T08:36:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.501+0000 7f293ee00640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29381f7bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f293000be00 tx=0x7f293000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.502+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930019070 con 0x7f29380d4b30 2024-09-13T08:36:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.502+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29381f85f0 con 0x7f29380d4b30 2024-09-13T08:36:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.502+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293814f760 con 0x7f29380d4b30 2024-09-13T08:36:44.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.506+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 msgr2=0x7f6e4c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.507+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2930014440 con 0x7f29380d4b30 2024-09-13T08:36:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.507+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930014910 con 0x7f29380d4b30 2024-09-13T08:36:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.507+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f293001c070 con 0x7f29380d4b30 2024-09-13T08:36:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.508+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f293001d6f0 con 0x7f29380d4b30 2024-09-13T08:36:44.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2914005290 con 0x7f29380d4b30 2024-09-13T08:36:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400b9c0 tx=0x7f6e4400be80 comp rx=0 tx=0).stop 2024-09-13T08:36:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 shutdown_connections 2024-09-13T08:36:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e4c0d2000 0x7f6e4c06e500 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e4c0b4ee0 0x7f6e4c208fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 --2- 172.21.15.31:0/2198959517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e4c0b4a30 0x7f6e4c0b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.509+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 >> 172.21.15.31:0/2198959517 conn(0x7f6e4c0d3e90 msgr2=0x7f6e4c160170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.511 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.510+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 shutdown_connections 2024-09-13T08:36:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.510+0000 7f6e4b200640 1 -- 172.21.15.31:0/2198959517 wait complete. 2024-09-13T08:36:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.520+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2930023080 con 0x7f29380d4b30 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.538+0000 7f7f07200640 1 -- 172.21.15.31:0/2688671713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 msgr2=0x7f7f080b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.538+0000 7f7f07200640 1 --2- 172.21.15.31:0/2688671713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080b3460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f0806cfe0 tx=0x7f7ef402a470 comp rx=0 tx=0).stop 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.538+0000 7f7f07200640 1 -- 172.21.15.31:0/2688671713 shutdown_connections 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.538+0000 7f7f07200640 1 --2- 172.21.15.31:0/2688671713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080b3460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.539+0000 7f7f07200640 1 --2- 172.21.15.31:0/2688671713 >> v2:172.21.15.45:3300/0 conn(0x7f7f080d1d90 0x7f7f080d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.539+0000 7f7f07200640 1 --2- 172.21.15.31:0/2688671713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f080b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.539+0000 7f7f07200640 1 -- 172.21.15.31:0/2688671713 >> 172.21.15.31:0/2688671713 conn(0x7f7f080d3e90 msgr2=0x7f7f080d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.539+0000 7f7f07200640 1 -- 172.21.15.31:0/2688671713 shutdown_connections 2024-09-13T08:36:44.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.542+0000 7f7f07200640 1 -- 172.21.15.31:0/2688671713 wait complete. 2024-09-13T08:36:44.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.544+0000 7f7f07200640 1 Processor -- start 2024-09-13T08:36:44.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.551+0000 7f7f07200640 1 -- start start 2024-09-13T08:36:44.553 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.553+0000 7f7f07200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f080d1d90 0x7f7f081f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f07200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f07200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f07200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080b3b20 con 0x7f7f080d2710 2024-09-13T08:36:44.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f07200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080b39a0 con 0x7f7f080d1d90 2024-09-13T08:36:44.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f07200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7f080b3ca0 con 0x7f7f08161470 2024-09-13T08:36:44.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f06800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f06800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46936/0 (socket says 172.21.15.31:46936) 2024-09-13T08:36:44.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.554+0000 7f7f06800640 1 -- 172.21.15.31:0/4258062996 learned_addr learned my addr 172.21.15.31:0/4258062996 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.555+0000 7f7f06800640 1 -- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f080d1d90 msgr2=0x7f7f081f4f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T08:36:44.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.555+0000 7f7f06800640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f080d1d90 0x7f7f081f4f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.558+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f5cdc005580 con 0x7f5d040cde80 2024-09-13T08:36:44.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.555+0000 7f7f06800640 1 -- 172.21.15.31:0/4258062996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 msgr2=0x7f7f080a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.555+0000 7f7f05400640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.556+0000 7f7f06800640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.556+0000 7f7f06800640 1 -- 172.21.15.31:0/4258062996 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f081f54d0 con 0x7f7f08161470 2024-09-13T08:36:44.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.556+0000 7f7f05400640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:36:44.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.557+0000 7f7f06800640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40098a0 tx=0x7f7ef4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.557+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40047a0 con 0x7f7f08161470 2024-09-13T08:36:44.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.557+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ef4004940 con 0x7f7f08161470 2024-09-13T08:36:44.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.557+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4004d70 con 0x7f7f08161470 2024-09-13T08:36:44.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.557+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f080ca230 con 0x7f7f08161470 2024-09-13T08:36:44.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.558+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7ef4003980 con 0x7f7f08161470 2024-09-13T08:36:44.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.567+0000 7f5cfaa00640 1 -- 172.21.15.31:0/629909646 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v10)=0 v10) ==== 115+0+0 (secure 0 0 0) 0x7f5cfc032030 con 0x7f5d040cde80 2024-09-13T08:36:44.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.559+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f080c7c10 con 0x7f7f08161470 2024-09-13T08:36:44.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.560+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7ef4003e00 con 0x7f7f08161470 2024-09-13T08:36:44.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.569+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f080b4150 con 0x7f7f08161470 2024-09-13T08:36:44.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 msgr2=0x7f5d040b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc032000 tx=0x7f5cfc004270 comp rx=0 tx=0).stop 2024-09-13T08:36:44.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 shutdown_connections 2024-09-13T08:36:44.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d0415df70 0x7f5d0415e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d040cde80 0x7f5d040b1900 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 --2- 172.21.15.31:0/629909646 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d040cbb40 0x7f5d040c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.570+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 >> 172.21.15.31:0/629909646 conn(0x7f5d040cfbb0 msgr2=0x7f5d040adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.572+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 shutdown_connections 2024-09-13T08:36:44.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.572+0000 7f5d09800640 1 -- 172.21.15.31:0/629909646 wait complete. 2024-09-13T08:36:44.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.576+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7ef4056070 con 0x7f7f08161470 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 -- 172.21.15.31:0/3169553603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4440d1d90 msgr2=0x7ff4440d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 --2- 172.21.15.31:0/3169553603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4440d1d90 0x7ff4440d21d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff430005fd0 tx=0x7ff43002a420 comp rx=0 tx=0).stop 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 -- 172.21.15.31:0/3169553603 shutdown_connections 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 --2- 172.21.15.31:0/3169553603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4440d1d90 0x7ff4440d21d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 --2- 172.21.15.31:0/3169553603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff444161470 0x7ff4440b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.605+0000 7ff449400640 1 -- 172.21.15.31:0/3169553603 >> 172.21.15.31:0/3169553603 conn(0x7ff4440d3e90 msgr2=0x7ff4440d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.608+0000 7ff449400640 1 -- 172.21.15.31:0/3169553603 shutdown_connections 2024-09-13T08:36:44.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.609+0000 7ff449400640 1 -- 172.21.15.31:0/3169553603 wait complete. 2024-09-13T08:36:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.611+0000 7ff449400640 1 Processor -- start 2024-09-13T08:36:44.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.614+0000 7ff449400640 1 -- start start 2024-09-13T08:36:44.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.616+0000 7ff449400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.617+0000 7ff449400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444161470 0x7ff4440caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.617+0000 7ff443e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.621+0000 7ff443e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46950/0 (socket says 172.21.15.31:46950) 2024-09-13T08:36:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.621+0000 7ff443e00640 1 -- 172.21.15.31:0/2357045616 learned_addr learned my addr 172.21.15.31:0/2357045616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.621+0000 7ff449400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 0x7ff44406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.622+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4440b5260 con 0x7ff44406da20 2024-09-13T08:36:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.622+0000 7ff448a00640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 0x7ff44406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.622+0000 7ff443400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444161470 0x7ff4440caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.622+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4440b26a0 con 0x7ff444161470 2024-09-13T08:36:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.622+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff4440b3b70 con 0x7ff4440d2710 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff448a00640 1 -- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 msgr2=0x7ff4441f4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff448a00640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff448a00640 1 -- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444161470 msgr2=0x7ff4440caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff448a00640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444161470 0x7ff4440caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff448a00640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4440cb4a0 con 0x7ff44406da20 2024-09-13T08:36:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.624+0000 7ff443e00640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.625+0000 7ff448a00640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 0x7ff44406deb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00be00 tx=0x7ff43c00bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.627+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c019070 con 0x7ff44406da20 2024-09-13T08:36:44.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.627+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff43c014440 con 0x7ff44406da20 2024-09-13T08:36:44.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.627+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0148d0 con 0x7ff44406da20 2024-09-13T08:36:44.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.628+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4441f4d90 con 0x7ff44406da20 2024-09-13T08:36:44.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.628+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4441f5350 con 0x7ff44406da20 2024-09-13T08:36:44.632 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 -- 172.21.15.31:0/3544294624 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980b1140 msgr2=0x7ff8980d23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 --2- 172.21.15.31:0/3544294624 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980b1140 0x7ff8980d23e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff890009be0 tx=0x7ff89002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.631+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41c005290 con 0x7ff44406da20 2024-09-13T08:36:44.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 -- 172.21.15.31:0/3544294624 shutdown_connections 2024-09-13T08:36:44.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 --2- 172.21.15.31:0/3544294624 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980b1140 0x7ff8980d23e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 -- 172.21.15.31:0/3544294624 >> 172.21.15.31:0/3544294624 conn(0x7ff8980d3e90 msgr2=0x7ff8980d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 -- 172.21.15.31:0/3544294624 shutdown_connections 2024-09-13T08:36:44.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff43c014a70 con 0x7ff44406da20 2024-09-13T08:36:44.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff43c004500 con 0x7ff44406da20 2024-09-13T08:36:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.636+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff43c004020 con 0x7ff44406da20 2024-09-13T08:36:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.634+0000 7ff89cc00640 1 -- 172.21.15.31:0/3544294624 wait complete. 2024-09-13T08:36:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.637+0000 7ff89cc00640 1 Processor -- start 2024-09-13T08:36:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.638+0000 7ff89cc00640 1 -- start start 2024-09-13T08:36:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.638+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7ff7ec005580 con 0x7ff8180d1d90 2024-09-13T08:36:44.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.642+0000 7ff89cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 0x7ff898153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.643+0000 7ff89cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 0x7ff8980cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.643+0000 7ff89cc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.644+0000 7ff89cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff89816d720 con 0x7ff8980d2000 2024-09-13T08:36:44.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.644+0000 7ff89cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8980b56f0 con 0x7ff8980b4ee0 2024-09-13T08:36:44.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.644+0000 7ff897e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.644+0000 7ff897e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46962/0 (socket says 172.21.15.31:46962) 2024-09-13T08:36:44.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.644+0000 7ff897e00640 1 -- 172.21.15.31:0/2217076926 learned_addr learned my addr 172.21.15.31:0/2217076926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.645+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff89816d8a0 con 0x7ff8980cfec0 2024-09-13T08:36:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.646+0000 7ff897400640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 0x7ff898153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.646+0000 7ff80e000640 1 -- 172.21.15.31:0/238480923 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7ff804003be0 con 0x7ff8180d1d90 2024-09-13T08:36:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.647+0000 7ff896a00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 0x7ff8980cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897400640 1 -- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 msgr2=0x7ff8980b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897400640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897400640 1 -- 172.21.15.31:0/2217076926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 msgr2=0x7ff8980cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897400640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 0x7ff8980cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897400640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8981fc970 con 0x7ff8980b4ee0 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.648+0000 7ff897e00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff896a00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 0x7ff8980cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:36:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff897400640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 0x7ff898153ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c3c0 tx=0x7ff88800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 msgr2=0x7ff8180b9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff80400c390 tx=0x7ff80400c850 comp rx=0 tx=0).stop 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 shutdown_connections 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8180d2170 0x7ff8180b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8180d1d90 0x7ff8180b9290 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 --2- 172.21.15.31:0/238480923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180b4ae0 0x7ff818201760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.649+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 >> 172.21.15.31:0/238480923 conn(0x7ff8180d3f50 msgr2=0x7ff81806d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.650+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 shutdown_connections 2024-09-13T08:36:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.650+0000 7ff803400640 1 -- 172.21.15.31:0/238480923 wait complete. 2024-09-13T08:36:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.654+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888011790 con 0x7ff8980b4ee0 2024-09-13T08:36:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.654+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8980b7d40 con 0x7ff8980b4ee0 2024-09-13T08:36:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.654+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8980b8300 con 0x7ff8980b4ee0 2024-09-13T08:36:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.656+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff888017070 con 0x7ff8980b4ee0 2024-09-13T08:36:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.656+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888011cc0 con 0x7ff8980b4ee0 2024-09-13T08:36:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.656+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff888005020 con 0x7ff8980b4ee0 2024-09-13T08:36:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.656+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff88801c460 con 0x7ff8980b4ee0 2024-09-13T08:36:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.656+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8980d2f40 con 0x7ff8980b4ee0 2024-09-13T08:36:44.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.662+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff8980d2f40 con 0x7ff8980b4ee0 2024-09-13T08:36:44.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.664+0000 7fd702e00640 1 -- 172.21.15.31:0/807449381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc16e820 msgr2=0x7fd6fc0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd6faa00640 1 -- 172.21.15.31:0/807449381 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4032630 con 0x7fd6fc16e820 2024-09-13T08:36:44.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 --2- 172.21.15.31:0/807449381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc16e820 0x7fd6fc0ceb50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4005fd0 tx=0x7fd6f402a780 comp rx=0 tx=0).stop 2024-09-13T08:36:44.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 -- 172.21.15.31:0/807449381 shutdown_connections 2024-09-13T08:36:44.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 --2- 172.21.15.31:0/807449381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc08b3d0 0x7fd6fc06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 --2- 172.21.15.31:0/807449381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc16e820 0x7fd6fc0ceb50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 -- 172.21.15.31:0/807449381 >> 172.21.15.31:0/807449381 conn(0x7fd6fc0cfbb0 msgr2=0x7fd6fc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 -- 172.21.15.31:0/807449381 shutdown_connections 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.665+0000 7fd702e00640 1 -- 172.21.15.31:0/807449381 wait complete. 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.666+0000 7fe54bc00640 1 -- 172.21.15.31:0/2286601641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440b4ae0 msgr2=0x7fe5440d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.666+0000 7fe54bc00640 1 --2- 172.21.15.31:0/2286601641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440b4ae0 0x7fe5440d1550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe5440b4a70 tx=0x7fe53002a530 comp rx=0 tx=0).stop 2024-09-13T08:36:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.666+0000 7fe54bc00640 1 -- 172.21.15.31:0/2286601641 shutdown_connections 2024-09-13T08:36:44.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fe54bc00640 1 --2- 172.21.15.31:0/2286601641 >> v2:172.21.15.31:3300/0 conn(0x7fe5440d2170 0x7fe5440b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fe54bc00640 1 --2- 172.21.15.31:0/2286601641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440b4ae0 0x7fe5440d1550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fe54bc00640 1 -- 172.21.15.31:0/2286601641 >> 172.21.15.31:0/2286601641 conn(0x7fe5440d3f50 msgr2=0x7fe5440d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fe54bc00640 1 -- 172.21.15.31:0/2286601641 shutdown_connections 2024-09-13T08:36:44.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fd702e00640 1 Processor -- start 2024-09-13T08:36:44.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fe54bc00640 1 -- 172.21.15.31:0/2286601641 wait complete. 2024-09-13T08:36:44.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.669+0000 7fd702e00640 1 -- start start 2024-09-13T08:36:44.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.671+0000 7fd702e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 0x7fd6fc0c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fd702e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc08b3d0 0x7fd6fc167140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fe54bc00640 1 Processor -- start 2024-09-13T08:36:44.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 0x7fd6fc0c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 0x7fd6fc0c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60432/0 (socket says 172.21.15.31:60432) 2024-09-13T08:36:44.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fd6fbe00640 1 -- 172.21.15.31:0/3150861811 learned_addr learned my addr 172.21.15.31:0/3150861811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fd6fb400640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc08b3d0 0x7fd6fc167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fd702e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 0x7fd6fc088ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc0cf210 con 0x7fd6fc08b3d0 2024-09-13T08:36:44.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fd700e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 0x7fd6fc088ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc0cf090 con 0x7fd6fc06f510 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd6fc0cf4e0 con 0x7fd6fc16e820 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fd700e00640 1 -- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 msgr2=0x7fd6fc0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fd700e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 0x7fd6fc0c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fd700e00640 1 -- 172.21.15.31:0/3150861811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc08b3d0 msgr2=0x7fd6fc167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fd700e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc08b3d0 0x7fd6fc167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fd700e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc0c5580 con 0x7fd6fc16e820 2024-09-13T08:36:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.675+0000 7fd700e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 0x7fd6fc088ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c3c0 tx=0x7fd6e800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8011790 con 0x7fd6fc16e820 2024-09-13T08:36:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6fc0c4900 con 0x7fd6fc16e820 2024-09-13T08:36:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6e8017070 con 0x7fd6fc16e820 2024-09-13T08:36:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8011790 con 0x7fd6fc16e820 2024-09-13T08:36:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6fc0c4ec0 con 0x7fd6fc16e820 2024-09-13T08:36:44.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd6e8011ac0 con 0x7fd6fc16e820 2024-09-13T08:36:44.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd6e80125e0 con 0x7fd6fc16e820 2024-09-13T08:36:44.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.672+0000 7fe54bc00640 1 -- start start 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fe54bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5440b4ae0 0x7fe5441ea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fe54bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fe54bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5440d2170 0x7fe5441fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fe54bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5440cedc0 con 0x7fe5440d2170 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.673+0000 7fe54bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5440cec40 con 0x7fe5440d1d90 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.674+0000 7fe54bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe5440cef40 con 0x7fe5440b4ae0 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fe543e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fe543e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60444/0 (socket says 172.21.15.31:60444) 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.676+0000 7fe543e00640 1 -- 172.21.15.31:0/741777074 learned_addr learned my addr 172.21.15.31:0/741777074 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fe543e00640 1 -- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5440b4ae0 msgr2=0x7fe5441ea6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fe543e00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5440b4ae0 0x7fe5441ea6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fe543e00640 1 -- 172.21.15.31:0/741777074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5440d2170 msgr2=0x7fe5441fc7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T08:36:44.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fe543e00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5440d2170 0x7fe5441fc7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.677+0000 7fe543e00640 1 -- 172.21.15.31:0/741777074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5441f8fd0 con 0x7fe5440d1d90 2024-09-13T08:36:44.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.679+0000 7fe543e00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe530009780 tx=0x7fe530032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d4005290 con 0x7fd6fc16e820 2024-09-13T08:36:44.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530040070 con 0x7fe5440d1d90 2024-09-13T08:36:44.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe530005310 con 0x7fe5440d1d90 2024-09-13T08:36:44.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5300047a0 con 0x7fe5440d1d90 2024-09-13T08:36:44.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5441f9250 con 0x7fe5440d1d90 2024-09-13T08:36:44.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.681+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5441f9810 con 0x7fe5440d1d90 2024-09-13T08:36:44.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.687+0000 7f078ca00640 1 -- 172.21.15.31:0/900053178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0780036540 msgr2=0x7f0780009a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.687+0000 7f078ca00640 1 --2- 172.21.15.31:0/900053178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0780036540 0x7f0780009a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0788069f40 tx=0x7f07880a5c70 comp rx=0 tx=0).stop 2024-09-13T08:36:44.689 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 -- 172.21.15.31:0/900053178 shutdown_connections 2024-09-13T08:36:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 --2- 172.21.15.31:0/900053178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0780036540 0x7f0780009a70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 --2- 172.21.15.31:0/900053178 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780005ad0 0x7f0780036000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 -- 172.21.15.31:0/900053178 >> 172.21.15.31:0/900053178 conn(0x7f0780054580 msgr2=0x7f0780054960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 -- 172.21.15.31:0/900053178 shutdown_connections 2024-09-13T08:36:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.686+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd6e8011db0 con 0x7fd6fc16e820 2024-09-13T08:36:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.690+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe530039040 con 0x7fe5440d1d90 2024-09-13T08:36:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.688+0000 7f078ca00640 1 -- 172.21.15.31:0/900053178 wait complete. 2024-09-13T08:36:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.690+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe530032020 con 0x7fe5440d1d90 2024-09-13T08:36:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.690+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5440b3ec0 con 0x7fe5440d1d90 2024-09-13T08:36:44.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.692+0000 7f078ca00640 1 Processor -- start 2024-09-13T08:36:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.692+0000 7f078ca00640 1 -- start start 2024-09-13T08:36:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.692+0000 7f9bd9c00640 1 -- 172.21.15.31:0/1995226479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40cdeb0 msgr2=0x7f9bd415e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.692+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/1995226479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40cdeb0 0x7f9bd415e210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40cef70 tx=0x7f9bcc02a870 comp rx=0 tx=0).stop 2024-09-13T08:36:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.693+0000 7f9bd9c00640 1 -- 172.21.15.31:0/1995226479 shutdown_connections 2024-09-13T08:36:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.693+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/1995226479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40cdeb0 0x7f9bd415e210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.693+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/1995226479 >> v2:172.21.15.45:3300/0 conn(0x7f9bd40ccaf0 0x7f9bd40ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.693+0000 7f9bd9c00640 1 -- 172.21.15.31:0/1995226479 >> 172.21.15.31:0/1995226479 conn(0x7f9bd40b1520 msgr2=0x7f9bd40b1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.693+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe5440b3ec0 con 0x7fe5440d1d90 2024-09-13T08:36:44.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.695+0000 7f078ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0780005ad0 0x7f07800dd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.695+0000 7f078ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.695+0000 7f9bd9c00640 1 -- 172.21.15.31:0/1995226479 shutdown_connections 2024-09-13T08:36:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.695+0000 7f078ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0785e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0780005ad0 0x7f07800dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0785400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0786800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0786800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46990/0 (socket says 172.21.15.31:46990) 2024-09-13T08:36:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0785400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36078/0 (socket says 172.21.15.31:36078) 2024-09-13T08:36:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f0786800640 1 -- 172.21.15.31:0/510133326 learned_addr learned my addr 172.21.15.31:0/510133326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.696+0000 7f078ca00640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0780009fb0 con 0x7f078001c8b0 2024-09-13T08:36:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.697+0000 7f078ca00640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f078000bcb0 con 0x7f0780005ad0 2024-09-13T08:36:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.697+0000 7f078ca00640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f078000a130 con 0x7f0780036540 2024-09-13T08:36:44.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.697+0000 7f9bd9c00640 1 -- 172.21.15.31:0/1995226479 wait complete. 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.699+0000 7f0786800640 1 -- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0780005ad0 msgr2=0x7f07800dd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0786800640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0780005ad0 0x7f07800dd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0786800640 1 -- 172.21.15.31:0/510133326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 msgr2=0x7f0780046c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0786800640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0786800640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07800de180 con 0x7f0780036540 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0785400640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:36:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.700+0000 7f0786800640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f078806b040 tx=0x7f07880ae9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.701+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07880b3050 con 0x7f0780036540 2024-09-13T08:36:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.702+0000 7f9bd9c00640 1 Processor -- start 2024-09-13T08:36:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.702+0000 7f9bd9c00640 1 -- start start 2024-09-13T08:36:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.704+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07880b4040 con 0x7f0780036540 2024-09-13T08:36:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.704+0000 7f078ca00640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07800de460 con 0x7f0780036540 2024-09-13T08:36:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.704+0000 7f078ca00640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07800ddd70 con 0x7f0780036540 2024-09-13T08:36:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.704+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07880ba450 con 0x7f0780036540 2024-09-13T08:36:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.705+0000 7f9bd9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 0x7f9bd414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.706+0000 7f9bd9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.706+0000 7f9bd9c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bd40cdeb0 0x7f9bd40b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:44.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.706+0000 7f9bcbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.706+0000 7f9bcbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60476/0 (socket says 172.21.15.31:60476) 2024-09-13T08:36:44.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.706+0000 7f9bcbe00640 1 -- 172.21.15.31:0/2015160410 learned_addr learned my addr 172.21.15.31:0/2015160410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.707+0000 7f9bd3e00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 0x7f9bd414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.707+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd406e7b0 con 0x7f9bd40ccaf0 2024-09-13T08:36:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.707+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd406e630 con 0x7f9bd40cdad0 2024-09-13T08:36:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.707+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd406e930 con 0x7f9bd40cdeb0 2024-09-13T08:36:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.707+0000 7f9bd9200640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bd40cdeb0 0x7f9bd40b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.708+0000 7f9bcbe00640 1 -- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bd40cdeb0 msgr2=0x7f9bd40b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.708+0000 7f9bcbe00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bd40cdeb0 0x7f9bd40b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.708+0000 7f9bcbe00640 1 -- 172.21.15.31:0/2015160410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 msgr2=0x7f9bd414b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.709+0000 7f9bcbe00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 0x7f9bd414b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.709+0000 7f9bcbe00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd4156be0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.709+0000 7f9bd3e00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 0x7f9bd414b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.709+0000 7f9bcbe00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400e790 tx=0x7f9bc400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.710+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f07880c1020 con 0x7f0780036540 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.710+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f07880badb0 con 0x7f0780036540 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.710+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075c005290 con 0x7f0780036540 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.710+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc400eea0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.710+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd414dba0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.711+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bc400cdb0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.711+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc400eea0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.711+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd4156e80 con 0x7f9bd40cdad0 2024-09-13T08:36:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.713+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9bc4010430 con 0x7f9bd40cdad0 2024-09-13T08:36:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.713+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9bc40188c0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.713+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f07880ba5f0 con 0x7f0780036540 2024-09-13T08:36:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.718+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd414e0c0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.719+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f2914005580 con 0x7f29380d4b30 2024-09-13T08:36:44.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.723+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f9bd414e0c0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.727+0000 7f2935600640 1 -- 172.21.15.31:0/1038414449 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v12)=0 v12) ==== 121+0+0 (secure 0 0 0) 0x7f29381504a0 con 0x7f29380d4b30 2024-09-13T08:36:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.731+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 msgr2=0x7f29381f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.731+0000 7f2940400640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29381f7bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f293000be00 tx=0x7f293000bef0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 shutdown_connections 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29380d4b30 0x7f29381f7bc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29380700a0 0x7f2938160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 --2- 172.21.15.31:0/1038414449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f293806fc60 0x7f293814ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 >> 172.21.15.31:0/1038414449 conn(0x7f29380d20e0 msgr2=0x7f293806eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 shutdown_connections 2024-09-13T08:36:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.732+0000 7f2940400640 1 -- 172.21.15.31:0/1038414449 wait complete. 2024-09-13T08:36:44.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.762+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f7f080a1d80 con 0x7f7f08161470 2024-09-13T08:36:44.763 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.770+0000 7f7efe000640 1 -- 172.21.15.31:0/4258062996 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v13)=0 v13) ==== 119+0+0 (secure 0 0 0) 0x7f7ef4056070 con 0x7f7f08161470 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 msgr2=0x7f7f0815dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40098a0 tx=0x7f7ef4004270 comp rx=0 tx=0).stop 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 shutdown_connections 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7f08161470 0x7f7f0815dae0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f080d2710 0x7f7f080a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 --2- 172.21.15.31:0/4258062996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f080d1d90 0x7f7f081f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.772+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 >> 172.21.15.31:0/4258062996 conn(0x7f7f080d3e90 msgr2=0x7f7f0806e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.774+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 shutdown_connections 2024-09-13T08:36:44.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.774+0000 7f7f07200640 1 -- 172.21.15.31:0/4258062996 wait complete. 2024-09-13T08:36:44.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.782+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7ff41c005580 con 0x7ff44406da20 2024-09-13T08:36:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.789+0000 7ff43be00640 1 -- 172.21.15.31:0/2357045616 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v14)=0 v14) ==== 121+0+0 (secure 0 0 0) 0x7ff43c014d60 con 0x7ff44406da20 2024-09-13T08:36:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.790+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 msgr2=0x7ff44406deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 0x7ff44406deb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00be00 tx=0x7ff43c00bef0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 shutdown_connections 2024-09-13T08:36:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff44406da20 0x7ff44406deb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444161470 0x7ff4440caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 --2- 172.21.15.31:0/2357045616 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4440d2710 0x7ff4441f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.791+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 >> 172.21.15.31:0/2357045616 conn(0x7ff4440d3e90 msgr2=0x7ff4440d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.792+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 shutdown_connections 2024-09-13T08:36:44.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.792+0000 7ff449400640 1 -- 172.21.15.31:0/2357045616 wait complete. 2024-09-13T08:36:44.800 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.815 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.822+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fe5440d1550 con 0x7fe5440d1d90 2024-09-13T08:36:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.827+0000 7fe540c00640 1 -- 172.21.15.31:0/741777074 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v15)=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7fe530049060 con 0x7fe5440d1d90 2024-09-13T08:36:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.828+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7ff8980d37a0 con 0x7ff8980b4ee0 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 msgr2=0x7fe5441eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe530009780 tx=0x7fe530032c10 comp rx=0 tx=0).stop 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 shutdown_connections 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5440d2170 0x7fe5441fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5440d1d90 0x7fe5441eabf0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe530009780 tx=0x7fe530032c10 comp rx=0 tx=0).stop 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 --2- 172.21.15.31:0/741777074 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5440b4ae0 0x7fe5441ea6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 >> 172.21.15.31:0/741777074 conn(0x7fe5440d3f50 msgr2=0x7fe5440b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.829+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 shutdown_connections 2024-09-13T08:36:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.830+0000 7fe54bc00640 1 -- 172.21.15.31:0/741777074 wait complete. 2024-09-13T08:36:44.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.834+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fd6d4005580 con 0x7fd6fc16e820 2024-09-13T08:36:44.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.835+0000 7ff88f400640 1 -- 172.21.15.31:0/2217076926 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v16)=0 v16) ==== 120+0+0 (secure 0 0 0) 0x7ff8980d37a0 con 0x7ff8980b4ee0 2024-09-13T08:36:44.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 msgr2=0x7ff898153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 0x7ff898153ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c3c0 tx=0x7ff88800c880 comp rx=0 tx=0).stop 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 shutdown_connections 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8980cfec0 0x7ff8980b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8980d2000 0x7ff8980cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 --2- 172.21.15.31:0/2217076926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8980b4ee0 0x7ff898153ee0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.836+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 >> 172.21.15.31:0/2217076926 conn(0x7ff8980d3e90 msgr2=0x7ff89816bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.837+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 shutdown_connections 2024-09-13T08:36:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.837+0000 7ff89cc00640 1 -- 172.21.15.31:0/2217076926 wait complete. 2024-09-13T08:36:44.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.843+0000 7fd6f3e00640 1 -- 172.21.15.31:0/3150861811 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v17)=0 v17) ==== 125+0+0 (secure 0 0 0) 0x7fd6e80127e0 con 0x7fd6fc16e820 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.844+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 msgr2=0x7fd6fc088ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.844+0000 7fd702e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 0x7fd6fc088ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c3c0 tx=0x7fd6e800c880 comp rx=0 tx=0).stop 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 shutdown_connections 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd6fc16e820 0x7fd6fc088ee0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc08b3d0 0x7fd6fc167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 --2- 172.21.15.31:0/3150861811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc06f510 0x7fd6fc0c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 >> 172.21.15.31:0/3150861811 conn(0x7fd6fc0cfbb0 msgr2=0x7fd6fc16a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 shutdown_connections 2024-09-13T08:36:44.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.845+0000 7fd702e00640 1 -- 172.21.15.31:0/3150861811 wait complete. 2024-09-13T08:36:44.856 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.856 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.872+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f075c005580 con 0x7f0780036540 2024-09-13T08:36:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.878+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f9bd40cc2e0 con 0x7f9bd40cdad0 2024-09-13T08:36:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.879+0000 7f077e000640 1 -- 172.21.15.31:0/510133326 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v18)=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f07880ba7d0 con 0x7f0780036540 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 msgr2=0x7f0780189480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f078806b040 tx=0x7f07880ae9e0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 shutdown_connections 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0780036540 0x7f0780189480 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078001c8b0 0x7f0780046c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 --2- 172.21.15.31:0/510133326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0780005ad0 0x7f07800dd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 >> 172.21.15.31:0/510133326 conn(0x7f0780054580 msgr2=0x7f0780054960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 shutdown_connections 2024-09-13T08:36:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.880+0000 7f0773400640 1 -- 172.21.15.31:0/510133326 wait complete. 2024-09-13T08:36:44.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.884+0000 7f9bd0c00640 1 -- 172.21.15.31:0/2015160410 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f9bc4018420 con 0x7f9bd40cdad0 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 msgr2=0x7f9bd4157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400e790 tx=0x7f9bc400ec50 comp rx=0 tx=0).stop 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 shutdown_connections 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bd40cdeb0 0x7f9bd40b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd40cdad0 0x7f9bd4157680 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 --2- 172.21.15.31:0/2015160410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd40ccaf0 0x7f9bd414b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 >> 172.21.15.31:0/2015160410 conn(0x7f9bd40b1520 msgr2=0x7f9bd406f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 shutdown_connections 2024-09-13T08:36:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:44.885+0000 7f9bd9c00640 1 -- 172.21.15.31:0/2015160410 wait complete. 2024-09-13T08:36:44.903 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.903 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-13T08:36:44.904 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T08:36:44.904 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 -- 172.21.15.31:0/2464928224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 msgr2=0x7fe9900d49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 --2- 172.21.15.31:0/2464928224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9900d49b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe980005fd0 tx=0x7fe98002a540 comp rx=0 tx=0).stop 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 -- 172.21.15.31:0/2464928224 shutdown_connections 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 --2- 172.21.15.31:0/2464928224 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 --2- 172.21.15.31:0/2464928224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9900d49b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 -- 172.21.15.31:0/2464928224 >> 172.21.15.31:0/2464928224 conn(0x7fe9900d3e90 msgr2=0x7fe9900d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 -- 172.21.15.31:0/2464928224 shutdown_connections 2024-09-13T08:36:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.098+0000 7fe98f200640 1 -- 172.21.15.31:0/2464928224 wait complete. 2024-09-13T08:36:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.100+0000 7fe98f200640 1 Processor -- start 2024-09-13T08:36:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.100+0000 7fe98f200640 1 -- start start 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.100+0000 7fe98f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 0x7fe9900b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9901f8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900d0d30 con 0x7fe99015e0a0 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900b5590 con 0x7fe9900d2450 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe9900d0eb0 con 0x7fe9900cf470 2024-09-13T08:36:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9901f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:54602/0 (socket says 172.21.15.31:54602) 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98de00640 1 -- 172.21.15.31:0/798735914 learned_addr learned my addr 172.21.15.31:0/798735914 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98d400640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 0x7fe9900b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 -- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 msgr2=0x7fe9900b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 -- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 msgr2=0x7fe9900b8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 0x7fe9900b8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98e800640 1 -- 172.21.15.31:0/798735914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9901f4870 con 0x7fe99015e0a0 2024-09-13T08:36:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98de00640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:45.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.101+0000 7fe98d400640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 0x7fe9900b8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:45.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe98e800640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9901f8550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe98800c680 tx=0x7fe98800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988003b70 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe988003d30 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880052d0 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9901f4b40 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9901f5100 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe988005470 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.102+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe98801b8b0 con 0x7fe99015e0a0 2024-09-13T08:36:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.103+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9900d1370 con 0x7fe99015e0a0 2024-09-13T08:36:45.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.106+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe98801bab0 con 0x7fe99015e0a0 2024-09-13T08:36:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.174+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe99015ca30 con 0x7fe99015e0a0 2024-09-13T08:36:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.177+0000 7fe986000640 1 -- 172.21.15.31:0/798735914 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fe98801bc90 con 0x7fe99015e0a0 2024-09-13T08:36:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-09-13T08:36:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 msgr2=0x7fe9901f8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9901f8550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe98800c680 tx=0x7fe98800cb40 comp rx=0 tx=0).stop 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 shutdown_connections 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe99015e0a0 0x7fe9901f8550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9900d2450 0x7fe9900b8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 --2- 172.21.15.31:0/798735914 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9900cf470 0x7fe9900b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.178+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 >> 172.21.15.31:0/798735914 conn(0x7fe9900d3e90 msgr2=0x7fe99006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.179+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 shutdown_connections 2024-09-13T08:36:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:45.179+0000 7fe98f200640 1 -- 172.21.15.31:0/798735914 wait complete. 2024-09-13T08:36:45.195 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T08:36:45.195 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T08:36:45.195 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:45.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T08:36:45.250 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:45.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T08:36:45.345 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:45.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T08:36:45.441 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:36:45.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T08:36:45.538 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:45.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T08:36:45.594 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:45.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T08:36:45.691 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:45.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T08:36:45.787 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:36:45.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T08:36:45.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new e7903a6a-0c16-44bc-ad3f-35e6d004dd97 0 2024-09-13T08:36:46.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 -- 172.21.15.45:0/3900368038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c0b33a0 msgr2=0x7f225c0b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 --2- 172.21.15.45:0/3900368038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c0b33a0 0x7f225c0b37e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f225c0b6900 tx=0x7f225402aac0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 -- 172.21.15.45:0/3900368038 shutdown_connections 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 --2- 172.21.15.45:0/3900368038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c0b33a0 0x7f225c0b37e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 --2- 172.21.15.45:0/3900368038 >> v2:172.21.15.31:3300/0 conn(0x7f225c161590 0x7f225c0d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 -- 172.21.15.45:0/3900368038 >> 172.21.15.45:0/3900368038 conn(0x7f225c0d3d90 msgr2=0x7f225c0d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 -- 172.21.15.45:0/3900368038 shutdown_connections 2024-09-13T08:36:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.078+0000 7f225be00640 1 -- 172.21.15.45:0/3900368038 wait complete. 2024-09-13T08:36:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.079+0000 7f225be00640 1 Processor -- start 2024-09-13T08:36:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.080+0000 7f225be00640 1 -- start start 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.080+0000 7f225be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.080+0000 7f225be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c161590 0x7f225c1f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f225c0ce9e0 con 0x7f225c0b33a0 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f225c0d2af0 con 0x7f225c161590 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f225c0b3d20 con 0x7f225c0d0160 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f2253e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.45:35494/0 (socket says 172.21.15.45:35494) 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 -- 172.21.15.45:0/2120333066 learned_addr learned my addr 172.21.15.45:0/2120333066 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f2253e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.45:41272/0 (socket says 172.21.15.45:41272) 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225b400640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c161590 0x7f225c1f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 -- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 msgr2=0x7f225c1f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 -- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c161590 msgr2=0x7f225c1f7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c161590 0x7f225c1f7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f225aa00640 1 -- 172.21.15.45:0/2120333066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225c15b910 con 0x7f225c0b33a0 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.081+0000 7f2253e00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:46.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f225aa00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f224c00e790 tx=0x7f224c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c00eea0 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f224c00cdb0 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c010690 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225c1f58d0 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225c0b70e0 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f224c010830 con 0x7f225c0b33a0 2024-09-13T08:36:46.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.082+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f224c018d50 con 0x7f225c0b33a0 2024-09-13T08:36:46.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.083+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f225c0b37e0 con 0x7f225c0b33a0 2024-09-13T08:36:46.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.086+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f225c1f58d0 con 0x7f225c0b33a0 2024-09-13T08:36:46.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.161+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e7903a6a-0c16-44bc-ad3f-35e6d004dd97", "id": 0} v 0) -- 0x7f225c15a450 con 0x7f225c0b33a0 2024-09-13T08:36:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.166+0000 7f2252a00640 1 -- 172.21.15.45:0/2120333066 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7903a6a-0c16-44bc-ad3f-35e6d004dd97", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f225c15b910 con 0x7f225c0b33a0 2024-09-13T08:36:46.167 INFO:teuthology.orchestra.run.smithi045.stdout:0 2024-09-13T08:36:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.167+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 msgr2=0x7f225c1f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.167+0000 7f225be00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f224c00e790 tx=0x7f224c00ec50 comp rx=0 tx=0).stop 2024-09-13T08:36:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 shutdown_connections 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f225c161590 0x7f225c1f7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f225c0d0160 0x7f225c1f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 --2- 172.21.15.45:0/2120333066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f225c0b33a0 0x7f225c1f1ee0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 >> 172.21.15.45:0/2120333066 conn(0x7f225c0d3d90 msgr2=0x7f225c06eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 shutdown_connections 2024-09-13T08:36:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.168+0000 7f225be00640 1 -- 172.21.15.45:0/2120333066 wait complete. 2024-09-13T08:36:46.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new 15a51edd-43e9-4b8c-9d97-70734e704624 1 2024-09-13T08:36:46.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.337+0000 7fa88c200640 1 -- 172.21.15.45:0/3745385979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 msgr2=0x7fa884072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.337+0000 7fa88c200640 1 --2- 172.21.15.45:0/3745385979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa884072560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009be0 tx=0x7fa87c02a920 comp rx=0 tx=0).stop 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.338+0000 7fa88c200640 1 -- 172.21.15.45:0/3745385979 shutdown_connections 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.338+0000 7fa88c200640 1 --2- 172.21.15.45:0/3745385979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa884072560 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.338+0000 7fa88c200640 1 -- 172.21.15.45:0/3745385979 >> 172.21.15.45:0/3745385979 conn(0x7fa8840cfb70 msgr2=0x7fa8840cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.338+0000 7fa88c200640 1 -- 172.21.15.45:0/3745385979 shutdown_connections 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.338+0000 7fa88c200640 1 -- 172.21.15.45:0/3745385979 wait complete. 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.339+0000 7fa88c200640 1 Processor -- start 2024-09-13T08:36:46.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.339+0000 7fa88c200640 1 -- start start 2024-09-13T08:36:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa88414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88415c040 0x7fa8841f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa88406fa20 con 0x7fa88415c040 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa88406f8a0 con 0x7fa8840d0690 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa88c200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa88406fba0 con 0x7fa8840cbc80 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.340+0000 7fa888e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37284/0 (socket says 172.21.15.45:37284) 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 -- 172.21.15.45:0/1180744592 learned_addr learned my addr 172.21.15.45:0/1180744592 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa88a200640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88415c040 0x7fa8841f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa889800640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa88414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 -- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 msgr2=0x7fa88414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa88414d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 -- 172.21.15.45:0/1180744592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88415c040 msgr2=0x7fa8841f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88415c040 0x7fa8841f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 -- 172.21.15.45:0/1180744592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88414da20 con 0x7fa8840d0690 2024-09-13T08:36:46.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa888e00640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa87000c500 tx=0x7fa87000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.341+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8700052d0 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa870005470 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870012870 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8840b1170 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8840b1730 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa870012a10 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.342+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fa87001b600 con 0x7fa8840d0690 2024-09-13T08:36:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.343+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884070090 con 0x7fa8840d0690 2024-09-13T08:36:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.347+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa87001b800 con 0x7fa8840d0690 2024-09-13T08:36:46.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.422+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "15a51edd-43e9-4b8c-9d97-70734e704624", "id": 1} v 0) -- 0x7fa884069850 con 0x7fa8840d0690 2024-09-13T08:36:46.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.426+0000 7fa881600640 1 -- 172.21.15.45:0/1180744592 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15a51edd-43e9-4b8c-9d97-70734e704624", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fa87001b9e0 con 0x7fa8840d0690 2024-09-13T08:36:46.428 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:36:46.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 msgr2=0x7fa8841f7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa87000c500 tx=0x7fa87000c9c0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 shutdown_connections 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88415c040 0x7fa8841f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8840d0690 0x7fa8841f7bf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 --2- 172.21.15.45:0/1180744592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa8840cbc80 0x7fa88414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 >> 172.21.15.45:0/1180744592 conn(0x7fa8840cfb70 msgr2=0x7fa88415a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 shutdown_connections 2024-09-13T08:36:46.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.428+0000 7fa88c200640 1 -- 172.21.15.45:0/1180744592 wait complete. 2024-09-13T08:36:46.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new 1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe 2 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 -- 172.21.15.45:0/532027201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c1616f0 msgr2=0x7f4b5c0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 --2- 172.21.15.45:0/532027201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c1616f0 0x7f4b5c0b4840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4b50005fd0 tx=0x7f4b5002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 -- 172.21.15.45:0/532027201 shutdown_connections 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 --2- 172.21.15.45:0/532027201 >> v2:172.21.15.31:3300/0 conn(0x7f4b5c0d4b10 0x7f4b5c0b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 --2- 172.21.15.45:0/532027201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c1616f0 0x7f4b5c0b4840 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 -- 172.21.15.45:0/532027201 >> 172.21.15.45:0/532027201 conn(0x7f4b5c0d3ff0 msgr2=0x7f4b5c0d4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 -- 172.21.15.45:0/532027201 shutdown_connections 2024-09-13T08:36:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.596+0000 7f4b64400640 1 -- 172.21.15.45:0/532027201 wait complete. 2024-09-13T08:36:46.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.598+0000 7f4b64400640 1 Processor -- start 2024-09-13T08:36:46.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.598+0000 7f4b64400640 1 -- start start 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b5c0d4b10 0x7f4b5c0cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b5c1616f0 0x7f4b5c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c162240 con 0x7f4b5c1616f0 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c1620c0 con 0x7f4b5c0d4b10 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b64400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4b5c1623c0 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b5c0d4b10 0x7f4b5c0cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.45:41318/0 (socket says 172.21.15.45:41318) 2024-09-13T08:36:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 -- 172.21.15.45:0/2350409632 learned_addr learned my addr 172.21.15.45:0/2350409632 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b62400640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b5c1616f0 0x7f4b5c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 -- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b5c0d4b10 msgr2=0x7f4b5c0cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b5c0d4b10 0x7f4b5c0cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 -- 172.21.15.45:0/2350409632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b5c1616f0 msgr2=0x7f4b5c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.599+0000 7f4b61a00640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b5c1616f0 0x7f4b5c15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b61a00640 1 -- 172.21.15.45:0/2350409632 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5c1f4e20 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b61a00640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4b50034040 tx=0x7f4b500038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5003e030 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b50003ca0 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50035040 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5c1f5100 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.600+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b5c1f56c0 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.601+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b50041030 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.601+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f4b50005190 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.601+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b5c162870 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.604+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f4b50004d70 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.678+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe", "id": 2} v 0) -- 0x7f4b5c0d23b0 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.682+0000 7f4b59600640 1 -- 172.21.15.45:0/2350409632 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f4b50004f90 con 0x7f4b5c0b16a0 2024-09-13T08:36:46.683 INFO:teuthology.orchestra.run.smithi045.stdout:2 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 msgr2=0x7f4b5c0a1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4b50034040 tx=0x7f4b500038a0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 shutdown_connections 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b5c1616f0 0x7f4b5c15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b5c0d4b10 0x7f4b5c0cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 --2- 172.21.15.45:0/2350409632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b5c0b16a0 0x7f4b5c0a1550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 >> 172.21.15.45:0/2350409632 conn(0x7f4b5c0d3ff0 msgr2=0x7f4b5c0d25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 shutdown_connections 2024-09-13T08:36:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.683+0000 7f4b64400640 1 -- 172.21.15.45:0/2350409632 wait complete. 2024-09-13T08:36:46.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new 022dffd3-c8d6-406e-a985-d47b5c8e8a91 3 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 -- 172.21.15.45:0/3315789954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d1d90 msgr2=0x7fc7bc0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/3315789954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc0d1550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005fd0 tx=0x7fc7ac02aaa0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 -- 172.21.15.45:0/3315789954 shutdown_connections 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/3315789954 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0d2170 0x7fc7bc0b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/3315789954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc0d1550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 -- 172.21.15.45:0/3315789954 >> 172.21.15.45:0/3315789954 conn(0x7fc7bc0d3f50 msgr2=0x7fc7bc0d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.854+0000 7fc7c2e00640 1 -- 172.21.15.45:0/3315789954 shutdown_connections 2024-09-13T08:36:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.855+0000 7fc7c2e00640 1 -- 172.21.15.45:0/3315789954 wait complete. 2024-09-13T08:36:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.856+0000 7fc7c2e00640 1 Processor -- start 2024-09-13T08:36:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.856+0000 7fc7c2e00640 1 -- start start 2024-09-13T08:36:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0b4ae0 0x7fc7bc1f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc0cedc0 con 0x7fc7bc0d1d90 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc0cec40 con 0x7fc7bc0d2170 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.857+0000 7fc7c2e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc0cef40 con 0x7fc7bc0b4ae0 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0b4ae0 0x7fc7bc1f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59110/0 (socket says 172.21.15.45:59110) 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 -- 172.21.15.45:0/2553668525 learned_addr learned my addr 172.21.15.45:0/2553668525 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7bb400640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 -- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0b4ae0 msgr2=0x7fc7bc1f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0b4ae0 0x7fc7bc1f7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 -- 172.21.15.45:0/2553668525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7bc0d1d90 msgr2=0x7fc7bc146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc146ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 -- 172.21.15.45:0/2553668525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc0b6ff0 con 0x7fc7bc0d2170 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.858+0000 7fc7c0e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400e740 tx=0x7fc7b400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:46.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400ccc0 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7b400ce80 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4005280 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc0b26b0 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc0b2c70 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc7b40047a0 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.859+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fc7b4004a90 con 0x7fc7bc0d2170 2024-09-13T08:36:46.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.860+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc0d1550 con 0x7fc7bc0d2170 2024-09-13T08:36:46.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.862+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc7b4004db0 con 0x7fc7bc0d2170 2024-09-13T08:36:46.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.939+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "022dffd3-c8d6-406e-a985-d47b5c8e8a91", "id": 3} v 0) -- 0x7fc7bc0b2e40 con 0x7fc7bc0d2170 2024-09-13T08:36:46.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.942+0000 7fc7b3e00640 1 -- 172.21.15.45:0/2553668525 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "022dffd3-c8d6-406e-a985-d47b5c8e8a91", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fc7b4021440 con 0x7fc7bc0d2170 2024-09-13T08:36:46.944 INFO:teuthology.orchestra.run.smithi045.stdout:3 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.943+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 msgr2=0x7fc7bc1470e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.943+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400e740 tx=0x7fc7b400ec00 comp rx=0 tx=0).stop 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 shutdown_connections 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0d2170 0x7fc7bc1470e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7bc0d1d90 0x7fc7bc146ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 --2- 172.21.15.45:0/2553668525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7bc0b4ae0 0x7fc7bc1f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 >> 172.21.15.45:0/2553668525 conn(0x7fc7bc0d3f50 msgr2=0x7fc7bc158470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:46.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 shutdown_connections 2024-09-13T08:36:46.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:46.944+0000 7fc7c2e00640 1 -- 172.21.15.45:0/2553668525 wait complete. 2024-09-13T08:36:46.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new dc59624a-a6b1-451d-b54c-a2b12eaed3f5 4 2024-09-13T08:36:47.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.112+0000 7fd5bc200640 1 -- 172.21.15.45:0/2167827707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 msgr2=0x7fd5b40b6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.112+0000 7fd5bc200640 1 --2- 172.21.15.45:0/2167827707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40b6d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0005fd0 tx=0x7fd5a002a560 comp rx=0 tx=0).stop 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.113+0000 7fd5bc200640 1 -- 172.21.15.45:0/2167827707 shutdown_connections 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.113+0000 7fd5bc200640 1 --2- 172.21.15.45:0/2167827707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40b6d80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.113+0000 7fd5bc200640 1 -- 172.21.15.45:0/2167827707 >> 172.21.15.45:0/2167827707 conn(0x7fd5b40d3f30 msgr2=0x7fd5b40d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.113+0000 7fd5bc200640 1 -- 172.21.15.45:0/2167827707 shutdown_connections 2024-09-13T08:36:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.113+0000 7fd5bc200640 1 -- 172.21.15.45:0/2167827707 wait complete. 2024-09-13T08:36:47.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.114+0000 7fd5bc200640 1 Processor -- start 2024-09-13T08:36:47.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.114+0000 7fd5bc200640 1 -- start start 2024-09-13T08:36:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 0x7fd5b414c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5b415e0b0 0x7fd5b40c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b40b7890 con 0x7fd5b40b72c0 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b40b7710 con 0x7fd5b40d4a20 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.115+0000 7fd5bc200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b40d0df0 con 0x7fd5b415e0b0 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5ba200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5b415e0b0 0x7fd5b40c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 0x7fd5b414c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59120/0 (socket says 172.21.15.45:59120) 2024-09-13T08:36:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b8e00640 1 -- 172.21.15.45:0/1073407783 learned_addr learned my addr 172.21.15.45:0/1073407783 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 -- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5b415e0b0 msgr2=0x7fd5b40c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5b415e0b0 0x7fd5b40c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 -- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 msgr2=0x7fd5b40c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.116+0000 7fd5b9800640 1 -- 172.21.15.45:0/1073407783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b414c890 con 0x7fd5b40b72c0 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b9800640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 0x7fd5b414c350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c390 tx=0x7fd5a800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8017070 con 0x7fd5b40b72c0 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5a80035f0 con 0x7fd5b40b72c0 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8003a40 con 0x7fd5b40b72c0 2024-09-13T08:36:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b4150120 con 0x7fd5b40b72c0 2024-09-13T08:36:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b41506e0 con 0x7fd5b40b72c0 2024-09-13T08:36:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5a801a070 con 0x7fd5b40b72c0 2024-09-13T08:36:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.117+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fd5a8003be0 con 0x7fd5b40b72c0 2024-09-13T08:36:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.118+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b4074720 con 0x7fd5b40b72c0 2024-09-13T08:36:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.120+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd5b4074720 con 0x7fd5b40b72c0 2024-09-13T08:36:47.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.196+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dc59624a-a6b1-451d-b54c-a2b12eaed3f5", "id": 4} v 0) -- 0x7fd5b40d1210 con 0x7fd5b40b72c0 2024-09-13T08:36:47.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.199+0000 7fd5b1600640 1 -- 172.21.15.45:0/1073407783 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc59624a-a6b1-451d-b54c-a2b12eaed3f5", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fd5a8021070 con 0x7fd5b40b72c0 2024-09-13T08:36:47.200 INFO:teuthology.orchestra.run.smithi045.stdout:4 2024-09-13T08:36:47.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 msgr2=0x7fd5b414c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 0x7fd5b414c350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c390 tx=0x7fd5a800c850 comp rx=0 tx=0).stop 2024-09-13T08:36:47.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 shutdown_connections 2024-09-13T08:36:47.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5b415e0b0 0x7fd5b40c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40d4a20 0x7fd5b40c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 --2- 172.21.15.45:0/1073407783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b40b72c0 0x7fd5b414c350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 >> 172.21.15.45:0/1073407783 conn(0x7fd5b40d3f30 msgr2=0x7fd5b4160650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 shutdown_connections 2024-09-13T08:36:47.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.200+0000 7fd5bc200640 1 -- 172.21.15.45:0/1073407783 wait complete. 2024-09-13T08:36:47.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new a4268a77-31dd-4270-958a-14dc24be7bb4 5 2024-09-13T08:36:47.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 -- 172.21.15.45:0/1843604620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0cf470 msgr2=0x7f01cc0d49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 --2- 172.21.15.45:0/1843604620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0cf470 0x7f01cc0d49b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005fd0 tx=0x7f01c402a940 comp rx=0 tx=0).stop 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 -- 172.21.15.45:0/1843604620 shutdown_connections 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 --2- 172.21.15.45:0/1843604620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0cf470 0x7f01cc0d49b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 -- 172.21.15.45:0/1843604620 >> 172.21.15.45:0/1843604620 conn(0x7f01cc0d3e90 msgr2=0x7f01cc0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 -- 172.21.15.45:0/1843604620 shutdown_connections 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.371+0000 7f01cb200640 1 -- 172.21.15.45:0/1843604620 wait complete. 2024-09-13T08:36:47.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.372+0000 7f01cb200640 1 Processor -- start 2024-09-13T08:36:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.372+0000 7f01cb200640 1 -- start start 2024-09-13T08:36:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 0x7f01cc0c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15e0a0 0x7f01cc0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc0b4f70 con 0x7f01cc15e0a0 2024-09-13T08:36:47.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc0b4df0 con 0x7f01cc0d2620 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.373+0000 7f01cb200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f01cc0b50f0 con 0x7f01cc0cf470 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 0x7f01cc0c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.45:40798/0 (socket says 172.21.15.45:40798) 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15e0a0 0x7f01cc0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 -- 172.21.15.45:0/776787507 learned_addr learned my addr 172.21.15.45:0/776787507 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 0x7f01cc0c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59144/0 (socket says 172.21.15.45:59144) 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 -- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 msgr2=0x7f01cc0c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 0x7f01cc0c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 -- 172.21.15.45:0/776787507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15e0a0 msgr2=0x7f01cc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15e0a0 0x7f01cc0ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.374+0000 7f01c9e00640 1 -- 172.21.15.45:0/776787507 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc1f4d40 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c9e00640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c400 tx=0x7f01bc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc011790 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01bc017070 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01cc1f5020 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc011ce0 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01cc1f55e0 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f01bc01c460 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.375+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f01bc012d10 con 0x7f01cc0cf470 2024-09-13T08:36:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.376+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01cc0b55a0 con 0x7f01cc0cf470 2024-09-13T08:36:47.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.378+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f01bc011e80 con 0x7f01cc0cf470 2024-09-13T08:36:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.453+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a4268a77-31dd-4270-958a-14dc24be7bb4", "id": 5} v 0) -- 0x7f01cc1f5820 con 0x7f01cc0cf470 2024-09-13T08:36:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.457+0000 7f01c2000640 1 -- 172.21.15.45:0/776787507 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4268a77-31dd-4270-958a-14dc24be7bb4", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f01bc012790 con 0x7f01cc0cf470 2024-09-13T08:36:47.459 INFO:teuthology.orchestra.run.smithi045.stdout:5 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.458+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 msgr2=0x7f01cc0cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.458+0000 7f01cb200640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c400 tx=0x7f01bc00c8c0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 shutdown_connections 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01cc15e0a0 0x7f01cc0ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01cc0d2620 0x7f01cc0c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 --2- 172.21.15.45:0/776787507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01cc0cf470 0x7f01cc0cc850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 >> 172.21.15.45:0/776787507 conn(0x7f01cc0d3e90 msgr2=0x7f01cc0d1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 shutdown_connections 2024-09-13T08:36:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.459+0000 7f01cb200640 1 -- 172.21.15.45:0/776787507 wait complete. 2024-09-13T08:36:47.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new eb828891-dcac-4b30-9dfe-139f49927ca2 6 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 -- 172.21.15.45:0/980085390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700cf470 msgr2=0x7f44700d1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 --2- 172.21.15.45:0/980085390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700cf470 0x7f44700d1f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44700b36b0 tx=0x7f445c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 -- 172.21.15.45:0/980085390 shutdown_connections 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 --2- 172.21.15.45:0/980085390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700cf470 0x7f44700d1f10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 --2- 172.21.15.45:0/980085390 >> v2:172.21.15.31:3300/0 conn(0x7f447015e0a0 0x7f44700d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 -- 172.21.15.45:0/980085390 >> 172.21.15.45:0/980085390 conn(0x7f44700d3e90 msgr2=0x7f44700d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 -- 172.21.15.45:0/980085390 shutdown_connections 2024-09-13T08:36:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.632+0000 7f446f200640 1 -- 172.21.15.45:0/980085390 wait complete. 2024-09-13T08:36:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.633+0000 7f446f200640 1 Processor -- start 2024-09-13T08:36:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.634+0000 7f446f200640 1 -- start start 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.634+0000 7f446f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 0x7f44700b9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.634+0000 7f446f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.634+0000 7f446f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 0x7f44700b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f44700d0d30 con 0x7f44700cf470 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44700b5590 con 0x7f44700d2450 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f44700d0eb0 con 0x7f447015e0a0 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 0x7f44700b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59160/0 (socket says 172.21.15.45:59160) 2024-09-13T08:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446d400640 1 -- 172.21.15.45:0/2276390111 learned_addr learned my addr 172.21.15.45:0/2276390111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:47.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446e800640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 0x7f44700b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 -- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 msgr2=0x7f44700b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 0x7f44700b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 -- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 msgr2=0x7f44700b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446de00640 1 -- 172.21.15.45:0/2276390111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44701f5a90 con 0x7f44700cf470 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446e800640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 0x7f44700b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.635+0000 7f446d400640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f446de00640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 0x7f44700b9180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f446000e6d0 tx=0x7f446000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446000cab0 con 0x7f44700cf470 2024-09-13T08:36:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f446000cd40 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460010690 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4470160950 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447015ea50 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4460010830 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.636+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f4460018ba0 con 0x7f44700cf470 2024-09-13T08:36:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.637+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44700d13d0 con 0x7f44700cf470 2024-09-13T08:36:47.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.639+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f4460010b20 con 0x7f44700cf470 2024-09-13T08:36:47.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.715+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eb828891-dcac-4b30-9dfe-139f49927ca2", "id": 6} v 0) -- 0x7f44700d01a0 con 0x7f44700cf470 2024-09-13T08:36:47.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.717+0000 7f4466000640 1 -- 172.21.15.45:0/2276390111 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb828891-dcac-4b30-9dfe-139f49927ca2", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f4460010d00 con 0x7f44700cf470 2024-09-13T08:36:47.718 INFO:teuthology.orchestra.run.smithi045.stdout:6 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 msgr2=0x7f44700b9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 0x7f44700b9180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f446000e6d0 tx=0x7f446000eb90 comp rx=0 tx=0).stop 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 shutdown_connections 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f447015e0a0 0x7f44700b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44700d2450 0x7f44700b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 --2- 172.21.15.45:0/2276390111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44700cf470 0x7f44700b9180 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.718+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 >> 172.21.15.45:0/2276390111 conn(0x7f44700d3e90 msgr2=0x7f44700b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.719+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 shutdown_connections 2024-09-13T08:36:47.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.719+0000 7f446f200640 1 -- 172.21.15.45:0/2276390111 wait complete. 2024-09-13T08:36:47.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b 7 2024-09-13T08:36:47.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 -- 172.21.15.45:0/2140621275 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380cee90 msgr2=0x7f2d380d28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 --2- 172.21.15.45:0/2140621275 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380cee90 0x7f2d380d28b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009830 tx=0x7f2d2402a510 comp rx=0 tx=0).stop 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 -- 172.21.15.45:0/2140621275 shutdown_connections 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 --2- 172.21.15.45:0/2140621275 >> v2:172.21.15.31:3300/0 conn(0x7f2d380d0370 0x7f2d380b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 --2- 172.21.15.45:0/2140621275 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380cee90 0x7f2d380d28b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 -- 172.21.15.45:0/2140621275 >> 172.21.15.45:0/2140621275 conn(0x7f2d380d1d90 msgr2=0x7f2d380d21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 -- 172.21.15.45:0/2140621275 shutdown_connections 2024-09-13T08:36:47.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.891+0000 7f2d3d800640 1 -- 172.21.15.45:0/2140621275 wait complete. 2024-09-13T08:36:47.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.893+0000 7f2d3d800640 1 Processor -- start 2024-09-13T08:36:47.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.893+0000 7f2d3d800640 1 -- start start 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380d0370 0x7f2d380c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d3815bfa0 0x7f2d380c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d380b61b0 con 0x7f2d3815bfa0 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d380b6030 con 0x7f2d380cee90 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d3d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2d380b6330 con 0x7f2d380d0370 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380d0370 0x7f2d380c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59164/0 (socket says 172.21.15.45:59164) 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 -- 172.21.15.45:0/1846488031 learned_addr learned my addr 172.21.15.45:0/1846488031 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-13T08:36:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d37400640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d3815bfa0 0x7f2d380c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 -- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380d0370 msgr2=0x7f2d380c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380d0370 0x7f2d380c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 -- 172.21.15.45:0/1846488031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d3815bfa0 msgr2=0x7f2d380c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d3815bfa0 0x7f2d380c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.894+0000 7f2d36a00640 1 -- 172.21.15.45:0/1846488031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d380c8f00 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d36a00640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c730 tx=0x7f2d2800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28016020 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d2800a950 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d380c9170 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2800ada0 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3815a920 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.895+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d28019020 con 0x7f2d380cee90 2024-09-13T08:36:47.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.896+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f2d2801b630 con 0x7f2d380cee90 2024-09-13T08:36:47.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.896+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d380d28b0 con 0x7f2d380cee90 2024-09-13T08:36:47.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.898+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2d28020020 con 0x7f2d380cee90 2024-09-13T08:36:47.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.973+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b", "id": 7} v 0) -- 0x7f2d380761d0 con 0x7f2d380cee90 2024-09-13T08:36:47.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.978+0000 7f2d2ea00640 1 -- 172.21.15.45:0/1846488031 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f2d3815c380 con 0x7f2d380cee90 2024-09-13T08:36:47.979 INFO:teuthology.orchestra.run.smithi045.stdout:7 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 msgr2=0x7f2d380c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c730 tx=0x7f2d2800cbf0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 shutdown_connections 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d3815bfa0 0x7f2d380c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d380d0370 0x7f2d380c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 --2- 172.21.15.45:0/1846488031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d380cee90 0x7f2d380c63a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 >> 172.21.15.45:0/1846488031 conn(0x7f2d380d1d90 msgr2=0x7f2d380b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 shutdown_connections 2024-09-13T08:36:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:36:47.979+0000 7f2d3d800640 1 -- 172.21.15.45:0/1846488031 wait complete. 2024-09-13T08:36:47.994 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T08:36:47.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T08:36:47.998 INFO:tasks.ceph.osd.0:Started 2024-09-13T08:36:47.998 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T08:36:47.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T08:36:48.002 INFO:tasks.ceph.osd.1:Started 2024-09-13T08:36:48.002 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T08:36:48.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T08:36:48.007 INFO:tasks.ceph.osd.2:Started 2024-09-13T08:36:48.007 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T08:36:48.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T08:36:48.012 INFO:tasks.ceph.osd.3:Started 2024-09-13T08:36:48.013 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T08:36:48.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T08:36:48.016 INFO:tasks.ceph.osd.4:Started 2024-09-13T08:36:48.017 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T08:36:48.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T08:36:48.020 INFO:tasks.ceph.osd.5:Started 2024-09-13T08:36:48.020 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T08:36:48.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T08:36:48.025 INFO:tasks.ceph.osd.6:Started 2024-09-13T08:36:48.025 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T08:36:48.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T08:36:48.030 INFO:tasks.ceph.osd.7:Started 2024-09-13T08:36:48.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:36:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 -- 172.21.15.31:0/2313416312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 msgr2=0x7f055c0ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 --2- 172.21.15.31:0/2313416312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c0ce980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f054800a970 tx=0x7f054802b8d0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 -- 172.21.15.31:0/2313416312 shutdown_connections 2024-09-13T08:36:48.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 --2- 172.21.15.31:0/2313416312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c0ce980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 -- 172.21.15.31:0/2313416312 >> 172.21.15.31:0/2313416312 conn(0x7f055c0cc910 msgr2=0x7f055c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 -- 172.21.15.31:0/2313416312 shutdown_connections 2024-09-13T08:36:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.229+0000 7f0561a00640 1 -- 172.21.15.31:0/2313416312 wait complete. 2024-09-13T08:36:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.231+0000 7f0561a00640 1 Processor -- start 2024-09-13T08:36:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.231+0000 7f0561a00640 1 -- start start 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f055c0cc270 0x7f055c09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f055c15a480 con 0x7f055c0aef70 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f055c15a300 con 0x7f055c0cc270 2024-09-13T08:36:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f0561a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f055c0b1910 con 0x7f055c070860 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f055a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f055a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38650/0 (socket says 172.21.15.31:38650) 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f055a000640 1 -- 172.21.15.31:0/2767665916 learned_addr learned my addr 172.21.15.31:0/2767665916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f055aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.232+0000 7f055b400640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f055c0cc270 0x7f055c09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 -- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f055c0cc270 msgr2=0x7f055c09bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f055c0cc270 0x7f055c09bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 -- 172.21.15.31:0/2767665916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 msgr2=0x7f055c15c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 -- 172.21.15.31:0/2767665916 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c09b870 con 0x7f055c070860 2024-09-13T08:36:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055a000640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:36:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f055aa00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c165fa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f054c00c3c0 tx=0x7f054c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c017070 con 0x7f055c070860 2024-09-13T08:36:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f054c0053e0 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.233+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c003850 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.234+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c06ab80 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.234+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c0ae510 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.234+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f054c01a070 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.234+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f054c011cc0 con 0x7f055c070860 2024-09-13T08:36:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.235+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c0ce980 con 0x7f055c070860 2024-09-13T08:36:48.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.239+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f054c021080 con 0x7f055c070860 2024-09-13T08:36:48.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.303+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f055c0b5e10 con 0x7f055c070860 2024-09-13T08:36:48.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.304+0000 7f0552a00640 1 -- 172.21.15.31:0/2767665916 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f055c09b870 con 0x7f055c070860 2024-09-13T08:36:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:36:48.305 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:36:47.976057+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 msgr2=0x7f055c165fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c165fa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f054c00c3c0 tx=0x7f054c00c880 comp rx=0 tx=0).stop 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 shutdown_connections 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f055c0cc270 0x7f055c09bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f055c0aef70 0x7f055c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 --2- 172.21.15.31:0/2767665916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f055c070860 0x7f055c165fa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 >> 172.21.15.31:0/2767665916 conn(0x7f055c0cc910 msgr2=0x7f055c072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:48.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 shutdown_connections 2024-09-13T08:36:48.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:36:48.305+0000 7f0561a00640 1 -- 172.21.15.31:0/2767665916 wait complete. 2024-09-13T08:36:48.321 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T08:36:48.321 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T08:36:49.127 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T08:36:49.270 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 -- 172.21.15.31:0/1913818567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d0fd0 msgr2=0x7f65280b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:49.270 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 --2- 172.21.15.31:0/1913818567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d0fd0 0x7f65280b40b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6514009830 tx=0x7f651402a5c0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.270 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 -- 172.21.15.31:0/1913818567 shutdown_connections 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 --2- 172.21.15.31:0/1913818567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d0fd0 0x7f65280b40b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 -- 172.21.15.31:0/1913818567 >> 172.21.15.31:0/1913818567 conn(0x7f652809d0a0 msgr2=0x7f65280d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 -- 172.21.15.31:0/1913818567 shutdown_connections 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.269+0000 7f652ea00640 1 -- 172.21.15.31:0/1913818567 wait complete. 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.271+0000 7f652ea00640 1 Processor -- start 2024-09-13T08:36:49.271 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.271+0000 7f652ea00640 1 -- start start 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65280d0fd0 0x7f652806e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f652815dfa0 0x7f65280c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65280b4d00 con 0x7f65280d2260 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65280d1c90 con 0x7f652815dfa0 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652ea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65280b4e80 con 0x7f65280d0fd0 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38664/0 (socket says 172.21.15.31:38664) 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652cc00640 1 -- 172.21.15.31:0/253408654 learned_addr learned my addr 172.21.15.31:0/253408654 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652d600640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65280d0fd0 0x7f652806e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.272+0000 7f652e000640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f652815dfa0 0x7f65280c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:49.274 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 -- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65280d0fd0 msgr2=0x7f652806e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:49.275 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65280d0fd0 0x7f652806e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.275 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 -- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f652815dfa0 msgr2=0x7f65280c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:49.275 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f652815dfa0 0x7f65280c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.275 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 -- 172.21.15.31:0/253408654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65280c8b40 con 0x7f65280d2260 2024-09-13T08:36:49.275 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652cc00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f651402abb0 tx=0x7f6514032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514040070 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6514005310 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514005c90 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.273+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65280c8e20 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.274+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65280c93e0 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.275+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65281e7650 con 0x7f65280d2260 2024-09-13T08:36:49.276 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.275+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 2) ==== 50665+0+0 (secure 0 0 0) 0x7f6514039040 con 0x7f65280d2260 2024-09-13T08:36:49.277 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.276+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6514073150 con 0x7f65280d2260 2024-09-13T08:36:49.278 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.278+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f651404ecf0 con 0x7f65280d2260 2024-09-13T08:36:49.343 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.343+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65281e9ea0 con 0x7f65280d2260 2024-09-13T08:36:49.344 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.343+0000 7f6525600640 1 -- 172.21.15.31:0/253408654 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f65140035f0 con 0x7f65280d2260 2024-09-13T08:36:49.344 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-13T08:36:49.344 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:36:47.976057+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:36:49.345 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 msgr2=0x7f65280b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:49.345 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f651402abb0 tx=0x7f6514032c10 comp rx=0 tx=0).stop 2024-09-13T08:36:49.345 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 shutdown_connections 2024-09-13T08:36:49.345 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f652815dfa0 0x7f65280c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.346 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65280d2260 0x7f65280b9430 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.346 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 --2- 172.21.15.31:0/253408654 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65280d0fd0 0x7f652806e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:49.346 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.344+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 >> 172.21.15.31:0/253408654 conn(0x7f652809d0a0 msgr2=0x7f65281ea6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:49.346 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.345+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 shutdown_connections 2024-09-13T08:36:49.346 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:49.345+0000 7f652ea00640 1 -- 172.21.15.31:0/253408654 wait complete. 2024-09-13T08:36:49.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-13T08:36:49.555 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-13T08:36:49.555+0000 7f9360d3e640 -1 Falling back to public interface 2024-09-13T08:36:49.566 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-09-13T08:36:49.565+0000 7f2302811640 -1 Falling back to public interface 2024-09-13T08:36:49.570 INFO:tasks.ceph.osd.6.smithi045.stderr:2024-09-13T08:36:49.569+0000 7fdc443bb640 -1 Falling back to public interface 2024-09-13T08:36:49.581 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-13T08:36:49.581+0000 7f7a6240d640 -1 Falling back to public interface 2024-09-13T08:36:49.583 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-09-13T08:36:49.582+0000 7f77f5811640 -1 Falling back to public interface 2024-09-13T08:36:49.811 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-13T08:36:49.811+0000 7f7e3b20f640 -1 Falling back to public interface 2024-09-13T08:36:49.909 INFO:tasks.ceph.osd.7.smithi045.stderr:2024-09-13T08:36:49.908+0000 7f21bca42640 -1 Falling back to public interface 2024-09-13T08:36:50.046 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:36:50.045+0000 7fe292400640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T08:36:50.047 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:36:50.045+0000 7fe292400640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T08:36:50.088 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-13T08:36:50.087+0000 7f5c3fd29640 -1 Falling back to public interface 2024-09-13T08:36:53.311 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-09-13T08:36:53.309+0000 7f2302811640 -1 osd.4 0 log_to_monitors true 2024-09-13T08:36:53.527 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-13T08:36:53.525+0000 7f7e3b20f640 -1 osd.2 0 log_to_monitors true 2024-09-13T08:36:53.554 INFO:tasks.ceph.osd.6.smithi045.stderr:2024-09-13T08:36:53.553+0000 7fdc443bb640 -1 osd.6 0 log_to_monitors true 2024-09-13T08:36:53.812 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-13T08:36:53.810+0000 7f9360d3e640 -1 osd.3 0 log_to_monitors true 2024-09-13T08:36:53.819 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-09-13T08:36:53.818+0000 7f77f5811640 -1 osd.5 0 log_to_monitors true 2024-09-13T08:36:53.885 INFO:tasks.ceph.osd.7.smithi045.stderr:2024-09-13T08:36:53.884+0000 7f21bca42640 -1 osd.7 0 log_to_monitors true 2024-09-13T08:36:54.015 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-13T08:36:54.013+0000 7f5c3fd29640 -1 osd.1 0 log_to_monitors true 2024-09-13T08:36:54.080 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-13T08:36:54.078+0000 7f7a6240d640 -1 osd.0 0 log_to_monitors true 2024-09-13T08:36:55.439 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-09-13T08:36:55.438+0000 7f22fd000640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T08:36:55.446 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-09-13T08:36:55.445+0000 7f22f3000640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:56.164 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T08:36:56.348 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 -- 172.21.15.31:0/2899675578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24300b50c0 msgr2=0x7f24300d0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:56.348 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 --2- 172.21.15.31:0/2899675578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24300b50c0 0x7f24300d0e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2428005fd0 tx=0x7f242802aa90 comp rx=0 tx=0).stop 2024-09-13T08:36:56.348 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 -- 172.21.15.31:0/2899675578 shutdown_connections 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 --2- 172.21.15.31:0/2899675578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24300b50c0 0x7f24300d0e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 --2- 172.21.15.31:0/2899675578 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f243015dfa0 0x7f24300d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 -- 172.21.15.31:0/2899675578 >> 172.21.15.31:0/2899675578 conn(0x7f243009d0a0 msgr2=0x7f24300d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 -- 172.21.15.31:0/2899675578 shutdown_connections 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.346+0000 7f2435a00640 1 -- 172.21.15.31:0/2899675578 wait complete. 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.348+0000 7f2435a00640 1 Processor -- start 2024-09-13T08:36:56.349 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.348+0000 7f2435a00640 1 -- start start 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24300b50c0 0x7f2430150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24300d2260 0x7f243009f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24300cf060 con 0x7f243015dfa0 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24300d1c90 con 0x7f24300d2260 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f24300cf1e0 con 0x7f24300b50c0 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34352/0 (socket says 172.21.15.31:34352) 2024-09-13T08:36:56.351 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2435000640 1 -- 172.21.15.31:0/2906027166 learned_addr learned my addr 172.21.15.31:0/2906027166 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:36:56.352 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f242fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24300b50c0 0x7f2430150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:56.352 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.349+0000 7f2427e00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24300d2260 0x7f243009f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 -- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24300b50c0 msgr2=0x7f2430150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24300b50c0 0x7f2430150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 -- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24300d2260 msgr2=0x7f243009f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24300d2260 0x7f243009f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 -- 172.21.15.31:0/2906027166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2430151a60 con 0x7f243015dfa0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435000640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2428009d90 tx=0x7f2428003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2428046070 con 0x7f243015dfa0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2428003e50 con 0x7f243015dfa0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2430151d40 con 0x7f243015dfa0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.350+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24280044b0 con 0x7f243015dfa0 2024-09-13T08:36:56.353 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.351+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2430152300 con 0x7f243015dfa0 2024-09-13T08:36:56.354 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.352+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24300cf770 con 0x7f243015dfa0 2024-09-13T08:36:56.354 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.352+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 101231+0+0 (secure 0 0 0) 0x7f2428049070 con 0x7f243015dfa0 2024-09-13T08:36:56.355 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.353+0000 7f242cc00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 0x7f240c07ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:36:56.355 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.354+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7f24280ba830 con 0x7f243015dfa0 2024-09-13T08:36:56.355 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.354+0000 7f242fe00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 0x7f240c07ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:36:56.356 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.355+0000 7f242fe00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 0x7f240c07ac80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f242000bb50 tx=0x7f242000b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:36:56.360 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.359+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2428032030 con 0x7f243015dfa0 2024-09-13T08:36:56.610 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.609+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24300d4d10 con 0x7f243015dfa0 2024-09-13T08:36:56.611 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.609+0000 7f242cc00640 1 -- 172.21.15.31:0/2906027166 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+7555 (secure 0 0 0) 0x7f24280836b0 con 0x7f243015dfa0 2024-09-13T08:36:56.611 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-13T08:36:56.611 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":14,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:36:56.410255+0000","last_up_change":"2024-09-13T08:36:56.410255+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:36:56.614 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 msgr2=0x7f240c07ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:56.614 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 0x7f240c07ac80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f242000bb50 tx=0x7f242000b2e0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 msgr2=0x7f2430152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2428009d90 tx=0x7f2428003ce0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 shutdown_connections 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f243015dfa0 0x7f2430152b00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24300d2260 0x7f243009f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f240c0787c0 0x7f240c07ac80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 --2- 172.21.15.31:0/2906027166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24300b50c0 0x7f2430150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.612+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 >> 172.21.15.31:0/2906027166 conn(0x7f243009d0a0 msgr2=0x7f24300b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.613+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 shutdown_connections 2024-09-13T08:36:56.615 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:36:56.613+0000 7f2435a00640 1 -- 172.21.15.31:0/2906027166 wait complete. 2024-09-13T08:36:56.630 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-09-13T08:36:56.864 INFO:tasks.ceph.osd.6.smithi045.stderr:2024-09-13T08:36:56.863+0000 7fdc3fc00640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T08:36:56.872 INFO:tasks.ceph.osd.6.smithi045.stderr:2024-09-13T08:36:56.871+0000 7fdc35200640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:56.885 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-09-13T08:36:56.884+0000 7f77f0c00640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T08:36:56.886 INFO:tasks.ceph.osd.7.smithi045.stderr:2024-09-13T08:36:56.885+0000 7f21b7800640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T08:36:56.891 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-09-13T08:36:56.890+0000 7f77e6200640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:56.894 INFO:tasks.ceph.osd.7.smithi045.stderr:2024-09-13T08:36:56.893+0000 7f21ad800640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:57.616 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-13T08:36:57.614+0000 7f7e36400640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T08:36:57.632 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-13T08:36:57.630+0000 7f935c600640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T08:36:57.632 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-13T08:36:57.628+0000 7f7e2ba00640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:57.639 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-13T08:36:57.637+0000 7f9351c00640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:57.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-13T08:36:57.676+0000 7f5c3b600640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T08:36:57.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-13T08:36:57.685+0000 7f5c30c00640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:36:57.734 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-13T08:36:57.732+0000 7f7a5d600640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T08:36:57.744 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-13T08:36:57.742+0000 7f7a52c00640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T08:37:03.438 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 -- 172.21.15.31:0/1224517208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114152d50 msgr2=0x7f7114070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 --2- 172.21.15.31:0/1224517208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114152d50 0x7f7114070250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7104005fd0 tx=0x7f710402a540 comp rx=0 tx=0).stop 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 -- 172.21.15.31:0/1224517208 shutdown_connections 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 --2- 172.21.15.31:0/1224517208 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7114073a20 0x7f7114070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 --2- 172.21.15.31:0/1224517208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114152d50 0x7f7114070250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.590 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 -- 172.21.15.31:0/1224517208 >> 172.21.15.31:0/1224517208 conn(0x7f71140693f0 msgr2=0x7f711406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:03.591 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 -- 172.21.15.31:0/1224517208 shutdown_connections 2024-09-13T08:37:03.591 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.588+0000 7f7119c00640 1 -- 172.21.15.31:0/1224517208 wait complete. 2024-09-13T08:37:03.591 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.589+0000 7f7119c00640 1 Processor -- start 2024-09-13T08:37:03.591 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.590+0000 7f7119c00640 1 -- start start 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114152d50 0x7f711413a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f711414aa50 0x7f71140b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7114068f40 con 0x7f7114153130 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71140cd200 con 0x7f7114152d50 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7119c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f711406f740 con 0x7f711414aa50 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34368/0 (socket says 172.21.15.31:34368) 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 -- 172.21.15.31:0/2041334803 learned_addr learned my addr 172.21.15.31:0/2041334803 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:03.594 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7112a00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114152d50 0x7f711413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f7113400640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f711414aa50 0x7f71140b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 -- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f711414aa50 msgr2=0x7f71140b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f711414aa50 0x7f71140b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 -- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114152d50 msgr2=0x7f711413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114152d50 0x7f711413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.591+0000 7f710a000640 1 -- 172.21.15.31:0/2041334803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711414d6f0 con 0x7f7114153130 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f710a000640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00c400 tx=0x7f70fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:03.595 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc017070 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70fc0053c0 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc003850 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71141564e0 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.592+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711414dc70 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.593+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711406fc60 con 0x7f7114153130 2024-09-13T08:37:03.596 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.595+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 101231+0+0 (secure 0 0 0) 0x7f70fc01a070 con 0x7f7114153130 2024-09-13T08:37:03.597 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.595+0000 7f710b400640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 0x7f70f007acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:03.597 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.596+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f70fc099bc0 con 0x7f7114153130 2024-09-13T08:37:03.597 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.596+0000 7f7112a00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 0x7f70f007acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:03.598 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.597+0000 7f7112a00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 0x7f70f007acb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7104033000 tx=0x7f7104004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:03.601 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.599+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f70fc021020 con 0x7f7114153130 2024-09-13T08:37:03.796 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.795+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71140cf290 con 0x7f7114153130 2024-09-13T08:37:03.797 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.796+0000 7f710b400640 1 -- 172.21.15.31:0/2041334803 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12848 (secure 0 0 0) 0x7f70fc061ff0 con 0x7f7114153130 2024-09-13T08:37:03.798 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-13T08:37:03.798 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":19,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:01.897926+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 msgr2=0x7f70f007acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 0x7f70f007acb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7104033000 tx=0x7f7104004730 comp rx=0 tx=0).stop 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 msgr2=0x7f711413b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00c400 tx=0x7f70fc00c8c0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 shutdown_connections 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f70f00787f0 0x7f70f007acb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f711414aa50 0x7f71140b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7114153130 0x7f711413b060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.800 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 --2- 172.21.15.31:0/2041334803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114152d50 0x7f711413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:03.801 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 >> 172.21.15.31:0/2041334803 conn(0x7f71140693f0 msgr2=0x7f711415eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:03.801 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 shutdown_connections 2024-09-13T08:37:03.801 INFO:teuthology.misc.health.smithi031.stderr:2024-09-13T08:37:03.798+0000 7f7119c00640 1 -- 172.21.15.31:0/2041334803 wait complete. 2024-09-13T08:37:03.813 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-13T08:37:03.813 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T08:37:03.814 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:37:03.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:37:03.818 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:37:03.818 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:37:03.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:37:03.823 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:37:03.823 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:37:03.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:37:03.828 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:37:03.828 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:37:03.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:37:03.832 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:37:03.832 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-13T08:37:03.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:03.957 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:37:03.957 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:37:03.974 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:37:03.975 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 -- 172.21.15.31:0/3225134421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cbc50 msgr2=0x7f91f80cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 --2- 172.21.15.31:0/3225134421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cbc50 0x7f91f80cc0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009be0 tx=0x7f91f002a920 comp rx=0 tx=0).stop 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 -- 172.21.15.31:0/3225134421 shutdown_connections 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 --2- 172.21.15.31:0/3225134421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cbc50 0x7f91f80cc0a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 -- 172.21.15.31:0/3225134421 >> 172.21.15.31:0/3225134421 conn(0x7f91f80cc910 msgr2=0x7f91f80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 -- 172.21.15.31:0/3225134421 shutdown_connections 2024-09-13T08:37:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.003+0000 7f91fd800640 1 -- 172.21.15.31:0/3225134421 wait complete. 2024-09-13T08:37:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.004+0000 7f91fd800640 1 Processor -- start 2024-09-13T08:37:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.005+0000 7f91fd800640 1 -- start start 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cf850 0x7f91f81f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f815bd70 0x7f91f80b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91f81600f0 con 0x7f91f80cbc50 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91f80cead0 con 0x7f91f815bd70 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91fd800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f91f809cd80 con 0x7f91f80cf850 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34414/0 (socket says 172.21.15.31:34414) 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 -- 172.21.15.31:0/2174884863 learned_addr learned my addr 172.21.15.31:0/2174884863 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f7400640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f815bd70 0x7f91f80b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6000640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cf850 0x7f91f81f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 -- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cf850 msgr2=0x7f91f81f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cf850 0x7f91f81f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 -- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f815bd70 msgr2=0x7f91f80b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f815bd70 0x7f91f80b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.006+0000 7f91f6a00640 1 -- 172.21.15.31:0/2174884863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f81f5910 con 0x7f91f80cbc50 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91f6a00640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c430 tx=0x7f91e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8011850 con 0x7f91f80cbc50 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91e8017070 con 0x7f91f80cbc50 2024-09-13T08:37:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8011da0 con 0x7f91f80cbc50 2024-09-13T08:37:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f809c170 con 0x7f91f80cbc50 2024-09-13T08:37:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.007+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f809c730 con 0x7f91f80cbc50 2024-09-13T08:37:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.008+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f80cb620 con 0x7f91f80cbc50 2024-09-13T08:37:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.009+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f91e801c460 con 0x7f91f80cbc50 2024-09-13T08:37:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.011+0000 7f91eea00640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 0x7f91d807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.011+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f91e809a230 con 0x7f91f80cbc50 2024-09-13T08:37:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.011+0000 7f91f6000640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 0x7f91d807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.012+0000 7f91f6000640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 0x7f91d807ad10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000c00 tx=0x7f91e4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:04.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.015+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f91f81f5910 con 0x7f91f80cbc50 2024-09-13T08:37:04.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.248+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f80cb850 con 0x7f91f80cbc50 2024-09-13T08:37:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.248+0000 7f91eea00640 1 -- 172.21.15.31:0/2174884863 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1428 (secure 0 0 0) 0x7f91e8062590 con 0x7f91f80cbc50 2024-09-13T08:37:04.250 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"btime":"2024-09-13T08:37:03:965334+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-13T08:37:04.251 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 msgr2=0x7f91d807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 0x7f91d807ad10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000c00 tx=0x7f91e4009040 comp rx=0 tx=0).stop 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 msgr2=0x7f91f809d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c430 tx=0x7f91e800c8f0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 shutdown_connections 2024-09-13T08:37:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f815bd70 0x7f91f80b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f91d8078850 0x7f91d807ad10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91f80cf850 0x7f91f81f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 --2- 172.21.15.31:0/2174884863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80cbc50 0x7f91f809d210 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 >> 172.21.15.31:0/2174884863 conn(0x7f91f80cc910 msgr2=0x7f91f80ae450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 shutdown_connections 2024-09-13T08:37:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:04.251+0000 7f91fd800640 1 -- 172.21.15.31:0/2174884863 wait complete. 2024-09-13T08:37:05.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:37:05.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:37:06.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:06.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 -- 172.21.15.31:0/4292749511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640cc720 msgr2=0x7f476406f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 --2- 172.21.15.31:0/4292749511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640cc720 0x7f476406f9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4754009830 tx=0x7f475402a510 comp rx=0 tx=0).stop 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 -- 172.21.15.31:0/4292749511 shutdown_connections 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 --2- 172.21.15.31:0/4292749511 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640ccb00 0x7f47640b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 --2- 172.21.15.31:0/4292749511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640cc720 0x7f476406f9b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 -- 172.21.15.31:0/4292749511 >> 172.21.15.31:0/4292749511 conn(0x7f476406cb00 msgr2=0x7f476406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 -- 172.21.15.31:0/4292749511 shutdown_connections 2024-09-13T08:37:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.439+0000 7f4769800640 1 -- 172.21.15.31:0/4292749511 wait complete. 2024-09-13T08:37:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.441+0000 7f4769800640 1 Processor -- start 2024-09-13T08:37:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.441+0000 7f4769800640 1 -- start start 2024-09-13T08:37:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 0x7f476415e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640ccb00 0x7f4764154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47640b5490 con 0x7f47640701a0 2024-09-13T08:37:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47640b5310 con 0x7f47640ccb00 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4769800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4764152d90 con 0x7f47640cc720 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4763e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4763e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58946/0 (socket says 172.21.15.31:58946) 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4763e00640 1 -- 172.21.15.31:0/107519763 learned_addr learned my addr 172.21.15.31:0/107519763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.442+0000 7f4768e00640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640ccb00 0x7f4764154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f475b400640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 0x7f476415e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 -- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 msgr2=0x7f476415e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 0x7f476415e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 -- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640ccb00 msgr2=0x7f4764154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640ccb00 0x7f4764154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 -- 172.21.15.31:0/107519763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4764155100 con 0x7f47640701a0 2024-09-13T08:37:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f475b400640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 0x7f476415e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4763e00640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4754009950 tx=0x7f4754032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4754040070 con 0x7f47640701a0 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47540038e0 con 0x7f47640701a0 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4754005c90 con 0x7f47640701a0 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47641553a0 con 0x7f47640701a0 2024-09-13T08:37:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.443+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47641f86c0 con 0x7f47640701a0 2024-09-13T08:37:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.444+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4764153260 con 0x7f47640701a0 2024-09-13T08:37:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.445+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4754039040 con 0x7f47640701a0 2024-09-13T08:37:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.446+0000 7f4760c00640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 0x7f474007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.446+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f47540bbbb0 con 0x7f47640701a0 2024-09-13T08:37:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.447+0000 7f475b400640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 0x7f474007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.447+0000 7f475b400640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 0x7f474007ad40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47640cec50 tx=0x7f474c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:06.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.451+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4754084560 con 0x7f47640701a0 2024-09-13T08:37:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.682+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47640b4ee0 con 0x7f47640701a0 2024-09-13T08:37:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.682+0000 7f4760c00640 1 -- 172.21.15.31:0/107519763 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3897 (secure 0 0 0) 0x7f4754083e60 con 0x7f47640701a0 2024-09-13T08:37:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:06.685 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"btime":"2024-09-13T08:37:04:968589+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4366,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-13T08:37:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 msgr2=0x7f474007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 0x7f474007ad40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47640cec50 tx=0x7f474c005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 msgr2=0x7f47640cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4754009950 tx=0x7f4754032c10 comp rx=0 tx=0).stop 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 shutdown_connections 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47640ccb00 0x7f4764154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4740078880 0x7f474007ad40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47640cc720 0x7f476415e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 --2- 172.21.15.31:0/107519763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47640701a0 0x7f47640cba10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 >> 172.21.15.31:0/107519763 conn(0x7f476406cb00 msgr2=0x7f476415ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 shutdown_connections 2024-09-13T08:37:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.685+0000 7f4769800640 1 -- 172.21.15.31:0/107519763 wait complete. 2024-09-13T08:37:06.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 -- 172.21.15.31:0/2607881173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2620003a80 msgr2=0x7f2620003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 --2- 172.21.15.31:0/2607881173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2620003a80 0x7f2620003ea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f262c00c4a0 tx=0x7f2620005830 comp rx=0 tx=0).stop 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263b400640 1 -- 172.21.15.31:0/2607881173 reap_dead start 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 -- 172.21.15.31:0/2607881173 shutdown_connections 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 -- 172.21.15.31:0/2607881173 >> 172.21.15.31:0/2607881173 conn(0x7f263c0d1c70 msgr2=0x7f263c0d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 -- 172.21.15.31:0/2607881173 shutdown_connections 2024-09-13T08:37:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.862+0000 7f263be00640 1 -- 172.21.15.31:0/2607881173 wait complete. 2024-09-13T08:37:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.864+0000 7f263be00640 1 Processor -- start 2024-09-13T08:37:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.864+0000 7f263be00640 1 -- start start 2024-09-13T08:37:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2620003a80 0x7f263c1f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f263c070c30 0x7f263c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f263c0b3210 con 0x7f263c0d02b0 2024-09-13T08:37:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f263c160840 con 0x7f2620003a80 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f263c0b3390 con 0x7f263c070c30 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58960/0 (socket says 172.21.15.31:58960) 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263b400640 1 -- 172.21.15.31:0/3134930008 learned_addr learned my addr 172.21.15.31:0/3134930008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f263aa00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2620003a80 0x7f263c1f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.865+0000 7f2632000640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f263c070c30 0x7f263c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 -- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f263c070c30 msgr2=0x7f263c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f263c070c30 0x7f263c0cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 -- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2620003a80 msgr2=0x7f263c1f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2620003a80 0x7f263c1f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 -- 172.21.15.31:0/3134930008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c1f57b0 con 0x7f263c0d02b0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263b400640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f263c160560 tx=0x7f2634001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634011840 con 0x7f263c0d02b0 2024-09-13T08:37:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2634018070 con 0x7f263c0d02b0 2024-09-13T08:37:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634011ca0 con 0x7f263c0d02b0 2024-09-13T08:37:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263c0c9e30 con 0x7f263c0d02b0 2024-09-13T08:37:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.866+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263c0ca3f0 con 0x7f263c0d02b0 2024-09-13T08:37:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.867+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263c15fa70 con 0x7f263c0d02b0 2024-09-13T08:37:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.868+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f263401d460 con 0x7f263c0d02b0 2024-09-13T08:37:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.870+0000 7f2633400640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 0x7f261807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.870+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f263409ace0 con 0x7f263c0d02b0 2024-09-13T08:37:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.870+0000 7f263aa00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 0x7f261807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.871+0000 7f263aa00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 0x7f261807ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f262c033000 tx=0x7f263c0d2b70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:06.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:06.875+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2634063600 con 0x7f263c0d02b0 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.073+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f263c15b1c0 con 0x7f263c0d02b0 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.073+0000 7f2633400640 1 -- 172.21.15.31:0/3134930008 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) ==== 84+0+65 (secure 0 0 0) 0x7f2634062f00 con 0x7f263c0d02b0 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:07.075 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:37:07.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 msgr2=0x7f261807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 0x7f261807ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f262c033000 tx=0x7f263c0d2b70 comp rx=0 tx=0).stop 2024-09-13T08:37:07.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 msgr2=0x7f263c159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f263c160560 tx=0x7f2634001c10 comp rx=0 tx=0).stop 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 shutdown_connections 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f263c0d02b0 0x7f263c159d60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2618078880 0x7f261807ad40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f263c070c30 0x7f263c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 --2- 172.21.15.31:0/3134930008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2620003a80 0x7f263c1f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 >> 172.21.15.31:0/3134930008 conn(0x7f263c0d1c70 msgr2=0x7f263c0b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 shutdown_connections 2024-09-13T08:37:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.075+0000 7f263be00640 1 -- 172.21.15.31:0/3134930008 wait complete. 2024-09-13T08:37:07.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:37:07.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 -- 172.21.15.31:0/3458371813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 msgr2=0x7f1fd40b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/3458371813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40b29e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc005fd0 tx=0x7f1fcc02a940 comp rx=0 tx=0).stop 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 -- 172.21.15.31:0/3458371813 shutdown_connections 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/3458371813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40b29e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 -- 172.21.15.31:0/3458371813 >> 172.21.15.31:0/3458371813 conn(0x7f1fd40cfbb0 msgr2=0x7f1fd40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 -- 172.21.15.31:0/3458371813 shutdown_connections 2024-09-13T08:37:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.256+0000 7f1fd9c00640 1 -- 172.21.15.31:0/3458371813 wait complete. 2024-09-13T08:37:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.258+0000 7f1fd9c00640 1 Processor -- start 2024-09-13T08:37:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.258+0000 7f1fd9c00640 1 -- start start 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 0x7f1fd415d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd415bfc0 0x7f1fd40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40b2f20 con 0x7f1fd40cbe50 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40d0a00 con 0x7f1fd415bfc0 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd9c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd40b30a0 con 0x7f1fd4072560 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58986/0 (socket says 172.21.15.31:58986) 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 -- 172.21.15.31:0/1547106623 learned_addr learned my addr 172.21.15.31:0/1547106623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd2a00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 0x7f1fd415d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd3400640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd415bfc0 0x7f1fd40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 -- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 msgr2=0x7f1fd415d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 0x7f1fd415d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 -- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd415bfc0 msgr2=0x7f1fd40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd415bfc0 0x7f1fd40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fca000640 1 -- 172.21.15.31:0/1547106623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd40c5ef0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.259+0000 7f1fd2a00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 0x7f1fd415d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fca000640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c500 tx=0x7f1fbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc0052d0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fbc005470 con 0x7f1fd40cbe50 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc012870 con 0x7f1fd40cbe50 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd41f58a0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.260+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd415c970 con 0x7f1fd40cbe50 2024-09-13T08:37:07.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.261+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd40b29e0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.262+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f1fbc0035f0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.263+0000 7f1fcb400640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 0x7f1fb007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:07.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.263+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f1fbc09a170 con 0x7f1fd40cbe50 2024-09-13T08:37:07.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.264+0000 7f1fd2a00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 0x7f1fb007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:07.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.264+0000 7f1fd2a00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 0x7f1fb007ad10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40046b0 tx=0x7f1fc4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:07.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.267+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1fbc062b40 con 0x7f1fd40cbe50 2024-09-13T08:37:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.463+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1fd40c9fc0 con 0x7f1fd40cbe50 2024-09-13T08:37:07.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.983+0000 7f1fcb400640 1 -- 172.21.15.31:0/1547106623 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) ==== 138+0+0 (secure 0 0 0) 0x7f1fbc062440 con 0x7f1fd40cbe50 2024-09-13T08:37:07.985 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:37:07.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 msgr2=0x7f1fb007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 0x7f1fb007ad10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40046b0 tx=0x7f1fc4009040 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 msgr2=0x7f1fd40c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c500 tx=0x7f1fbc00c9c0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 shutdown_connections 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fd415bfc0 0x7f1fd40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd40cbe50 0x7f1fd40c8ca0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.986+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1fb0078850 0x7f1fb007ad10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.987+0000 7f1fd9c00640 1 --2- 172.21.15.31:0/1547106623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1fd4072560 0x7f1fd415d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.987+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 >> 172.21.15.31:0/1547106623 conn(0x7f1fd40cfbb0 msgr2=0x7f1fd40cda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.987+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 shutdown_connections 2024-09-13T08:37:07.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:07.987+0000 7f1fd9c00640 1 -- 172.21.15.31:0/1547106623 wait complete. 2024-09-13T08:37:08.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:37:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 -- 172.21.15.31:0/83923693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 msgr2=0x7f53480cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 --2- 172.21.15.31:0/83923693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f53480cb740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5348069dd0 tx=0x7f533002a560 comp rx=0 tx=0).stop 2024-09-13T08:37:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 -- 172.21.15.31:0/83923693 shutdown_connections 2024-09-13T08:37:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 --2- 172.21.15.31:0/83923693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f53480cb740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 --2- 172.21.15.31:0/83923693 >> v2:172.21.15.45:3300/0 conn(0x7f534815bfc0 0x7f5348072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 -- 172.21.15.31:0/83923693 >> 172.21.15.31:0/83923693 conn(0x7f53480cfbb0 msgr2=0x7f53480cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 -- 172.21.15.31:0/83923693 shutdown_connections 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.180+0000 7f534f000640 1 -- 172.21.15.31:0/83923693 wait complete. 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.182+0000 7f534f000640 1 Processor -- start 2024-09-13T08:37:08.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.182+0000 7f534f000640 1 -- start start 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f534815bfc0 0x7f53480c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53480c36e0 0x7f53480c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53480b3270 con 0x7f534806fa60 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53480b30f0 con 0x7f53480c36e0 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f53480b33f0 con 0x7f534815bfc0 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f5347e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:08.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f5347e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59008/0 (socket says 172.21.15.31:59008) 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f5347e00640 1 -- 172.21.15.31:0/1096478403 learned_addr learned my addr 172.21.15.31:0/1096478403 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f533fe00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f534815bfc0 0x7f53480c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.183+0000 7f534d000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53480c36e0 0x7f53480c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 -- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f534815bfc0 msgr2=0x7f53480c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f534815bfc0 0x7f53480c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 -- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53480c36e0 msgr2=0x7f53480c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53480c36e0 0x7f53480c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 -- 172.21.15.31:0/1096478403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53480cf170 con 0x7f534806fa60 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5347e00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f533800d700 tx=0x7f533800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338017070 con 0x7f534806fa60 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5338005310 con 0x7f534806fa60 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338009820 con 0x7f534806fa60 2024-09-13T08:37:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53480cee90 con 0x7f534806fa60 2024-09-13T08:37:08.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.184+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5348153e40 con 0x7f534806fa60 2024-09-13T08:37:08.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.185+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53480cbe80 con 0x7f534806fa60 2024-09-13T08:37:08.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.187+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f53380099c0 con 0x7f534806fa60 2024-09-13T08:37:08.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.188+0000 7f5344c00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 0x7f532407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:08.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.188+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5227+0+0 (secure 0 0 0) 0x7f533809a7a0 con 0x7f534806fa60 2024-09-13T08:37:08.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.188+0000 7f533fe00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 0x7f532407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:08.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.189+0000 7f533fe00640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 0x7f532407ad10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5330033000 tx=0x7f5330004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:08.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.193+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5338062990 con 0x7f534806fa60 2024-09-13T08:37:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.391+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f53480cb310 con 0x7f534806fa60 2024-09-13T08:37:08.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.986+0000 7f5344c00640 1 -- 172.21.15.31:0/1096478403 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) ==== 170+0+0 (secure 0 0 0) 0x7f53380673c0 con 0x7f534806fa60 2024-09-13T08:37:08.989 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 msgr2=0x7f532407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 0x7f532407ad10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5330033000 tx=0x7f5330004730 comp rx=0 tx=0).stop 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 msgr2=0x7f534814b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f533800d700 tx=0x7f533800dbc0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 shutdown_connections 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53480c36e0 0x7f53480c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5324078850 0x7f532407ad10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f534815bfc0 0x7f53480c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 --2- 172.21.15.31:0/1096478403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f534806fa60 0x7f534814b390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 >> 172.21.15.31:0/1096478403 conn(0x7f53480cfbb0 msgr2=0x7f5348157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.990+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 shutdown_connections 2024-09-13T08:37:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:08.991+0000 7f534f000640 1 -- 172.21.15.31:0/1096478403 wait complete. 2024-09-13T08:37:09.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:37:09.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 -- 172.21.15.31:0/427672409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb840cbe50 msgr2=0x7feb840b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:09.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 --2- 172.21.15.31:0/427672409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb840cbe50 0x7feb840b29e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb7c005fd0 tx=0x7feb7c02a8d0 comp rx=0 tx=0).stop 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 -- 172.21.15.31:0/427672409 shutdown_connections 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 --2- 172.21.15.31:0/427672409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb840cbe50 0x7feb840b29e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 -- 172.21.15.31:0/427672409 >> 172.21.15.31:0/427672409 conn(0x7feb840cfbb0 msgr2=0x7feb840cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 -- 172.21.15.31:0/427672409 shutdown_connections 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.199+0000 7feb8b600640 1 -- 172.21.15.31:0/427672409 wait complete. 2024-09-13T08:37:09.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.201+0000 7feb8b600640 1 Processor -- start 2024-09-13T08:37:09.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.201+0000 7feb8b600640 1 -- start start 2024-09-13T08:37:09.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:09.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feb840cbe50 0x7feb840c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:09.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8415bfc0 0x7feb841f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:09.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feb840b30a0 con 0x7feb84072560 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feb840b2f20 con 0x7feb8415bfc0 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb8b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7feb840b3220 con 0x7feb840cbe50 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb88c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb88c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59010/0 (socket says 172.21.15.31:59010) 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb88c00640 1 -- 172.21.15.31:0/1756419399 learned_addr learned my addr 172.21.15.31:0/1756419399 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb7be00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feb840cbe50 0x7feb840c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.202+0000 7feb89600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8415bfc0 0x7feb841f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 -- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feb840cbe50 msgr2=0x7feb840c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feb840cbe50 0x7feb840c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 -- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8415bfc0 msgr2=0x7feb841f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8415bfc0 0x7feb841f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:09.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 -- 172.21.15.31:0/1756419399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb840c8020 con 0x7feb84072560 2024-09-13T08:37:09.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb88c00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c4b0 tx=0x7feb7400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:09.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74016020 con 0x7feb84072560 2024-09-13T08:37:09.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb74012920 con 0x7feb84072560 2024-09-13T08:37:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74012d70 con 0x7feb84072560 2024-09-13T08:37:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb840c8300 con 0x7feb84072560 2024-09-13T08:37:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.203+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb840c88c0 con 0x7feb84072560 2024-09-13T08:37:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.204+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb840b29e0 con 0x7feb84072560 2024-09-13T08:37:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.206+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7feb74019020 con 0x7feb84072560 2024-09-13T08:37:09.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.207+0000 7feb81600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 0x7feb6007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:09.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.207+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5609+0+0 (secure 0 0 0) 0x7feb7409a580 con 0x7feb84072560 2024-09-13T08:37:09.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.207+0000 7feb7be00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 0x7feb6007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:09.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.208+0000 7feb7be00640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 0x7feb6007ad10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb8415e6e0 tx=0x7feb6c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:09.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.211+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7feb74020030 con 0x7feb84072560 2024-09-13T08:37:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:09.444+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7feb840cba60 con 0x7feb84072560 2024-09-13T08:37:10.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.005+0000 7feb81600640 1 -- 172.21.15.31:0/1756419399 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7feb74062560 con 0x7feb84072560 2024-09-13T08:37:10.010 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-09-13T08:37:10.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.012+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 msgr2=0x7feb6007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.012+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 0x7feb6007ad10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb8415e6e0 tx=0x7feb6c009040 comp rx=0 tx=0).stop 2024-09-13T08:37:10.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.012+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 msgr2=0x7feb840c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.012+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c4b0 tx=0x7feb7400c970 comp rx=0 tx=0).stop 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 shutdown_connections 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8415bfc0 0x7feb841f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7feb60078850 0x7feb6007ad10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feb840cbe50 0x7feb840c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 --2- 172.21.15.31:0/1756419399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb84072560 0x7feb840c5ed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.015+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 >> 172.21.15.31:0/1756419399 conn(0x7feb840cfbb0 msgr2=0x7feb84157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.016+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 shutdown_connections 2024-09-13T08:37:10.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.017+0000 7feb8b600640 1 -- 172.21.15.31:0/1756419399 wait complete. 2024-09-13T08:37:10.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 -- 172.21.15.31:0/219770832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500b9d30 msgr2=0x7fdc500d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 --2- 172.21.15.31:0/219770832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500b9d30 0x7fdc500d0a80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005fd0 tx=0x7fdc3c02a670 comp rx=0 tx=0).stop 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 -- 172.21.15.31:0/219770832 shutdown_connections 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 --2- 172.21.15.31:0/219770832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500b9d30 0x7fdc500d0a80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 -- 172.21.15.31:0/219770832 >> 172.21.15.31:0/219770832 conn(0x7fdc500d4070 msgr2=0x7fdc500d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 -- 172.21.15.31:0/219770832 shutdown_connections 2024-09-13T08:37:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.208+0000 7fdc58600640 1 -- 172.21.15.31:0/219770832 wait complete. 2024-09-13T08:37:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.209+0000 7fdc58600640 1 Processor -- start 2024-09-13T08:37:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.209+0000 7fdc58600640 1 -- start start 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdc500b9d30 0x7fdc500a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc5015e310 0x7fdc500cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501696b0 con 0x7fdc500d4c00 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc500d13d0 con 0x7fdc5015e310 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.210+0000 7fdc58600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdc50169830 con 0x7fdc500b9d30 2024-09-13T08:37:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59024/0 (socket says 172.21.15.31:59024) 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 -- 172.21.15.31:0/3745117024 learned_addr learned my addr 172.21.15.31:0/3745117024 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55c00640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdc500b9d30 0x7fdc500a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc56600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc5015e310 0x7fdc500cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 -- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdc500b9d30 msgr2=0x7fdc500a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdc500b9d30 0x7fdc500a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 -- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc5015e310 msgr2=0x7fdc500cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc5015e310 0x7fdc500cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 -- 172.21.15.31:0/3745117024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc500a1690 con 0x7fdc500d4c00 2024-09-13T08:37:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.211+0000 7fdc55200640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c006010 tx=0x7fdc3c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.212+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c03f070 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.212+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc3c004890 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.212+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c035030 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.212+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc500a1970 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.212+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc500a1360 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.213+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc500008d0 con 0x7fdc500d4c00 2024-09-13T08:37:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.214+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fdc3c042070 con 0x7fdc500d4c00 2024-09-13T08:37:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.215+0000 7fdc4d600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 0x7fdc3007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.215+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdc3c0bb790 con 0x7fdc500d4c00 2024-09-13T08:37:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.216+0000 7fdc55c00640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 0x7fdc3007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.216+0000 7fdc55c00640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 0x7fdc3007ad10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc44001330 tx=0x7fdc44009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:10.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.219+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdc3c0837a0 con 0x7fdc500d4c00 2024-09-13T08:37:10.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.417+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc500a1b40 con 0x7fdc500d4c00 2024-09-13T08:37:10.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.418+0000 7fdc4d600640 1 -- 172.21.15.31:0/3745117024 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16634 (secure 0 0 0) 0x7fdc3c0881d0 con 0x7fdc500d4c00 2024-09-13T08:37:10.420 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:10.420 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:09.446311+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 msgr2=0x7fdc3007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 0x7fdc3007ad10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc44001330 tx=0x7fdc44009040 comp rx=0 tx=0).stop 2024-09-13T08:37:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 msgr2=0x7fdc500cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c006010 tx=0x7fdc3c0039e0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 shutdown_connections 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc5015e310 0x7fdc500cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc500d4c00 0x7fdc500cb9e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdc30078850 0x7fdc3007ad10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 --2- 172.21.15.31:0/3745117024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdc500b9d30 0x7fdc500a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 >> 172.21.15.31:0/3745117024 conn(0x7fdc500d4070 msgr2=0x7fdc500d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 shutdown_connections 2024-09-13T08:37:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.420+0000 7fdc58600640 1 -- 172.21.15.31:0/3745117024 wait complete. 2024-09-13T08:37:10.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 -- 172.21.15.31:0/1861113630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc15dfc0 msgr2=0x7f67bc0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 --2- 172.21.15.31:0/1861113630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc15dfc0 0x7f67bc0cc300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009860 tx=0x7f67ac02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 -- 172.21.15.31:0/1861113630 shutdown_connections 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 --2- 172.21.15.31:0/1861113630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc15dfc0 0x7f67bc0cc300 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 -- 172.21.15.31:0/1861113630 >> 172.21.15.31:0/1861113630 conn(0x7f67bc0cc910 msgr2=0x7f67bc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 -- 172.21.15.31:0/1861113630 shutdown_connections 2024-09-13T08:37:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.596+0000 7f67c2400640 1 -- 172.21.15.31:0/1861113630 wait complete. 2024-09-13T08:37:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.597+0000 7f67c2400640 1 Processor -- start 2024-09-13T08:37:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.598+0000 7f67c2400640 1 -- start start 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.598+0000 7f67c2400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc0b1910 0x7f67bc1f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.598+0000 7f67c2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67c2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc15dfc0 0x7f67bc0ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67c2400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0b1da0 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67c2400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0740a0 con 0x7f67bc15dfc0 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67c2400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0b1f20 con 0x7f67bc0b1910 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67bb400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc0b1910 0x7f67bc1f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59026/0 (socket says 172.21.15.31:59026) 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 -- 172.21.15.31:0/3181850585 learned_addr learned my addr 172.21.15.31:0/3181850585 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67bbe00640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc15dfc0 0x7f67bc0ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 -- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc0b1910 msgr2=0x7f67bc1f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc0b1910 0x7f67bc1f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 -- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc15dfc0 msgr2=0x7f67bc0ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc15dfc0 0x7f67bc0ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 -- 172.21.15.31:0/3181850585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc09aab0 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.599+0000 7f67b2a00640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c760 tx=0x7f67a400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.600+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4016020 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.600+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67a400a930 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.600+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a400ad80 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.600+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc09d750 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.600+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc0b3990 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.601+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc09dc50 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.602+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f67a4019020 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.604+0000 7f67b3e00640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 0x7f679807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.604+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f67a409a8e0 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.604+0000 7f67bb400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 0x7f679807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.605+0000 7f67bb400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 0x7f679807ad10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67ac034000 tx=0x7f67ac004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:10.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.608+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f67bc09dc50 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.804+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67bc09ac80 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.805+0000 7f67b3e00640 1 -- 172.21.15.31:0/3181850585 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16634 (secure 0 0 0) 0x7f67a4020020 con 0x7f67bc0ca5c0 2024-09-13T08:37:10.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:10.808 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:09.446311+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 msgr2=0x7f679807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 0x7f679807ad10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67ac034000 tx=0x7f67ac004730 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 msgr2=0x7f67bc150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c760 tx=0x7f67a400cc20 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 shutdown_connections 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67bc15dfc0 0x7f67bc0ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67bc0ca5c0 0x7f67bc150cc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6798078850 0x7f679807ad10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 --2- 172.21.15.31:0/3181850585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67bc0b1910 0x7f67bc1f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 >> 172.21.15.31:0/3181850585 conn(0x7f67bc0cc910 msgr2=0x7f67bc073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 shutdown_connections 2024-09-13T08:37:10.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.808+0000 7f67c2400640 1 -- 172.21.15.31:0/3181850585 wait complete. 2024-09-13T08:37:10.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:37:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 -- 172.21.15.31:0/3378204223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a40cc8c0 msgr2=0x7f48a40ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 --2- 172.21.15.31:0/3378204223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a40cc8c0 0x7f48a40ce8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4890005fd0 tx=0x7f489002a530 comp rx=0 tx=0).stop 2024-09-13T08:37:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 -- 172.21.15.31:0/3378204223 shutdown_connections 2024-09-13T08:37:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 --2- 172.21.15.31:0/3378204223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a40cc8c0 0x7f48a40ce8e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 --2- 172.21.15.31:0/3378204223 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a4161420 0x7f48a40ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 -- 172.21.15.31:0/3378204223 >> 172.21.15.31:0/3378204223 conn(0x7f48a40d1c00 msgr2=0x7f48a40d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 -- 172.21.15.31:0/3378204223 shutdown_connections 2024-09-13T08:37:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.988+0000 7f48a3200640 1 -- 172.21.15.31:0/3378204223 wait complete. 2024-09-13T08:37:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.990+0000 7f48a3200640 1 Processor -- start 2024-09-13T08:37:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.990+0000 7f48a3200640 1 -- start start 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a40cfc80 0x7f48a40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48a40d09c0 con 0x7f48a4161420 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48a40d0840 con 0x7f48a40cfc80 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f48a40b2fa0 con 0x7f48a40cc8c0 2024-09-13T08:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:44158/0 (socket says 172.21.15.31:44158) 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a40cfc80 0x7f48a40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 -- 172.21.15.31:0/629799001 learned_addr learned my addr 172.21.15.31:0/629799001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a2800640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59036/0 (socket says 172.21.15.31:59036) 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 -- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a40cfc80 msgr2=0x7f48a40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a40cfc80 0x7f48a40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 -- 172.21.15.31:0/629799001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 msgr2=0x7f48a41f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a1e00640 1 -- 172.21.15.31:0/629799001 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a40c93a0 con 0x7f48a40cc8c0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.991+0000 7f48a2800640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f48a1e00640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f489400c9a0 tx=0x7f489400ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4894007bb0 con 0x7f48a40cc8c0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4894005020 con 0x7f48a40cc8c0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4894007e60 con 0x7f48a40cc8c0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a40c9660 con 0x7f48a40cc8c0 2024-09-13T08:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.992+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a40c9c20 con 0x7f48a40cc8c0 2024-09-13T08:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.993+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a40d0130 con 0x7f48a40cc8c0 2024-09-13T08:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.995+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4894005470 con 0x7f48a40cc8c0 2024-09-13T08:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.996+0000 7f489a000640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 0x7f488407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.996+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f489409b1f0 con 0x7f48a40cc8c0 2024-09-13T08:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.996+0000 7f48a1400640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 0x7f488407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:10.997+0000 7f48a1400640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 0x7f488407ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4890033000 tx=0x7f4890004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:11.000+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4894014030 con 0x7f48a40cc8c0 2024-09-13T08:37:11.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:11.234+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f48a40cd4a0 con 0x7f48a40cc8c0 2024-09-13T08:37:12.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.020+0000 7f489a000640 1 -- 172.21.15.31:0/629799001 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f48940631f0 con 0x7f48a40cc8c0 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 msgr2=0x7f488407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 0x7f488407ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4890033000 tx=0x7f4890004730 comp rx=0 tx=0).stop 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 msgr2=0x7f48a40b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f489400c9a0 tx=0x7f489400ce60 comp rx=0 tx=0).stop 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 shutdown_connections 2024-09-13T08:37:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a4161420 0x7f48a41f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a40cfc80 0x7f48a40cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4884078850 0x7f488407ad10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 --2- 172.21.15.31:0/629799001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f48a40cc8c0 0x7f48a40b56b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 >> 172.21.15.31:0/629799001 conn(0x7f48a40d1c00 msgr2=0x7f48a415faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 shutdown_connections 2024-09-13T08:37:12.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.023+0000 7f48a3200640 1 -- 172.21.15.31:0/629799001 wait complete. 2024-09-13T08:37:12.043 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:37:12.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 -- 172.21.15.31:0/2853613439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec4073ac0 msgr2=0x7f0ec40b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 --2- 172.21.15.31:0/2853613439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec4073ac0 0x7f0ec40b2820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0005fd0 tx=0x7f0eb002a5f0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 -- 172.21.15.31:0/2853613439 shutdown_connections 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 --2- 172.21.15.31:0/2853613439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec4073ac0 0x7f0ec40b2820 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 -- 172.21.15.31:0/2853613439 >> 172.21.15.31:0/2853613439 conn(0x7f0ec40b1520 msgr2=0x7f0ec40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 -- 172.21.15.31:0/2853613439 shutdown_connections 2024-09-13T08:37:12.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.210+0000 7f0eca200640 1 -- 172.21.15.31:0/2853613439 wait complete. 2024-09-13T08:37:12.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.212+0000 7f0eca200640 1 Processor -- start 2024-09-13T08:37:12.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.212+0000 7f0eca200640 1 -- start start 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ec4073ac0 0x7f0ec40c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 0x7f0ec40cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40caed0 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40cad50 con 0x7f0ec415bc00 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0eca200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec40cb050 con 0x7f0ec4073ac0 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59056/0 (socket says 172.21.15.31:59056) 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 -- 172.21.15.31:0/3568241584 learned_addr learned my addr 172.21.15.31:0/3568241584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec8e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ec4073ac0 0x7f0ec40c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:12.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec9800640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 0x7f0ec40cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 -- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ec4073ac0 msgr2=0x7f0ec40c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ec4073ac0 0x7f0ec40c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 -- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 msgr2=0x7f0ec40cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 0x7f0ec40cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec3e00640 1 -- 172.21.15.31:0/3568241584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec41f5110 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.213+0000 7f0ec9800640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 0x7f0ec40cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0ec3e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb003f070 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb0004890 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec41f5670 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0035030 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.214+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec40c5cb0 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.215+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec40cb570 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.216+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0eb0042070 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.217+0000 7f0ec0c00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 0x7f0ea407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.217+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0eb00bb830 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.218+0000 7f0ec8e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 0x7f0ea407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.219+0000 7f0ec8e00640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 0x7f0ea407ad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40d08c0 tx=0x7f0eb8009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:12.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.221+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0ec41f5110 con 0x7f0ec40cc3e0 2024-09-13T08:37:12.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:12.459+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0ec415d910 con 0x7f0ec40cc3e0 2024-09-13T08:37:13.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.022+0000 7f0ec0c00640 1 -- 172.21.15.31:0/3568241584 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f0ec415d910 con 0x7f0ec40cc3e0 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 msgr2=0x7f0ea407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 0x7f0ea407ad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40d08c0 tx=0x7f0eb8009040 comp rx=0 tx=0).stop 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 msgr2=0x7f0ec41565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb00039e0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 shutdown_connections 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ec415bc00 0x7f0ec40cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ec40cc3e0 0x7f0ec41565f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ea4078850 0x7f0ea407ad10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 --2- 172.21.15.31:0/3568241584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ec4073ac0 0x7f0ec40c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 >> 172.21.15.31:0/3568241584 conn(0x7f0ec40b1520 msgr2=0x7f0ec406fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 shutdown_connections 2024-09-13T08:37:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.026+0000 7f0eca200640 1 -- 172.21.15.31:0/3568241584 wait complete. 2024-09-13T08:37:13.045 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:37:13.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 -- 172.21.15.31:0/514419070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780d1190 msgr2=0x7f9478167d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 --2- 172.21.15.31:0/514419070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780d1190 0x7f9478167d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9470005fd0 tx=0x7f947002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 -- 172.21.15.31:0/514419070 shutdown_connections 2024-09-13T08:37:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 --2- 172.21.15.31:0/514419070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780d1190 0x7f9478167d80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 --2- 172.21.15.31:0/514419070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 0x7f94780cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 -- 172.21.15.31:0/514419070 >> 172.21.15.31:0/514419070 conn(0x7f94780d4070 msgr2=0x7f94780d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 -- 172.21.15.31:0/514419070 shutdown_connections 2024-09-13T08:37:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.207+0000 7f947cc00640 1 -- 172.21.15.31:0/514419070 wait complete. 2024-09-13T08:37:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.209+0000 7f947cc00640 1 Processor -- start 2024-09-13T08:37:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.209+0000 7f947cc00640 1 -- start start 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 0x7f947815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 0x7f94780b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9478168440 con 0x7f94780b9d30 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94781682c0 con 0x7f94780d1190 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f947cc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f94781685c0 con 0x7f947815e310 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59068/0 (socket says 172.21.15.31:59068) 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 -- 172.21.15.31:0/3333205670 learned_addr learned my addr 172.21.15.31:0/3333205670 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475400640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 0x7f947815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9476800640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 0x7f94780b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 -- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 msgr2=0x7f94780b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 0x7f94780b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 -- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 msgr2=0x7f947815a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 0x7f947815a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475e00640 1 -- 172.21.15.31:0/3333205670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94780b1cf0 con 0x7f94780b9d30 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.210+0000 7f9475400640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 0x7f947815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f9475e00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f946800e7c0 tx=0x7f946800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946800ccc0 con 0x7f94780b9d30 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9468010430 con 0x7f94780b9d30 2024-09-13T08:37:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468010840 con 0x7f94780b9d30 2024-09-13T08:37:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947815abd0 con 0x7f94780b9d30 2024-09-13T08:37:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.211+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947815b190 con 0x7f94780b9d30 2024-09-13T08:37:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.212+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94780cf690 con 0x7f94780b9d30 2024-09-13T08:37:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.213+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f94680109e0 con 0x7f94780b9d30 2024-09-13T08:37:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.214+0000 7f946e000640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 0x7f945807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.214+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f946809b440 con 0x7f94780b9d30 2024-09-13T08:37:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.214+0000 7f9475400640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 0x7f945807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.215+0000 7f9475400640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 0x7f945807ad10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9478167bb0 tx=0x7f9464009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.218+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9468063a10 con 0x7f94780b9d30 2024-09-13T08:37:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.452+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94780d0860 con 0x7f94780b9d30 2024-09-13T08:37:13.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.452+0000 7f946e000640 1 -- 172.21.15.31:0/3333205670 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7f9468063310 con 0x7f94780b9d30 2024-09-13T08:37:13.454 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:13.454 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:13.455 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 msgr2=0x7f945807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 0x7f945807ad10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9478167bb0 tx=0x7f9464009040 comp rx=0 tx=0).stop 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 msgr2=0x7f94781598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f946800e7c0 tx=0x7f946800ec80 comp rx=0 tx=0).stop 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 shutdown_connections 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9458078850 0x7f945807ad10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f947815e310 0x7f94780b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94780d1190 0x7f947815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 --2- 172.21.15.31:0/3333205670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94780b9d30 0x7f94781598f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 >> 172.21.15.31:0/3333205670 conn(0x7f94780d4070 msgr2=0x7f94780cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 shutdown_connections 2024-09-13T08:37:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.455+0000 7f947cc00640 1 -- 172.21.15.31:0/3333205670 wait complete. 2024-09-13T08:37:13.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 -- 172.21.15.31:0/1124105860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0d0e80 msgr2=0x7f99fc0cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 --2- 172.21.15.31:0/1124105860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0d0e80 0x7f99fc0cbb00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009830 tx=0x7f99ec02a510 comp rx=0 tx=0).stop 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 -- 172.21.15.31:0/1124105860 shutdown_connections 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 --2- 172.21.15.31:0/1124105860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0cc040 0x7f99fc0b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 --2- 172.21.15.31:0/1124105860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0d0e80 0x7f99fc0cbb00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 -- 172.21.15.31:0/1124105860 >> 172.21.15.31:0/1124105860 conn(0x7f99fc0d0360 msgr2=0x7f99fc0d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 -- 172.21.15.31:0/1124105860 shutdown_connections 2024-09-13T08:37:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.631+0000 7f9a03200640 1 -- 172.21.15.31:0/1124105860 wait complete. 2024-09-13T08:37:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.633+0000 7f9a03200640 1 Processor -- start 2024-09-13T08:37:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.633+0000 7f9a03200640 1 -- start start 2024-09-13T08:37:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 0x7f99fc15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc15e2d0 0x7f99fc0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc15e830 con 0x7f99fc0cc040 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc15e6b0 con 0x7f99fc15e2d0 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a03200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc0cca20 con 0x7f99fc0d0e80 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59084/0 (socket says 172.21.15.31:59084) 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.634+0000 7f9a01e00640 1 -- 172.21.15.31:0/2084286846 learned_addr learned my addr 172.21.15.31:0/2084286846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01400640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 0x7f99fc15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a02800640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc15e2d0 0x7f99fc0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 -- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 msgr2=0x7f99fc15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 0x7f99fc15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 -- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc15e2d0 msgr2=0x7f99fc0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc15e2d0 0x7f99fc0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 -- 172.21.15.31:0/2084286846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc0c5230 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01400640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 0x7f99fc15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a01e00640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c450 tx=0x7f99f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0016020 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99f0012920 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc0c5510 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0012d70 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.635+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc0c5ad0 con 0x7f99fc0cc040 2024-09-13T08:37:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.637+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99fc0b3ba0 con 0x7f99fc0cc040 2024-09-13T08:37:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.637+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f99f0019020 con 0x7f99fc0cc040 2024-09-13T08:37:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.638+0000 7f99f6000640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 0x7f99e007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.638+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f99f009a940 con 0x7f99fc0cc040 2024-09-13T08:37:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.639+0000 7f9a01400640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 0x7f99e007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.639+0000 7f9a01400640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 0x7f99e007ad40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99ec034000 tx=0x7f99ec004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.642+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f99f0020020 con 0x7f99fc0cc040 2024-09-13T08:37:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.842+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99fc15d930 con 0x7f99fc0cc040 2024-09-13T08:37:13.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.843+0000 7f99f6000640 1 -- 172.21.15.31:0/2084286846 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7f99f0062f10 con 0x7f99fc0cc040 2024-09-13T08:37:13.845 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:13.845 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 msgr2=0x7f99e007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 0x7f99e007ad40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99ec034000 tx=0x7f99ec004730 comp rx=0 tx=0).stop 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 msgr2=0x7f99fc1f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c450 tx=0x7f99f000c910 comp rx=0 tx=0).stop 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 shutdown_connections 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc15e2d0 0x7f99fc0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f99e0078880 0x7f99e007ad40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f99fc0d0e80 0x7f99fc15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 --2- 172.21.15.31:0/2084286846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99fc0cc040 0x7f99fc1f2f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:13.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 >> 172.21.15.31:0/2084286846 conn(0x7f99fc0d0360 msgr2=0x7f99fc15c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:13.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 shutdown_connections 2024-09-13T08:37:13.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:13.846+0000 7f9a03200640 1 -- 172.21.15.31:0/2084286846 wait complete. 2024-09-13T08:37:13.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:37:13.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 -- 172.21.15.31:0/4213795253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d40b2f80 msgr2=0x7f04d40b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 --2- 172.21.15.31:0/4213795253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d40b2f80 0x7f04d40b3a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f04cc005fd0 tx=0x7f04cc02aa90 comp rx=0 tx=0).stop 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 -- 172.21.15.31:0/4213795253 shutdown_connections 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 --2- 172.21.15.31:0/4213795253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d40b2f80 0x7f04d40b3a10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 --2- 172.21.15.31:0/4213795253 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d0240 0x7f04d40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 -- 172.21.15.31:0/4213795253 >> 172.21.15.31:0/4213795253 conn(0x7f04d40d3e90 msgr2=0x7f04d40d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.021+0000 7f04d3200640 1 -- 172.21.15.31:0/4213795253 shutdown_connections 2024-09-13T08:37:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.022+0000 7f04d3200640 1 -- 172.21.15.31:0/4213795253 wait complete. 2024-09-13T08:37:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.023+0000 7f04d3200640 1 Processor -- start 2024-09-13T08:37:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.023+0000 7f04d3200640 1 -- start start 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 0x7f04d41f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d08a0 0x7f04d414d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04d40b3f50 con 0x7f04d414d620 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04d40d15a0 con 0x7f04d40b2f80 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.024+0000 7f04d3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f04d40b40d0 con 0x7f04d40d08a0 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 0x7f04d41f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04c9400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d08a0 0x7f04d414d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 0x7f04d41f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:59822/0 (socket says 172.21.15.31:59822) 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59100/0 (socket says 172.21.15.31:59100) 2024-09-13T08:37:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d1e00640 1 -- 172.21.15.31:0/1220019441 learned_addr learned my addr 172.21.15.31:0/1220019441 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 -- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d08a0 msgr2=0x7f04d414d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d08a0 0x7f04d414d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 -- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 msgr2=0x7f04d41f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 0x7f04d41f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 -- 172.21.15.31:0/1220019441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d40cac30 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.025+0000 7f04d2800640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d408c4f0 tx=0x7f04cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.026+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc046070 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.026+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04cc003e00 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.026+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0044b0 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.026+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d40ca950 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.026+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d40b8090 con 0x7f04d414d620 2024-09-13T08:37:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.027+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d40b4660 con 0x7f04d414d620 2024-09-13T08:37:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.028+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f04cc049070 con 0x7f04d414d620 2024-09-13T08:37:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.029+0000 7f04caa00640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 0x7f04b007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.029+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f04cc0bb650 con 0x7f04d414d620 2024-09-13T08:37:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.030+0000 7f04d1e00640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 0x7f04b007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.030+0000 7f04d1e00640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 0x7f04b007ad10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000f60 tx=0x7f04c4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.035+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f04cc032030 con 0x7f04d414d620 2024-09-13T08:37:14.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.227+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04d40b3910 con 0x7f04b0078850 2024-09-13T08:37:14.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.233+0000 7f04caa00640 1 -- 172.21.15.31:0/1220019441 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241528 (secure 0 0 0) 0x7f04d40b3910 con 0x7f04b0078850 2024-09-13T08:37:14.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:14.239 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:14.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 msgr2=0x7f04b007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 0x7f04b007ad10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000f60 tx=0x7f04c4009040 comp rx=0 tx=0).stop 2024-09-13T08:37:14.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 msgr2=0x7f04d40c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d408c4f0 tx=0x7f04cc003c10 comp rx=0 tx=0).stop 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 shutdown_connections 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04d414d620 0x7f04d40c9c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f04b0078850 0x7f04b007ad10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04d40d08a0 0x7f04d414d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 --2- 172.21.15.31:0/1220019441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d40b2f80 0x7f04d41f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.237+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 >> 172.21.15.31:0/1220019441 conn(0x7f04d40d3e90 msgr2=0x7f04d40b5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.238+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 shutdown_connections 2024-09-13T08:37:14.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.238+0000 7f04d3200640 1 -- 172.21.15.31:0/1220019441 wait complete. 2024-09-13T08:37:14.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T08:37:13.037289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590950,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":121,"num_write_kb":1629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236060,"kb_used_data":2680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738660,"statfs":{"total":773094113280,"available":772852387840,"internally_reserved":0,"allocated":2744320,"data_stored":1660308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":582,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.050741"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994505+0000","last_change":"2024-09-13T08:37:07.994505+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994505+0000","last_undegraded":"2024-09-13T08:37:07.994505+0000","last_fullsized":"2024-09-13T08:37:07.994505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.001152+0000","last_change":"2024-09-13T08:37:08.001152+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.001152+0000","last_undegraded":"2024-09-13T08:37:08.001152+0000","last_fullsized":"2024-09-13T08:37:08.001152+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997926+0000","last_change":"2024-09-13T08:37:07.997926+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997926+0000","last_undegraded":"2024-09-13T08:37:07.997926+0000","last_fullsized":"2024-09-13T08:37:07.997926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.000156+0000","last_change":"2024-09-13T08:37:08.000156+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.000156+0000","last_undegraded":"2024-09-13T08:37:08.000156+0000","last_fullsized":"2024-09-13T08:37:08.000156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.993476+0000","last_change":"2024-09-13T08:37:07.993476+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.993476+0000","last_undegraded":"2024-09-13T08:37:07.993476+0000","last_fullsized":"2024-09-13T08:37:07.993476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991160+0000","last_change":"2024-09-13T08:37:07.991160+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991160+0000","last_undegraded":"2024-09-13T08:37:07.991160+0000","last_fullsized":"2024-09-13T08:37:07.991160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991362+0000","last_change":"2024-09-13T08:37:07.991362+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991362+0000","last_undegraded":"2024-09-13T08:37:07.991362+0000","last_fullsized":"2024-09-13T08:37:07.991362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.995305+0000","last_change":"2024-09-13T08:37:07.995305+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.995305+0000","last_undegraded":"2024-09-13T08:37:07.995305+0000","last_fullsized":"2024-09-13T08:37:07.995305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.989862+0000","last_change":"2024-09-13T08:37:07.989862+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.989862+0000","last_undegraded":"2024-09-13T08:37:07.989862+0000","last_fullsized":"2024-09-13T08:37:07.989862+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.000116+0000","last_change":"2024-09-13T08:37:08.000116+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.000116+0000","last_undegraded":"2024-09-13T08:37:08.000116+0000","last_fullsized":"2024-09-13T08:37:08.000116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990230+0000","last_change":"2024-09-13T08:37:07.990230+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990230+0000","last_undegraded":"2024-09-13T08:37:07.990230+0000","last_fullsized":"2024-09-13T08:37:07.990230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990135+0000","last_change":"2024-09-13T08:37:07.990135+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990135+0000","last_undegraded":"2024-09-13T08:37:07.990135+0000","last_fullsized":"2024-09-13T08:37:07.990135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994486+0000","last_change":"2024-09-13T08:37:07.994486+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994486+0000","last_undegraded":"2024-09-13T08:37:07.994486+0000","last_fullsized":"2024-09-13T08:37:07.994486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997853+0000","last_change":"2024-09-13T08:37:07.997853+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997853+0000","last_undegraded":"2024-09-13T08:37:07.997853+0000","last_fullsized":"2024-09-13T08:37:07.997853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":105,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T08:37:07.993837+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:07.993837+0000","last_peered":"2024-09-13T08:37:07.993837+0000","last_clean":"2024-09-13T08:37:07.993837+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:07.993837+0000","last_undegraded":"2024-09-13T08:37:07.993837+0000","last_fullsized":"2024-09-13T08:37:07.993837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998362+0000","last_change":"2024-09-13T08:37:07.998362+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998362+0000","last_undegraded":"2024-09-13T08:37:07.998362+0000","last_fullsized":"2024-09-13T08:37:07.998362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998337+0000","last_change":"2024-09-13T08:37:07.998337+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998337+0000","last_undegraded":"2024-09-13T08:37:07.998337+0000","last_fullsized":"2024-09-13T08:37:07.998337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991221+0000","last_change":"2024-09-13T08:37:07.991221+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991221+0000","last_undegraded":"2024-09-13T08:37:07.991221+0000","last_fullsized":"2024-09-13T08:37:07.991221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998457+0000","last_change":"2024-09-13T08:37:07.998457+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998457+0000","last_undegraded":"2024-09-13T08:37:07.998457+0000","last_fullsized":"2024-09-13T08:37:07.998457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994499+0000","last_change":"2024-09-13T08:37:07.994499+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994499+0000","last_undegraded":"2024-09-13T08:37:07.994499+0000","last_fullsized":"2024-09-13T08:37:07.994499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999603+0000","last_change":"2024-09-13T08:37:07.999603+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999603+0000","last_undegraded":"2024-09-13T08:37:07.999603+0000","last_fullsized":"2024-09-13T08:37:07.999603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999370+0000","last_change":"2024-09-13T08:37:07.999370+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999370+0000","last_undegraded":"2024-09-13T08:37:07.999370+0000","last_fullsized":"2024-09-13T08:37:07.999370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990197+0000","last_change":"2024-09-13T08:37:07.990197+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990197+0000","last_undegraded":"2024-09-13T08:37:07.990197+0000","last_fullsized":"2024-09-13T08:37:07.990197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997875+0000","last_change":"2024-09-13T08:37:07.997875+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997875+0000","last_undegraded":"2024-09-13T08:37:07.997875+0000","last_fullsized":"2024-09-13T08:37:07.997875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999965+0000","last_change":"2024-09-13T08:37:07.999965+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999965+0000","last_undegraded":"2024-09-13T08:37:07.999965+0000","last_fullsized":"2024-09-13T08:37:07.999965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990298+0000","last_change":"2024-09-13T08:37:07.990298+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990298+0000","last_undegraded":"2024-09-13T08:37:07.990298+0000","last_fullsized":"2024-09-13T08:37:07.990298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990876+0000","last_change":"2024-09-13T08:37:07.990876+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990876+0000","last_undegraded":"2024-09-13T08:37:07.990876+0000","last_fullsized":"2024-09-13T08:37:07.990876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994548+0000","last_change":"2024-09-13T08:37:07.994548+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994548+0000","last_undegraded":"2024-09-13T08:37:07.994548+0000","last_fullsized":"2024-09-13T08:37:07.994548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":582,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":778240,"data_stored":646156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":778240,"data_stored":646156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":61605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:14.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:14.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 -- 172.21.15.31:0/3310522899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0cf280 msgr2=0x7fcc3c0ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 --2- 172.21.15.31:0/3310522899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0cf280 0x7fcc3c0ca4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400a970 tx=0x7fcc2402b8d0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 -- 172.21.15.31:0/3310522899 shutdown_connections 2024-09-13T08:37:14.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 --2- 172.21.15.31:0/3310522899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0cf280 0x7fcc3c0ca4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 -- 172.21.15.31:0/3310522899 >> 172.21.15.31:0/3310522899 conn(0x7fcc3c0cc910 msgr2=0x7fcc3c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:14.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 -- 172.21.15.31:0/3310522899 shutdown_connections 2024-09-13T08:37:14.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.455+0000 7fcc41a00640 1 -- 172.21.15.31:0/3310522899 wait complete. 2024-09-13T08:37:14.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.458+0000 7fcc41a00640 1 Processor -- start 2024-09-13T08:37:14.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.458+0000 7fcc41a00640 1 -- start start 2024-09-13T08:37:14.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 0x7fcc3c073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c15af30 0x7fcc3c155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c0cdfa0 con 0x7fcc3c0cf280 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c0ad450 con 0x7fcc3c15af30 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcc3c0ad5d0 con 0x7fcc3c0caa20 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59112/0 (socket says 172.21.15.31:59112) 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 -- 172.21.15.31:0/427990296 learned_addr learned my addr 172.21.15.31:0/427990296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc3b400640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c15af30 0x7fcc3c155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc3aa00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 0x7fcc3c073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 -- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 msgr2=0x7fcc3c073670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 0x7fcc3c073670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 -- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c15af30 msgr2=0x7fcc3c155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c15af30 0x7fcc3c155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 -- 172.21.15.31:0/427990296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3c155900 con 0x7fcc3c0cf280 2024-09-13T08:37:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc3aa00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 0x7fcc3c073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:14.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc33e00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc24000c00 tx=0x7fcc24004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240035f0 con 0x7fcc3c0cf280 2024-09-13T08:37:14.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc24003790 con 0x7fcc3c0cf280 2024-09-13T08:37:14.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24003be0 con 0x7fcc3c0cf280 2024-09-13T08:37:14.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3c073420 con 0x7fcc3c0cf280 2024-09-13T08:37:14.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.459+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3c09b940 con 0x7fcc3c0cf280 2024-09-13T08:37:14.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.460+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3c0ca4e0 con 0x7fcc3c0cf280 2024-09-13T08:37:14.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.461+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fcc24032460 con 0x7fcc3c0cf280 2024-09-13T08:37:14.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.463+0000 7fcc32a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 0x7fcc1807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.463+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fcc240bc130 con 0x7fcc3c0cf280 2024-09-13T08:37:14.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.463+0000 7fcc3aa00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 0x7fcc1807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.463+0000 7fcc3aa00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 0x7fcc1807ad10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c001120 tx=0x7fcc2c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.466+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcc24084790 con 0x7fcc3c0cf280 2024-09-13T08:37:14.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.657+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc3c155570 con 0x7fcc18078850 2024-09-13T08:37:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.663+0000 7fcc32a00640 1 -- 172.21.15.31:0/427990296 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241528 (secure 0 0 0) 0x7fcc2c001c40 con 0x7fcc18078850 2024-09-13T08:37:14.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:14.667 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 msgr2=0x7fcc1807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 0x7fcc1807ad10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c001120 tx=0x7fcc2c009040 comp rx=0 tx=0).stop 2024-09-13T08:37:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 msgr2=0x7fcc3c072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc24000c00 tx=0x7fcc24004270 comp rx=0 tx=0).stop 2024-09-13T08:37:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 shutdown_connections 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c15af30 0x7fcc3c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3c0cf280 0x7fcc3c072d20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc18078850 0x7fcc1807ad10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 --2- 172.21.15.31:0/427990296 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc3c0caa20 0x7fcc3c073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 >> 172.21.15.31:0/427990296 conn(0x7fcc3c0cc910 msgr2=0x7fcc3c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 shutdown_connections 2024-09-13T08:37:14.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.667+0000 7fcc41a00640 1 -- 172.21.15.31:0/427990296 wait complete. 2024-09-13T08:37:14.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T08:37:13.037289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590950,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":121,"num_write_kb":1629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236060,"kb_used_data":2680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738660,"statfs":{"total":773094113280,"available":772852387840,"internally_reserved":0,"allocated":2744320,"data_stored":1660308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":582,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.050741"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994505+0000","last_change":"2024-09-13T08:37:07.994505+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994505+0000","last_undegraded":"2024-09-13T08:37:07.994505+0000","last_fullsized":"2024-09-13T08:37:07.994505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.001152+0000","last_change":"2024-09-13T08:37:08.001152+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.001152+0000","last_undegraded":"2024-09-13T08:37:08.001152+0000","last_fullsized":"2024-09-13T08:37:08.001152+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997926+0000","last_change":"2024-09-13T08:37:07.997926+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997926+0000","last_undegraded":"2024-09-13T08:37:07.997926+0000","last_fullsized":"2024-09-13T08:37:07.997926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.000156+0000","last_change":"2024-09-13T08:37:08.000156+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.000156+0000","last_undegraded":"2024-09-13T08:37:08.000156+0000","last_fullsized":"2024-09-13T08:37:08.000156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.993476+0000","last_change":"2024-09-13T08:37:07.993476+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.993476+0000","last_undegraded":"2024-09-13T08:37:07.993476+0000","last_fullsized":"2024-09-13T08:37:07.993476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991160+0000","last_change":"2024-09-13T08:37:07.991160+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991160+0000","last_undegraded":"2024-09-13T08:37:07.991160+0000","last_fullsized":"2024-09-13T08:37:07.991160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991362+0000","last_change":"2024-09-13T08:37:07.991362+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991362+0000","last_undegraded":"2024-09-13T08:37:07.991362+0000","last_fullsized":"2024-09-13T08:37:07.991362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.995305+0000","last_change":"2024-09-13T08:37:07.995305+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.995305+0000","last_undegraded":"2024-09-13T08:37:07.995305+0000","last_fullsized":"2024-09-13T08:37:07.995305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.989862+0000","last_change":"2024-09-13T08:37:07.989862+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.989862+0000","last_undegraded":"2024-09-13T08:37:07.989862+0000","last_fullsized":"2024-09-13T08:37:07.989862+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:08.000116+0000","last_change":"2024-09-13T08:37:08.000116+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.000116+0000","last_undegraded":"2024-09-13T08:37:08.000116+0000","last_fullsized":"2024-09-13T08:37:08.000116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990230+0000","last_change":"2024-09-13T08:37:07.990230+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990230+0000","last_undegraded":"2024-09-13T08:37:07.990230+0000","last_fullsized":"2024-09-13T08:37:07.990230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990135+0000","last_change":"2024-09-13T08:37:07.990135+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990135+0000","last_undegraded":"2024-09-13T08:37:07.990135+0000","last_fullsized":"2024-09-13T08:37:07.990135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994486+0000","last_change":"2024-09-13T08:37:07.994486+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994486+0000","last_undegraded":"2024-09-13T08:37:07.994486+0000","last_fullsized":"2024-09-13T08:37:07.994486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997853+0000","last_change":"2024-09-13T08:37:07.997853+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997853+0000","last_undegraded":"2024-09-13T08:37:07.997853+0000","last_fullsized":"2024-09-13T08:37:07.997853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":105,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T08:37:07.993837+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:07.993837+0000","last_peered":"2024-09-13T08:37:07.993837+0000","last_clean":"2024-09-13T08:37:07.993837+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:07.993837+0000","last_undegraded":"2024-09-13T08:37:07.993837+0000","last_fullsized":"2024-09-13T08:37:07.993837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998362+0000","last_change":"2024-09-13T08:37:07.998362+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998362+0000","last_undegraded":"2024-09-13T08:37:07.998362+0000","last_fullsized":"2024-09-13T08:37:07.998362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998337+0000","last_change":"2024-09-13T08:37:07.998337+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998337+0000","last_undegraded":"2024-09-13T08:37:07.998337+0000","last_fullsized":"2024-09-13T08:37:07.998337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.991221+0000","last_change":"2024-09-13T08:37:07.991221+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.991221+0000","last_undegraded":"2024-09-13T08:37:07.991221+0000","last_fullsized":"2024-09-13T08:37:07.991221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.998457+0000","last_change":"2024-09-13T08:37:07.998457+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.998457+0000","last_undegraded":"2024-09-13T08:37:07.998457+0000","last_fullsized":"2024-09-13T08:37:07.998457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994499+0000","last_change":"2024-09-13T08:37:07.994499+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994499+0000","last_undegraded":"2024-09-13T08:37:07.994499+0000","last_fullsized":"2024-09-13T08:37:07.994499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999603+0000","last_change":"2024-09-13T08:37:07.999603+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999603+0000","last_undegraded":"2024-09-13T08:37:07.999603+0000","last_fullsized":"2024-09-13T08:37:07.999603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999370+0000","last_change":"2024-09-13T08:37:07.999370+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999370+0000","last_undegraded":"2024-09-13T08:37:07.999370+0000","last_fullsized":"2024-09-13T08:37:07.999370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990197+0000","last_change":"2024-09-13T08:37:07.990197+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990197+0000","last_undegraded":"2024-09-13T08:37:07.990197+0000","last_fullsized":"2024-09-13T08:37:07.990197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.997875+0000","last_change":"2024-09-13T08:37:07.997875+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.997875+0000","last_undegraded":"2024-09-13T08:37:07.997875+0000","last_fullsized":"2024-09-13T08:37:07.997875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.999965+0000","last_change":"2024-09-13T08:37:07.999965+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.999965+0000","last_undegraded":"2024-09-13T08:37:07.999965+0000","last_fullsized":"2024-09-13T08:37:07.999965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990298+0000","last_change":"2024-09-13T08:37:07.990298+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990298+0000","last_undegraded":"2024-09-13T08:37:07.990298+0000","last_fullsized":"2024-09-13T08:37:07.990298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.990876+0000","last_change":"2024-09-13T08:37:07.990876+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.990876+0000","last_undegraded":"2024-09-13T08:37:07.990876+0000","last_fullsized":"2024-09-13T08:37:07.990876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-09-13T08:37:07.994548+0000","last_change":"2024-09-13T08:37:07.994548+0000","last_active":"2024-09-13T08:37:07.979801+0000","last_peered":"2024-09-13T08:37:07.979801+0000","last_clean":"2024-09-13T08:37:07.979801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:07.994548+0000","last_undegraded":"2024-09-13T08:37:07.994548+0000","last_fullsized":"2024-09-13T08:37:07.994548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:37:08.985399+0000","last_change":"2024-09-13T08:37:08.985399+0000","last_active":"2024-09-13T08:37:08.985399+0000","last_peered":"2024-09-13T08:37:08.985399+0000","last_clean":"2024-09-13T08:37:08.985399+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:37:08.985399+0000","last_undegraded":"2024-09-13T08:37:08.985399+0000","last_fullsized":"2024-09-13T08:37:08.985399+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":582,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":778240,"data_stored":646156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":55788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":778240,"data_stored":646156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":61605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:14.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.881+0000 7f0e0ae00640 1 -- 172.21.15.31:0/850625760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e040cbc50 msgr2=0x7f0e040cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.881+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/850625760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e040cbc50 0x7f0e040cc0a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009be0 tx=0x7f0dfc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.882+0000 7f0e0ae00640 1 -- 172.21.15.31:0/850625760 shutdown_connections 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.882+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/850625760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e040cbc50 0x7f0e040cc0a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.882+0000 7f0e0ae00640 1 -- 172.21.15.31:0/850625760 >> 172.21.15.31:0/850625760 conn(0x7f0e040cc910 msgr2=0x7f0e040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.882+0000 7f0e0ae00640 1 -- 172.21.15.31:0/850625760 shutdown_connections 2024-09-13T08:37:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.882+0000 7f0e0ae00640 1 -- 172.21.15.31:0/850625760 wait complete. 2024-09-13T08:37:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.883+0000 7f0e0ae00640 1 Processor -- start 2024-09-13T08:37:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.883+0000 7f0e0ae00640 1 -- start start 2024-09-13T08:37:14.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e040cbc50 0x7f0e040b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e0415bd70 0x7f0e040c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e041600f0 con 0x7f0e040cf850 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040cead0 con 0x7f0e040cbc50 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e0ae00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0e0415ce40 con 0x7f0e0415bd70 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e03400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e040cbc50 0x7f0e040b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e03400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34374/0 (socket says 172.21.15.31:34374) 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.884+0000 7f0e03400640 1 -- 172.21.15.31:0/2493856444 learned_addr learned my addr 172.21.15.31:0/2493856444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e08e00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e0415bd70 0x7f0e040c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 -- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e0415bd70 msgr2=0x7f0e040c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e0415bd70 0x7f0e040c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 -- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e040cbc50 msgr2=0x7f0e040b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e040cbc50 0x7f0e040b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 -- 172.21.15.31:0/2493856444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0415d120 con 0x7f0e040cf850 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e03400640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c730 tx=0x7f0df000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df000a840 con 0x7f0e040cf850 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0df0017070 con 0x7f0e040cf850 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0415d400 con 0x7f0e040cf850 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.885+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df000ad90 con 0x7f0e040cf850 2024-09-13T08:37:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.886+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0415d9c0 con 0x7f0e040cf850 2024-09-13T08:37:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.887+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e0415db90 con 0x7f0e040cf850 2024-09-13T08:37:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.888+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0df001c460 con 0x7f0e040cf850 2024-09-13T08:37:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.888+0000 7f0dfbe00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 0x7f0de407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.889+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0df009a6c0 con 0x7f0e040cf850 2024-09-13T08:37:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.889+0000 7f0e03e00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 0x7f0de407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.889+0000 7f0e03e00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 0x7f0de407ad10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0df4001120 tx=0x7f0df4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:14.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:14.893+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0df00626d0 con 0x7f0e040cf850 2024-09-13T08:37:14.979 INFO:tasks.ceph.mds.b.smithi045.stderr:2024-09-13T08:37:14.978+0000 7f59cda00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:37:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.087+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e040ae730 con 0x7f0de4078850 2024-09-13T08:37:15.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.093+0000 7f0dfbe00640 1 -- 172.21.15.31:0/2493856444 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251932 (secure 0 0 0) 0x7f0e040ae730 con 0x7f0de4078850 2024-09-13T08:37:15.098 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:15.099 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 msgr2=0x7f0de407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 0x7f0de407ad10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0df4001120 tx=0x7f0df4009040 comp rx=0 tx=0).stop 2024-09-13T08:37:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 msgr2=0x7f0e0415c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c730 tx=0x7f0df000cbf0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 shutdown_connections 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0de4078850 0x7f0de407ad10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e0415bd70 0x7f0e040c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e040cf850 0x7f0e0415c460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 --2- 172.21.15.31:0/2493856444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e040cbc50 0x7f0e040b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 >> 172.21.15.31:0/2493856444 conn(0x7f0e040cc910 msgr2=0x7f0e040af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 shutdown_connections 2024-09-13T08:37:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.097+0000 7f0e0ae00640 1 -- 172.21.15.31:0/2493856444 wait complete. 2024-09-13T08:37:15.114 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-13T08:37:15.037781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235920,"kb_used_data":3024,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738800,"statfs":{"total":773094113280,"available":772852531200,"internally_reserved":0,"allocated":3096576,"data_stored":1820596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.047925"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021976+0000","last_change":"2024-09-13T08:37:10.038242+0000","last_active":"2024-09-13T08:37:11.021976+0000","last_peered":"2024-09-13T08:37:11.021976+0000","last_clean":"2024-09-13T08:37:11.021976+0000","last_became_active":"2024-09-13T08:37:10.037984+0000","last_became_peered":"2024-09-13T08:37:10.037984+0000","last_unstale":"2024-09-13T08:37:11.021976+0000","last_undegraded":"2024-09-13T08:37:11.021976+0000","last_fullsized":"2024-09-13T08:37:11.021976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:05.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328871+0000","last_change":"2024-09-13T08:37:10.043402+0000","last_active":"2024-09-13T08:37:11.328871+0000","last_peered":"2024-09-13T08:37:11.328871+0000","last_clean":"2024-09-13T08:37:11.328871+0000","last_became_active":"2024-09-13T08:37:10.041028+0000","last_became_peered":"2024-09-13T08:37:10.041028+0000","last_unstale":"2024-09-13T08:37:11.328871+0000","last_undegraded":"2024-09-13T08:37:11.328871+0000","last_fullsized":"2024-09-13T08:37:11.328871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328967+0000","last_change":"2024-09-13T08:37:10.038931+0000","last_active":"2024-09-13T08:37:11.328967+0000","last_peered":"2024-09-13T08:37:11.328967+0000","last_clean":"2024-09-13T08:37:11.328967+0000","last_became_active":"2024-09-13T08:37:10.038676+0000","last_became_peered":"2024-09-13T08:37:10.038676+0000","last_unstale":"2024-09-13T08:37:11.328967+0000","last_undegraded":"2024-09-13T08:37:11.328967+0000","last_fullsized":"2024-09-13T08:37:11.328967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015339+0000","last_change":"2024-09-13T08:37:10.018106+0000","last_active":"2024-09-13T08:37:11.015339+0000","last_peered":"2024-09-13T08:37:11.015339+0000","last_clean":"2024-09-13T08:37:11.015339+0000","last_became_active":"2024-09-13T08:37:10.017812+0000","last_became_peered":"2024-09-13T08:37:10.017812+0000","last_unstale":"2024-09-13T08:37:11.015339+0000","last_undegraded":"2024-09-13T08:37:11.015339+0000","last_fullsized":"2024-09-13T08:37:11.015339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021892+0000","last_change":"2024-09-13T08:37:10.035463+0000","last_active":"2024-09-13T08:37:11.021892+0000","last_peered":"2024-09-13T08:37:11.021892+0000","last_clean":"2024-09-13T08:37:11.021892+0000","last_became_active":"2024-09-13T08:37:10.032852+0000","last_became_peered":"2024-09-13T08:37:10.032852+0000","last_unstale":"2024-09-13T08:37:11.021892+0000","last_undegraded":"2024-09-13T08:37:11.021892+0000","last_fullsized":"2024-09-13T08:37:11.021892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:34.731391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015062+0000","last_change":"2024-09-13T08:37:10.030116+0000","last_active":"2024-09-13T08:37:11.015062+0000","last_peered":"2024-09-13T08:37:11.015062+0000","last_clean":"2024-09-13T08:37:11.015062+0000","last_became_active":"2024-09-13T08:37:10.029773+0000","last_became_peered":"2024-09-13T08:37:10.029773+0000","last_unstale":"2024-09-13T08:37:11.015062+0000","last_undegraded":"2024-09-13T08:37:11.015062+0000","last_fullsized":"2024-09-13T08:37:11.015062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021640+0000","last_change":"2024-09-13T08:37:10.025604+0000","last_active":"2024-09-13T08:37:11.021640+0000","last_peered":"2024-09-13T08:37:11.021640+0000","last_clean":"2024-09-13T08:37:11.021640+0000","last_became_active":"2024-09-13T08:37:10.020828+0000","last_became_peered":"2024-09-13T08:37:10.020828+0000","last_unstale":"2024-09-13T08:37:11.021640+0000","last_undegraded":"2024-09-13T08:37:11.021640+0000","last_fullsized":"2024-09-13T08:37:11.021640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328358+0000","last_change":"2024-09-13T08:37:10.041296+0000","last_active":"2024-09-13T08:37:11.328358+0000","last_peered":"2024-09-13T08:37:11.328358+0000","last_clean":"2024-09-13T08:37:11.328358+0000","last_became_active":"2024-09-13T08:37:10.035151+0000","last_became_peered":"2024-09-13T08:37:10.035151+0000","last_unstale":"2024-09-13T08:37:11.328358+0000","last_undegraded":"2024-09-13T08:37:11.328358+0000","last_fullsized":"2024-09-13T08:37:11.328358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:28.830741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723967+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:12.723967+0000","last_peered":"2024-09-13T08:37:12.723967+0000","last_clean":"2024-09-13T08:37:12.723967+0000","last_became_active":"2024-09-13T08:37:10.026743+0000","last_became_peered":"2024-09-13T08:37:10.026743+0000","last_unstale":"2024-09-13T08:37:12.723967+0000","last_undegraded":"2024-09-13T08:37:12.723967+0000","last_fullsized":"2024-09-13T08:37:12.723967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022008+0000","last_change":"2024-09-13T08:37:10.020548+0000","last_active":"2024-09-13T08:37:11.022008+0000","last_peered":"2024-09-13T08:37:11.022008+0000","last_clean":"2024-09-13T08:37:11.022008+0000","last_became_active":"2024-09-13T08:37:10.020144+0000","last_became_peered":"2024-09-13T08:37:10.020144+0000","last_unstale":"2024-09-13T08:37:11.022008+0000","last_undegraded":"2024-09-13T08:37:11.022008+0000","last_fullsized":"2024-09-13T08:37:11.022008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:13.408879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022140+0000","last_change":"2024-09-13T08:37:10.038191+0000","last_active":"2024-09-13T08:37:11.022140+0000","last_peered":"2024-09-13T08:37:11.022140+0000","last_clean":"2024-09-13T08:37:11.022140+0000","last_became_active":"2024-09-13T08:37:10.037935+0000","last_became_peered":"2024-09-13T08:37:10.037935+0000","last_unstale":"2024-09-13T08:37:11.022140+0000","last_undegraded":"2024-09-13T08:37:11.022140+0000","last_fullsized":"2024-09-13T08:37:11.022140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021867+0000","last_change":"2024-09-13T08:37:10.025910+0000","last_active":"2024-09-13T08:37:11.021867+0000","last_peered":"2024-09-13T08:37:11.021867+0000","last_clean":"2024-09-13T08:37:11.021867+0000","last_became_active":"2024-09-13T08:37:10.023349+0000","last_became_peered":"2024-09-13T08:37:10.023349+0000","last_unstale":"2024-09-13T08:37:11.021867+0000","last_undegraded":"2024-09-13T08:37:11.021867+0000","last_fullsized":"2024-09-13T08:37:11.021867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:02.005454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863687+0000","last_change":"2024-09-13T08:37:10.033914+0000","last_active":"2024-09-13T08:37:12.863687+0000","last_peered":"2024-09-13T08:37:12.863687+0000","last_clean":"2024-09-13T08:37:12.863687+0000","last_became_active":"2024-09-13T08:37:10.030947+0000","last_became_peered":"2024-09-13T08:37:10.030947+0000","last_unstale":"2024-09-13T08:37:12.863687+0000","last_undegraded":"2024-09-13T08:37:12.863687+0000","last_fullsized":"2024-09-13T08:37:12.863687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:49.714886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863461+0000","last_change":"2024-09-13T08:37:10.024803+0000","last_active":"2024-09-13T08:37:12.863461+0000","last_peered":"2024-09-13T08:37:12.863461+0000","last_clean":"2024-09-13T08:37:12.863461+0000","last_became_active":"2024-09-13T08:37:10.024512+0000","last_became_peered":"2024-09-13T08:37:10.024512+0000","last_unstale":"2024-09-13T08:37:12.863461+0000","last_undegraded":"2024-09-13T08:37:12.863461+0000","last_fullsized":"2024-09-13T08:37:12.863461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328365+0000","last_change":"2024-09-13T08:37:10.035650+0000","last_active":"2024-09-13T08:37:11.328365+0000","last_peered":"2024-09-13T08:37:11.328365+0000","last_clean":"2024-09-13T08:37:11.328365+0000","last_became_active":"2024-09-13T08:37:10.035331+0000","last_became_peered":"2024-09-13T08:37:10.035331+0000","last_unstale":"2024-09-13T08:37:11.328365+0000","last_undegraded":"2024-09-13T08:37:11.328365+0000","last_fullsized":"2024-09-13T08:37:11.328365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.703160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.024424+0000","last_change":"2024-09-13T08:37:10.022368+0000","last_active":"2024-09-13T08:37:10.024424+0000","last_peered":"2024-09-13T08:37:10.024424+0000","last_clean":"2024-09-13T08:37:10.024424+0000","last_became_active":"2024-09-13T08:37:10.022095+0000","last_became_peered":"2024-09-13T08:37:10.022095+0000","last_unstale":"2024-09-13T08:37:10.024424+0000","last_undegraded":"2024-09-13T08:37:10.024424+0000","last_fullsized":"2024-09-13T08:37:10.024424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039163+0000","last_change":"2024-09-13T08:37:10.032355+0000","last_active":"2024-09-13T08:37:10.039163+0000","last_peered":"2024-09-13T08:37:10.039163+0000","last_clean":"2024-09-13T08:37:10.039163+0000","last_became_active":"2024-09-13T08:37:10.032098+0000","last_became_peered":"2024-09-13T08:37:10.032098+0000","last_unstale":"2024-09-13T08:37:10.039163+0000","last_undegraded":"2024-09-13T08:37:10.039163+0000","last_fullsized":"2024-09-13T08:37:10.039163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017318+0000","last_change":"2024-09-13T08:37:10.022000+0000","last_active":"2024-09-13T08:37:11.017318+0000","last_peered":"2024-09-13T08:37:11.017318+0000","last_clean":"2024-09-13T08:37:11.017318+0000","last_became_active":"2024-09-13T08:37:10.021622+0000","last_became_peered":"2024-09-13T08:37:10.021622+0000","last_unstale":"2024-09-13T08:37:11.017318+0000","last_undegraded":"2024-09-13T08:37:11.017318+0000","last_fullsized":"2024-09-13T08:37:11.017318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034062+0000","last_change":"2024-09-13T08:37:10.026234+0000","last_active":"2024-09-13T08:37:10.034062+0000","last_peered":"2024-09-13T08:37:10.034062+0000","last_clean":"2024-09-13T08:37:10.034062+0000","last_became_active":"2024-09-13T08:37:10.025970+0000","last_became_peered":"2024-09-13T08:37:10.025970+0000","last_unstale":"2024-09-13T08:37:10.034062+0000","last_undegraded":"2024-09-13T08:37:10.034062+0000","last_fullsized":"2024-09-13T08:37:10.034062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:34.396268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021932+0000","last_change":"2024-09-13T08:37:10.028101+0000","last_active":"2024-09-13T08:37:11.021932+0000","last_peered":"2024-09-13T08:37:11.021932+0000","last_clean":"2024-09-13T08:37:11.021932+0000","last_became_active":"2024-09-13T08:37:10.021666+0000","last_became_peered":"2024-09-13T08:37:10.021666+0000","last_unstale":"2024-09-13T08:37:11.021932+0000","last_undegraded":"2024-09-13T08:37:11.021932+0000","last_fullsized":"2024-09-13T08:37:11.021932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:43.682175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723677+0000","last_change":"2024-09-13T08:37:10.020945+0000","last_active":"2024-09-13T08:37:12.723677+0000","last_peered":"2024-09-13T08:37:12.723677+0000","last_clean":"2024-09-13T08:37:12.723677+0000","last_became_active":"2024-09-13T08:37:10.020675+0000","last_became_peered":"2024-09-13T08:37:10.020675+0000","last_unstale":"2024-09-13T08:37:12.723677+0000","last_undegraded":"2024-09-13T08:37:12.723677+0000","last_fullsized":"2024-09-13T08:37:12.723677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:41.501115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328354+0000","last_change":"2024-09-13T08:37:10.036002+0000","last_active":"2024-09-13T08:37:11.328354+0000","last_peered":"2024-09-13T08:37:11.328354+0000","last_clean":"2024-09-13T08:37:11.328354+0000","last_became_active":"2024-09-13T08:37:10.035684+0000","last_became_peered":"2024-09-13T08:37:10.035684+0000","last_unstale":"2024-09-13T08:37:11.328354+0000","last_undegraded":"2024-09-13T08:37:11.328354+0000","last_fullsized":"2024-09-13T08:37:11.328354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039036+0000","last_change":"2024-09-13T08:37:10.032065+0000","last_active":"2024-09-13T08:37:10.039036+0000","last_peered":"2024-09-13T08:37:10.039036+0000","last_clean":"2024-09-13T08:37:10.039036+0000","last_became_active":"2024-09-13T08:37:10.031802+0000","last_became_peered":"2024-09-13T08:37:10.031802+0000","last_unstale":"2024-09-13T08:37:10.039036+0000","last_undegraded":"2024-09-13T08:37:10.039036+0000","last_fullsized":"2024-09-13T08:37:10.039036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328937+0000","last_change":"2024-09-13T08:37:10.036040+0000","last_active":"2024-09-13T08:37:11.328937+0000","last_peered":"2024-09-13T08:37:11.328937+0000","last_clean":"2024-09-13T08:37:11.328937+0000","last_became_active":"2024-09-13T08:37:10.035698+0000","last_became_peered":"2024-09-13T08:37:10.035698+0000","last_unstale":"2024-09-13T08:37:11.328937+0000","last_undegraded":"2024-09-13T08:37:11.328937+0000","last_fullsized":"2024-09-13T08:37:11.328937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015904+0000","last_change":"2024-09-13T08:37:10.023312+0000","last_active":"2024-09-13T08:37:11.015904+0000","last_peered":"2024-09-13T08:37:11.015904+0000","last_clean":"2024-09-13T08:37:11.015904+0000","last_became_active":"2024-09-13T08:37:10.022193+0000","last_became_peered":"2024-09-13T08:37:10.022193+0000","last_unstale":"2024-09-13T08:37:11.015904+0000","last_undegraded":"2024-09-13T08:37:11.015904+0000","last_fullsized":"2024-09-13T08:37:11.015904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863802+0000","last_change":"2024-09-13T08:37:10.028354+0000","last_active":"2024-09-13T08:37:12.863802+0000","last_peered":"2024-09-13T08:37:12.863802+0000","last_clean":"2024-09-13T08:37:12.863802+0000","last_became_active":"2024-09-13T08:37:10.027989+0000","last_became_peered":"2024-09-13T08:37:10.027989+0000","last_unstale":"2024-09-13T08:37:12.863802+0000","last_undegraded":"2024-09-13T08:37:12.863802+0000","last_fullsized":"2024-09-13T08:37:12.863802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863758+0000","last_change":"2024-09-13T08:37:10.039029+0000","last_active":"2024-09-13T08:37:12.863758+0000","last_peered":"2024-09-13T08:37:12.863758+0000","last_clean":"2024-09-13T08:37:12.863758+0000","last_became_active":"2024-09-13T08:37:10.038689+0000","last_became_peered":"2024-09-13T08:37:10.038689+0000","last_unstale":"2024-09-13T08:37:12.863758+0000","last_undegraded":"2024-09-13T08:37:12.863758+0000","last_fullsized":"2024-09-13T08:37:12.863758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328458+0000","last_change":"2024-09-13T08:37:10.035713+0000","last_active":"2024-09-13T08:37:11.328458+0000","last_peered":"2024-09-13T08:37:11.328458+0000","last_clean":"2024-09-13T08:37:11.328458+0000","last_became_active":"2024-09-13T08:37:10.035292+0000","last_became_peered":"2024-09-13T08:37:10.035292+0000","last_unstale":"2024-09-13T08:37:11.328458+0000","last_undegraded":"2024-09-13T08:37:11.328458+0000","last_fullsized":"2024-09-13T08:37:11.328458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.038709+0000","last_change":"2024-09-13T08:37:10.029548+0000","last_active":"2024-09-13T08:37:10.038709+0000","last_peered":"2024-09-13T08:37:10.038709+0000","last_clean":"2024-09-13T08:37:10.038709+0000","last_became_active":"2024-09-13T08:37:10.029292+0000","last_became_peered":"2024-09-13T08:37:10.029292+0000","last_unstale":"2024-09-13T08:37:10.038709+0000","last_undegraded":"2024-09-13T08:37:10.038709+0000","last_fullsized":"2024-09-13T08:37:10.038709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016762+0000","last_change":"2024-09-13T08:37:10.012936+0000","last_active":"2024-09-13T08:37:11.016762+0000","last_peered":"2024-09-13T08:37:11.016762+0000","last_clean":"2024-09-13T08:37:11.016762+0000","last_became_active":"2024-09-13T08:37:10.012582+0000","last_became_peered":"2024-09-13T08:37:10.012582+0000","last_unstale":"2024-09-13T08:37:11.016762+0000","last_undegraded":"2024-09-13T08:37:11.016762+0000","last_fullsized":"2024-09-13T08:37:11.016762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015064+0000","last_change":"2024-09-13T08:37:10.017195+0000","last_active":"2024-09-13T08:37:11.015064+0000","last_peered":"2024-09-13T08:37:11.015064+0000","last_clean":"2024-09-13T08:37:11.015064+0000","last_became_active":"2024-09-13T08:37:10.016877+0000","last_became_peered":"2024-09-13T08:37:10.016877+0000","last_unstale":"2024-09-13T08:37:11.015064+0000","last_undegraded":"2024-09-13T08:37:11.015064+0000","last_fullsized":"2024-09-13T08:37:11.015064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.474080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723910+0000","last_change":"2024-09-13T08:37:10.023947+0000","last_active":"2024-09-13T08:37:12.723910+0000","last_peered":"2024-09-13T08:37:12.723910+0000","last_clean":"2024-09-13T08:37:12.723910+0000","last_became_active":"2024-09-13T08:37:10.023659+0000","last_became_peered":"2024-09-13T08:37:10.023659+0000","last_unstale":"2024-09-13T08:37:12.723910+0000","last_undegraded":"2024-09-13T08:37:12.723910+0000","last_fullsized":"2024-09-13T08:37:12.723910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328520+0000","last_change":"2024-09-13T08:37:09.014843+0000","last_active":"2024-09-13T08:37:11.328520+0000","last_peered":"2024-09-13T08:37:11.328520+0000","last_clean":"2024-09-13T08:37:11.328520+0000","last_became_active":"2024-09-13T08:37:09.014454+0000","last_became_peered":"2024-09-13T08:37:09.014454+0000","last_unstale":"2024-09-13T08:37:11.328520+0000","last_undegraded":"2024-09-13T08:37:11.328520+0000","last_fullsized":"2024-09-13T08:37:11.328520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016091+0000","last_change":"2024-09-13T08:37:09.024672+0000","last_active":"2024-09-13T08:37:10.016091+0000","last_peered":"2024-09-13T08:37:10.016091+0000","last_clean":"2024-09-13T08:37:10.016091+0000","last_became_active":"2024-09-13T08:37:09.024363+0000","last_became_peered":"2024-09-13T08:37:09.024363+0000","last_unstale":"2024-09-13T08:37:10.016091+0000","last_undegraded":"2024-09-13T08:37:10.016091+0000","last_fullsized":"2024-09-13T08:37:10.016091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:55.612953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863222+0000","last_change":"2024-09-13T08:37:09.014986+0000","last_active":"2024-09-13T08:37:12.863222+0000","last_peered":"2024-09-13T08:37:12.863222+0000","last_clean":"2024-09-13T08:37:12.863222+0000","last_became_active":"2024-09-13T08:37:09.014658+0000","last_became_peered":"2024-09-13T08:37:09.014658+0000","last_unstale":"2024-09-13T08:37:12.863222+0000","last_undegraded":"2024-09-13T08:37:12.863222+0000","last_fullsized":"2024-09-13T08:37:12.863222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328628+0000","last_change":"2024-09-13T08:37:10.030653+0000","last_active":"2024-09-13T08:37:11.328628+0000","last_peered":"2024-09-13T08:37:11.328628+0000","last_clean":"2024-09-13T08:37:11.328628+0000","last_became_active":"2024-09-13T08:37:10.030383+0000","last_became_peered":"2024-09-13T08:37:10.030383+0000","last_unstale":"2024-09-13T08:37:11.328628+0000","last_undegraded":"2024-09-13T08:37:11.328628+0000","last_fullsized":"2024-09-13T08:37:11.328628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.010377+0000","last_change":"2024-09-13T08:37:09.023571+0000","last_active":"2024-09-13T08:37:10.010377+0000","last_peered":"2024-09-13T08:37:10.010377+0000","last_clean":"2024-09-13T08:37:10.010377+0000","last_became_active":"2024-09-13T08:37:09.023048+0000","last_became_peered":"2024-09-13T08:37:09.023048+0000","last_unstale":"2024-09-13T08:37:10.010377+0000","last_undegraded":"2024-09-13T08:37:10.010377+0000","last_fullsized":"2024-09-13T08:37:10.010377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:14.567695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723890+0000","last_change":"2024-09-13T08:37:09.010512+0000","last_active":"2024-09-13T08:37:12.723890+0000","last_peered":"2024-09-13T08:37:12.723890+0000","last_clean":"2024-09-13T08:37:12.723890+0000","last_became_active":"2024-09-13T08:37:09.010229+0000","last_became_peered":"2024-09-13T08:37:09.010229+0000","last_unstale":"2024-09-13T08:37:12.723890+0000","last_undegraded":"2024-09-13T08:37:12.723890+0000","last_fullsized":"2024-09-13T08:37:12.723890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015969+0000","last_change":"2024-09-13T08:37:10.020061+0000","last_active":"2024-09-13T08:37:11.015969+0000","last_peered":"2024-09-13T08:37:11.015969+0000","last_clean":"2024-09-13T08:37:11.015969+0000","last_became_active":"2024-09-13T08:37:10.019663+0000","last_became_peered":"2024-09-13T08:37:10.019663+0000","last_unstale":"2024-09-13T08:37:11.015969+0000","last_undegraded":"2024-09-13T08:37:11.015969+0000","last_fullsized":"2024-09-13T08:37:11.015969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:48.021040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021832+0000","last_change":"2024-09-13T08:37:09.011936+0000","last_active":"2024-09-13T08:37:11.021832+0000","last_peered":"2024-09-13T08:37:11.021832+0000","last_clean":"2024-09-13T08:37:11.021832+0000","last_became_active":"2024-09-13T08:37:09.011670+0000","last_became_peered":"2024-09-13T08:37:09.011670+0000","last_unstale":"2024-09-13T08:37:11.021832+0000","last_undegraded":"2024-09-13T08:37:11.021832+0000","last_fullsized":"2024-09-13T08:37:11.021832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:31.341815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329184+0000","last_change":"2024-09-13T08:37:10.027709+0000","last_active":"2024-09-13T08:37:11.329184+0000","last_peered":"2024-09-13T08:37:11.329184+0000","last_clean":"2024-09-13T08:37:11.329184+0000","last_became_active":"2024-09-13T08:37:10.027410+0000","last_became_peered":"2024-09-13T08:37:10.027410+0000","last_unstale":"2024-09-13T08:37:11.329184+0000","last_undegraded":"2024-09-13T08:37:11.329184+0000","last_fullsized":"2024-09-13T08:37:11.329184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:25.100540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017898+0000","last_change":"2024-09-13T08:37:09.003083+0000","last_active":"2024-09-13T08:37:11.017898+0000","last_peered":"2024-09-13T08:37:11.017898+0000","last_clean":"2024-09-13T08:37:11.017898+0000","last_became_active":"2024-09-13T08:37:09.002697+0000","last_became_peered":"2024-09-13T08:37:09.002697+0000","last_unstale":"2024-09-13T08:37:11.017898+0000","last_undegraded":"2024-09-13T08:37:11.017898+0000","last_fullsized":"2024-09-13T08:37:11.017898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:56.231941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.025144+0000","last_change":"2024-09-13T08:37:10.021360+0000","last_active":"2024-09-13T08:37:10.025144+0000","last_peered":"2024-09-13T08:37:10.025144+0000","last_clean":"2024-09-13T08:37:10.025144+0000","last_became_active":"2024-09-13T08:37:10.021080+0000","last_became_peered":"2024-09-13T08:37:10.021080+0000","last_unstale":"2024-09-13T08:37:10.025144+0000","last_undegraded":"2024-09-13T08:37:10.025144+0000","last_fullsized":"2024-09-13T08:37:10.025144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015415+0000","last_change":"2024-09-13T08:37:09.014879+0000","last_active":"2024-09-13T08:37:11.015415+0000","last_peered":"2024-09-13T08:37:11.015415+0000","last_clean":"2024-09-13T08:37:11.015415+0000","last_became_active":"2024-09-13T08:37:09.014572+0000","last_became_peered":"2024-09-13T08:37:09.014572+0000","last_unstale":"2024-09-13T08:37:11.015415+0000","last_undegraded":"2024-09-13T08:37:11.015415+0000","last_fullsized":"2024-09-13T08:37:11.015415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:14.033544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022331+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:11.022331+0000","last_peered":"2024-09-13T08:37:11.022331+0000","last_clean":"2024-09-13T08:37:11.022331+0000","last_became_active":"2024-09-13T08:37:10.026781+0000","last_became_peered":"2024-09-13T08:37:10.026781+0000","last_unstale":"2024-09-13T08:37:11.022331+0000","last_undegraded":"2024-09-13T08:37:11.022331+0000","last_fullsized":"2024-09-13T08:37:11.022331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017361+0000","last_change":"2024-09-13T08:37:09.015199+0000","last_active":"2024-09-13T08:37:11.017361+0000","last_peered":"2024-09-13T08:37:11.017361+0000","last_clean":"2024-09-13T08:37:11.017361+0000","last_became_active":"2024-09-13T08:37:09.014848+0000","last_became_peered":"2024-09-13T08:37:09.014848+0000","last_unstale":"2024-09-13T08:37:11.017361+0000","last_undegraded":"2024-09-13T08:37:11.017361+0000","last_fullsized":"2024-09-13T08:37:11.017361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:01.967164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863695+0000","last_change":"2024-09-13T08:37:10.027034+0000","last_active":"2024-09-13T08:37:12.863695+0000","last_peered":"2024-09-13T08:37:12.863695+0000","last_clean":"2024-09-13T08:37:12.863695+0000","last_became_active":"2024-09-13T08:37:10.026723+0000","last_became_peered":"2024-09-13T08:37:10.026723+0000","last_unstale":"2024-09-13T08:37:12.863695+0000","last_undegraded":"2024-09-13T08:37:12.863695+0000","last_fullsized":"2024-09-13T08:37:12.863695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016198+0000","last_change":"2024-09-13T08:37:09.021683+0000","last_active":"2024-09-13T08:37:10.016198+0000","last_peered":"2024-09-13T08:37:10.016198+0000","last_clean":"2024-09-13T08:37:10.016198+0000","last_became_active":"2024-09-13T08:37:09.019630+0000","last_became_peered":"2024-09-13T08:37:09.019630+0000","last_unstale":"2024-09-13T08:37:10.016198+0000","last_undegraded":"2024-09-13T08:37:10.016198+0000","last_fullsized":"2024-09-13T08:37:10.016198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:47.015443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039364+0000","last_change":"2024-09-13T08:37:10.032446+0000","last_active":"2024-09-13T08:37:10.039364+0000","last_peered":"2024-09-13T08:37:10.039364+0000","last_clean":"2024-09-13T08:37:10.039364+0000","last_became_active":"2024-09-13T08:37:10.031448+0000","last_became_peered":"2024-09-13T08:37:10.031448+0000","last_unstale":"2024-09-13T08:37:10.039364+0000","last_undegraded":"2024-09-13T08:37:10.039364+0000","last_fullsized":"2024-09-13T08:37:10.039364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021502+0000","last_change":"2024-09-13T08:37:09.025565+0000","last_active":"2024-09-13T08:37:11.021502+0000","last_peered":"2024-09-13T08:37:11.021502+0000","last_clean":"2024-09-13T08:37:11.021502+0000","last_became_active":"2024-09-13T08:37:09.025242+0000","last_became_peered":"2024-09-13T08:37:09.025242+0000","last_unstale":"2024-09-13T08:37:11.021502+0000","last_undegraded":"2024-09-13T08:37:11.021502+0000","last_fullsized":"2024-09-13T08:37:11.021502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:08.638798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723679+0000","last_change":"2024-09-13T08:37:10.030817+0000","last_active":"2024-09-13T08:37:12.723679+0000","last_peered":"2024-09-13T08:37:12.723679+0000","last_clean":"2024-09-13T08:37:12.723679+0000","last_became_active":"2024-09-13T08:37:10.030554+0000","last_became_peered":"2024-09-13T08:37:10.030554+0000","last_unstale":"2024-09-13T08:37:12.723679+0000","last_undegraded":"2024-09-13T08:37:12.723679+0000","last_fullsized":"2024-09-13T08:37:12.723679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.027753+0000","last_change":"2024-09-13T08:37:10.023766+0000","last_active":"2024-09-13T08:37:10.027753+0000","last_peered":"2024-09-13T08:37:10.027753+0000","last_clean":"2024-09-13T08:37:10.027753+0000","last_became_active":"2024-09-13T08:37:10.022889+0000","last_became_peered":"2024-09-13T08:37:10.022889+0000","last_unstale":"2024-09-13T08:37:10.027753+0000","last_undegraded":"2024-09-13T08:37:10.027753+0000","last_fullsized":"2024-09-13T08:37:10.027753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:50.095913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022117+0000","last_change":"2024-09-13T08:37:09.023974+0000","last_active":"2024-09-13T08:37:11.022117+0000","last_peered":"2024-09-13T08:37:11.022117+0000","last_clean":"2024-09-13T08:37:11.022117+0000","last_became_active":"2024-09-13T08:37:09.023471+0000","last_became_peered":"2024-09-13T08:37:09.023471+0000","last_unstale":"2024-09-13T08:37:11.022117+0000","last_undegraded":"2024-09-13T08:37:11.022117+0000","last_fullsized":"2024-09-13T08:37:11.022117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:38.190525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016062+0000","last_change":"2024-09-13T08:37:10.023008+0000","last_active":"2024-09-13T08:37:11.016062+0000","last_peered":"2024-09-13T08:37:11.016062+0000","last_clean":"2024-09-13T08:37:11.016062+0000","last_became_active":"2024-09-13T08:37:10.021695+0000","last_became_peered":"2024-09-13T08:37:10.021695+0000","last_unstale":"2024-09-13T08:37:11.016062+0000","last_undegraded":"2024-09-13T08:37:11.016062+0000","last_fullsized":"2024-09-13T08:37:11.016062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:43.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022182+0000","last_change":"2024-09-13T08:37:10.031929+0000","last_active":"2024-09-13T08:37:11.022182+0000","last_peered":"2024-09-13T08:37:11.022182+0000","last_clean":"2024-09-13T08:37:11.022182+0000","last_became_active":"2024-09-13T08:37:10.031598+0000","last_became_peered":"2024-09-13T08:37:10.031598+0000","last_unstale":"2024-09-13T08:37:11.022182+0000","last_undegraded":"2024-09-13T08:37:11.022182+0000","last_fullsized":"2024-09-13T08:37:11.022182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328623+0000","last_change":"2024-09-13T08:37:09.011613+0000","last_active":"2024-09-13T08:37:11.328623+0000","last_peered":"2024-09-13T08:37:11.328623+0000","last_clean":"2024-09-13T08:37:11.328623+0000","last_became_active":"2024-09-13T08:37:09.011071+0000","last_became_peered":"2024-09-13T08:37:09.011071+0000","last_unstale":"2024-09-13T08:37:11.328623+0000","last_undegraded":"2024-09-13T08:37:11.328623+0000","last_fullsized":"2024-09-13T08:37:11.328623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022072+0000","last_change":"2024-09-13T08:37:10.028420+0000","last_active":"2024-09-13T08:37:11.022072+0000","last_peered":"2024-09-13T08:37:11.022072+0000","last_clean":"2024-09-13T08:37:11.022072+0000","last_became_active":"2024-09-13T08:37:10.028166+0000","last_became_peered":"2024-09-13T08:37:10.028166+0000","last_unstale":"2024-09-13T08:37:11.022072+0000","last_undegraded":"2024-09-13T08:37:11.022072+0000","last_fullsized":"2024-09-13T08:37:11.022072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:58.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863837+0000","last_change":"2024-09-13T08:37:09.015072+0000","last_active":"2024-09-13T08:37:12.863837+0000","last_peered":"2024-09-13T08:37:12.863837+0000","last_clean":"2024-09-13T08:37:12.863837+0000","last_became_active":"2024-09-13T08:37:09.014597+0000","last_became_peered":"2024-09-13T08:37:09.014597+0000","last_unstale":"2024-09-13T08:37:12.863837+0000","last_undegraded":"2024-09-13T08:37:12.863837+0000","last_fullsized":"2024-09-13T08:37:12.863837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.022427+0000","last_change":"2024-09-13T08:37:10.018219+0000","last_active":"2024-09-13T08:37:10.022427+0000","last_peered":"2024-09-13T08:37:10.022427+0000","last_clean":"2024-09-13T08:37:10.022427+0000","last_became_active":"2024-09-13T08:37:10.017946+0000","last_became_peered":"2024-09-13T08:37:10.017946+0000","last_unstale":"2024-09-13T08:37:10.022427+0000","last_undegraded":"2024-09-13T08:37:10.022427+0000","last_fullsized":"2024-09-13T08:37:10.022427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723548+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:12.723548+0000","last_peered":"2024-09-13T08:37:12.723548+0000","last_clean":"2024-09-13T08:37:12.723548+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:12.723548+0000","last_undegraded":"2024-09-13T08:37:12.723548+0000","last_fullsized":"2024-09-13T08:37:12.723548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.013248+0000","last_change":"2024-09-13T08:37:09.021195+0000","last_active":"2024-09-13T08:37:10.013248+0000","last_peered":"2024-09-13T08:37:10.013248+0000","last_clean":"2024-09-13T08:37:10.013248+0000","last_became_active":"2024-09-13T08:37:09.019692+0000","last_became_peered":"2024-09-13T08:37:09.019692+0000","last_unstale":"2024-09-13T08:37:10.013248+0000","last_undegraded":"2024-09-13T08:37:10.013248+0000","last_fullsized":"2024-09-13T08:37:10.013248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.430584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.010874+0000","last_change":"2024-09-13T08:37:09.019756+0000","last_active":"2024-09-13T08:37:10.010874+0000","last_peered":"2024-09-13T08:37:10.010874+0000","last_clean":"2024-09-13T08:37:10.010874+0000","last_became_active":"2024-09-13T08:37:09.019375+0000","last_became_peered":"2024-09-13T08:37:09.019375+0000","last_unstale":"2024-09-13T08:37:10.010874+0000","last_undegraded":"2024-09-13T08:37:10.010874+0000","last_fullsized":"2024-09-13T08:37:10.010874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:40.495518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863575+0000","last_change":"2024-09-13T08:37:10.020491+0000","last_active":"2024-09-13T08:37:12.863575+0000","last_peered":"2024-09-13T08:37:12.863575+0000","last_clean":"2024-09-13T08:37:12.863575+0000","last_became_active":"2024-09-13T08:37:10.020271+0000","last_became_peered":"2024-09-13T08:37:10.020271+0000","last_unstale":"2024-09-13T08:37:12.863575+0000","last_undegraded":"2024-09-13T08:37:12.863575+0000","last_fullsized":"2024-09-13T08:37:12.863575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:29.393985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016620+0000","last_change":"2024-09-13T08:37:09.015889+0000","last_active":"2024-09-13T08:37:11.016620+0000","last_peered":"2024-09-13T08:37:11.016620+0000","last_clean":"2024-09-13T08:37:11.016620+0000","last_became_active":"2024-09-13T08:37:09.015587+0000","last_became_peered":"2024-09-13T08:37:09.015587+0000","last_unstale":"2024-09-13T08:37:11.016620+0000","last_undegraded":"2024-09-13T08:37:11.016620+0000","last_fullsized":"2024-09-13T08:37:11.016620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328457+0000","last_change":"2024-09-13T08:37:10.036912+0000","last_active":"2024-09-13T08:37:11.328457+0000","last_peered":"2024-09-13T08:37:11.328457+0000","last_clean":"2024-09-13T08:37:11.328457+0000","last_became_active":"2024-09-13T08:37:10.036665+0000","last_became_peered":"2024-09-13T08:37:10.036665+0000","last_unstale":"2024-09-13T08:37:11.328457+0000","last_undegraded":"2024-09-13T08:37:11.328457+0000","last_fullsized":"2024-09-13T08:37:11.328457+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016380+0000","last_change":"2024-09-13T08:37:09.023866+0000","last_active":"2024-09-13T08:37:10.016380+0000","last_peered":"2024-09-13T08:37:10.016380+0000","last_clean":"2024-09-13T08:37:10.016380+0000","last_became_active":"2024-09-13T08:37:09.021825+0000","last_became_peered":"2024-09-13T08:37:09.021825+0000","last_unstale":"2024-09-13T08:37:10.016380+0000","last_undegraded":"2024-09-13T08:37:10.016380+0000","last_fullsized":"2024-09-13T08:37:10.016380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:42.779836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022032+0000","last_change":"2024-09-13T08:37:10.027315+0000","last_active":"2024-09-13T08:37:11.022032+0000","last_peered":"2024-09-13T08:37:11.022032+0000","last_clean":"2024-09-13T08:37:11.022032+0000","last_became_active":"2024-09-13T08:37:10.027018+0000","last_became_peered":"2024-09-13T08:37:10.027018+0000","last_unstale":"2024-09-13T08:37:11.022032+0000","last_undegraded":"2024-09-13T08:37:11.022032+0000","last_fullsized":"2024-09-13T08:37:11.022032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328769+0000","last_change":"2024-09-13T08:37:09.018684+0000","last_active":"2024-09-13T08:37:11.328769+0000","last_peered":"2024-09-13T08:37:11.328769+0000","last_clean":"2024-09-13T08:37:11.328769+0000","last_became_active":"2024-09-13T08:37:09.016570+0000","last_became_peered":"2024-09-13T08:37:09.016570+0000","last_unstale":"2024-09-13T08:37:11.328769+0000","last_undegraded":"2024-09-13T08:37:11.328769+0000","last_fullsized":"2024-09-13T08:37:11.328769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015606+0000","last_change":"2024-09-13T08:37:10.018230+0000","last_active":"2024-09-13T08:37:11.015606+0000","last_peered":"2024-09-13T08:37:11.015606+0000","last_clean":"2024-09-13T08:37:11.015606+0000","last_became_active":"2024-09-13T08:37:10.017427+0000","last_became_peered":"2024-09-13T08:37:10.017427+0000","last_unstale":"2024-09-13T08:37:11.015606+0000","last_undegraded":"2024-09-13T08:37:11.015606+0000","last_fullsized":"2024-09-13T08:37:11.015606+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.015515+0000","last_change":"2024-09-13T08:37:09.018807+0000","last_active":"2024-09-13T08:37:10.015515+0000","last_peered":"2024-09-13T08:37:10.015515+0000","last_clean":"2024-09-13T08:37:10.015515+0000","last_became_active":"2024-09-13T08:37:09.018298+0000","last_became_peered":"2024-09-13T08:37:09.018298+0000","last_unstale":"2024-09-13T08:37:10.015515+0000","last_undegraded":"2024-09-13T08:37:10.015515+0000","last_fullsized":"2024-09-13T08:37:10.015515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.295316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863880+0000","last_change":"2024-09-13T08:37:10.028730+0000","last_active":"2024-09-13T08:37:12.863880+0000","last_peered":"2024-09-13T08:37:12.863880+0000","last_clean":"2024-09-13T08:37:12.863880+0000","last_became_active":"2024-09-13T08:37:10.028481+0000","last_became_peered":"2024-09-13T08:37:10.028481+0000","last_unstale":"2024-09-13T08:37:12.863880+0000","last_undegraded":"2024-09-13T08:37:12.863880+0000","last_fullsized":"2024-09-13T08:37:12.863880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.044490+0000","last_change":"2024-09-13T08:37:09.024256+0000","last_active":"2024-09-13T08:37:10.044490+0000","last_peered":"2024-09-13T08:37:10.044490+0000","last_clean":"2024-09-13T08:37:10.044490+0000","last_became_active":"2024-09-13T08:37:09.023967+0000","last_became_peered":"2024-09-13T08:37:09.023967+0000","last_unstale":"2024-09-13T08:37:10.044490+0000","last_undegraded":"2024-09-13T08:37:10.044490+0000","last_fullsized":"2024-09-13T08:37:10.044490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:41.577735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021357+0000","last_change":"2024-09-13T08:37:09.011069+0000","last_active":"2024-09-13T08:37:11.021357+0000","last_peered":"2024-09-13T08:37:11.021357+0000","last_clean":"2024-09-13T08:37:11.021357+0000","last_became_active":"2024-09-13T08:37:09.010687+0000","last_became_peered":"2024-09-13T08:37:09.010687+0000","last_unstale":"2024-09-13T08:37:11.021357+0000","last_undegraded":"2024-09-13T08:37:11.021357+0000","last_fullsized":"2024-09-13T08:37:11.021357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034067+0000","last_change":"2024-09-13T08:37:10.026288+0000","last_active":"2024-09-13T08:37:10.034067+0000","last_peered":"2024-09-13T08:37:10.034067+0000","last_clean":"2024-09-13T08:37:10.034067+0000","last_became_active":"2024-09-13T08:37:10.026029+0000","last_became_peered":"2024-09-13T08:37:10.026029+0000","last_unstale":"2024-09-13T08:37:10.034067+0000","last_undegraded":"2024-09-13T08:37:10.034067+0000","last_fullsized":"2024-09-13T08:37:10.034067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863577+0000","last_change":"2024-09-13T08:37:09.019714+0000","last_active":"2024-09-13T08:37:12.863577+0000","last_peered":"2024-09-13T08:37:12.863577+0000","last_clean":"2024-09-13T08:37:12.863577+0000","last_became_active":"2024-09-13T08:37:09.018957+0000","last_became_peered":"2024-09-13T08:37:09.018957+0000","last_unstale":"2024-09-13T08:37:12.863577+0000","last_undegraded":"2024-09-13T08:37:12.863577+0000","last_fullsized":"2024-09-13T08:37:12.863577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017312+0000","last_change":"2024-09-13T08:37:10.026722+0000","last_active":"2024-09-13T08:37:11.017312+0000","last_peered":"2024-09-13T08:37:11.017312+0000","last_clean":"2024-09-13T08:37:11.017312+0000","last_became_active":"2024-09-13T08:37:10.025267+0000","last_became_peered":"2024-09-13T08:37:10.025267+0000","last_unstale":"2024-09-13T08:37:11.017312+0000","last_undegraded":"2024-09-13T08:37:11.017312+0000","last_fullsized":"2024-09-13T08:37:11.017312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039159+0000","last_change":"2024-09-13T08:37:09.022982+0000","last_active":"2024-09-13T08:37:10.039159+0000","last_peered":"2024-09-13T08:37:10.039159+0000","last_clean":"2024-09-13T08:37:10.039159+0000","last_became_active":"2024-09-13T08:37:09.021746+0000","last_became_peered":"2024-09-13T08:37:09.021746+0000","last_unstale":"2024-09-13T08:37:10.039159+0000","last_undegraded":"2024-09-13T08:37:10.039159+0000","last_fullsized":"2024-09-13T08:37:10.039159+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.468482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.025322+0000","last_change":"2024-09-13T08:37:10.016999+0000","last_active":"2024-09-13T08:37:10.025322+0000","last_peered":"2024-09-13T08:37:10.025322+0000","last_clean":"2024-09-13T08:37:10.025322+0000","last_became_active":"2024-09-13T08:37:10.016738+0000","last_became_peered":"2024-09-13T08:37:10.016738+0000","last_unstale":"2024-09-13T08:37:10.025322+0000","last_undegraded":"2024-09-13T08:37:10.025322+0000","last_fullsized":"2024-09-13T08:37:10.025322+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034085+0000","last_change":"2024-09-13T08:37:10.026898+0000","last_active":"2024-09-13T08:37:10.034085+0000","last_peered":"2024-09-13T08:37:10.034085+0000","last_clean":"2024-09-13T08:37:10.034085+0000","last_became_active":"2024-09-13T08:37:10.026602+0000","last_became_peered":"2024-09-13T08:37:10.026602+0000","last_unstale":"2024-09-13T08:37:10.034085+0000","last_undegraded":"2024-09-13T08:37:10.034085+0000","last_fullsized":"2024-09-13T08:37:10.034085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:27.242730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022006+0000","last_change":"2024-09-13T08:37:09.011138+0000","last_active":"2024-09-13T08:37:11.022006+0000","last_peered":"2024-09-13T08:37:11.022006+0000","last_clean":"2024-09-13T08:37:11.022006+0000","last_became_active":"2024-09-13T08:37:09.010863+0000","last_became_peered":"2024-09-13T08:37:09.010863+0000","last_unstale":"2024-09-13T08:37:11.022006+0000","last_undegraded":"2024-09-13T08:37:11.022006+0000","last_fullsized":"2024-09-13T08:37:11.022006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:14.227052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723804+0000","last_change":"2024-09-13T08:37:10.029092+0000","last_active":"2024-09-13T08:37:12.723804+0000","last_peered":"2024-09-13T08:37:12.723804+0000","last_clean":"2024-09-13T08:37:12.723804+0000","last_became_active":"2024-09-13T08:37:10.028796+0000","last_became_peered":"2024-09-13T08:37:10.028796+0000","last_unstale":"2024-09-13T08:37:12.723804+0000","last_undegraded":"2024-09-13T08:37:12.723804+0000","last_fullsized":"2024-09-13T08:37:12.723804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022178+0000","last_change":"2024-09-13T08:37:09.013579+0000","last_active":"2024-09-13T08:37:11.022178+0000","last_peered":"2024-09-13T08:37:11.022178+0000","last_clean":"2024-09-13T08:37:11.022178+0000","last_became_active":"2024-09-13T08:37:09.013327+0000","last_became_peered":"2024-09-13T08:37:09.013327+0000","last_unstale":"2024-09-13T08:37:11.022178+0000","last_undegraded":"2024-09-13T08:37:11.022178+0000","last_fullsized":"2024-09-13T08:37:11.022178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:34.207907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328992+0000","last_change":"2024-09-13T08:37:10.032358+0000","last_active":"2024-09-13T08:37:11.328992+0000","last_peered":"2024-09-13T08:37:11.328992+0000","last_clean":"2024-09-13T08:37:11.328992+0000","last_became_active":"2024-09-13T08:37:10.032047+0000","last_became_peered":"2024-09-13T08:37:10.032047+0000","last_unstale":"2024-09-13T08:37:11.328992+0000","last_undegraded":"2024-09-13T08:37:11.328992+0000","last_fullsized":"2024-09-13T08:37:11.328992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:19.283020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329066+0000","last_change":"2024-09-13T08:37:09.015906+0000","last_active":"2024-09-13T08:37:11.329066+0000","last_peered":"2024-09-13T08:37:11.329066+0000","last_clean":"2024-09-13T08:37:11.329066+0000","last_became_active":"2024-09-13T08:37:09.015500+0000","last_became_peered":"2024-09-13T08:37:09.015500+0000","last_unstale":"2024-09-13T08:37:11.329066+0000","last_undegraded":"2024-09-13T08:37:11.329066+0000","last_fullsized":"2024-09-13T08:37:11.329066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:51.846602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329023+0000","last_change":"2024-09-13T08:37:10.043965+0000","last_active":"2024-09-13T08:37:11.329023+0000","last_peered":"2024-09-13T08:37:11.329023+0000","last_clean":"2024-09-13T08:37:11.329023+0000","last_became_active":"2024-09-13T08:37:10.041509+0000","last_became_peered":"2024-09-13T08:37:10.041509+0000","last_unstale":"2024-09-13T08:37:11.329023+0000","last_undegraded":"2024-09-13T08:37:11.329023+0000","last_fullsized":"2024-09-13T08:37:11.329023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":241664,"data_stored":81079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":221184,"data_stored":74655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":835584,"data_stored":671415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":75237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":245760,"data_stored":82567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":671393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":81011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:15.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 -- 172.21.15.31:0/3288213781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 msgr2=0x7fd554004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 --2- 172.21.15.31:0/3288213781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd554004060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd560006e50 tx=0x7fd554005830 comp rx=0 tx=0).stop 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd56f400640 1 -- 172.21.15.31:0/3288213781 reap_dead start 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 -- 172.21.15.31:0/3288213781 shutdown_connections 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 -- 172.21.15.31:0/3288213781 >> 172.21.15.31:0/3288213781 conn(0x7fd5700cc910 msgr2=0x7fd5700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 -- 172.21.15.31:0/3288213781 shutdown_connections 2024-09-13T08:37:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.313+0000 7fd575e00640 1 -- 172.21.15.31:0/3288213781 wait complete. 2024-09-13T08:37:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.314+0000 7fd575e00640 1 Processor -- start 2024-09-13T08:37:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.314+0000 7fd575e00640 1 -- start start 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd57014e660 0x7fd57009cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd57009d120 0x7fd5701f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701600f0 con 0x7fd554003c60 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700cb850 con 0x7fd57009d120 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.315+0000 7fd575e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5701f2fa0 con 0x7fd57014e660 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34398/0 (socket says 172.21.15.31:34398) 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 -- 172.21.15.31:0/4091002518 learned_addr learned my addr 172.21.15.31:0/4091002518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56f400640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd57009d120 0x7fd5701f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56e000640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd57014e660 0x7fd57009cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 -- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd57014e660 msgr2=0x7fd57009cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd57014e660 0x7fd57009cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 -- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd57009d120 msgr2=0x7fd5701f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd57009d120 0x7fd5701f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 -- 172.21.15.31:0/4091002518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57014db80 con 0x7fd554003c60 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.316+0000 7fd56ea00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd560031040 tx=0x7fd560006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.317+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600037f0 con 0x7fd554003c60 2024-09-13T08:37:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.317+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd56003f070 con 0x7fd554003c60 2024-09-13T08:37:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.317+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560003ce0 con 0x7fd554003c60 2024-09-13T08:37:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.317+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd57014de60 con 0x7fd554003c60 2024-09-13T08:37:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.317+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd57014b4f0 con 0x7fd554003c60 2024-09-13T08:37:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.318+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd57014d970 con 0x7fd554003c60 2024-09-13T08:37:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.319+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd560044470 con 0x7fd554003c60 2024-09-13T08:37:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.320+0000 7fd567400640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 0x7fd55007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.320+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fd5600bb700 con 0x7fd554003c60 2024-09-13T08:37:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.320+0000 7fd56e000640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 0x7fd55007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.321+0000 7fd56e000640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 0x7fd55007ad10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd57009dbc0 tx=0x7fd55c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:15.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.324+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd560083680 con 0x7fd554003c60 2024-09-13T08:37:15.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.592+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fd5700cc490 con 0x7fd554003c60 2024-09-13T08:37:15.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.592+0000 7fd567400640 1 -- 172.21.15.31:0/4091002518 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1214 (secure 0 0 0) 0x7fd5600880b0 con 0x7fd554003c60 2024-09-13T08:37:15.593 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:15.594 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"0251b593-a02a-47fa-b938-c065b7284822","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":31,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1726216617,"num_in_osds":8,"osd_in_since":1726216607,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":57},{"state_name":"creating+peering","count":27},{"state_name":"active+clean","count":13}],"num_pgs":97,"num_pools":3,"num_objects":10,"data_bytes":590950,"bytes_used":241725440,"bytes_avail":772852387840,"bytes_total":773094113280,"unknown_pgs_ratio":0.58762884140014648,"inactive_pgs_ratio":0.27835050225257874,"write_bytes_sec":202,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":4366}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-13T08:36:43.730982+0000","services":{}},"progress_events":{}} 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 msgr2=0x7fd55007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 0x7fd55007ad10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd57009dbc0 tx=0x7fd55c005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 msgr2=0x7fd57014afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd560031040 tx=0x7fd560006310 comp rx=0 tx=0).stop 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 shutdown_connections 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd57009d120 0x7fd5701f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd550078850 0x7fd55007ad10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd57014e660 0x7fd57009cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 --2- 172.21.15.31:0/4091002518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd554003c60 0x7fd57014afb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 >> 172.21.15.31:0/4091002518 conn(0x7fd5700cc910 msgr2=0x7fd5700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:15.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 shutdown_connections 2024-09-13T08:37:15.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.595+0000 7fd575e00640 1 -- 172.21.15.31:0/4091002518 wait complete. 2024-09-13T08:37:15.611 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T08:37:15.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 -- 172.21.15.31:0/3209245031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600ccb00 msgr2=0x7f22600b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 --2- 172.21.15.31:0/3209245031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600ccb00 0x7f22600b4bc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f224c009830 tx=0x7f224c02a510 comp rx=0 tx=0).stop 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 -- 172.21.15.31:0/3209245031 shutdown_connections 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 --2- 172.21.15.31:0/3209245031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600ccb00 0x7f22600b4bc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 --2- 172.21.15.31:0/3209245031 >> v2:172.21.15.45:3300/0 conn(0x7f22600cc720 0x7f226006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 -- 172.21.15.31:0/3209245031 >> 172.21.15.31:0/3209245031 conn(0x7f226006cb00 msgr2=0x7f226006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 -- 172.21.15.31:0/3209245031 shutdown_connections 2024-09-13T08:37:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.772+0000 7f2266e00640 1 -- 172.21.15.31:0/3209245031 wait complete. 2024-09-13T08:37:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.773+0000 7f2266e00640 1 Processor -- start 2024-09-13T08:37:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.774+0000 7f2266e00640 1 -- start start 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.774+0000 7f2266e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600701a0 0x7f22600c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.774+0000 7f2266e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2266e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22600ccb00 0x7f22600b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2266e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22600b5490 con 0x7f22600cc720 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2266e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f22600b5310 con 0x7f22600ccb00 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2266e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2260152d90 con 0x7f22600701a0 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34406/0 (socket says 172.21.15.31:34406) 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 -- 172.21.15.31:0/681792911 learned_addr learned my addr 172.21.15.31:0/681792911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265a00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600701a0 0x7f22600c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2266400640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22600ccb00 0x7f22600b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 -- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600701a0 msgr2=0x7f22600c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600701a0 0x7f22600c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 -- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22600ccb00 msgr2=0x7f22600b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22600ccb00 0x7f22600b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 -- 172.21.15.31:0/681792911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226014d760 con 0x7f22600cc720 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.775+0000 7f2265000640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f224c000c00 tx=0x7f224c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.776+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c005c90 con 0x7f22600cc720 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.776+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f224c031e50 con 0x7f22600cc720 2024-09-13T08:37:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.776+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c033290 con 0x7f22600cc720 2024-09-13T08:37:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.776+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22600c7890 con 0x7f22600cc720 2024-09-13T08:37:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.776+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22600c8fb0 con 0x7f22600cc720 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.777+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2260153260 con 0x7f22600cc720 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.778+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f224c033430 con 0x7f22600cc720 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.779+0000 7f225d600640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 0x7f224007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.779+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f224c0bba20 con 0x7f22600cc720 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.779+0000 7f2265a00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 0x7f224007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.780+0000 7f2265a00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 0x7f224007ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2254009bd0 tx=0x7f2254005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.783+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f224c03f030 con 0x7f22600cc720 2024-09-13T08:37:15.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.976+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2260152b00 con 0x7f2240078880 2024-09-13T08:37:15.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.981+0000 7f225d600640 1 -- 172.21.15.31:0/681792911 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251932 (secure 0 0 0) 0x7f2260152b00 con 0x7f2240078880 2024-09-13T08:37:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 msgr2=0x7f224007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 0x7f224007ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2254009bd0 tx=0x7f2254005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 msgr2=0x7f2260146c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f224c000c00 tx=0x7f224c033c20 comp rx=0 tx=0).stop 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 shutdown_connections 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22600ccb00 0x7f22600b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22600cc720 0x7f2260146c80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2240078880 0x7f224007ad40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 --2- 172.21.15.31:0/681792911 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f22600701a0 0x7f22600c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 >> 172.21.15.31:0/681792911 conn(0x7f226006cb00 msgr2=0x7f22600c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 shutdown_connections 2024-09-13T08:37:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:15.985+0000 7f2266e00640 1 -- 172.21.15.31:0/681792911 wait complete. 2024-09-13T08:37:16.002 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-13T08:37:15.037781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235920,"kb_used_data":3024,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738800,"statfs":{"total":773094113280,"available":772852531200,"internally_reserved":0,"allocated":3096576,"data_stored":1820596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.047925"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021976+0000","last_change":"2024-09-13T08:37:10.038242+0000","last_active":"2024-09-13T08:37:11.021976+0000","last_peered":"2024-09-13T08:37:11.021976+0000","last_clean":"2024-09-13T08:37:11.021976+0000","last_became_active":"2024-09-13T08:37:10.037984+0000","last_became_peered":"2024-09-13T08:37:10.037984+0000","last_unstale":"2024-09-13T08:37:11.021976+0000","last_undegraded":"2024-09-13T08:37:11.021976+0000","last_fullsized":"2024-09-13T08:37:11.021976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:05.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328871+0000","last_change":"2024-09-13T08:37:10.043402+0000","last_active":"2024-09-13T08:37:11.328871+0000","last_peered":"2024-09-13T08:37:11.328871+0000","last_clean":"2024-09-13T08:37:11.328871+0000","last_became_active":"2024-09-13T08:37:10.041028+0000","last_became_peered":"2024-09-13T08:37:10.041028+0000","last_unstale":"2024-09-13T08:37:11.328871+0000","last_undegraded":"2024-09-13T08:37:11.328871+0000","last_fullsized":"2024-09-13T08:37:11.328871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328967+0000","last_change":"2024-09-13T08:37:10.038931+0000","last_active":"2024-09-13T08:37:11.328967+0000","last_peered":"2024-09-13T08:37:11.328967+0000","last_clean":"2024-09-13T08:37:11.328967+0000","last_became_active":"2024-09-13T08:37:10.038676+0000","last_became_peered":"2024-09-13T08:37:10.038676+0000","last_unstale":"2024-09-13T08:37:11.328967+0000","last_undegraded":"2024-09-13T08:37:11.328967+0000","last_fullsized":"2024-09-13T08:37:11.328967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015339+0000","last_change":"2024-09-13T08:37:10.018106+0000","last_active":"2024-09-13T08:37:11.015339+0000","last_peered":"2024-09-13T08:37:11.015339+0000","last_clean":"2024-09-13T08:37:11.015339+0000","last_became_active":"2024-09-13T08:37:10.017812+0000","last_became_peered":"2024-09-13T08:37:10.017812+0000","last_unstale":"2024-09-13T08:37:11.015339+0000","last_undegraded":"2024-09-13T08:37:11.015339+0000","last_fullsized":"2024-09-13T08:37:11.015339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021892+0000","last_change":"2024-09-13T08:37:10.035463+0000","last_active":"2024-09-13T08:37:11.021892+0000","last_peered":"2024-09-13T08:37:11.021892+0000","last_clean":"2024-09-13T08:37:11.021892+0000","last_became_active":"2024-09-13T08:37:10.032852+0000","last_became_peered":"2024-09-13T08:37:10.032852+0000","last_unstale":"2024-09-13T08:37:11.021892+0000","last_undegraded":"2024-09-13T08:37:11.021892+0000","last_fullsized":"2024-09-13T08:37:11.021892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:34.731391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015062+0000","last_change":"2024-09-13T08:37:10.030116+0000","last_active":"2024-09-13T08:37:11.015062+0000","last_peered":"2024-09-13T08:37:11.015062+0000","last_clean":"2024-09-13T08:37:11.015062+0000","last_became_active":"2024-09-13T08:37:10.029773+0000","last_became_peered":"2024-09-13T08:37:10.029773+0000","last_unstale":"2024-09-13T08:37:11.015062+0000","last_undegraded":"2024-09-13T08:37:11.015062+0000","last_fullsized":"2024-09-13T08:37:11.015062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021640+0000","last_change":"2024-09-13T08:37:10.025604+0000","last_active":"2024-09-13T08:37:11.021640+0000","last_peered":"2024-09-13T08:37:11.021640+0000","last_clean":"2024-09-13T08:37:11.021640+0000","last_became_active":"2024-09-13T08:37:10.020828+0000","last_became_peered":"2024-09-13T08:37:10.020828+0000","last_unstale":"2024-09-13T08:37:11.021640+0000","last_undegraded":"2024-09-13T08:37:11.021640+0000","last_fullsized":"2024-09-13T08:37:11.021640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328358+0000","last_change":"2024-09-13T08:37:10.041296+0000","last_active":"2024-09-13T08:37:11.328358+0000","last_peered":"2024-09-13T08:37:11.328358+0000","last_clean":"2024-09-13T08:37:11.328358+0000","last_became_active":"2024-09-13T08:37:10.035151+0000","last_became_peered":"2024-09-13T08:37:10.035151+0000","last_unstale":"2024-09-13T08:37:11.328358+0000","last_undegraded":"2024-09-13T08:37:11.328358+0000","last_fullsized":"2024-09-13T08:37:11.328358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:28.830741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723967+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:12.723967+0000","last_peered":"2024-09-13T08:37:12.723967+0000","last_clean":"2024-09-13T08:37:12.723967+0000","last_became_active":"2024-09-13T08:37:10.026743+0000","last_became_peered":"2024-09-13T08:37:10.026743+0000","last_unstale":"2024-09-13T08:37:12.723967+0000","last_undegraded":"2024-09-13T08:37:12.723967+0000","last_fullsized":"2024-09-13T08:37:12.723967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022008+0000","last_change":"2024-09-13T08:37:10.020548+0000","last_active":"2024-09-13T08:37:11.022008+0000","last_peered":"2024-09-13T08:37:11.022008+0000","last_clean":"2024-09-13T08:37:11.022008+0000","last_became_active":"2024-09-13T08:37:10.020144+0000","last_became_peered":"2024-09-13T08:37:10.020144+0000","last_unstale":"2024-09-13T08:37:11.022008+0000","last_undegraded":"2024-09-13T08:37:11.022008+0000","last_fullsized":"2024-09-13T08:37:11.022008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:13.408879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022140+0000","last_change":"2024-09-13T08:37:10.038191+0000","last_active":"2024-09-13T08:37:11.022140+0000","last_peered":"2024-09-13T08:37:11.022140+0000","last_clean":"2024-09-13T08:37:11.022140+0000","last_became_active":"2024-09-13T08:37:10.037935+0000","last_became_peered":"2024-09-13T08:37:10.037935+0000","last_unstale":"2024-09-13T08:37:11.022140+0000","last_undegraded":"2024-09-13T08:37:11.022140+0000","last_fullsized":"2024-09-13T08:37:11.022140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021867+0000","last_change":"2024-09-13T08:37:10.025910+0000","last_active":"2024-09-13T08:37:11.021867+0000","last_peered":"2024-09-13T08:37:11.021867+0000","last_clean":"2024-09-13T08:37:11.021867+0000","last_became_active":"2024-09-13T08:37:10.023349+0000","last_became_peered":"2024-09-13T08:37:10.023349+0000","last_unstale":"2024-09-13T08:37:11.021867+0000","last_undegraded":"2024-09-13T08:37:11.021867+0000","last_fullsized":"2024-09-13T08:37:11.021867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:02.005454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863687+0000","last_change":"2024-09-13T08:37:10.033914+0000","last_active":"2024-09-13T08:37:12.863687+0000","last_peered":"2024-09-13T08:37:12.863687+0000","last_clean":"2024-09-13T08:37:12.863687+0000","last_became_active":"2024-09-13T08:37:10.030947+0000","last_became_peered":"2024-09-13T08:37:10.030947+0000","last_unstale":"2024-09-13T08:37:12.863687+0000","last_undegraded":"2024-09-13T08:37:12.863687+0000","last_fullsized":"2024-09-13T08:37:12.863687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:49.714886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863461+0000","last_change":"2024-09-13T08:37:10.024803+0000","last_active":"2024-09-13T08:37:12.863461+0000","last_peered":"2024-09-13T08:37:12.863461+0000","last_clean":"2024-09-13T08:37:12.863461+0000","last_became_active":"2024-09-13T08:37:10.024512+0000","last_became_peered":"2024-09-13T08:37:10.024512+0000","last_unstale":"2024-09-13T08:37:12.863461+0000","last_undegraded":"2024-09-13T08:37:12.863461+0000","last_fullsized":"2024-09-13T08:37:12.863461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328365+0000","last_change":"2024-09-13T08:37:10.035650+0000","last_active":"2024-09-13T08:37:11.328365+0000","last_peered":"2024-09-13T08:37:11.328365+0000","last_clean":"2024-09-13T08:37:11.328365+0000","last_became_active":"2024-09-13T08:37:10.035331+0000","last_became_peered":"2024-09-13T08:37:10.035331+0000","last_unstale":"2024-09-13T08:37:11.328365+0000","last_undegraded":"2024-09-13T08:37:11.328365+0000","last_fullsized":"2024-09-13T08:37:11.328365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.703160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.024424+0000","last_change":"2024-09-13T08:37:10.022368+0000","last_active":"2024-09-13T08:37:10.024424+0000","last_peered":"2024-09-13T08:37:10.024424+0000","last_clean":"2024-09-13T08:37:10.024424+0000","last_became_active":"2024-09-13T08:37:10.022095+0000","last_became_peered":"2024-09-13T08:37:10.022095+0000","last_unstale":"2024-09-13T08:37:10.024424+0000","last_undegraded":"2024-09-13T08:37:10.024424+0000","last_fullsized":"2024-09-13T08:37:10.024424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039163+0000","last_change":"2024-09-13T08:37:10.032355+0000","last_active":"2024-09-13T08:37:10.039163+0000","last_peered":"2024-09-13T08:37:10.039163+0000","last_clean":"2024-09-13T08:37:10.039163+0000","last_became_active":"2024-09-13T08:37:10.032098+0000","last_became_peered":"2024-09-13T08:37:10.032098+0000","last_unstale":"2024-09-13T08:37:10.039163+0000","last_undegraded":"2024-09-13T08:37:10.039163+0000","last_fullsized":"2024-09-13T08:37:10.039163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017318+0000","last_change":"2024-09-13T08:37:10.022000+0000","last_active":"2024-09-13T08:37:11.017318+0000","last_peered":"2024-09-13T08:37:11.017318+0000","last_clean":"2024-09-13T08:37:11.017318+0000","last_became_active":"2024-09-13T08:37:10.021622+0000","last_became_peered":"2024-09-13T08:37:10.021622+0000","last_unstale":"2024-09-13T08:37:11.017318+0000","last_undegraded":"2024-09-13T08:37:11.017318+0000","last_fullsized":"2024-09-13T08:37:11.017318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034062+0000","last_change":"2024-09-13T08:37:10.026234+0000","last_active":"2024-09-13T08:37:10.034062+0000","last_peered":"2024-09-13T08:37:10.034062+0000","last_clean":"2024-09-13T08:37:10.034062+0000","last_became_active":"2024-09-13T08:37:10.025970+0000","last_became_peered":"2024-09-13T08:37:10.025970+0000","last_unstale":"2024-09-13T08:37:10.034062+0000","last_undegraded":"2024-09-13T08:37:10.034062+0000","last_fullsized":"2024-09-13T08:37:10.034062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:34.396268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021932+0000","last_change":"2024-09-13T08:37:10.028101+0000","last_active":"2024-09-13T08:37:11.021932+0000","last_peered":"2024-09-13T08:37:11.021932+0000","last_clean":"2024-09-13T08:37:11.021932+0000","last_became_active":"2024-09-13T08:37:10.021666+0000","last_became_peered":"2024-09-13T08:37:10.021666+0000","last_unstale":"2024-09-13T08:37:11.021932+0000","last_undegraded":"2024-09-13T08:37:11.021932+0000","last_fullsized":"2024-09-13T08:37:11.021932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:43.682175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723677+0000","last_change":"2024-09-13T08:37:10.020945+0000","last_active":"2024-09-13T08:37:12.723677+0000","last_peered":"2024-09-13T08:37:12.723677+0000","last_clean":"2024-09-13T08:37:12.723677+0000","last_became_active":"2024-09-13T08:37:10.020675+0000","last_became_peered":"2024-09-13T08:37:10.020675+0000","last_unstale":"2024-09-13T08:37:12.723677+0000","last_undegraded":"2024-09-13T08:37:12.723677+0000","last_fullsized":"2024-09-13T08:37:12.723677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:41.501115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328354+0000","last_change":"2024-09-13T08:37:10.036002+0000","last_active":"2024-09-13T08:37:11.328354+0000","last_peered":"2024-09-13T08:37:11.328354+0000","last_clean":"2024-09-13T08:37:11.328354+0000","last_became_active":"2024-09-13T08:37:10.035684+0000","last_became_peered":"2024-09-13T08:37:10.035684+0000","last_unstale":"2024-09-13T08:37:11.328354+0000","last_undegraded":"2024-09-13T08:37:11.328354+0000","last_fullsized":"2024-09-13T08:37:11.328354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039036+0000","last_change":"2024-09-13T08:37:10.032065+0000","last_active":"2024-09-13T08:37:10.039036+0000","last_peered":"2024-09-13T08:37:10.039036+0000","last_clean":"2024-09-13T08:37:10.039036+0000","last_became_active":"2024-09-13T08:37:10.031802+0000","last_became_peered":"2024-09-13T08:37:10.031802+0000","last_unstale":"2024-09-13T08:37:10.039036+0000","last_undegraded":"2024-09-13T08:37:10.039036+0000","last_fullsized":"2024-09-13T08:37:10.039036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328937+0000","last_change":"2024-09-13T08:37:10.036040+0000","last_active":"2024-09-13T08:37:11.328937+0000","last_peered":"2024-09-13T08:37:11.328937+0000","last_clean":"2024-09-13T08:37:11.328937+0000","last_became_active":"2024-09-13T08:37:10.035698+0000","last_became_peered":"2024-09-13T08:37:10.035698+0000","last_unstale":"2024-09-13T08:37:11.328937+0000","last_undegraded":"2024-09-13T08:37:11.328937+0000","last_fullsized":"2024-09-13T08:37:11.328937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015904+0000","last_change":"2024-09-13T08:37:10.023312+0000","last_active":"2024-09-13T08:37:11.015904+0000","last_peered":"2024-09-13T08:37:11.015904+0000","last_clean":"2024-09-13T08:37:11.015904+0000","last_became_active":"2024-09-13T08:37:10.022193+0000","last_became_peered":"2024-09-13T08:37:10.022193+0000","last_unstale":"2024-09-13T08:37:11.015904+0000","last_undegraded":"2024-09-13T08:37:11.015904+0000","last_fullsized":"2024-09-13T08:37:11.015904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863802+0000","last_change":"2024-09-13T08:37:10.028354+0000","last_active":"2024-09-13T08:37:12.863802+0000","last_peered":"2024-09-13T08:37:12.863802+0000","last_clean":"2024-09-13T08:37:12.863802+0000","last_became_active":"2024-09-13T08:37:10.027989+0000","last_became_peered":"2024-09-13T08:37:10.027989+0000","last_unstale":"2024-09-13T08:37:12.863802+0000","last_undegraded":"2024-09-13T08:37:12.863802+0000","last_fullsized":"2024-09-13T08:37:12.863802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863758+0000","last_change":"2024-09-13T08:37:10.039029+0000","last_active":"2024-09-13T08:37:12.863758+0000","last_peered":"2024-09-13T08:37:12.863758+0000","last_clean":"2024-09-13T08:37:12.863758+0000","last_became_active":"2024-09-13T08:37:10.038689+0000","last_became_peered":"2024-09-13T08:37:10.038689+0000","last_unstale":"2024-09-13T08:37:12.863758+0000","last_undegraded":"2024-09-13T08:37:12.863758+0000","last_fullsized":"2024-09-13T08:37:12.863758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328458+0000","last_change":"2024-09-13T08:37:10.035713+0000","last_active":"2024-09-13T08:37:11.328458+0000","last_peered":"2024-09-13T08:37:11.328458+0000","last_clean":"2024-09-13T08:37:11.328458+0000","last_became_active":"2024-09-13T08:37:10.035292+0000","last_became_peered":"2024-09-13T08:37:10.035292+0000","last_unstale":"2024-09-13T08:37:11.328458+0000","last_undegraded":"2024-09-13T08:37:11.328458+0000","last_fullsized":"2024-09-13T08:37:11.328458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.038709+0000","last_change":"2024-09-13T08:37:10.029548+0000","last_active":"2024-09-13T08:37:10.038709+0000","last_peered":"2024-09-13T08:37:10.038709+0000","last_clean":"2024-09-13T08:37:10.038709+0000","last_became_active":"2024-09-13T08:37:10.029292+0000","last_became_peered":"2024-09-13T08:37:10.029292+0000","last_unstale":"2024-09-13T08:37:10.038709+0000","last_undegraded":"2024-09-13T08:37:10.038709+0000","last_fullsized":"2024-09-13T08:37:10.038709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016762+0000","last_change":"2024-09-13T08:37:10.012936+0000","last_active":"2024-09-13T08:37:11.016762+0000","last_peered":"2024-09-13T08:37:11.016762+0000","last_clean":"2024-09-13T08:37:11.016762+0000","last_became_active":"2024-09-13T08:37:10.012582+0000","last_became_peered":"2024-09-13T08:37:10.012582+0000","last_unstale":"2024-09-13T08:37:11.016762+0000","last_undegraded":"2024-09-13T08:37:11.016762+0000","last_fullsized":"2024-09-13T08:37:11.016762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015064+0000","last_change":"2024-09-13T08:37:10.017195+0000","last_active":"2024-09-13T08:37:11.015064+0000","last_peered":"2024-09-13T08:37:11.015064+0000","last_clean":"2024-09-13T08:37:11.015064+0000","last_became_active":"2024-09-13T08:37:10.016877+0000","last_became_peered":"2024-09-13T08:37:10.016877+0000","last_unstale":"2024-09-13T08:37:11.015064+0000","last_undegraded":"2024-09-13T08:37:11.015064+0000","last_fullsized":"2024-09-13T08:37:11.015064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.474080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723910+0000","last_change":"2024-09-13T08:37:10.023947+0000","last_active":"2024-09-13T08:37:12.723910+0000","last_peered":"2024-09-13T08:37:12.723910+0000","last_clean":"2024-09-13T08:37:12.723910+0000","last_became_active":"2024-09-13T08:37:10.023659+0000","last_became_peered":"2024-09-13T08:37:10.023659+0000","last_unstale":"2024-09-13T08:37:12.723910+0000","last_undegraded":"2024-09-13T08:37:12.723910+0000","last_fullsized":"2024-09-13T08:37:12.723910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328520+0000","last_change":"2024-09-13T08:37:09.014843+0000","last_active":"2024-09-13T08:37:11.328520+0000","last_peered":"2024-09-13T08:37:11.328520+0000","last_clean":"2024-09-13T08:37:11.328520+0000","last_became_active":"2024-09-13T08:37:09.014454+0000","last_became_peered":"2024-09-13T08:37:09.014454+0000","last_unstale":"2024-09-13T08:37:11.328520+0000","last_undegraded":"2024-09-13T08:37:11.328520+0000","last_fullsized":"2024-09-13T08:37:11.328520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016091+0000","last_change":"2024-09-13T08:37:09.024672+0000","last_active":"2024-09-13T08:37:10.016091+0000","last_peered":"2024-09-13T08:37:10.016091+0000","last_clean":"2024-09-13T08:37:10.016091+0000","last_became_active":"2024-09-13T08:37:09.024363+0000","last_became_peered":"2024-09-13T08:37:09.024363+0000","last_unstale":"2024-09-13T08:37:10.016091+0000","last_undegraded":"2024-09-13T08:37:10.016091+0000","last_fullsized":"2024-09-13T08:37:10.016091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:55.612953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863222+0000","last_change":"2024-09-13T08:37:09.014986+0000","last_active":"2024-09-13T08:37:12.863222+0000","last_peered":"2024-09-13T08:37:12.863222+0000","last_clean":"2024-09-13T08:37:12.863222+0000","last_became_active":"2024-09-13T08:37:09.014658+0000","last_became_peered":"2024-09-13T08:37:09.014658+0000","last_unstale":"2024-09-13T08:37:12.863222+0000","last_undegraded":"2024-09-13T08:37:12.863222+0000","last_fullsized":"2024-09-13T08:37:12.863222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328628+0000","last_change":"2024-09-13T08:37:10.030653+0000","last_active":"2024-09-13T08:37:11.328628+0000","last_peered":"2024-09-13T08:37:11.328628+0000","last_clean":"2024-09-13T08:37:11.328628+0000","last_became_active":"2024-09-13T08:37:10.030383+0000","last_became_peered":"2024-09-13T08:37:10.030383+0000","last_unstale":"2024-09-13T08:37:11.328628+0000","last_undegraded":"2024-09-13T08:37:11.328628+0000","last_fullsized":"2024-09-13T08:37:11.328628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.010377+0000","last_change":"2024-09-13T08:37:09.023571+0000","last_active":"2024-09-13T08:37:10.010377+0000","last_peered":"2024-09-13T08:37:10.010377+0000","last_clean":"2024-09-13T08:37:10.010377+0000","last_became_active":"2024-09-13T08:37:09.023048+0000","last_became_peered":"2024-09-13T08:37:09.023048+0000","last_unstale":"2024-09-13T08:37:10.010377+0000","last_undegraded":"2024-09-13T08:37:10.010377+0000","last_fullsized":"2024-09-13T08:37:10.010377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:14.567695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723890+0000","last_change":"2024-09-13T08:37:09.010512+0000","last_active":"2024-09-13T08:37:12.723890+0000","last_peered":"2024-09-13T08:37:12.723890+0000","last_clean":"2024-09-13T08:37:12.723890+0000","last_became_active":"2024-09-13T08:37:09.010229+0000","last_became_peered":"2024-09-13T08:37:09.010229+0000","last_unstale":"2024-09-13T08:37:12.723890+0000","last_undegraded":"2024-09-13T08:37:12.723890+0000","last_fullsized":"2024-09-13T08:37:12.723890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015969+0000","last_change":"2024-09-13T08:37:10.020061+0000","last_active":"2024-09-13T08:37:11.015969+0000","last_peered":"2024-09-13T08:37:11.015969+0000","last_clean":"2024-09-13T08:37:11.015969+0000","last_became_active":"2024-09-13T08:37:10.019663+0000","last_became_peered":"2024-09-13T08:37:10.019663+0000","last_unstale":"2024-09-13T08:37:11.015969+0000","last_undegraded":"2024-09-13T08:37:11.015969+0000","last_fullsized":"2024-09-13T08:37:11.015969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:48.021040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021832+0000","last_change":"2024-09-13T08:37:09.011936+0000","last_active":"2024-09-13T08:37:11.021832+0000","last_peered":"2024-09-13T08:37:11.021832+0000","last_clean":"2024-09-13T08:37:11.021832+0000","last_became_active":"2024-09-13T08:37:09.011670+0000","last_became_peered":"2024-09-13T08:37:09.011670+0000","last_unstale":"2024-09-13T08:37:11.021832+0000","last_undegraded":"2024-09-13T08:37:11.021832+0000","last_fullsized":"2024-09-13T08:37:11.021832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:31.341815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329184+0000","last_change":"2024-09-13T08:37:10.027709+0000","last_active":"2024-09-13T08:37:11.329184+0000","last_peered":"2024-09-13T08:37:11.329184+0000","last_clean":"2024-09-13T08:37:11.329184+0000","last_became_active":"2024-09-13T08:37:10.027410+0000","last_became_peered":"2024-09-13T08:37:10.027410+0000","last_unstale":"2024-09-13T08:37:11.329184+0000","last_undegraded":"2024-09-13T08:37:11.329184+0000","last_fullsized":"2024-09-13T08:37:11.329184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:25.100540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017898+0000","last_change":"2024-09-13T08:37:09.003083+0000","last_active":"2024-09-13T08:37:11.017898+0000","last_peered":"2024-09-13T08:37:11.017898+0000","last_clean":"2024-09-13T08:37:11.017898+0000","last_became_active":"2024-09-13T08:37:09.002697+0000","last_became_peered":"2024-09-13T08:37:09.002697+0000","last_unstale":"2024-09-13T08:37:11.017898+0000","last_undegraded":"2024-09-13T08:37:11.017898+0000","last_fullsized":"2024-09-13T08:37:11.017898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:56.231941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.025144+0000","last_change":"2024-09-13T08:37:10.021360+0000","last_active":"2024-09-13T08:37:10.025144+0000","last_peered":"2024-09-13T08:37:10.025144+0000","last_clean":"2024-09-13T08:37:10.025144+0000","last_became_active":"2024-09-13T08:37:10.021080+0000","last_became_peered":"2024-09-13T08:37:10.021080+0000","last_unstale":"2024-09-13T08:37:10.025144+0000","last_undegraded":"2024-09-13T08:37:10.025144+0000","last_fullsized":"2024-09-13T08:37:10.025144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015415+0000","last_change":"2024-09-13T08:37:09.014879+0000","last_active":"2024-09-13T08:37:11.015415+0000","last_peered":"2024-09-13T08:37:11.015415+0000","last_clean":"2024-09-13T08:37:11.015415+0000","last_became_active":"2024-09-13T08:37:09.014572+0000","last_became_peered":"2024-09-13T08:37:09.014572+0000","last_unstale":"2024-09-13T08:37:11.015415+0000","last_undegraded":"2024-09-13T08:37:11.015415+0000","last_fullsized":"2024-09-13T08:37:11.015415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:14.033544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022331+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:11.022331+0000","last_peered":"2024-09-13T08:37:11.022331+0000","last_clean":"2024-09-13T08:37:11.022331+0000","last_became_active":"2024-09-13T08:37:10.026781+0000","last_became_peered":"2024-09-13T08:37:10.026781+0000","last_unstale":"2024-09-13T08:37:11.022331+0000","last_undegraded":"2024-09-13T08:37:11.022331+0000","last_fullsized":"2024-09-13T08:37:11.022331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017361+0000","last_change":"2024-09-13T08:37:09.015199+0000","last_active":"2024-09-13T08:37:11.017361+0000","last_peered":"2024-09-13T08:37:11.017361+0000","last_clean":"2024-09-13T08:37:11.017361+0000","last_became_active":"2024-09-13T08:37:09.014848+0000","last_became_peered":"2024-09-13T08:37:09.014848+0000","last_unstale":"2024-09-13T08:37:11.017361+0000","last_undegraded":"2024-09-13T08:37:11.017361+0000","last_fullsized":"2024-09-13T08:37:11.017361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:01.967164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863695+0000","last_change":"2024-09-13T08:37:10.027034+0000","last_active":"2024-09-13T08:37:12.863695+0000","last_peered":"2024-09-13T08:37:12.863695+0000","last_clean":"2024-09-13T08:37:12.863695+0000","last_became_active":"2024-09-13T08:37:10.026723+0000","last_became_peered":"2024-09-13T08:37:10.026723+0000","last_unstale":"2024-09-13T08:37:12.863695+0000","last_undegraded":"2024-09-13T08:37:12.863695+0000","last_fullsized":"2024-09-13T08:37:12.863695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016198+0000","last_change":"2024-09-13T08:37:09.021683+0000","last_active":"2024-09-13T08:37:10.016198+0000","last_peered":"2024-09-13T08:37:10.016198+0000","last_clean":"2024-09-13T08:37:10.016198+0000","last_became_active":"2024-09-13T08:37:09.019630+0000","last_became_peered":"2024-09-13T08:37:09.019630+0000","last_unstale":"2024-09-13T08:37:10.016198+0000","last_undegraded":"2024-09-13T08:37:10.016198+0000","last_fullsized":"2024-09-13T08:37:10.016198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:47.015443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039364+0000","last_change":"2024-09-13T08:37:10.032446+0000","last_active":"2024-09-13T08:37:10.039364+0000","last_peered":"2024-09-13T08:37:10.039364+0000","last_clean":"2024-09-13T08:37:10.039364+0000","last_became_active":"2024-09-13T08:37:10.031448+0000","last_became_peered":"2024-09-13T08:37:10.031448+0000","last_unstale":"2024-09-13T08:37:10.039364+0000","last_undegraded":"2024-09-13T08:37:10.039364+0000","last_fullsized":"2024-09-13T08:37:10.039364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021502+0000","last_change":"2024-09-13T08:37:09.025565+0000","last_active":"2024-09-13T08:37:11.021502+0000","last_peered":"2024-09-13T08:37:11.021502+0000","last_clean":"2024-09-13T08:37:11.021502+0000","last_became_active":"2024-09-13T08:37:09.025242+0000","last_became_peered":"2024-09-13T08:37:09.025242+0000","last_unstale":"2024-09-13T08:37:11.021502+0000","last_undegraded":"2024-09-13T08:37:11.021502+0000","last_fullsized":"2024-09-13T08:37:11.021502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:08.638798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723679+0000","last_change":"2024-09-13T08:37:10.030817+0000","last_active":"2024-09-13T08:37:12.723679+0000","last_peered":"2024-09-13T08:37:12.723679+0000","last_clean":"2024-09-13T08:37:12.723679+0000","last_became_active":"2024-09-13T08:37:10.030554+0000","last_became_peered":"2024-09-13T08:37:10.030554+0000","last_unstale":"2024-09-13T08:37:12.723679+0000","last_undegraded":"2024-09-13T08:37:12.723679+0000","last_fullsized":"2024-09-13T08:37:12.723679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.027753+0000","last_change":"2024-09-13T08:37:10.023766+0000","last_active":"2024-09-13T08:37:10.027753+0000","last_peered":"2024-09-13T08:37:10.027753+0000","last_clean":"2024-09-13T08:37:10.027753+0000","last_became_active":"2024-09-13T08:37:10.022889+0000","last_became_peered":"2024-09-13T08:37:10.022889+0000","last_unstale":"2024-09-13T08:37:10.027753+0000","last_undegraded":"2024-09-13T08:37:10.027753+0000","last_fullsized":"2024-09-13T08:37:10.027753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:50.095913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022117+0000","last_change":"2024-09-13T08:37:09.023974+0000","last_active":"2024-09-13T08:37:11.022117+0000","last_peered":"2024-09-13T08:37:11.022117+0000","last_clean":"2024-09-13T08:37:11.022117+0000","last_became_active":"2024-09-13T08:37:09.023471+0000","last_became_peered":"2024-09-13T08:37:09.023471+0000","last_unstale":"2024-09-13T08:37:11.022117+0000","last_undegraded":"2024-09-13T08:37:11.022117+0000","last_fullsized":"2024-09-13T08:37:11.022117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:38.190525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016062+0000","last_change":"2024-09-13T08:37:10.023008+0000","last_active":"2024-09-13T08:37:11.016062+0000","last_peered":"2024-09-13T08:37:11.016062+0000","last_clean":"2024-09-13T08:37:11.016062+0000","last_became_active":"2024-09-13T08:37:10.021695+0000","last_became_peered":"2024-09-13T08:37:10.021695+0000","last_unstale":"2024-09-13T08:37:11.016062+0000","last_undegraded":"2024-09-13T08:37:11.016062+0000","last_fullsized":"2024-09-13T08:37:11.016062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:43.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022182+0000","last_change":"2024-09-13T08:37:10.031929+0000","last_active":"2024-09-13T08:37:11.022182+0000","last_peered":"2024-09-13T08:37:11.022182+0000","last_clean":"2024-09-13T08:37:11.022182+0000","last_became_active":"2024-09-13T08:37:10.031598+0000","last_became_peered":"2024-09-13T08:37:10.031598+0000","last_unstale":"2024-09-13T08:37:11.022182+0000","last_undegraded":"2024-09-13T08:37:11.022182+0000","last_fullsized":"2024-09-13T08:37:11.022182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328623+0000","last_change":"2024-09-13T08:37:09.011613+0000","last_active":"2024-09-13T08:37:11.328623+0000","last_peered":"2024-09-13T08:37:11.328623+0000","last_clean":"2024-09-13T08:37:11.328623+0000","last_became_active":"2024-09-13T08:37:09.011071+0000","last_became_peered":"2024-09-13T08:37:09.011071+0000","last_unstale":"2024-09-13T08:37:11.328623+0000","last_undegraded":"2024-09-13T08:37:11.328623+0000","last_fullsized":"2024-09-13T08:37:11.328623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022072+0000","last_change":"2024-09-13T08:37:10.028420+0000","last_active":"2024-09-13T08:37:11.022072+0000","last_peered":"2024-09-13T08:37:11.022072+0000","last_clean":"2024-09-13T08:37:11.022072+0000","last_became_active":"2024-09-13T08:37:10.028166+0000","last_became_peered":"2024-09-13T08:37:10.028166+0000","last_unstale":"2024-09-13T08:37:11.022072+0000","last_undegraded":"2024-09-13T08:37:11.022072+0000","last_fullsized":"2024-09-13T08:37:11.022072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:58.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863837+0000","last_change":"2024-09-13T08:37:09.015072+0000","last_active":"2024-09-13T08:37:12.863837+0000","last_peered":"2024-09-13T08:37:12.863837+0000","last_clean":"2024-09-13T08:37:12.863837+0000","last_became_active":"2024-09-13T08:37:09.014597+0000","last_became_peered":"2024-09-13T08:37:09.014597+0000","last_unstale":"2024-09-13T08:37:12.863837+0000","last_undegraded":"2024-09-13T08:37:12.863837+0000","last_fullsized":"2024-09-13T08:37:12.863837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.022427+0000","last_change":"2024-09-13T08:37:10.018219+0000","last_active":"2024-09-13T08:37:10.022427+0000","last_peered":"2024-09-13T08:37:10.022427+0000","last_clean":"2024-09-13T08:37:10.022427+0000","last_became_active":"2024-09-13T08:37:10.017946+0000","last_became_peered":"2024-09-13T08:37:10.017946+0000","last_unstale":"2024-09-13T08:37:10.022427+0000","last_undegraded":"2024-09-13T08:37:10.022427+0000","last_fullsized":"2024-09-13T08:37:10.022427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723548+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:12.723548+0000","last_peered":"2024-09-13T08:37:12.723548+0000","last_clean":"2024-09-13T08:37:12.723548+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:12.723548+0000","last_undegraded":"2024-09-13T08:37:12.723548+0000","last_fullsized":"2024-09-13T08:37:12.723548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.013248+0000","last_change":"2024-09-13T08:37:09.021195+0000","last_active":"2024-09-13T08:37:10.013248+0000","last_peered":"2024-09-13T08:37:10.013248+0000","last_clean":"2024-09-13T08:37:10.013248+0000","last_became_active":"2024-09-13T08:37:09.019692+0000","last_became_peered":"2024-09-13T08:37:09.019692+0000","last_unstale":"2024-09-13T08:37:10.013248+0000","last_undegraded":"2024-09-13T08:37:10.013248+0000","last_fullsized":"2024-09-13T08:37:10.013248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.430584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.010874+0000","last_change":"2024-09-13T08:37:09.019756+0000","last_active":"2024-09-13T08:37:10.010874+0000","last_peered":"2024-09-13T08:37:10.010874+0000","last_clean":"2024-09-13T08:37:10.010874+0000","last_became_active":"2024-09-13T08:37:09.019375+0000","last_became_peered":"2024-09-13T08:37:09.019375+0000","last_unstale":"2024-09-13T08:37:10.010874+0000","last_undegraded":"2024-09-13T08:37:10.010874+0000","last_fullsized":"2024-09-13T08:37:10.010874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:40.495518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863575+0000","last_change":"2024-09-13T08:37:10.020491+0000","last_active":"2024-09-13T08:37:12.863575+0000","last_peered":"2024-09-13T08:37:12.863575+0000","last_clean":"2024-09-13T08:37:12.863575+0000","last_became_active":"2024-09-13T08:37:10.020271+0000","last_became_peered":"2024-09-13T08:37:10.020271+0000","last_unstale":"2024-09-13T08:37:12.863575+0000","last_undegraded":"2024-09-13T08:37:12.863575+0000","last_fullsized":"2024-09-13T08:37:12.863575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:29.393985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016620+0000","last_change":"2024-09-13T08:37:09.015889+0000","last_active":"2024-09-13T08:37:11.016620+0000","last_peered":"2024-09-13T08:37:11.016620+0000","last_clean":"2024-09-13T08:37:11.016620+0000","last_became_active":"2024-09-13T08:37:09.015587+0000","last_became_peered":"2024-09-13T08:37:09.015587+0000","last_unstale":"2024-09-13T08:37:11.016620+0000","last_undegraded":"2024-09-13T08:37:11.016620+0000","last_fullsized":"2024-09-13T08:37:11.016620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328457+0000","last_change":"2024-09-13T08:37:10.036912+0000","last_active":"2024-09-13T08:37:11.328457+0000","last_peered":"2024-09-13T08:37:11.328457+0000","last_clean":"2024-09-13T08:37:11.328457+0000","last_became_active":"2024-09-13T08:37:10.036665+0000","last_became_peered":"2024-09-13T08:37:10.036665+0000","last_unstale":"2024-09-13T08:37:11.328457+0000","last_undegraded":"2024-09-13T08:37:11.328457+0000","last_fullsized":"2024-09-13T08:37:11.328457+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.016380+0000","last_change":"2024-09-13T08:37:09.023866+0000","last_active":"2024-09-13T08:37:10.016380+0000","last_peered":"2024-09-13T08:37:10.016380+0000","last_clean":"2024-09-13T08:37:10.016380+0000","last_became_active":"2024-09-13T08:37:09.021825+0000","last_became_peered":"2024-09-13T08:37:09.021825+0000","last_unstale":"2024-09-13T08:37:10.016380+0000","last_undegraded":"2024-09-13T08:37:10.016380+0000","last_fullsized":"2024-09-13T08:37:10.016380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:42.779836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022032+0000","last_change":"2024-09-13T08:37:10.027315+0000","last_active":"2024-09-13T08:37:11.022032+0000","last_peered":"2024-09-13T08:37:11.022032+0000","last_clean":"2024-09-13T08:37:11.022032+0000","last_became_active":"2024-09-13T08:37:10.027018+0000","last_became_peered":"2024-09-13T08:37:10.027018+0000","last_unstale":"2024-09-13T08:37:11.022032+0000","last_undegraded":"2024-09-13T08:37:11.022032+0000","last_fullsized":"2024-09-13T08:37:11.022032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328769+0000","last_change":"2024-09-13T08:37:09.018684+0000","last_active":"2024-09-13T08:37:11.328769+0000","last_peered":"2024-09-13T08:37:11.328769+0000","last_clean":"2024-09-13T08:37:11.328769+0000","last_became_active":"2024-09-13T08:37:09.016570+0000","last_became_peered":"2024-09-13T08:37:09.016570+0000","last_unstale":"2024-09-13T08:37:11.328769+0000","last_undegraded":"2024-09-13T08:37:11.328769+0000","last_fullsized":"2024-09-13T08:37:11.328769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015606+0000","last_change":"2024-09-13T08:37:10.018230+0000","last_active":"2024-09-13T08:37:11.015606+0000","last_peered":"2024-09-13T08:37:11.015606+0000","last_clean":"2024-09-13T08:37:11.015606+0000","last_became_active":"2024-09-13T08:37:10.017427+0000","last_became_peered":"2024-09-13T08:37:10.017427+0000","last_unstale":"2024-09-13T08:37:11.015606+0000","last_undegraded":"2024-09-13T08:37:11.015606+0000","last_fullsized":"2024-09-13T08:37:11.015606+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.015515+0000","last_change":"2024-09-13T08:37:09.018807+0000","last_active":"2024-09-13T08:37:10.015515+0000","last_peered":"2024-09-13T08:37:10.015515+0000","last_clean":"2024-09-13T08:37:10.015515+0000","last_became_active":"2024-09-13T08:37:09.018298+0000","last_became_peered":"2024-09-13T08:37:09.018298+0000","last_unstale":"2024-09-13T08:37:10.015515+0000","last_undegraded":"2024-09-13T08:37:10.015515+0000","last_fullsized":"2024-09-13T08:37:10.015515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.295316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863880+0000","last_change":"2024-09-13T08:37:10.028730+0000","last_active":"2024-09-13T08:37:12.863880+0000","last_peered":"2024-09-13T08:37:12.863880+0000","last_clean":"2024-09-13T08:37:12.863880+0000","last_became_active":"2024-09-13T08:37:10.028481+0000","last_became_peered":"2024-09-13T08:37:10.028481+0000","last_unstale":"2024-09-13T08:37:12.863880+0000","last_undegraded":"2024-09-13T08:37:12.863880+0000","last_fullsized":"2024-09-13T08:37:12.863880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.044490+0000","last_change":"2024-09-13T08:37:09.024256+0000","last_active":"2024-09-13T08:37:10.044490+0000","last_peered":"2024-09-13T08:37:10.044490+0000","last_clean":"2024-09-13T08:37:10.044490+0000","last_became_active":"2024-09-13T08:37:09.023967+0000","last_became_peered":"2024-09-13T08:37:09.023967+0000","last_unstale":"2024-09-13T08:37:10.044490+0000","last_undegraded":"2024-09-13T08:37:10.044490+0000","last_fullsized":"2024-09-13T08:37:10.044490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:41.577735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021357+0000","last_change":"2024-09-13T08:37:09.011069+0000","last_active":"2024-09-13T08:37:11.021357+0000","last_peered":"2024-09-13T08:37:11.021357+0000","last_clean":"2024-09-13T08:37:11.021357+0000","last_became_active":"2024-09-13T08:37:09.010687+0000","last_became_peered":"2024-09-13T08:37:09.010687+0000","last_unstale":"2024-09-13T08:37:11.021357+0000","last_undegraded":"2024-09-13T08:37:11.021357+0000","last_fullsized":"2024-09-13T08:37:11.021357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034067+0000","last_change":"2024-09-13T08:37:10.026288+0000","last_active":"2024-09-13T08:37:10.034067+0000","last_peered":"2024-09-13T08:37:10.034067+0000","last_clean":"2024-09-13T08:37:10.034067+0000","last_became_active":"2024-09-13T08:37:10.026029+0000","last_became_peered":"2024-09-13T08:37:10.026029+0000","last_unstale":"2024-09-13T08:37:10.034067+0000","last_undegraded":"2024-09-13T08:37:10.034067+0000","last_fullsized":"2024-09-13T08:37:10.034067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863577+0000","last_change":"2024-09-13T08:37:09.019714+0000","last_active":"2024-09-13T08:37:12.863577+0000","last_peered":"2024-09-13T08:37:12.863577+0000","last_clean":"2024-09-13T08:37:12.863577+0000","last_became_active":"2024-09-13T08:37:09.018957+0000","last_became_peered":"2024-09-13T08:37:09.018957+0000","last_unstale":"2024-09-13T08:37:12.863577+0000","last_undegraded":"2024-09-13T08:37:12.863577+0000","last_fullsized":"2024-09-13T08:37:12.863577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017312+0000","last_change":"2024-09-13T08:37:10.026722+0000","last_active":"2024-09-13T08:37:11.017312+0000","last_peered":"2024-09-13T08:37:11.017312+0000","last_clean":"2024-09-13T08:37:11.017312+0000","last_became_active":"2024-09-13T08:37:10.025267+0000","last_became_peered":"2024-09-13T08:37:10.025267+0000","last_unstale":"2024-09-13T08:37:11.017312+0000","last_undegraded":"2024-09-13T08:37:11.017312+0000","last_fullsized":"2024-09-13T08:37:11.017312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.039159+0000","last_change":"2024-09-13T08:37:09.022982+0000","last_active":"2024-09-13T08:37:10.039159+0000","last_peered":"2024-09-13T08:37:10.039159+0000","last_clean":"2024-09-13T08:37:10.039159+0000","last_became_active":"2024-09-13T08:37:09.021746+0000","last_became_peered":"2024-09-13T08:37:09.021746+0000","last_unstale":"2024-09-13T08:37:10.039159+0000","last_undegraded":"2024-09-13T08:37:10.039159+0000","last_fullsized":"2024-09-13T08:37:10.039159+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.468482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.025322+0000","last_change":"2024-09-13T08:37:10.016999+0000","last_active":"2024-09-13T08:37:10.025322+0000","last_peered":"2024-09-13T08:37:10.025322+0000","last_clean":"2024-09-13T08:37:10.025322+0000","last_became_active":"2024-09-13T08:37:10.016738+0000","last_became_peered":"2024-09-13T08:37:10.016738+0000","last_unstale":"2024-09-13T08:37:10.025322+0000","last_undegraded":"2024-09-13T08:37:10.025322+0000","last_fullsized":"2024-09-13T08:37:10.025322+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T08:37:10.034085+0000","last_change":"2024-09-13T08:37:10.026898+0000","last_active":"2024-09-13T08:37:10.034085+0000","last_peered":"2024-09-13T08:37:10.034085+0000","last_clean":"2024-09-13T08:37:10.034085+0000","last_became_active":"2024-09-13T08:37:10.026602+0000","last_became_peered":"2024-09-13T08:37:10.026602+0000","last_unstale":"2024-09-13T08:37:10.034085+0000","last_undegraded":"2024-09-13T08:37:10.034085+0000","last_fullsized":"2024-09-13T08:37:10.034085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:27.242730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022006+0000","last_change":"2024-09-13T08:37:09.011138+0000","last_active":"2024-09-13T08:37:11.022006+0000","last_peered":"2024-09-13T08:37:11.022006+0000","last_clean":"2024-09-13T08:37:11.022006+0000","last_became_active":"2024-09-13T08:37:09.010863+0000","last_became_peered":"2024-09-13T08:37:09.010863+0000","last_unstale":"2024-09-13T08:37:11.022006+0000","last_undegraded":"2024-09-13T08:37:11.022006+0000","last_fullsized":"2024-09-13T08:37:11.022006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:14.227052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723804+0000","last_change":"2024-09-13T08:37:10.029092+0000","last_active":"2024-09-13T08:37:12.723804+0000","last_peered":"2024-09-13T08:37:12.723804+0000","last_clean":"2024-09-13T08:37:12.723804+0000","last_became_active":"2024-09-13T08:37:10.028796+0000","last_became_peered":"2024-09-13T08:37:10.028796+0000","last_unstale":"2024-09-13T08:37:12.723804+0000","last_undegraded":"2024-09-13T08:37:12.723804+0000","last_fullsized":"2024-09-13T08:37:12.723804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022178+0000","last_change":"2024-09-13T08:37:09.013579+0000","last_active":"2024-09-13T08:37:11.022178+0000","last_peered":"2024-09-13T08:37:11.022178+0000","last_clean":"2024-09-13T08:37:11.022178+0000","last_became_active":"2024-09-13T08:37:09.013327+0000","last_became_peered":"2024-09-13T08:37:09.013327+0000","last_unstale":"2024-09-13T08:37:11.022178+0000","last_undegraded":"2024-09-13T08:37:11.022178+0000","last_fullsized":"2024-09-13T08:37:11.022178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:34.207907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328992+0000","last_change":"2024-09-13T08:37:10.032358+0000","last_active":"2024-09-13T08:37:11.328992+0000","last_peered":"2024-09-13T08:37:11.328992+0000","last_clean":"2024-09-13T08:37:11.328992+0000","last_became_active":"2024-09-13T08:37:10.032047+0000","last_became_peered":"2024-09-13T08:37:10.032047+0000","last_unstale":"2024-09-13T08:37:11.328992+0000","last_undegraded":"2024-09-13T08:37:11.328992+0000","last_fullsized":"2024-09-13T08:37:11.328992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:19.283020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329066+0000","last_change":"2024-09-13T08:37:09.015906+0000","last_active":"2024-09-13T08:37:11.329066+0000","last_peered":"2024-09-13T08:37:11.329066+0000","last_clean":"2024-09-13T08:37:11.329066+0000","last_became_active":"2024-09-13T08:37:09.015500+0000","last_became_peered":"2024-09-13T08:37:09.015500+0000","last_unstale":"2024-09-13T08:37:11.329066+0000","last_undegraded":"2024-09-13T08:37:11.329066+0000","last_fullsized":"2024-09-13T08:37:11.329066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:51.846602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329023+0000","last_change":"2024-09-13T08:37:10.043965+0000","last_active":"2024-09-13T08:37:11.329023+0000","last_peered":"2024-09-13T08:37:11.329023+0000","last_clean":"2024-09-13T08:37:11.329023+0000","last_became_active":"2024-09-13T08:37:10.041509+0000","last_became_peered":"2024-09-13T08:37:10.041509+0000","last_unstale":"2024-09-13T08:37:11.329023+0000","last_undegraded":"2024-09-13T08:37:11.329023+0000","last_fullsized":"2024-09-13T08:37:11.329023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":241664,"data_stored":81079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":221184,"data_stored":74655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":835584,"data_stored":671415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":75237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":245760,"data_stored":82567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":671393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":81011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:19.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.178+0000 7f415fe00640 1 -- 172.21.15.31:0/1213834494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4144004800 msgr2=0x7f4144004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.178+0000 7f415fe00640 1 --2- 172.21.15.31:0/1213834494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4144004800 0x7f4144004c80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f414800c470 tx=0x7f4144005830 comp rx=0 tx=0).stop 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.179+0000 7f415fe00640 1 -- 172.21.15.31:0/1213834494 shutdown_connections 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.179+0000 7f415fe00640 1 --2- 172.21.15.31:0/1213834494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4144004800 0x7f4144004c80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.179+0000 7f415fe00640 1 -- 172.21.15.31:0/1213834494 >> 172.21.15.31:0/1213834494 conn(0x7f41600d0360 msgr2=0x7f41600d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.179+0000 7f415fe00640 1 -- 172.21.15.31:0/1213834494 shutdown_connections 2024-09-13T08:37:19.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.179+0000 7f415fe00640 1 -- 172.21.15.31:0/1213834494 wait complete. 2024-09-13T08:37:19.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.180+0000 7f415fe00640 1 Processor -- start 2024-09-13T08:37:19.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 -- start start 2024-09-13T08:37:19.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4144004800 0x7f4160099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 0x7f416015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41600ccba0 con 0x7f416015c370 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41600cca20 con 0x7f41601f5350 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.181+0000 7f415fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f41600b3ba0 con 0x7f4144004800 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34422/0 (socket says 172.21.15.31:34422) 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 -- 172.21.15.31:0/2522670101 learned_addr learned my addr 172.21.15.31:0/2522670101 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415ea00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4144004800 0x7f4160099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f4156000640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 0x7f416015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 -- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4144004800 msgr2=0x7f4160099af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4144004800 0x7f4160099af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 -- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 msgr2=0x7f416015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 0x7f416015be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 -- 172.21.15.31:0/2522670101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41601e7320 con 0x7f416015c370 2024-09-13T08:37:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f4156000640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 0x7f416015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.182+0000 7f415f400640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f41600cc920 tx=0x7f4158006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.183+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158015070 con 0x7f416015c370 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.183+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4158011860 con 0x7f416015c370 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.183+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158011cb0 con 0x7f416015c370 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.183+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41601e6dd0 con 0x7f416015c370 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.183+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41601e9810 con 0x7f416015c370 2024-09-13T08:37:19.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.184+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41601e9340 con 0x7f416015c370 2024-09-13T08:37:19.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.185+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4158018070 con 0x7f416015c370 2024-09-13T08:37:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.186+0000 7f4157400640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 0x7f413c07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.186+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f415809b1a0 con 0x7f416015c370 2024-09-13T08:37:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.187+0000 7f415ea00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 0x7f413c07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.187+0000 7f415ea00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 0x7f413c07ad10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4160052890 tx=0x7f4150005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.190+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4158063800 con 0x7f416015c370 2024-09-13T08:37:19.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.381+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f41600b4070 con 0x7f413c078850 2024-09-13T08:37:19.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.385+0000 7f4157400640 1 -- 172.21.15.31:0/2522670101 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251845 (secure 0 0 0) 0x7f41600b4070 con 0x7f413c078850 2024-09-13T08:37:19.390 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:19.391 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.389+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 msgr2=0x7f413c07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.389+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 0x7f413c07ad10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4160052890 tx=0x7f4150005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.389+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 msgr2=0x7f416006f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.389+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f41600cc920 tx=0x7f4158006430 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 shutdown_connections 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416015c370 0x7f416006f890 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41601f5350 0x7f416015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f413c078850 0x7f413c07ad10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 --2- 172.21.15.31:0/2522670101 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4144004800 0x7f4160099af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 >> 172.21.15.31:0/2522670101 conn(0x7f41600d0360 msgr2=0x7f416006ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 shutdown_connections 2024-09-13T08:37:19.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.390+0000 7f415fe00640 1 -- 172.21.15.31:0/2522670101 wait complete. 2024-09-13T08:37:19.406 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-13T08:37:19.038933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235936,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738784,"statfs":{"total":773094113280,"available":772852514816,"internally_reserved":0,"allocated":3121152,"data_stored":1833196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.037126"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021976+0000","last_change":"2024-09-13T08:37:10.038242+0000","last_active":"2024-09-13T08:37:11.021976+0000","last_peered":"2024-09-13T08:37:11.021976+0000","last_clean":"2024-09-13T08:37:11.021976+0000","last_became_active":"2024-09-13T08:37:10.037984+0000","last_became_peered":"2024-09-13T08:37:10.037984+0000","last_unstale":"2024-09-13T08:37:11.021976+0000","last_undegraded":"2024-09-13T08:37:11.021976+0000","last_fullsized":"2024-09-13T08:37:11.021976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:05.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328871+0000","last_change":"2024-09-13T08:37:10.043402+0000","last_active":"2024-09-13T08:37:11.328871+0000","last_peered":"2024-09-13T08:37:11.328871+0000","last_clean":"2024-09-13T08:37:11.328871+0000","last_became_active":"2024-09-13T08:37:10.041028+0000","last_became_peered":"2024-09-13T08:37:10.041028+0000","last_unstale":"2024-09-13T08:37:11.328871+0000","last_undegraded":"2024-09-13T08:37:11.328871+0000","last_fullsized":"2024-09-13T08:37:11.328871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328967+0000","last_change":"2024-09-13T08:37:10.038931+0000","last_active":"2024-09-13T08:37:11.328967+0000","last_peered":"2024-09-13T08:37:11.328967+0000","last_clean":"2024-09-13T08:37:11.328967+0000","last_became_active":"2024-09-13T08:37:10.038676+0000","last_became_peered":"2024-09-13T08:37:10.038676+0000","last_unstale":"2024-09-13T08:37:11.328967+0000","last_undegraded":"2024-09-13T08:37:11.328967+0000","last_fullsized":"2024-09-13T08:37:11.328967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015339+0000","last_change":"2024-09-13T08:37:10.018106+0000","last_active":"2024-09-13T08:37:11.015339+0000","last_peered":"2024-09-13T08:37:11.015339+0000","last_clean":"2024-09-13T08:37:11.015339+0000","last_became_active":"2024-09-13T08:37:10.017812+0000","last_became_peered":"2024-09-13T08:37:10.017812+0000","last_unstale":"2024-09-13T08:37:11.015339+0000","last_undegraded":"2024-09-13T08:37:11.015339+0000","last_fullsized":"2024-09-13T08:37:11.015339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021892+0000","last_change":"2024-09-13T08:37:10.035463+0000","last_active":"2024-09-13T08:37:11.021892+0000","last_peered":"2024-09-13T08:37:11.021892+0000","last_clean":"2024-09-13T08:37:11.021892+0000","last_became_active":"2024-09-13T08:37:10.032852+0000","last_became_peered":"2024-09-13T08:37:10.032852+0000","last_unstale":"2024-09-13T08:37:11.021892+0000","last_undegraded":"2024-09-13T08:37:11.021892+0000","last_fullsized":"2024-09-13T08:37:11.021892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:34.731391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015062+0000","last_change":"2024-09-13T08:37:10.030116+0000","last_active":"2024-09-13T08:37:11.015062+0000","last_peered":"2024-09-13T08:37:11.015062+0000","last_clean":"2024-09-13T08:37:11.015062+0000","last_became_active":"2024-09-13T08:37:10.029773+0000","last_became_peered":"2024-09-13T08:37:10.029773+0000","last_unstale":"2024-09-13T08:37:11.015062+0000","last_undegraded":"2024-09-13T08:37:11.015062+0000","last_fullsized":"2024-09-13T08:37:11.015062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021640+0000","last_change":"2024-09-13T08:37:10.025604+0000","last_active":"2024-09-13T08:37:11.021640+0000","last_peered":"2024-09-13T08:37:11.021640+0000","last_clean":"2024-09-13T08:37:11.021640+0000","last_became_active":"2024-09-13T08:37:10.020828+0000","last_became_peered":"2024-09-13T08:37:10.020828+0000","last_unstale":"2024-09-13T08:37:11.021640+0000","last_undegraded":"2024-09-13T08:37:11.021640+0000","last_fullsized":"2024-09-13T08:37:11.021640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328358+0000","last_change":"2024-09-13T08:37:10.041296+0000","last_active":"2024-09-13T08:37:11.328358+0000","last_peered":"2024-09-13T08:37:11.328358+0000","last_clean":"2024-09-13T08:37:11.328358+0000","last_became_active":"2024-09-13T08:37:10.035151+0000","last_became_peered":"2024-09-13T08:37:10.035151+0000","last_unstale":"2024-09-13T08:37:11.328358+0000","last_undegraded":"2024-09-13T08:37:11.328358+0000","last_fullsized":"2024-09-13T08:37:11.328358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:28.830741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723967+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:12.723967+0000","last_peered":"2024-09-13T08:37:12.723967+0000","last_clean":"2024-09-13T08:37:12.723967+0000","last_became_active":"2024-09-13T08:37:10.026743+0000","last_became_peered":"2024-09-13T08:37:10.026743+0000","last_unstale":"2024-09-13T08:37:12.723967+0000","last_undegraded":"2024-09-13T08:37:12.723967+0000","last_fullsized":"2024-09-13T08:37:12.723967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022008+0000","last_change":"2024-09-13T08:37:10.020548+0000","last_active":"2024-09-13T08:37:11.022008+0000","last_peered":"2024-09-13T08:37:11.022008+0000","last_clean":"2024-09-13T08:37:11.022008+0000","last_became_active":"2024-09-13T08:37:10.020144+0000","last_became_peered":"2024-09-13T08:37:10.020144+0000","last_unstale":"2024-09-13T08:37:11.022008+0000","last_undegraded":"2024-09-13T08:37:11.022008+0000","last_fullsized":"2024-09-13T08:37:11.022008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:13.408879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022140+0000","last_change":"2024-09-13T08:37:10.038191+0000","last_active":"2024-09-13T08:37:11.022140+0000","last_peered":"2024-09-13T08:37:11.022140+0000","last_clean":"2024-09-13T08:37:11.022140+0000","last_became_active":"2024-09-13T08:37:10.037935+0000","last_became_peered":"2024-09-13T08:37:10.037935+0000","last_unstale":"2024-09-13T08:37:11.022140+0000","last_undegraded":"2024-09-13T08:37:11.022140+0000","last_fullsized":"2024-09-13T08:37:11.022140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021867+0000","last_change":"2024-09-13T08:37:10.025910+0000","last_active":"2024-09-13T08:37:11.021867+0000","last_peered":"2024-09-13T08:37:11.021867+0000","last_clean":"2024-09-13T08:37:11.021867+0000","last_became_active":"2024-09-13T08:37:10.023349+0000","last_became_peered":"2024-09-13T08:37:10.023349+0000","last_unstale":"2024-09-13T08:37:11.021867+0000","last_undegraded":"2024-09-13T08:37:11.021867+0000","last_fullsized":"2024-09-13T08:37:11.021867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:02.005454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863687+0000","last_change":"2024-09-13T08:37:10.033914+0000","last_active":"2024-09-13T08:37:12.863687+0000","last_peered":"2024-09-13T08:37:12.863687+0000","last_clean":"2024-09-13T08:37:12.863687+0000","last_became_active":"2024-09-13T08:37:10.030947+0000","last_became_peered":"2024-09-13T08:37:10.030947+0000","last_unstale":"2024-09-13T08:37:12.863687+0000","last_undegraded":"2024-09-13T08:37:12.863687+0000","last_fullsized":"2024-09-13T08:37:12.863687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:49.714886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863461+0000","last_change":"2024-09-13T08:37:10.024803+0000","last_active":"2024-09-13T08:37:12.863461+0000","last_peered":"2024-09-13T08:37:12.863461+0000","last_clean":"2024-09-13T08:37:12.863461+0000","last_became_active":"2024-09-13T08:37:10.024512+0000","last_became_peered":"2024-09-13T08:37:10.024512+0000","last_unstale":"2024-09-13T08:37:12.863461+0000","last_undegraded":"2024-09-13T08:37:12.863461+0000","last_fullsized":"2024-09-13T08:37:12.863461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328365+0000","last_change":"2024-09-13T08:37:10.035650+0000","last_active":"2024-09-13T08:37:11.328365+0000","last_peered":"2024-09-13T08:37:11.328365+0000","last_clean":"2024-09-13T08:37:11.328365+0000","last_became_active":"2024-09-13T08:37:10.035331+0000","last_became_peered":"2024-09-13T08:37:10.035331+0000","last_unstale":"2024-09-13T08:37:11.328365+0000","last_undegraded":"2024-09-13T08:37:11.328365+0000","last_fullsized":"2024-09-13T08:37:11.328365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.703160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.022368+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.022095+0000","last_became_peered":"2024-09-13T08:37:10.022095+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596810+0000","last_change":"2024-09-13T08:37:10.032355+0000","last_active":"2024-09-13T08:37:13.596810+0000","last_peered":"2024-09-13T08:37:13.596810+0000","last_clean":"2024-09-13T08:37:13.596810+0000","last_became_active":"2024-09-13T08:37:10.032098+0000","last_became_peered":"2024-09-13T08:37:10.032098+0000","last_unstale":"2024-09-13T08:37:13.596810+0000","last_undegraded":"2024-09-13T08:37:13.596810+0000","last_fullsized":"2024-09-13T08:37:13.596810+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:32:27.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017318+0000","last_change":"2024-09-13T08:37:10.022000+0000","last_active":"2024-09-13T08:37:11.017318+0000","last_peered":"2024-09-13T08:37:11.017318+0000","last_clean":"2024-09-13T08:37:11.017318+0000","last_became_active":"2024-09-13T08:37:10.021622+0000","last_became_peered":"2024-09-13T08:37:10.021622+0000","last_unstale":"2024-09-13T08:37:11.017318+0000","last_undegraded":"2024-09-13T08:37:11.017318+0000","last_fullsized":"2024-09-13T08:37:11.017318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596836+0000","last_change":"2024-09-13T08:37:10.026234+0000","last_active":"2024-09-13T08:37:13.596836+0000","last_peered":"2024-09-13T08:37:13.596836+0000","last_clean":"2024-09-13T08:37:13.596836+0000","last_became_active":"2024-09-13T08:37:10.025970+0000","last_became_peered":"2024-09-13T08:37:10.025970+0000","last_unstale":"2024-09-13T08:37:13.596836+0000","last_undegraded":"2024-09-13T08:37:13.596836+0000","last_fullsized":"2024-09-13T08:37:13.596836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:52.574964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021932+0000","last_change":"2024-09-13T08:37:10.028101+0000","last_active":"2024-09-13T08:37:11.021932+0000","last_peered":"2024-09-13T08:37:11.021932+0000","last_clean":"2024-09-13T08:37:11.021932+0000","last_became_active":"2024-09-13T08:37:10.021666+0000","last_became_peered":"2024-09-13T08:37:10.021666+0000","last_unstale":"2024-09-13T08:37:11.021932+0000","last_undegraded":"2024-09-13T08:37:11.021932+0000","last_fullsized":"2024-09-13T08:37:11.021932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:43.682175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723677+0000","last_change":"2024-09-13T08:37:10.020945+0000","last_active":"2024-09-13T08:37:12.723677+0000","last_peered":"2024-09-13T08:37:12.723677+0000","last_clean":"2024-09-13T08:37:12.723677+0000","last_became_active":"2024-09-13T08:37:10.020675+0000","last_became_peered":"2024-09-13T08:37:10.020675+0000","last_unstale":"2024-09-13T08:37:12.723677+0000","last_undegraded":"2024-09-13T08:37:12.723677+0000","last_fullsized":"2024-09-13T08:37:12.723677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:41.501115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328354+0000","last_change":"2024-09-13T08:37:10.036002+0000","last_active":"2024-09-13T08:37:11.328354+0000","last_peered":"2024-09-13T08:37:11.328354+0000","last_clean":"2024-09-13T08:37:11.328354+0000","last_became_active":"2024-09-13T08:37:10.035684+0000","last_became_peered":"2024-09-13T08:37:10.035684+0000","last_unstale":"2024-09-13T08:37:11.328354+0000","last_undegraded":"2024-09-13T08:37:11.328354+0000","last_fullsized":"2024-09-13T08:37:11.328354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.032065+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.031802+0000","last_became_peered":"2024-09-13T08:37:10.031802+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328937+0000","last_change":"2024-09-13T08:37:10.036040+0000","last_active":"2024-09-13T08:37:11.328937+0000","last_peered":"2024-09-13T08:37:11.328937+0000","last_clean":"2024-09-13T08:37:11.328937+0000","last_became_active":"2024-09-13T08:37:10.035698+0000","last_became_peered":"2024-09-13T08:37:10.035698+0000","last_unstale":"2024-09-13T08:37:11.328937+0000","last_undegraded":"2024-09-13T08:37:11.328937+0000","last_fullsized":"2024-09-13T08:37:11.328937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015904+0000","last_change":"2024-09-13T08:37:10.023312+0000","last_active":"2024-09-13T08:37:11.015904+0000","last_peered":"2024-09-13T08:37:11.015904+0000","last_clean":"2024-09-13T08:37:11.015904+0000","last_became_active":"2024-09-13T08:37:10.022193+0000","last_became_peered":"2024-09-13T08:37:10.022193+0000","last_unstale":"2024-09-13T08:37:11.015904+0000","last_undegraded":"2024-09-13T08:37:11.015904+0000","last_fullsized":"2024-09-13T08:37:11.015904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863802+0000","last_change":"2024-09-13T08:37:10.028354+0000","last_active":"2024-09-13T08:37:12.863802+0000","last_peered":"2024-09-13T08:37:12.863802+0000","last_clean":"2024-09-13T08:37:12.863802+0000","last_became_active":"2024-09-13T08:37:10.027989+0000","last_became_peered":"2024-09-13T08:37:10.027989+0000","last_unstale":"2024-09-13T08:37:12.863802+0000","last_undegraded":"2024-09-13T08:37:12.863802+0000","last_fullsized":"2024-09-13T08:37:12.863802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863758+0000","last_change":"2024-09-13T08:37:10.039029+0000","last_active":"2024-09-13T08:37:12.863758+0000","last_peered":"2024-09-13T08:37:12.863758+0000","last_clean":"2024-09-13T08:37:12.863758+0000","last_became_active":"2024-09-13T08:37:10.038689+0000","last_became_peered":"2024-09-13T08:37:10.038689+0000","last_unstale":"2024-09-13T08:37:12.863758+0000","last_undegraded":"2024-09-13T08:37:12.863758+0000","last_fullsized":"2024-09-13T08:37:12.863758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328458+0000","last_change":"2024-09-13T08:37:10.035713+0000","last_active":"2024-09-13T08:37:11.328458+0000","last_peered":"2024-09-13T08:37:11.328458+0000","last_clean":"2024-09-13T08:37:11.328458+0000","last_became_active":"2024-09-13T08:37:10.035292+0000","last_became_peered":"2024-09-13T08:37:10.035292+0000","last_unstale":"2024-09-13T08:37:11.328458+0000","last_undegraded":"2024-09-13T08:37:11.328458+0000","last_fullsized":"2024-09-13T08:37:11.328458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596170+0000","last_change":"2024-09-13T08:37:10.029548+0000","last_active":"2024-09-13T08:37:13.596170+0000","last_peered":"2024-09-13T08:37:13.596170+0000","last_clean":"2024-09-13T08:37:13.596170+0000","last_became_active":"2024-09-13T08:37:10.029292+0000","last_became_peered":"2024-09-13T08:37:10.029292+0000","last_unstale":"2024-09-13T08:37:13.596170+0000","last_undegraded":"2024-09-13T08:37:13.596170+0000","last_fullsized":"2024-09-13T08:37:13.596170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016762+0000","last_change":"2024-09-13T08:37:10.012936+0000","last_active":"2024-09-13T08:37:11.016762+0000","last_peered":"2024-09-13T08:37:11.016762+0000","last_clean":"2024-09-13T08:37:11.016762+0000","last_became_active":"2024-09-13T08:37:10.012582+0000","last_became_peered":"2024-09-13T08:37:10.012582+0000","last_unstale":"2024-09-13T08:37:11.016762+0000","last_undegraded":"2024-09-13T08:37:11.016762+0000","last_fullsized":"2024-09-13T08:37:11.016762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015064+0000","last_change":"2024-09-13T08:37:10.017195+0000","last_active":"2024-09-13T08:37:11.015064+0000","last_peered":"2024-09-13T08:37:11.015064+0000","last_clean":"2024-09-13T08:37:11.015064+0000","last_became_active":"2024-09-13T08:37:10.016877+0000","last_became_peered":"2024-09-13T08:37:10.016877+0000","last_unstale":"2024-09-13T08:37:11.015064+0000","last_undegraded":"2024-09-13T08:37:11.015064+0000","last_fullsized":"2024-09-13T08:37:11.015064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.474080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723910+0000","last_change":"2024-09-13T08:37:10.023947+0000","last_active":"2024-09-13T08:37:12.723910+0000","last_peered":"2024-09-13T08:37:12.723910+0000","last_clean":"2024-09-13T08:37:12.723910+0000","last_became_active":"2024-09-13T08:37:10.023659+0000","last_became_peered":"2024-09-13T08:37:10.023659+0000","last_unstale":"2024-09-13T08:37:12.723910+0000","last_undegraded":"2024-09-13T08:37:12.723910+0000","last_fullsized":"2024-09-13T08:37:12.723910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328520+0000","last_change":"2024-09-13T08:37:09.014843+0000","last_active":"2024-09-13T08:37:11.328520+0000","last_peered":"2024-09-13T08:37:11.328520+0000","last_clean":"2024-09-13T08:37:11.328520+0000","last_became_active":"2024-09-13T08:37:09.014454+0000","last_became_peered":"2024-09-13T08:37:09.014454+0000","last_unstale":"2024-09-13T08:37:11.328520+0000","last_undegraded":"2024-09-13T08:37:11.328520+0000","last_fullsized":"2024-09-13T08:37:11.328520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596441+0000","last_change":"2024-09-13T08:37:09.024672+0000","last_active":"2024-09-13T08:37:13.596441+0000","last_peered":"2024-09-13T08:37:13.596441+0000","last_clean":"2024-09-13T08:37:13.596441+0000","last_became_active":"2024-09-13T08:37:09.024363+0000","last_became_peered":"2024-09-13T08:37:09.024363+0000","last_unstale":"2024-09-13T08:37:13.596441+0000","last_undegraded":"2024-09-13T08:37:13.596441+0000","last_fullsized":"2024-09-13T08:37:13.596441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:55.612953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863222+0000","last_change":"2024-09-13T08:37:09.014986+0000","last_active":"2024-09-13T08:37:12.863222+0000","last_peered":"2024-09-13T08:37:12.863222+0000","last_clean":"2024-09-13T08:37:12.863222+0000","last_became_active":"2024-09-13T08:37:09.014658+0000","last_became_peered":"2024-09-13T08:37:09.014658+0000","last_unstale":"2024-09-13T08:37:12.863222+0000","last_undegraded":"2024-09-13T08:37:12.863222+0000","last_fullsized":"2024-09-13T08:37:12.863222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328628+0000","last_change":"2024-09-13T08:37:10.030653+0000","last_active":"2024-09-13T08:37:11.328628+0000","last_peered":"2024-09-13T08:37:11.328628+0000","last_clean":"2024-09-13T08:37:11.328628+0000","last_became_active":"2024-09-13T08:37:10.030383+0000","last_became_peered":"2024-09-13T08:37:10.030383+0000","last_unstale":"2024-09-13T08:37:11.328628+0000","last_undegraded":"2024-09-13T08:37:11.328628+0000","last_fullsized":"2024-09-13T08:37:11.328628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596255+0000","last_change":"2024-09-13T08:37:09.023571+0000","last_active":"2024-09-13T08:37:13.596255+0000","last_peered":"2024-09-13T08:37:13.596255+0000","last_clean":"2024-09-13T08:37:13.596255+0000","last_became_active":"2024-09-13T08:37:09.023048+0000","last_became_peered":"2024-09-13T08:37:09.023048+0000","last_unstale":"2024-09-13T08:37:13.596255+0000","last_undegraded":"2024-09-13T08:37:13.596255+0000","last_fullsized":"2024-09-13T08:37:13.596255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:14.567695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723890+0000","last_change":"2024-09-13T08:37:09.010512+0000","last_active":"2024-09-13T08:37:12.723890+0000","last_peered":"2024-09-13T08:37:12.723890+0000","last_clean":"2024-09-13T08:37:12.723890+0000","last_became_active":"2024-09-13T08:37:09.010229+0000","last_became_peered":"2024-09-13T08:37:09.010229+0000","last_unstale":"2024-09-13T08:37:12.723890+0000","last_undegraded":"2024-09-13T08:37:12.723890+0000","last_fullsized":"2024-09-13T08:37:12.723890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015969+0000","last_change":"2024-09-13T08:37:10.020061+0000","last_active":"2024-09-13T08:37:11.015969+0000","last_peered":"2024-09-13T08:37:11.015969+0000","last_clean":"2024-09-13T08:37:11.015969+0000","last_became_active":"2024-09-13T08:37:10.019663+0000","last_became_peered":"2024-09-13T08:37:10.019663+0000","last_unstale":"2024-09-13T08:37:11.015969+0000","last_undegraded":"2024-09-13T08:37:11.015969+0000","last_fullsized":"2024-09-13T08:37:11.015969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:48.021040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021832+0000","last_change":"2024-09-13T08:37:09.011936+0000","last_active":"2024-09-13T08:37:11.021832+0000","last_peered":"2024-09-13T08:37:11.021832+0000","last_clean":"2024-09-13T08:37:11.021832+0000","last_became_active":"2024-09-13T08:37:09.011670+0000","last_became_peered":"2024-09-13T08:37:09.011670+0000","last_unstale":"2024-09-13T08:37:11.021832+0000","last_undegraded":"2024-09-13T08:37:11.021832+0000","last_fullsized":"2024-09-13T08:37:11.021832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:31.341815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329184+0000","last_change":"2024-09-13T08:37:10.027709+0000","last_active":"2024-09-13T08:37:11.329184+0000","last_peered":"2024-09-13T08:37:11.329184+0000","last_clean":"2024-09-13T08:37:11.329184+0000","last_became_active":"2024-09-13T08:37:10.027410+0000","last_became_peered":"2024-09-13T08:37:10.027410+0000","last_unstale":"2024-09-13T08:37:11.329184+0000","last_undegraded":"2024-09-13T08:37:11.329184+0000","last_fullsized":"2024-09-13T08:37:11.329184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:25.100540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017898+0000","last_change":"2024-09-13T08:37:09.003083+0000","last_active":"2024-09-13T08:37:11.017898+0000","last_peered":"2024-09-13T08:37:11.017898+0000","last_clean":"2024-09-13T08:37:11.017898+0000","last_became_active":"2024-09-13T08:37:09.002697+0000","last_became_peered":"2024-09-13T08:37:09.002697+0000","last_unstale":"2024-09-13T08:37:11.017898+0000","last_undegraded":"2024-09-13T08:37:11.017898+0000","last_fullsized":"2024-09-13T08:37:11.017898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:56.231941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596339+0000","last_change":"2024-09-13T08:37:10.021360+0000","last_active":"2024-09-13T08:37:13.596339+0000","last_peered":"2024-09-13T08:37:13.596339+0000","last_clean":"2024-09-13T08:37:13.596339+0000","last_became_active":"2024-09-13T08:37:10.021080+0000","last_became_peered":"2024-09-13T08:37:10.021080+0000","last_unstale":"2024-09-13T08:37:13.596339+0000","last_undegraded":"2024-09-13T08:37:13.596339+0000","last_fullsized":"2024-09-13T08:37:13.596339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:32.591794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015415+0000","last_change":"2024-09-13T08:37:09.014879+0000","last_active":"2024-09-13T08:37:11.015415+0000","last_peered":"2024-09-13T08:37:11.015415+0000","last_clean":"2024-09-13T08:37:11.015415+0000","last_became_active":"2024-09-13T08:37:09.014572+0000","last_became_peered":"2024-09-13T08:37:09.014572+0000","last_unstale":"2024-09-13T08:37:11.015415+0000","last_undegraded":"2024-09-13T08:37:11.015415+0000","last_fullsized":"2024-09-13T08:37:11.015415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:14.033544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022331+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:11.022331+0000","last_peered":"2024-09-13T08:37:11.022331+0000","last_clean":"2024-09-13T08:37:11.022331+0000","last_became_active":"2024-09-13T08:37:10.026781+0000","last_became_peered":"2024-09-13T08:37:10.026781+0000","last_unstale":"2024-09-13T08:37:11.022331+0000","last_undegraded":"2024-09-13T08:37:11.022331+0000","last_fullsized":"2024-09-13T08:37:11.022331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017361+0000","last_change":"2024-09-13T08:37:09.015199+0000","last_active":"2024-09-13T08:37:11.017361+0000","last_peered":"2024-09-13T08:37:11.017361+0000","last_clean":"2024-09-13T08:37:11.017361+0000","last_became_active":"2024-09-13T08:37:09.014848+0000","last_became_peered":"2024-09-13T08:37:09.014848+0000","last_unstale":"2024-09-13T08:37:11.017361+0000","last_undegraded":"2024-09-13T08:37:11.017361+0000","last_fullsized":"2024-09-13T08:37:11.017361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:01.967164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863695+0000","last_change":"2024-09-13T08:37:10.027034+0000","last_active":"2024-09-13T08:37:12.863695+0000","last_peered":"2024-09-13T08:37:12.863695+0000","last_clean":"2024-09-13T08:37:12.863695+0000","last_became_active":"2024-09-13T08:37:10.026723+0000","last_became_peered":"2024-09-13T08:37:10.026723+0000","last_unstale":"2024-09-13T08:37:12.863695+0000","last_undegraded":"2024-09-13T08:37:12.863695+0000","last_fullsized":"2024-09-13T08:37:12.863695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596545+0000","last_change":"2024-09-13T08:37:09.021683+0000","last_active":"2024-09-13T08:37:13.596545+0000","last_peered":"2024-09-13T08:37:13.596545+0000","last_clean":"2024-09-13T08:37:13.596545+0000","last_became_active":"2024-09-13T08:37:09.019630+0000","last_became_peered":"2024-09-13T08:37:09.019630+0000","last_unstale":"2024-09-13T08:37:13.596545+0000","last_undegraded":"2024-09-13T08:37:13.596545+0000","last_fullsized":"2024-09-13T08:37:13.596545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:47.015443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596308+0000","last_change":"2024-09-13T08:37:10.032446+0000","last_active":"2024-09-13T08:37:13.596308+0000","last_peered":"2024-09-13T08:37:13.596308+0000","last_clean":"2024-09-13T08:37:13.596308+0000","last_became_active":"2024-09-13T08:37:10.031448+0000","last_became_peered":"2024-09-13T08:37:10.031448+0000","last_unstale":"2024-09-13T08:37:13.596308+0000","last_undegraded":"2024-09-13T08:37:13.596308+0000","last_fullsized":"2024-09-13T08:37:13.596308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:38:36.201616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021502+0000","last_change":"2024-09-13T08:37:09.025565+0000","last_active":"2024-09-13T08:37:11.021502+0000","last_peered":"2024-09-13T08:37:11.021502+0000","last_clean":"2024-09-13T08:37:11.021502+0000","last_became_active":"2024-09-13T08:37:09.025242+0000","last_became_peered":"2024-09-13T08:37:09.025242+0000","last_unstale":"2024-09-13T08:37:11.021502+0000","last_undegraded":"2024-09-13T08:37:11.021502+0000","last_fullsized":"2024-09-13T08:37:11.021502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:08.638798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723679+0000","last_change":"2024-09-13T08:37:10.030817+0000","last_active":"2024-09-13T08:37:12.723679+0000","last_peered":"2024-09-13T08:37:12.723679+0000","last_clean":"2024-09-13T08:37:12.723679+0000","last_became_active":"2024-09-13T08:37:10.030554+0000","last_became_peered":"2024-09-13T08:37:10.030554+0000","last_unstale":"2024-09-13T08:37:12.723679+0000","last_undegraded":"2024-09-13T08:37:12.723679+0000","last_fullsized":"2024-09-13T08:37:12.723679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596103+0000","last_change":"2024-09-13T08:37:10.023766+0000","last_active":"2024-09-13T08:37:13.596103+0000","last_peered":"2024-09-13T08:37:13.596103+0000","last_clean":"2024-09-13T08:37:13.596103+0000","last_became_active":"2024-09-13T08:37:10.022889+0000","last_became_peered":"2024-09-13T08:37:10.022889+0000","last_unstale":"2024-09-13T08:37:13.596103+0000","last_undegraded":"2024-09-13T08:37:13.596103+0000","last_fullsized":"2024-09-13T08:37:13.596103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022117+0000","last_change":"2024-09-13T08:37:09.023974+0000","last_active":"2024-09-13T08:37:11.022117+0000","last_peered":"2024-09-13T08:37:11.022117+0000","last_clean":"2024-09-13T08:37:11.022117+0000","last_became_active":"2024-09-13T08:37:09.023471+0000","last_became_peered":"2024-09-13T08:37:09.023471+0000","last_unstale":"2024-09-13T08:37:11.022117+0000","last_undegraded":"2024-09-13T08:37:11.022117+0000","last_fullsized":"2024-09-13T08:37:11.022117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:38.190525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016062+0000","last_change":"2024-09-13T08:37:10.023008+0000","last_active":"2024-09-13T08:37:11.016062+0000","last_peered":"2024-09-13T08:37:11.016062+0000","last_clean":"2024-09-13T08:37:11.016062+0000","last_became_active":"2024-09-13T08:37:10.021695+0000","last_became_peered":"2024-09-13T08:37:10.021695+0000","last_unstale":"2024-09-13T08:37:11.016062+0000","last_undegraded":"2024-09-13T08:37:11.016062+0000","last_fullsized":"2024-09-13T08:37:11.016062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:43.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022182+0000","last_change":"2024-09-13T08:37:10.031929+0000","last_active":"2024-09-13T08:37:11.022182+0000","last_peered":"2024-09-13T08:37:11.022182+0000","last_clean":"2024-09-13T08:37:11.022182+0000","last_became_active":"2024-09-13T08:37:10.031598+0000","last_became_peered":"2024-09-13T08:37:10.031598+0000","last_unstale":"2024-09-13T08:37:11.022182+0000","last_undegraded":"2024-09-13T08:37:11.022182+0000","last_fullsized":"2024-09-13T08:37:11.022182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328623+0000","last_change":"2024-09-13T08:37:09.011613+0000","last_active":"2024-09-13T08:37:11.328623+0000","last_peered":"2024-09-13T08:37:11.328623+0000","last_clean":"2024-09-13T08:37:11.328623+0000","last_became_active":"2024-09-13T08:37:09.011071+0000","last_became_peered":"2024-09-13T08:37:09.011071+0000","last_unstale":"2024-09-13T08:37:11.328623+0000","last_undegraded":"2024-09-13T08:37:11.328623+0000","last_fullsized":"2024-09-13T08:37:11.328623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022072+0000","last_change":"2024-09-13T08:37:10.028420+0000","last_active":"2024-09-13T08:37:11.022072+0000","last_peered":"2024-09-13T08:37:11.022072+0000","last_clean":"2024-09-13T08:37:11.022072+0000","last_became_active":"2024-09-13T08:37:10.028166+0000","last_became_peered":"2024-09-13T08:37:10.028166+0000","last_unstale":"2024-09-13T08:37:11.022072+0000","last_undegraded":"2024-09-13T08:37:11.022072+0000","last_fullsized":"2024-09-13T08:37:11.022072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:58.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863837+0000","last_change":"2024-09-13T08:37:09.015072+0000","last_active":"2024-09-13T08:37:12.863837+0000","last_peered":"2024-09-13T08:37:12.863837+0000","last_clean":"2024-09-13T08:37:12.863837+0000","last_became_active":"2024-09-13T08:37:09.014597+0000","last_became_peered":"2024-09-13T08:37:09.014597+0000","last_unstale":"2024-09-13T08:37:12.863837+0000","last_undegraded":"2024-09-13T08:37:12.863837+0000","last_fullsized":"2024-09-13T08:37:12.863837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596652+0000","last_change":"2024-09-13T08:37:10.018219+0000","last_active":"2024-09-13T08:37:13.596652+0000","last_peered":"2024-09-13T08:37:13.596652+0000","last_clean":"2024-09-13T08:37:13.596652+0000","last_became_active":"2024-09-13T08:37:10.017946+0000","last_became_peered":"2024-09-13T08:37:10.017946+0000","last_unstale":"2024-09-13T08:37:13.596652+0000","last_undegraded":"2024-09-13T08:37:13.596652+0000","last_fullsized":"2024-09-13T08:37:13.596652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:12.941476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723548+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:12.723548+0000","last_peered":"2024-09-13T08:37:12.723548+0000","last_clean":"2024-09-13T08:37:12.723548+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:12.723548+0000","last_undegraded":"2024-09-13T08:37:12.723548+0000","last_fullsized":"2024-09-13T08:37:12.723548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596411+0000","last_change":"2024-09-13T08:37:09.021195+0000","last_active":"2024-09-13T08:37:13.596411+0000","last_peered":"2024-09-13T08:37:13.596411+0000","last_clean":"2024-09-13T08:37:13.596411+0000","last_became_active":"2024-09-13T08:37:09.019692+0000","last_became_peered":"2024-09-13T08:37:09.019692+0000","last_unstale":"2024-09-13T08:37:13.596411+0000","last_undegraded":"2024-09-13T08:37:13.596411+0000","last_fullsized":"2024-09-13T08:37:13.596411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.430584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596362+0000","last_change":"2024-09-13T08:37:09.019756+0000","last_active":"2024-09-13T08:37:13.596362+0000","last_peered":"2024-09-13T08:37:13.596362+0000","last_clean":"2024-09-13T08:37:13.596362+0000","last_became_active":"2024-09-13T08:37:09.019375+0000","last_became_peered":"2024-09-13T08:37:09.019375+0000","last_unstale":"2024-09-13T08:37:13.596362+0000","last_undegraded":"2024-09-13T08:37:13.596362+0000","last_fullsized":"2024-09-13T08:37:13.596362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:40.495518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863575+0000","last_change":"2024-09-13T08:37:10.020491+0000","last_active":"2024-09-13T08:37:12.863575+0000","last_peered":"2024-09-13T08:37:12.863575+0000","last_clean":"2024-09-13T08:37:12.863575+0000","last_became_active":"2024-09-13T08:37:10.020271+0000","last_became_peered":"2024-09-13T08:37:10.020271+0000","last_unstale":"2024-09-13T08:37:12.863575+0000","last_undegraded":"2024-09-13T08:37:12.863575+0000","last_fullsized":"2024-09-13T08:37:12.863575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:29.393985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016620+0000","last_change":"2024-09-13T08:37:09.015889+0000","last_active":"2024-09-13T08:37:11.016620+0000","last_peered":"2024-09-13T08:37:11.016620+0000","last_clean":"2024-09-13T08:37:11.016620+0000","last_became_active":"2024-09-13T08:37:09.015587+0000","last_became_peered":"2024-09-13T08:37:09.015587+0000","last_unstale":"2024-09-13T08:37:11.016620+0000","last_undegraded":"2024-09-13T08:37:11.016620+0000","last_fullsized":"2024-09-13T08:37:11.016620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328457+0000","last_change":"2024-09-13T08:37:10.036912+0000","last_active":"2024-09-13T08:37:11.328457+0000","last_peered":"2024-09-13T08:37:11.328457+0000","last_clean":"2024-09-13T08:37:11.328457+0000","last_became_active":"2024-09-13T08:37:10.036665+0000","last_became_peered":"2024-09-13T08:37:10.036665+0000","last_unstale":"2024-09-13T08:37:11.328457+0000","last_undegraded":"2024-09-13T08:37:11.328457+0000","last_fullsized":"2024-09-13T08:37:11.328457+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596672+0000","last_change":"2024-09-13T08:37:09.023866+0000","last_active":"2024-09-13T08:37:13.596672+0000","last_peered":"2024-09-13T08:37:13.596672+0000","last_clean":"2024-09-13T08:37:13.596672+0000","last_became_active":"2024-09-13T08:37:09.021825+0000","last_became_peered":"2024-09-13T08:37:09.021825+0000","last_unstale":"2024-09-13T08:37:13.596672+0000","last_undegraded":"2024-09-13T08:37:13.596672+0000","last_fullsized":"2024-09-13T08:37:13.596672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:42.779836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022032+0000","last_change":"2024-09-13T08:37:10.027315+0000","last_active":"2024-09-13T08:37:11.022032+0000","last_peered":"2024-09-13T08:37:11.022032+0000","last_clean":"2024-09-13T08:37:11.022032+0000","last_became_active":"2024-09-13T08:37:10.027018+0000","last_became_peered":"2024-09-13T08:37:10.027018+0000","last_unstale":"2024-09-13T08:37:11.022032+0000","last_undegraded":"2024-09-13T08:37:11.022032+0000","last_fullsized":"2024-09-13T08:37:11.022032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328769+0000","last_change":"2024-09-13T08:37:09.018684+0000","last_active":"2024-09-13T08:37:11.328769+0000","last_peered":"2024-09-13T08:37:11.328769+0000","last_clean":"2024-09-13T08:37:11.328769+0000","last_became_active":"2024-09-13T08:37:09.016570+0000","last_became_peered":"2024-09-13T08:37:09.016570+0000","last_unstale":"2024-09-13T08:37:11.328769+0000","last_undegraded":"2024-09-13T08:37:11.328769+0000","last_fullsized":"2024-09-13T08:37:11.328769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015606+0000","last_change":"2024-09-13T08:37:10.018230+0000","last_active":"2024-09-13T08:37:11.015606+0000","last_peered":"2024-09-13T08:37:11.015606+0000","last_clean":"2024-09-13T08:37:11.015606+0000","last_became_active":"2024-09-13T08:37:10.017427+0000","last_became_peered":"2024-09-13T08:37:10.017427+0000","last_unstale":"2024-09-13T08:37:11.015606+0000","last_undegraded":"2024-09-13T08:37:11.015606+0000","last_fullsized":"2024-09-13T08:37:11.015606+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596769+0000","last_change":"2024-09-13T08:37:09.018807+0000","last_active":"2024-09-13T08:37:13.596769+0000","last_peered":"2024-09-13T08:37:13.596769+0000","last_clean":"2024-09-13T08:37:13.596769+0000","last_became_active":"2024-09-13T08:37:09.018298+0000","last_became_peered":"2024-09-13T08:37:09.018298+0000","last_unstale":"2024-09-13T08:37:13.596769+0000","last_undegraded":"2024-09-13T08:37:13.596769+0000","last_fullsized":"2024-09-13T08:37:13.596769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.295316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863880+0000","last_change":"2024-09-13T08:37:10.028730+0000","last_active":"2024-09-13T08:37:12.863880+0000","last_peered":"2024-09-13T08:37:12.863880+0000","last_clean":"2024-09-13T08:37:12.863880+0000","last_became_active":"2024-09-13T08:37:10.028481+0000","last_became_peered":"2024-09-13T08:37:10.028481+0000","last_unstale":"2024-09-13T08:37:12.863880+0000","last_undegraded":"2024-09-13T08:37:12.863880+0000","last_fullsized":"2024-09-13T08:37:12.863880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596783+0000","last_change":"2024-09-13T08:37:09.024256+0000","last_active":"2024-09-13T08:37:13.596783+0000","last_peered":"2024-09-13T08:37:13.596783+0000","last_clean":"2024-09-13T08:37:13.596783+0000","last_became_active":"2024-09-13T08:37:09.023967+0000","last_became_peered":"2024-09-13T08:37:09.023967+0000","last_unstale":"2024-09-13T08:37:13.596783+0000","last_undegraded":"2024-09-13T08:37:13.596783+0000","last_fullsized":"2024-09-13T08:37:13.596783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:41.577735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021357+0000","last_change":"2024-09-13T08:37:09.011069+0000","last_active":"2024-09-13T08:37:11.021357+0000","last_peered":"2024-09-13T08:37:11.021357+0000","last_clean":"2024-09-13T08:37:11.021357+0000","last_became_active":"2024-09-13T08:37:09.010687+0000","last_became_peered":"2024-09-13T08:37:09.010687+0000","last_unstale":"2024-09-13T08:37:11.021357+0000","last_undegraded":"2024-09-13T08:37:11.021357+0000","last_fullsized":"2024-09-13T08:37:11.021357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596198+0000","last_change":"2024-09-13T08:37:10.026288+0000","last_active":"2024-09-13T08:37:13.596198+0000","last_peered":"2024-09-13T08:37:13.596198+0000","last_clean":"2024-09-13T08:37:13.596198+0000","last_became_active":"2024-09-13T08:37:10.026029+0000","last_became_peered":"2024-09-13T08:37:10.026029+0000","last_unstale":"2024-09-13T08:37:13.596198+0000","last_undegraded":"2024-09-13T08:37:13.596198+0000","last_fullsized":"2024-09-13T08:37:13.596198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863577+0000","last_change":"2024-09-13T08:37:09.019714+0000","last_active":"2024-09-13T08:37:12.863577+0000","last_peered":"2024-09-13T08:37:12.863577+0000","last_clean":"2024-09-13T08:37:12.863577+0000","last_became_active":"2024-09-13T08:37:09.018957+0000","last_became_peered":"2024-09-13T08:37:09.018957+0000","last_unstale":"2024-09-13T08:37:12.863577+0000","last_undegraded":"2024-09-13T08:37:12.863577+0000","last_fullsized":"2024-09-13T08:37:12.863577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017312+0000","last_change":"2024-09-13T08:37:10.026722+0000","last_active":"2024-09-13T08:37:11.017312+0000","last_peered":"2024-09-13T08:37:11.017312+0000","last_clean":"2024-09-13T08:37:11.017312+0000","last_became_active":"2024-09-13T08:37:10.025267+0000","last_became_peered":"2024-09-13T08:37:10.025267+0000","last_unstale":"2024-09-13T08:37:11.017312+0000","last_undegraded":"2024-09-13T08:37:11.017312+0000","last_fullsized":"2024-09-13T08:37:11.017312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596295+0000","last_change":"2024-09-13T08:37:09.022982+0000","last_active":"2024-09-13T08:37:13.596295+0000","last_peered":"2024-09-13T08:37:13.596295+0000","last_clean":"2024-09-13T08:37:13.596295+0000","last_became_active":"2024-09-13T08:37:09.021746+0000","last_became_peered":"2024-09-13T08:37:09.021746+0000","last_unstale":"2024-09-13T08:37:13.596295+0000","last_undegraded":"2024-09-13T08:37:13.596295+0000","last_fullsized":"2024-09-13T08:37:13.596295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.468482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596487+0000","last_change":"2024-09-13T08:37:10.016999+0000","last_active":"2024-09-13T08:37:13.596487+0000","last_peered":"2024-09-13T08:37:13.596487+0000","last_clean":"2024-09-13T08:37:13.596487+0000","last_became_active":"2024-09-13T08:37:10.016738+0000","last_became_peered":"2024-09-13T08:37:10.016738+0000","last_unstale":"2024-09-13T08:37:13.596487+0000","last_undegraded":"2024-09-13T08:37:13.596487+0000","last_fullsized":"2024-09-13T08:37:13.596487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:16:16.734699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596582+0000","last_change":"2024-09-13T08:37:10.026898+0000","last_active":"2024-09-13T08:37:13.596582+0000","last_peered":"2024-09-13T08:37:13.596582+0000","last_clean":"2024-09-13T08:37:13.596582+0000","last_became_active":"2024-09-13T08:37:10.026602+0000","last_became_peered":"2024-09-13T08:37:10.026602+0000","last_unstale":"2024-09-13T08:37:13.596582+0000","last_undegraded":"2024-09-13T08:37:13.596582+0000","last_fullsized":"2024-09-13T08:37:13.596582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:42.330227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022006+0000","last_change":"2024-09-13T08:37:09.011138+0000","last_active":"2024-09-13T08:37:11.022006+0000","last_peered":"2024-09-13T08:37:11.022006+0000","last_clean":"2024-09-13T08:37:11.022006+0000","last_became_active":"2024-09-13T08:37:09.010863+0000","last_became_peered":"2024-09-13T08:37:09.010863+0000","last_unstale":"2024-09-13T08:37:11.022006+0000","last_undegraded":"2024-09-13T08:37:11.022006+0000","last_fullsized":"2024-09-13T08:37:11.022006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:14.227052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723804+0000","last_change":"2024-09-13T08:37:10.029092+0000","last_active":"2024-09-13T08:37:12.723804+0000","last_peered":"2024-09-13T08:37:12.723804+0000","last_clean":"2024-09-13T08:37:12.723804+0000","last_became_active":"2024-09-13T08:37:10.028796+0000","last_became_peered":"2024-09-13T08:37:10.028796+0000","last_unstale":"2024-09-13T08:37:12.723804+0000","last_undegraded":"2024-09-13T08:37:12.723804+0000","last_fullsized":"2024-09-13T08:37:12.723804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022178+0000","last_change":"2024-09-13T08:37:09.013579+0000","last_active":"2024-09-13T08:37:11.022178+0000","last_peered":"2024-09-13T08:37:11.022178+0000","last_clean":"2024-09-13T08:37:11.022178+0000","last_became_active":"2024-09-13T08:37:09.013327+0000","last_became_peered":"2024-09-13T08:37:09.013327+0000","last_unstale":"2024-09-13T08:37:11.022178+0000","last_undegraded":"2024-09-13T08:37:11.022178+0000","last_fullsized":"2024-09-13T08:37:11.022178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:34.207907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328992+0000","last_change":"2024-09-13T08:37:10.032358+0000","last_active":"2024-09-13T08:37:11.328992+0000","last_peered":"2024-09-13T08:37:11.328992+0000","last_clean":"2024-09-13T08:37:11.328992+0000","last_became_active":"2024-09-13T08:37:10.032047+0000","last_became_peered":"2024-09-13T08:37:10.032047+0000","last_unstale":"2024-09-13T08:37:11.328992+0000","last_undegraded":"2024-09-13T08:37:11.328992+0000","last_fullsized":"2024-09-13T08:37:11.328992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:19.283020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329066+0000","last_change":"2024-09-13T08:37:09.015906+0000","last_active":"2024-09-13T08:37:11.329066+0000","last_peered":"2024-09-13T08:37:11.329066+0000","last_clean":"2024-09-13T08:37:11.329066+0000","last_became_active":"2024-09-13T08:37:09.015500+0000","last_became_peered":"2024-09-13T08:37:09.015500+0000","last_unstale":"2024-09-13T08:37:11.329066+0000","last_undegraded":"2024-09-13T08:37:11.329066+0000","last_fullsized":"2024-09-13T08:37:11.329066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:51.846602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329023+0000","last_change":"2024-09-13T08:37:10.043965+0000","last_active":"2024-09-13T08:37:11.329023+0000","last_peered":"2024-09-13T08:37:11.329023+0000","last_clean":"2024-09-13T08:37:11.329023+0000","last_became_active":"2024-09-13T08:37:10.041509+0000","last_became_peered":"2024-09-13T08:37:10.041509+0000","last_unstale":"2024-09-13T08:37:11.329023+0000","last_undegraded":"2024-09-13T08:37:11.329023+0000","last_fullsized":"2024-09-13T08:37:11.329023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":241664,"data_stored":81079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":80955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":835584,"data_stored":671415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":81537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":245760,"data_stored":82567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":671393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":81011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:19.413 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:37:19.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T08:37:19.414 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 -- 172.21.15.31:0/781115813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f808815e160 msgr2=0x7f80880d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 --2- 172.21.15.31:0/781115813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f808815e160 0x7f80880d0790 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8078009890 tx=0x7f807802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 -- 172.21.15.31:0/781115813 shutdown_connections 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 --2- 172.21.15.31:0/781115813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f808815e160 0x7f80880d0790 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 -- 172.21.15.31:0/781115813 >> 172.21.15.31:0/781115813 conn(0x7f80880d3f50 msgr2=0x7f80880d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 -- 172.21.15.31:0/781115813 shutdown_connections 2024-09-13T08:37:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.608+0000 7f808de00640 1 -- 172.21.15.31:0/781115813 wait complete. 2024-09-13T08:37:19.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.609+0000 7f808de00640 1 Processor -- start 2024-09-13T08:37:19.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 -- start start 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80880cfa70 0x7f80880b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f80880d0cd0 0x7f80880a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f80880d14a0 con 0x7f808815e160 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80880d1320 con 0x7f80880cfa70 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.610+0000 7f808de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f80880d1620 con 0x7f80880d0cd0 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34432/0 (socket says 172.21.15.31:34432) 2024-09-13T08:37:19.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 -- 172.21.15.31:0/2190150702 learned_addr learned my addr 172.21.15.31:0/2190150702 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808ca00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80880cfa70 0x7f80880b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f807fe00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f80880d0cd0 0x7f80880a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 -- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f80880d0cd0 msgr2=0x7f80880a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f80880d0cd0 0x7f80880a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 -- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80880cfa70 msgr2=0x7f80880b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80880cfa70 0x7f80880b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.611+0000 7f808d400640 1 -- 172.21.15.31:0/2190150702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80880a1c20 con 0x7f808815e160 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f808d400640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f808000c7a0 tx=0x7f808000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080003b70 con 0x7f808815e160 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8080003d10 con 0x7f808815e160 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808000aab0 con 0x7f808815e160 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80881533b0 con 0x7f808815e160 2024-09-13T08:37:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.612+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8088153970 con 0x7f808815e160 2024-09-13T08:37:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.613+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80880ca5b0 con 0x7f808815e160 2024-09-13T08:37:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.614+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f808000ac50 con 0x7f808815e160 2024-09-13T08:37:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.615+0000 7f8085600640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 0x7f806407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.615+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f808009a820 con 0x7f808815e160 2024-09-13T08:37:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.615+0000 7f808ca00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 0x7f806407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.616+0000 7f808ca00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 0x7f806407ad10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8078034000 tx=0x7f8078004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.619+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8080062e00 con 0x7f808815e160 2024-09-13T08:37:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.815+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f80880760f0 con 0x7f808815e160 2024-09-13T08:37:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.816+0000 7f8085600640 1 -- 172.21.15.31:0/2190150702 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7f8080062700 con 0x7f808815e160 2024-09-13T08:37:19.817 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 msgr2=0x7f806407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 0x7f806407ad10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8078034000 tx=0x7f8078004730 comp rx=0 tx=0).stop 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 msgr2=0x7f80880cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f808000c7a0 tx=0x7f808000cc60 comp rx=0 tx=0).stop 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 shutdown_connections 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808815e160 0x7f80880cc5f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8064078850 0x7f806407ad10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f80880d0cd0 0x7f80880a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 --2- 172.21.15.31:0/2190150702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80880cfa70 0x7f80880b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 >> 172.21.15.31:0/2190150702 conn(0x7f80880d3f50 msgr2=0x7f80880b8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:19.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 shutdown_connections 2024-09-13T08:37:19.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.818+0000 7f808de00640 1 -- 172.21.15.31:0/2190150702 wait complete. 2024-09-13T08:37:19.835 INFO:teuthology.orchestra.run.smithi031.stdout:[{"version":27,"timestamp":"2024-09-13T08:36:57.734514+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"3636.621475"}]}] 2024-09-13T08:37:19.835 INFO:tasks.ceph_manager:config epoch is 27 2024-09-13T08:37:19.835 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T08:37:19.835 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T08:37:19.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T08:37:19.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 -- 172.21.15.31:0/1855340447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815af30 msgr2=0x7fe9480cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 --2- 172.21.15.31:0/1855340447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815af30 0x7fe9480cf280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe938009830 tx=0x7fe93802a510 comp rx=0 tx=0).stop 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 -- 172.21.15.31:0/1855340447 shutdown_connections 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 --2- 172.21.15.31:0/1855340447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815b310 0x7fe9480ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 --2- 172.21.15.31:0/1855340447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815af30 0x7fe9480cf280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 -- 172.21.15.31:0/1855340447 >> 172.21.15.31:0/1855340447 conn(0x7fe9480cc910 msgr2=0x7fe9480cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 -- 172.21.15.31:0/1855340447 shutdown_connections 2024-09-13T08:37:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.994+0000 7fe94d200640 1 -- 172.21.15.31:0/1855340447 wait complete. 2024-09-13T08:37:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.996+0000 7fe94d200640 1 Processor -- start 2024-09-13T08:37:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.996+0000 7fe94d200640 1 -- start start 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815af30 0x7fe9480b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815b310 0x7fe9480b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9480caba0 con 0x7fe948072c90 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9480caa20 con 0x7fe94815b310 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe94d200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe9480cad20 con 0x7fe94815af30 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.997+0000 7fe946800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34444/0 (socket says 172.21.15.31:34444) 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe945e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815af30 0x7fe9480b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe93d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815b310 0x7fe9480b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 -- 172.21.15.31:0/3040279810 learned_addr learned my addr 172.21.15.31:0/3040279810 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 -- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815af30 msgr2=0x7fe9480b2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815af30 0x7fe9480b2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 -- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815b310 msgr2=0x7fe9480b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815b310 0x7fe9480b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 -- 172.21.15.31:0/3040279810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9480b54d0 con 0x7fe948072c90 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe946800640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe94000cae0 tx=0x7fe94000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.998+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940016020 con 0x7fe948072c90 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.999+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe94000ad30 con 0x7fe948072c90 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.999+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9480b5770 con 0x7fe948072c90 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.999+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940012680 con 0x7fe948072c90 2024-09-13T08:37:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:19.999+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe94809be70 con 0x7fe948072c90 2024-09-13T08:37:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.000+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9480ad680 con 0x7fe948072c90 2024-09-13T08:37:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.001+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe940019020 con 0x7fe948072c90 2024-09-13T08:37:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.002+0000 7fe93ea00640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 0x7fe92407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.002+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe94009aaf0 con 0x7fe948072c90 2024-09-13T08:37:20.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.002+0000 7fe945e00640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 0x7fe92407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.003+0000 7fe945e00640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 0x7fe92407ad10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe938034000 tx=0x7fe938004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.006+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe940020030 con 0x7fe948072c90 2024-09-13T08:37:20.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.256+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe9480b5e10 con 0x7fe948072c90 2024-09-13T08:37:20.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.259+0000 7fe93ea00640 1 -- 172.21.15.31:0/3040279810 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+194338 (secure 0 0 0) 0x7fe940062b00 con 0x7fe948072c90 2024-09-13T08:37:20.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:20.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 msgr2=0x7fe92407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 0x7fe92407ad10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe938034000 tx=0x7fe938004730 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 msgr2=0x7fe948073120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe94000cae0 tx=0x7fe94000a060 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 shutdown_connections 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe948072c90 0x7fe948073120 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94815b310 0x7fe9480b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe924078850 0x7fe92407ad10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 --2- 172.21.15.31:0/3040279810 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe94815af30 0x7fe9480b2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 >> 172.21.15.31:0/3040279810 conn(0x7fe9480cc910 msgr2=0x7fe9480cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 shutdown_connections 2024-09-13T08:37:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.263+0000 7fe94d200640 1 -- 172.21.15.31:0/3040279810 wait complete. 2024-09-13T08:37:20.279 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"flags":0,"active_gid":4112,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4176094944},{"type":"v1","addr":"172.21.15.45:6801","nonce":4176094944}]},"active_addr":"172.21.15.45:6801/4176094944","active_change":"2024-09-13T08:36:49.024454+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4118,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":224785773}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":462607798}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2115686593}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":583368432}]}]} 2024-09-13T08:37:20.283 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T08:37:20.283 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T08:37:20.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 -- 172.21.15.31:0/3708635866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180ca5c0 msgr2=0x7f7c180ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 --2- 172.21.15.31:0/3708635866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180ca5c0 0x7f7c180ce600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c1809de00 tx=0x7f7c0002a560 comp rx=0 tx=0).stop 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 -- 172.21.15.31:0/3708635866 shutdown_connections 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 --2- 172.21.15.31:0/3708635866 >> v2:172.21.15.45:3300/0 conn(0x7f7c180b1910 0x7f7c180b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 --2- 172.21.15.31:0/3708635866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180ca5c0 0x7f7c180ce600 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 --2- 172.21.15.31:0/3708635866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c1815dfc0 0x7f7c180cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 -- 172.21.15.31:0/3708635866 >> 172.21.15.31:0/3708635866 conn(0x7f7c180cc910 msgr2=0x7f7c180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 -- 172.21.15.31:0/3708635866 shutdown_connections 2024-09-13T08:37:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.442+0000 7f7c1da00640 1 -- 172.21.15.31:0/3708635866 wait complete. 2024-09-13T08:37:20.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.444+0000 7f7c1da00640 1 Processor -- start 2024-09-13T08:37:20.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.444+0000 7f7c1da00640 1 -- start start 2024-09-13T08:37:20.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 0x7f7c181f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c1815dfc0 0x7f7c1815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c18073900 con 0x7f7c180b1910 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c18073780 con 0x7f7c180ca5c0 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c1da00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7c18073a80 con 0x7f7c1815dfc0 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c16a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c16a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34468/0 (socket says 172.21.15.31:34468) 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.445+0000 7f7c16a00640 1 -- 172.21.15.31:0/2408032534 learned_addr learned my addr 172.21.15.31:0/2408032534 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c0e000640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 0x7f7c181f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 -- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c1815dfc0 msgr2=0x7f7c1815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c1815dfc0 0x7f7c1815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 -- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 msgr2=0x7f7c181f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 0x7f7c181f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 -- 172.21.15.31:0/2408032534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c180b4440 con 0x7f7c180b1910 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c0e000640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 0x7f7c181f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c16a00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800d6d0 tx=0x7f7c0800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08017070 con 0x7f7c180b1910 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c08005330 con 0x7f7c180b1910 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08009820 con 0x7f7c180b1910 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.446+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c180b4720 con 0x7f7c180b1910 2024-09-13T08:37:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.447+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c180b4ce0 con 0x7f7c180b1910 2024-09-13T08:37:20.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.447+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c180acdb0 con 0x7f7c180b1910 2024-09-13T08:37:20.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.449+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f7c08009bd0 con 0x7f7c180b1910 2024-09-13T08:37:20.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.450+0000 7f7c0f400640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 0x7f7bf407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.450+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f7c0809ac30 con 0x7f7c180b1910 2024-09-13T08:37:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.450+0000 7f7c0e000640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 0x7f7bf407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.451+0000 7f7c0e000640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 0x7f7bf407ad10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c00033000 tx=0x7f7c00004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.454+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7c180acdb0 con 0x7f7c180b1910 2024-09-13T08:37:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.650+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c18073fa0 con 0x7f7c180b1910 2024-09-13T08:37:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.651+0000 7f7c0f400640 1 -- 172.21.15.31:0/2408032534 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7f7c08063290 con 0x7f7c180b1910 2024-09-13T08:37:20.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 msgr2=0x7f7bf407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 0x7f7bf407ad10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c00033000 tx=0x7f7c00004730 comp rx=0 tx=0).stop 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 msgr2=0x7f7c180b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800d6d0 tx=0x7f7c0800db90 comp rx=0 tx=0).stop 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 shutdown_connections 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7bf4078850 0x7f7bf407ad10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c1815dfc0 0x7f7c1815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c180ca5c0 0x7f7c181f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 --2- 172.21.15.31:0/2408032534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c180b1910 0x7f7c180b3d10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 >> 172.21.15.31:0/2408032534 conn(0x7f7c180cc910 msgr2=0x7f7c180ad4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 shutdown_connections 2024-09-13T08:37:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.653+0000 7f7c1da00640 1 -- 172.21.15.31:0/2408032534 wait complete. 2024-09-13T08:37:20.670 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T08:37:20.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:20.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 -- 172.21.15.31:0/896709819 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80b52d0 msgr2=0x7f83b8152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 --2- 172.21.15.31:0/896709819 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80b52d0 0x7f83b8152680 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83b0005fd0 tx=0x7f83b002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 -- 172.21.15.31:0/896709819 shutdown_connections 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 --2- 172.21.15.31:0/896709819 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80b52d0 0x7f83b8152680 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 --2- 172.21.15.31:0/896709819 >> v2:172.21.15.45:3300/0 conn(0x7f83b80cc720 0x7f83b806f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 -- 172.21.15.31:0/896709819 >> 172.21.15.31:0/896709819 conn(0x7f83b806cb00 msgr2=0x7f83b806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 -- 172.21.15.31:0/896709819 shutdown_connections 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.830+0000 7f83bea00640 1 -- 172.21.15.31:0/896709819 wait complete. 2024-09-13T08:37:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.832+0000 7f83bea00640 1 Processor -- start 2024-09-13T08:37:20.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.832+0000 7f83bea00640 1 -- start start 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80cc720 0x7f83b815e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83b80ccb00 0x7f83b81f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8152bc0 con 0x7f83b80b52d0 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f83b80703b0 con 0x7f83b80ccb00 2024-09-13T08:37:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f83b8152d40 con 0x7f83b80cc720 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34472/0 (socket says 172.21.15.31:34472) 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bd600640 1 -- 172.21.15.31:0/2506785582 learned_addr learned my addr 172.21.15.31:0/2506785582 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83bcc00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80cc720 0x7f83b815e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.833+0000 7f83be000640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83b80ccb00 0x7f83b81f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 -- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80cc720 msgr2=0x7f83b815e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80cc720 0x7f83b815e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 -- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83b80ccb00 msgr2=0x7f83b81f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83b80ccb00 0x7f83b81f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 -- 172.21.15.31:0/2506785582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b81482d0 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bd600640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ea20 tx=0x7f83ac00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00cd20 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83ac010430 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac010840 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b815d7f0 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.834+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b815ddb0 con 0x7f83b80b52d0 2024-09-13T08:37:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.835+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b8152680 con 0x7f83b80b52d0 2024-09-13T08:37:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.836+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f83ac0109e0 con 0x7f83b80b52d0 2024-09-13T08:37:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.837+0000 7f83b5600640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 0x7f839807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.837+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f83ac09b370 con 0x7f83b80b52d0 2024-09-13T08:37:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.838+0000 7f83bcc00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 0x7f839807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.839+0000 7f83bcc00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 0x7f839807ad10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83a4007fd0 tx=0x7f83a4001370 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:20.842+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f83ac063380 con 0x7f83b80b52d0 2024-09-13T08:37:21.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.037+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83b80b58c0 con 0x7f83b80b52d0 2024-09-13T08:37:21.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.038+0000 7f83b5600640 1 -- 172.21.15.31:0/2506785582 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7f83ac067db0 con 0x7f83b80b52d0 2024-09-13T08:37:21.039 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:21.040 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 msgr2=0x7f839807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 0x7f839807ad10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83a4007fd0 tx=0x7f83a4001370 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 msgr2=0x7f83b8147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ea20 tx=0x7f83ac00eee0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 shutdown_connections 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83b80ccb00 0x7f83b81f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8398078850 0x7f839807ad10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f83b80cc720 0x7f83b815e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 --2- 172.21.15.31:0/2506785582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83b80b52d0 0x7f83b8147d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 >> 172.21.15.31:0/2506785582 conn(0x7f83b806cb00 msgr2=0x7f83b8151ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 shutdown_connections 2024-09-13T08:37:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.041+0000 7f83bea00640 1 -- 172.21.15.31:0/2506785582 wait complete. 2024-09-13T08:37:21.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T08:37:21.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T08:37:21.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T08:37:21.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T08:37:21.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T08:37:21.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T08:37:21.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T08:37:21.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T08:37:21.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.342+0000 7f49b9600640 1 -- 172.21.15.31:0/295607465 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b4150d20 msgr2=0x7f49b40ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.342+0000 7f49b9600640 1 --2- 172.21.15.31:0/295607465 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b4150d20 0x7f49b40ca310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f49a0005fd0 tx=0x7f49a002b110 comp rx=0 tx=0).stop 2024-09-13T08:37:21.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.343+0000 7f49b9600640 1 -- 172.21.15.31:0/295607465 shutdown_connections 2024-09-13T08:37:21.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.343+0000 7f49b9600640 1 --2- 172.21.15.31:0/295607465 >> v2:172.21.15.31:3300/0 conn(0x7f49b40ca850 0x7f49b40cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.343+0000 7f49b9600640 1 --2- 172.21.15.31:0/295607465 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b4150d20 0x7f49b40ca310 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.343+0000 7f49b9600640 1 -- 172.21.15.31:0/295607465 >> 172.21.15.31:0/295607465 conn(0x7f49b406cb00 msgr2=0x7f49b406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.343+0000 7f49b9600640 1 -- 172.21.15.31:0/295607465 shutdown_connections 2024-09-13T08:37:21.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.344+0000 7f49b9600640 1 -- 172.21.15.31:0/295607465 wait complete. 2024-09-13T08:37:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.346+0000 7f49b9600640 1 Processor -- start 2024-09-13T08:37:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.347+0000 7f49b9600640 1 -- start start 2024-09-13T08:37:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.350+0000 7f49b9600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 0x7f49b4150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b40ccc10 0x7f49b40c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b4150d20 0x7f49b4151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b40ccc10 0x7f49b40c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49b406e5c0 con 0x7f49b40ccc10 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49b406e440 con 0x7f49b4150d20 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b9600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f49b406d620 con 0x7f49b40ca850 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 0x7f49b4150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 0x7f49b4150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48036/0 (socket says 172.21.15.31:48036) 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.351+0000 7f49b2a00640 1 -- 172.21.15.31:0/1219268302 learned_addr learned my addr 172.21.15.31:0/1219268302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f49b2a00640 1 -- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b4150d20 msgr2=0x7f49b4151d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f49b2a00640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b4150d20 0x7f49b4151d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f49b2a00640 1 -- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b40ccc10 msgr2=0x7f49b40c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f0e4fa00640 1 -- 172.21.15.31:0/3427178820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e400e28c0 msgr2=0x7f0e400538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/3427178820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e400e28c0 0x7f0e400538c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0e40056da0 tx=0x7f0e480a5ba0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f49b2a00640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b40ccc10 0x7f49b40c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.352+0000 7f49b2a00640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b41515a0 con 0x7f49b40ca850 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f0e4fa00640 1 -- 172.21.15.31:0/3427178820 shutdown_connections 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/3427178820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e40006b30 0x7f0e4003ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/3427178820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e400e28c0 0x7f0e400538c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f0e4fa00640 1 -- 172.21.15.31:0/3427178820 >> 172.21.15.31:0/3427178820 conn(0x7f0e40055550 msgr2=0x7f0e40055930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f49b2000640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b40ccc10 0x7f49b40c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f41cea00640 1 -- 172.21.15.31:0/3088679330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 msgr2=0x7f41c80d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f41cea00640 1 --2- 172.21.15.31:0/3088679330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c80d4b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41c0005fd0 tx=0x7f41c002a8c0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.353+0000 7f49b2a00640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 0x7f49b4150140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49a00074b0 tx=0x7f49a0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f0e4fa00640 1 -- 172.21.15.31:0/3427178820 shutdown_connections 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f41cea00640 1 -- 172.21.15.31:0/3088679330 shutdown_connections 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f41cea00640 1 --2- 172.21.15.31:0/3088679330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c80d4b20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f41cea00640 1 -- 172.21.15.31:0/3088679330 >> 172.21.15.31:0/3088679330 conn(0x7f41c80d4030 msgr2=0x7f41c80d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f41cea00640 1 -- 172.21.15.31:0/3088679330 shutdown_connections 2024-09-13T08:37:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f41cea00640 1 -- 172.21.15.31:0/3088679330 wait complete. 2024-09-13T08:37:21.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f667ae00640 1 -- 172.21.15.31:0/1279797269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c037a00 msgr2=0x7f666c03a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f667ae00640 1 --2- 172.21.15.31:0/1279797269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c037a00 0x7f666c03a450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f666c0077c0 tx=0x7f666002a610 comp rx=0 tx=0).stop 2024-09-13T08:37:21.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 -- 172.21.15.31:0/1279797269 shutdown_connections 2024-09-13T08:37:21.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 --2- 172.21.15.31:0/1279797269 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c03a990 0x7f666c0e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 --2- 172.21.15.31:0/1279797269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c037a00 0x7f666c03a450 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 --2- 172.21.15.31:0/1279797269 >> v2:172.21.15.45:3300/0 conn(0x7f666c0e2bd0 0x7f666c006790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 -- 172.21.15.31:0/1279797269 >> 172.21.15.31:0/1279797269 conn(0x7f666c055590 msgr2=0x7f666c055970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.355+0000 7f667ae00640 1 -- 172.21.15.31:0/1279797269 shutdown_connections 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.356+0000 7f667ae00640 1 -- 172.21.15.31:0/1279797269 wait complete. 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.356+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0007e60 con 0x7f49b40ca850 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.356+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49a0003b00 con 0x7f49b40ca850 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.356+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0005020 con 0x7f49b40ca850 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.357+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b40c8670 con 0x7f49b40ca850 2024-09-13T08:37:21.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.357+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b40c8390 con 0x7f49b40ca850 2024-09-13T08:37:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.354+0000 7f0e4fa00640 1 -- 172.21.15.31:0/3427178820 wait complete. 2024-09-13T08:37:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.356+0000 7f0e4fa00640 1 Processor -- start 2024-09-13T08:37:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.357+0000 7f41cea00640 1 Processor -- start 2024-09-13T08:37:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.357+0000 7f41cea00640 1 -- start start 2024-09-13T08:37:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.357+0000 7f0e4fa00640 1 -- start start 2024-09-13T08:37:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.358+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49b40cc6d0 con 0x7f49b40ca850 2024-09-13T08:37:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.359+0000 7f5f00200640 1 -- 172.21.15.31:0/11466996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef8070820 msgr2=0x7f5ef80b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.359+0000 7f41cea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c81548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.359+0000 7f5f00200640 1 --2- 172.21.15.31:0/11466996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef8070820 0x7f5ef80b3dc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80cc310 tx=0x7f5eec02a560 comp rx=0 tx=0).stop 2024-09-13T08:37:21.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41cea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41c8162010 0x7f41c815d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 -- 172.21.15.31:0/11466996 shutdown_connections 2024-09-13T08:37:21.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 --2- 172.21.15.31:0/11466996 >> v2:172.21.15.31:3300/0 conn(0x7f5ef80cc930 0x7f5ef80ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 --2- 172.21.15.31:0/11466996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef8070820 0x7f5ef80b3dc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 -- 172.21.15.31:0/11466996 >> 172.21.15.31:0/11466996 conn(0x7f5ef80b18f0 msgr2=0x7f5ef80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41cea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c8162b00 0x7f41c815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 -- 172.21.15.31:0/11466996 shutdown_connections 2024-09-13T08:37:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f0e4fa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e40006b30 0x7f0e4004dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f0e4fa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4003bf60 0x7f0e400d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f667ae00640 1 Processor -- start 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41c7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c81548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41c7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c81548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60742/0 (socket says 172.21.15.31:60742) 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41cea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80cff60 con 0x7f41c8162b00 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41cea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80cfde0 con 0x7f41c80b6e80 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f41cea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f41c816d8c0 con 0x7f41c8162010 2024-09-13T08:37:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.360+0000 7f5f00200640 1 -- 172.21.15.31:0/11466996 wait complete. 2024-09-13T08:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41c8162010 0x7f41c815d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 -- 172.21.15.31:0/4225624033 learned_addr learned my addr 172.21.15.31:0/4225624033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4fa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e400e28c0 0x7f0e400e1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41cca00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c8162b00 0x7f41c815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f49a00051c0 con 0x7f49b40ca850 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e40006b30 0x7f0e4004dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e47e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4003bf60 0x7f0e400d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e40006b30 0x7f0e4004dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60746/0 (socket says 172.21.15.31:60746) 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4d000640 1 -- 172.21.15.31:0/4200484713 learned_addr learned my addr 172.21.15.31:0/4200484713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e40057db0 con 0x7f0e4003bf60 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e40057c30 con 0x7f0e40006b30 2024-09-13T08:37:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0e40057f30 con 0x7f0e400e28c0 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f667ae00640 1 -- start start 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e400e28c0 0x7f0e400e1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 -- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e40006b30 msgr2=0x7f0e4004dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e40006b30 0x7f0e4004dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 -- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4003bf60 msgr2=0x7f0e400d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4003bf60 0x7f0e400d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.362+0000 7f0e4da00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e400e1d90 con 0x7f0e400e28c0 2024-09-13T08:37:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c037a00 0x7f666c0509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 -- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41c8162010 msgr2=0x7f41c815d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41c8162010 0x7f41c815d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 -- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c8162b00 msgr2=0x7f41c815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c8162b00 0x7f41c815dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41c7e00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c815cb80 con 0x7f41c80b6e80 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.361+0000 7f41cca00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c8162b00 0x7f41c815dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41c7e00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c81548f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c4c0 tx=0x7f41b800c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8016020 con 0x7f41c80b6e80 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c815ce60 con 0x7f41c80b6e80 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c815bc40 con 0x7f41c80b6e80 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41b8012940 con 0x7f41c80b6e80 2024-09-13T08:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8012d90 con 0x7f41c80b6e80 2024-09-13T08:37:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f49aaa00640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4994078920 0x7f499407ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f49a00ba690 con 0x7f49b40ca850 2024-09-13T08:37:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f49aaa00640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7f4994080ba0 0x7f4994082fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49a0009e90 con 0x7f4994080ba0 2024-09-13T08:37:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f49a00baa80 con 0x7f49b40ca850 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 0x7f666c0dea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f666c0e2bd0 0x7f666c0da000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f666c0e6c60 con 0x7f666c03a990 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f666c0e6ae0 con 0x7f666c0e2bd0 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f667ae00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f666c0e6de0 con 0x7f666c037a00 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.363+0000 7f6679000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 0x7f666c0dea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f6679000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 0x7f666c0dea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34552/0 (socket says 172.21.15.31:34552) 2024-09-13T08:37:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f6679000640 1 -- 172.21.15.31:0/1975688308 learned_addr learned my addr 172.21.15.31:0/1975688308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f667a400640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f666c0e2bd0 0x7f666c0da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f6679a00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c037a00 0x7f666c0509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.364+0000 7f49b2000640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4994078920 0x7f499407ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f5f00200640 1 Processor -- start 2024-09-13T08:37:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 -- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f666c0e2bd0 msgr2=0x7f666c0da000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f666c0e2bd0 0x7f666c0da000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 -- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 msgr2=0x7f666c0dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41a0001030 con 0x7f41c80b6e80 2024-09-13T08:37:21.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f5f00200640 1 -- start start 2024-09-13T08:37:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 0x7f666c0dea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666c0e5ae0 con 0x7f666c037a00 2024-09-13T08:37:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f49b3400640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7f4994080ba0 0x7f4994082fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f6679a00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c037a00 0x7f666c0509c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f666800c8f0 tx=0x7f666800cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f49b2000640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4994078920 0x7f499407ade0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f49a4006fd0 tx=0x7f49a4008040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.365+0000 7f0e4da00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e400e28c0 0x7f0e400e1090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c4d0 tx=0x7f0e3c00c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f667a400640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f666c0e2bd0 0x7f666c0da000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f6679000640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666c03a990 0x7f666c0dea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:37:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668007bb0 con 0x7f666c037a00 2024-09-13T08:37:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f666800ae40 con 0x7f666c037a00 2024-09-13T08:37:21.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668005260 con 0x7f666c037a00 2024-09-13T08:37:21.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c017070 con 0x7f0e400e28c0 2024-09-13T08:37:21.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e3c0035f0 con 0x7f0e400e28c0 2024-09-13T08:37:21.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f49b3400640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7f4994080ba0 0x7f4994082fc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c003a40 con 0x7f0e400e28c0 2024-09-13T08:37:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f5f00200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ef8070820 0x7f5ef815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.366+0000 7f5f00200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef80cc930 0x7f5ef81ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5f00200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 0x7f5ef81f9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5f00200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef80cf8d0 con 0x7f5ef80cf060 2024-09-13T08:37:21.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5f00200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef80cf750 con 0x7f5ef80cc930 2024-09-13T08:37:21.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5f00200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef8153ef0 con 0x7f5ef8070820 2024-09-13T08:37:21.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f41b8019020 con 0x7f41c80b6e80 2024-09-13T08:37:21.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1721775280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ea8003b40 msgr2=0x7f5ea8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5ec6a00640 1 --2- 172.21.15.31:0/1721775280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ea8003b40 0x7f5ea8003f40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0066670 tx=0x7f5ea8005830 comp rx=0 tx=0).stop 2024-09-13T08:37:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e4004e2b0 con 0x7f0e400e28c0 2024-09-13T08:37:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4004e870 con 0x7f0e400e28c0 2024-09-13T08:37:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666c0e5dc0 con 0x7f666c037a00 2024-09-13T08:37:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f666c0e3290 con 0x7f666c037a00 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5efe200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 0x7f5ef81f9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.367+0000 7f5efd800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ef8070820 0x7f5ef815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5efe200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 0x7f5ef81f9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34560/0 (socket says 172.21.15.31:34560) 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5efe200640 1 -- 172.21.15.31:0/1446453126 learned_addr learned my addr 172.21.15.31:0/1446453126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5efce00640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef80cc930 0x7f5ef81ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== osd.0 v2:172.21.15.31:6808/1940612994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49a0009e90 con 0x7f4994080ba0 2024-09-13T08:37:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1721775280 shutdown_connections 2024-09-13T08:37:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5ec6a00640 1 --2- 172.21.15.31:0/1721775280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ea8003b40 0x7f5ea8003f40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.368+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1721775280 >> 172.21.15.31:0/1721775280 conn(0x7f5eb8054560 msgr2=0x7f5eb8054940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e400583e0 con 0x7f0e400e28c0 2024-09-13T08:37:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7fd28a000640 1 -- 172.21.15.31:0/2389383053 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28406f8f0 msgr2=0x7fd28406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7fd28a000640 1 --2- 172.21.15.31:0/2389383053 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28406f8f0 0x7fd28406e660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd270005fd0 tx=0x7fd27002a530 comp rx=0 tx=0).stop 2024-09-13T08:37:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1721775280 shutdown_connections 2024-09-13T08:37:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f41bfe00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f41a8078920 0x7f41a807ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f666c0e7330 con 0x7f666c037a00 2024-09-13T08:37:21.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5efe200640 1 -- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ef8070820 msgr2=0x7f5ef815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5efe200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ef8070820 0x7f5ef815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5efe200640 1 -- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef80cc930 msgr2=0x7f5ef81ed3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5efe200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ef80cc930 0x7f5ef81ed3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5efe200640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef815a9c0 con 0x7f5ef80cf060 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f5efe200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 0x7f5ef81f9810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000b9a0 tx=0x7f5ef000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0019070 con 0x7f5ef80cf060 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef815aca0 con 0x7f5ef80cf060 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f41b809a900 con 0x7f41c80b6e80 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f41c7400640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f41a8078920 0x7f41a807ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef81ede80 con 0x7f5ef80cf060 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f41bfe00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f41a8080b40 0x7f41a8082f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41a8083610 con 0x7f41a8080b40 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f41b809acf0 con 0x7f41c80b6e80 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7fd28a000640 1 -- 172.21.15.31:0/2389383053 shutdown_connections 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7fd28a000640 1 --2- 172.21.15.31:0/2389383053 >> v2:172.21.15.31:3300/0 conn(0x7fd28406ec90 0x7fd284072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7fd28a000640 1 --2- 172.21.15.31:0/2389383053 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28406f8f0 0x7fd28406e660 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd270005fd0 tx=0x7fd27002a530 comp rx=0 tx=0).stop 2024-09-13T08:37:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.370+0000 7fd28a000640 1 -- 172.21.15.31:0/2389383053 >> 172.21.15.31:0/2389383053 conn(0x7fd2840b18f0 msgr2=0x7fd2840b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7f41c7400640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f41a8078920 0x7f41a807ade0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f41c8074200 tx=0x7f41b4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.369+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1721775280 wait complete. 2024-09-13T08:37:21.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ef000cc90 con 0x7f5ef80cf060 2024-09-13T08:37:21.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00146a0 con 0x7f5ef80cf060 2024-09-13T08:37:21.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7fd28a000640 1 -- 172.21.15.31:0/2389383053 shutdown_connections 2024-09-13T08:37:21.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.372+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f6668005400 con 0x7f666c037a00 2024-09-13T08:37:21.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.372+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5ed4001030 con 0x7f5ef80cf060 2024-09-13T08:37:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.372+0000 7f41cca00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f41a8080b40 0x7f41a8082f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7f9bb3200640 1 -- 172.21.15.31:0/3043893307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 msgr2=0x7f9bac0cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.371+0000 7f9bb3200640 1 --2- 172.21.15.31:0/3043893307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac0cdb80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9bac06e610 tx=0x7f9ba402ee60 comp rx=0 tx=0).stop 2024-09-13T08:37:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f5ec6a00640 1 Processor -- start 2024-09-13T08:37:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f9bb3200640 1 -- 172.21.15.31:0/3043893307 shutdown_connections 2024-09-13T08:37:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f9bb3200640 1 --2- 172.21.15.31:0/3043893307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bac0ced20 0x7f9bac0ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f9bb3200640 1 --2- 172.21.15.31:0/3043893307 >> v2:172.21.15.31:3300/0 conn(0x7f9bac0ce600 0x7f9bac15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f9bb3200640 1 --2- 172.21.15.31:0/3043893307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac0cdb80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f9bb3200640 1 -- 172.21.15.31:0/3043893307 >> 172.21.15.31:0/3043893307 conn(0x7f9bac0b18f0 msgr2=0x7f9bac0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f5ec6a00640 1 -- start start 2024-09-13T08:37:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f41cca00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f41a8080b40 0x7f41a8082f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f6671600640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66540788a0 0x7f665407ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f666809aee0 con 0x7f666c037a00 2024-09-13T08:37:21.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f9bb3200640 1 -- 172.21.15.31:0/3043893307 shutdown_connections 2024-09-13T08:37:21.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f9bb3200640 1 -- 172.21.15.31:0/3043893307 wait complete. 2024-09-13T08:37:21.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f6679000640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66540788a0 0x7f665407ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f6671600640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f6654080ac0 0x7f6654082ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f666801ce50 con 0x7f6654080ac0 2024-09-13T08:37:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f666809b290 con 0x7f666c037a00 2024-09-13T08:37:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.374+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f5ef001c070 con 0x7f5ef80cf060 2024-09-13T08:37:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.375+0000 7f6679000640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66540788a0 0x7f665407ad60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f666c00dd10 tx=0x7f6660004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.375+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== osd.2 v2:172.21.15.31:6816/657376189 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41a8083610 con 0x7f41a8080b40 2024-09-13T08:37:21.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.373+0000 7fd28a000640 1 -- 172.21.15.31:0/2389383053 wait complete. 2024-09-13T08:37:21.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.375+0000 7fd28a000640 1 Processor -- start 2024-09-13T08:37:21.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.375+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0e3c01a070 con 0x7f0e400e28c0 2024-09-13T08:37:21.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7fd28a000640 1 -- start start 2024-09-13T08:37:21.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f0e44c00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e2c078950 0x7f0e2c07ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0e3c09a0c0 con 0x7f0e400e28c0 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 0x7f5eb8182030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5eb8021a10 0x7f5eb8034d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb8169630 0x7f5eb8169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb800bcd0 con 0x7f5ea8003b40 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f667a400640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f6654080ac0 0x7f6654082ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb801d380 con 0x7f5eb8169630 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f5ec6a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5eb80361f0 con 0x7f5eb8021a10 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.376+0000 7f9bb3200640 1 Processor -- start 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f0e44c00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f0e2c080b70 0x7f0e2c082f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e2c083640 con 0x7f0e2c080b70 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f0e4d000640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e2c078950 0x7f0e2c07ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0e3c09a4b0 con 0x7f0e400e28c0 2024-09-13T08:37:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f0e47e00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f0e2c080b70 0x7f0e2c082f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f667a400640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f6654080ac0 0x7f6654082ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ef5600640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5edc078850 0x7f5edc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f5ef009a550 con 0x7f5ef80cf060 2024-09-13T08:37:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ef5600640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f5edc0809f0 0x7f5edc082e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7fd28a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28406ec90 0x7fd2841f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ec5600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 0x7f5eb8182030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd28a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd284161560 0x7fd28409ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ef0012e50 con 0x7f5edc0809f0 2024-09-13T08:37:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5ef00a1050 con 0x7f5ef80cf060 2024-09-13T08:37:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd28a000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 0x7fd28414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.377+0000 7f5ec5600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 0x7f5eb8182030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34570/0 (socket says 172.21.15.31:34570) 2024-09-13T08:37:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7f5ec5600640 1 -- 172.21.15.31:0/1842033568 learned_addr learned my addr 172.21.15.31:0/1842033568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== osd.3 v2:172.21.15.31:6800/2845481020 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f666801ce50 con 0x7f6654080ac0 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd283400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28406ec90 0x7fd2841f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd28a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd284073860 con 0x7fd284161560 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd28a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2840736e0 con 0x7fd28406ec90 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd28a000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd2840739e0 con 0x7fd2840ca300 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd283e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 0x7fd28414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd283e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 0x7fd28414f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48086/0 (socket says 172.21.15.31:48086) 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7fd283e00640 1 -- 172.21.15.31:0/2738766043 learned_addr learned my addr 172.21.15.31:0/2738766043 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283e00640 1 -- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28406ec90 msgr2=0x7fd2841f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283e00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28406ec90 0x7fd2841f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd282a00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd284161560 0x7fd28409ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283e00640 1 -- 172.21.15.31:0/2738766043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd284161560 msgr2=0x7fd28409ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283e00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd284161560 0x7fd28409ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283e00640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2841f5870 con 0x7fd2840ca300 2024-09-13T08:37:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7f5ec5600640 1 -- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5eb8021a10 msgr2=0x7f5eb8034d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7f9bb3200640 1 -- start start 2024-09-13T08:37:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7fd283400640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28406ec90 0x7fd2841f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec6000640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb8169630 0x7f5eb8169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bac0ce600 0x7f9bac1f99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec4c00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5eb8021a10 0x7f5eb8034d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec5600640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5eb8021a10 0x7f5eb8034d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec5600640 1 -- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb8169630 msgr2=0x7f5eb8169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec5600640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb8169630 0x7f5eb8169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f5ec5600640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb817e3d0 con 0x7f5ea8003b40 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd283e00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 0x7fd28414f1f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd27400c970 tx=0x7fd27400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.378+0000 7f0e47e00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f0e2c080b70 0x7f0e2c082f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f0e4d000640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e2c078950 0x7f0e2c07ae10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e4806b690 tx=0x7f0e48075570 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7f5efd800640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5edc078850 0x7f5edc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7f5efce00640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f5edc0809f0 0x7f5edc082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== osd.1 v2:172.21.15.31:6824/3396863731 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e2c083640 con 0x7f0e2c080b70 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274007bb0 con 0x7fd2840ca300 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd27400f450 con 0x7fd2840ca300 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274007e60 con 0x7fd2840ca300 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2840ca7c0 con 0x7fd2840ca300 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd284152450 con 0x7fd2840ca300 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bac0ced20 0x7f9bac09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9bac160320 con 0x7f9bac0ce600 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9bac1601a0 con 0x7f9bac15be30 2024-09-13T08:37:21.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.379+0000 7f9bb3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9bac1604a0 con 0x7f9bac0ced20 2024-09-13T08:37:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7f5efd800640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5edc078850 0x7f5edc07ad10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5eec033000 tx=0x7f5eec00b040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7f5efce00640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f5edc0809f0 0x7f5edc082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.380+0000 7f5ec6000640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb8169630 0x7f5eb8169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd284152830 con 0x7fd2840ca300 2024-09-13T08:37:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7f5ec4c00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5eb8021a10 0x7f5eb8034d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7f5ec5600640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 0x7f5eb8182030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0066670 tx=0x7f5ec00afca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== osd.4 v2:172.21.15.45:6802/1087405525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5ef0012e50 con 0x7f5edc0809f0 2024-09-13T08:37:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00b4040 con 0x7f5ea8003b40 2024-09-13T08:37:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ec0063590 con 0x7f5ea8003b40 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec006aa20 con 0x7f5ea8003b40 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.381+0000 7f9bab400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bac0ced20 0x7f9bac09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f9bb1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f9babe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bac0ce600 0x7f9bac1f99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f9babe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bac0ce600 0x7f9bac1f99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34588/0 (socket says 172.21.15.31:34588) 2024-09-13T08:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.382+0000 7f9babe00640 1 -- 172.21.15.31:0/4119298009 learned_addr learned my addr 172.21.15.31:0/4119298009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb817e6b0 con 0x7f5ea8003b40 2024-09-13T08:37:21.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb8169fd0 con 0x7f5ea8003b40 2024-09-13T08:37:21.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f9bb1200640 1 -- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bac0ced20 msgr2=0x7f9bac09cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f9bb1200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9bac0ced20 0x7f9bac09cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f9bb1200640 1 -- 172.21.15.31:0/4119298009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bac0ce600 msgr2=0x7f9bac1f99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f9bb1200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bac0ce600 0x7f9bac1f99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.383+0000 7f9bb1200640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bac158090 con 0x7f9bac15be30 2024-09-13T08:37:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.386+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd27400f8a0 con 0x7fd2840ca300 2024-09-13T08:37:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.384+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e9c001030 con 0x7f5ea8003b40 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9bb1200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac157b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b440 tx=0x7f9b9c00b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c015020 con 0x7f9bac15be30 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b9c011920 con 0x7f9bac15be30 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c011d50 con 0x7f9bac15be30 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bac15c570 con 0x7f9bac15be30 2024-09-13T08:37:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bac15caf0 con 0x7f9bac15be30 2024-09-13T08:37:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.387+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f5ec0063740 con 0x7f5ea8003b40 2024-09-13T08:37:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd27b400640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd264078850 0x7fd26407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fd27409b100 con 0x7fd2840ca300 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd27b400640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7fd2640809f0 0x7fd264082e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2640834c0 con 0x7fd2640809f0 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd2740a2050 con 0x7fd2840ca300 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.388+0000 7fd283400640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd264078850 0x7fd26407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b84001030 con 0x7f9bac15be30 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f5ebd600640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5ea0078920 0x7f5ea007ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f5ec01361c0 con 0x7f5ea8003b40 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f5ebd600640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f5ea0080b40 0x7f5ea0082f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ec008ce00 con 0x7f5ea0080b40 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5ec013d050 con 0x7f5ea8003b40 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.390+0000 7f5ec4c00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5ea0078920 0x7f5ea007ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.389+0000 7fd282a00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7fd2640809f0 0x7fd264082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.390+0000 7f5ec6000640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f5ea0080b40 0x7f5ea0082f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.390+0000 7fd283400640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd264078850 0x7fd26407ad10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001d40 tx=0x7fd27c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.391+0000 7fd282a00640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7fd2640809f0 0x7fd264082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.391+0000 7f5ec4c00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5ea0078920 0x7f5ea007ade0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8053d70 tx=0x7f5eac006b30 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.391+0000 7f5ec6000640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f5ea0080b40 0x7f5ea0082f60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.392+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== osd.5 v2:172.21.15.45:6818/2092128205 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2640834c0 con 0x7fd2640809f0 2024-09-13T08:37:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.392+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== osd.6 v2:172.21.15.45:6810/636367962 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5ec008ce00 con 0x7f5ea0080b40 2024-09-13T08:37:21.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.392+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f9b9c018020 con 0x7f9bac15be30 2024-09-13T08:37:21.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.393+0000 7f9ba3e00640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b8c078850 0x7f9b8c07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.394+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f9b9c09a740 con 0x7f9bac15be30 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.394+0000 7f9ba3e00640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7f9b8c0809f0 0x7f9b8c082e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.394+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b9c00de50 con 0x7f9b8c0809f0 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.394+0000 7f9babe00640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b8c078850 0x7f9b8c07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.394+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9b9c0a1050 con 0x7f9bac15be30 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.395+0000 7f9bab400640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7f9b8c0809f0 0x7f9b8c082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.395+0000 7f9babe00640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b8c078850 0x7f9b8c07ad10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009bf0 tx=0x7f9ba4004ab0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.396+0000 7f9bab400640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7f9b8c0809f0 0x7f9b8c082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.397+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== osd.7 v2:172.21.15.45:6826/1264531367 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9c00de50 con 0x7f9b8c0809f0 2024-09-13T08:37:21.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.400+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 --> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f49b40b5c10 con 0x7f4994080ba0 2024-09-13T08:37:21.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.401+0000 7f49aaa00640 1 -- 172.21.15.31:0/1219268302 <== osd.0 v2:172.21.15.31:6808/1940612994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f49b40b5c10 con 0x7f4994080ba0 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.402+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7f4994080ba0 msgr2=0x7f4994082fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.402+0000 7f49b9600640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7f4994080ba0 0x7f4994082fc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.402+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4994078920 msgr2=0x7f499407ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.402+0000 7f49b9600640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4994078920 0x7f499407ade0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f49a4006fd0 tx=0x7f49a4008040 comp rx=0 tx=0).stop 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 msgr2=0x7f49b4150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 --2- 172.21.15.31:0/1219268302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b40ca850 0x7f49b4150140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49a00074b0 tx=0x7f49a0003820 comp rx=0 tx=0).stop 2024-09-13T08:37:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b3400640 1 -- 172.21.15.31:0/1219268302 reap_dead start 2024-09-13T08:37:21.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 shutdown_connections 2024-09-13T08:37:21.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 >> 172.21.15.31:0/1219268302 conn(0x7f49b406cb00 msgr2=0x7f49b4073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 shutdown_connections 2024-09-13T08:37:21.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.403+0000 7f49b9600640 1 -- 172.21.15.31:0/1219268302 wait complete. 2024-09-13T08:37:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.409+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 --> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41a0002dd0 con 0x7f41a8080b40 2024-09-13T08:37:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.409+0000 7f41bfe00640 1 -- 172.21.15.31:0/4225624033 <== osd.2 v2:172.21.15.31:6816/657376189 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41a0002dd0 con 0x7f41a8080b40 2024-09-13T08:37:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.410+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f41a8080b40 msgr2=0x7f41a8082f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.410+0000 7f41cea00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f41a8080b40 0x7f41a8082f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.411+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f41a8078920 msgr2=0x7f41a807ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.411+0000 7f41cea00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f41a8078920 0x7f41a807ade0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f41c8074200 tx=0x7f41b4009040 comp rx=0 tx=0).stop 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.411+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 msgr2=0x7f41c81548f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.411+0000 7f41cea00640 1 --2- 172.21.15.31:0/4225624033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c80b6e80 0x7f41c81548f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c4c0 tx=0x7f41b800c980 comp rx=0 tx=0).stop 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.411+0000 7f41cca00640 1 -- 172.21.15.31:0/4225624033 reap_dead start 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.413+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 shutdown_connections 2024-09-13T08:37:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.413+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 >> 172.21.15.31:0/4225624033 conn(0x7f41c80d4030 msgr2=0x7f41c806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.413+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 shutdown_connections 2024-09-13T08:37:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.413+0000 7f41cea00640 1 -- 172.21.15.31:0/4225624033 wait complete. 2024-09-13T08:37:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.416+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 --> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f666c0e3720 con 0x7f6654080ac0 2024-09-13T08:37:21.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.417+0000 7f6671600640 1 -- 172.21.15.31:0/1975688308 <== osd.3 v2:172.21.15.31:6800/2845481020 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f666c0e3720 con 0x7f6654080ac0 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.418+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f6654080ac0 msgr2=0x7f6654082ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.418+0000 7f667ae00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f6654080ac0 0x7f6654082ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.419+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66540788a0 msgr2=0x7f665407ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.419+0000 7f667ae00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66540788a0 0x7f665407ad60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f666c00dd10 tx=0x7f6660004730 comp rx=0 tx=0).stop 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.419+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c037a00 msgr2=0x7f666c0509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.419+0000 7f667ae00640 1 --2- 172.21.15.31:0/1975688308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f666c037a00 0x7f666c0509c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f666800c8f0 tx=0x7f666800cdb0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.419+0000 7f667a400640 1 -- 172.21.15.31:0/1975688308 reap_dead start 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.420+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 shutdown_connections 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.420+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 >> 172.21.15.31:0/1975688308 conn(0x7f666c055590 msgr2=0x7f666c00e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.420+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 shutdown_connections 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.421+0000 7f667ae00640 1 -- 172.21.15.31:0/1975688308 wait complete. 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.422+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 --> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e40055c10 con 0x7f0e2c080b70 2024-09-13T08:37:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.422+0000 7f0e44c00640 1 -- 172.21.15.31:0/4200484713 <== osd.1 v2:172.21.15.31:6824/3396863731 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e40055c10 con 0x7f0e2c080b70 2024-09-13T08:37:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 --> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b84002dd0 con 0x7f9b8c0809f0 2024-09-13T08:37:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 --> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ed4002dd0 con 0x7f5edc0809f0 2024-09-13T08:37:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f0e2c080b70 msgr2=0x7f0e2c082f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f0e2c080b70 0x7f0e2c082f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f9ba3e00640 1 -- 172.21.15.31:0/4119298009 <== osd.7 v2:172.21.15.45:6826/1264531367 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9b84002dd0 con 0x7f9b8c0809f0 2024-09-13T08:37:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e2c078950 msgr2=0x7f0e2c07ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e2c078950 0x7f0e2c07ae10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e4806b690 tx=0x7f0e48075570 comp rx=0 tx=0).stop 2024-09-13T08:37:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e400e28c0 msgr2=0x7f0e400e1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.423+0000 7f0e4fa00640 1 --2- 172.21.15.31:0/4200484713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e400e28c0 0x7f0e400e1090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c4d0 tx=0x7f0e3c00c990 comp rx=0 tx=0).stop 2024-09-13T08:37:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f0e4da00640 1 -- 172.21.15.31:0/4200484713 reap_dead start 2024-09-13T08:37:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f5ef5600640 1 -- 172.21.15.31:0/1446453126 <== osd.4 v2:172.21.15.45:6802/1087405525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5ed4002dd0 con 0x7f5edc0809f0 2024-09-13T08:37:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 shutdown_connections 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 >> 172.21.15.31:0/4200484713 conn(0x7f0e40055550 msgr2=0x7f0e400056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7f9b8c0809f0 msgr2=0x7f9b8c082e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7f9b8c0809f0 0x7f9b8c082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b8c078850 msgr2=0x7f9b8c07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b8c078850 0x7f9b8c07ad10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009bf0 tx=0x7f9ba4004ab0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 msgr2=0x7f9bac157b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 --> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e9c002dd0 con 0x7f5ea0080b40 2024-09-13T08:37:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f5edc0809f0 msgr2=0x7f5edc082e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f5f00200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f5edc0809f0 0x7f5edc082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 --2- 172.21.15.31:0/4119298009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bac15be30 0x7f9bac157b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b440 tx=0x7f9b9c00b900 comp rx=0 tx=0).stop 2024-09-13T08:37:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb1200640 1 -- 172.21.15.31:0/4119298009 reap_dead start 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 shutdown_connections 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.424+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 >> 172.21.15.31:0/4119298009 conn(0x7f9bac0b18f0 msgr2=0x7f9bac0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 shutdown_connections 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 shutdown_connections 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 --> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd284073f00 con 0x7fd2640809f0 2024-09-13T08:37:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5edc078850 msgr2=0x7f5edc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5edc078850 0x7f5edc07ad10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5eec033000 tx=0x7f5eec00b040 comp rx=0 tx=0).stop 2024-09-13T08:37:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5ebd600640 1 -- 172.21.15.31:0/1842033568 <== osd.6 v2:172.21.15.45:6810/636367962 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e9c002dd0 con 0x7f5ea0080b40 2024-09-13T08:37:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f9bb3200640 1 -- 172.21.15.31:0/4119298009 wait complete. 2024-09-13T08:37:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 msgr2=0x7f5ef81f9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 --2- 172.21.15.31:0/1446453126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef80cf060 0x7f5ef81f9810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000b9a0 tx=0x7f5ef000be60 comp rx=0 tx=0).stop 2024-09-13T08:37:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f0e4fa00640 1 -- 172.21.15.31:0/4200484713 wait complete. 2024-09-13T08:37:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5efe200640 1 -- 172.21.15.31:0/1446453126 reap_dead start 2024-09-13T08:37:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 shutdown_connections 2024-09-13T08:37:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 >> 172.21.15.31:0/1446453126 conn(0x7f5ef80b18f0 msgr2=0x7f5ef8073470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7fd27b400640 1 -- 172.21.15.31:0/2738766043 <== osd.5 v2:172.21.15.45:6818/2092128205 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd284073f00 con 0x7fd2640809f0 2024-09-13T08:37:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f5ea0080b40 msgr2=0x7f5ea0082f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5ec6a00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f5ea0080b40 0x7f5ea0082f60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.425+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 shutdown_connections 2024-09-13T08:37:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5ea0078920 msgr2=0x7f5ea007ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5ea0078920 0x7f5ea007ade0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8053d70 tx=0x7f5eac006b30 comp rx=0 tx=0).stop 2024-09-13T08:37:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 msgr2=0x7f5eb8182030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 --2- 172.21.15.31:0/1842033568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea8003b40 0x7f5eb8182030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0066670 tx=0x7f5ec00afca0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5f00200640 1 -- 172.21.15.31:0/1446453126 wait complete. 2024-09-13T08:37:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6000640 1 -- 172.21.15.31:0/1842033568 reap_dead start 2024-09-13T08:37:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7fd2640809f0 msgr2=0x7fd264082e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7fd2640809f0 0x7fd264082e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 shutdown_connections 2024-09-13T08:37:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 >> 172.21.15.31:0/1842033568 conn(0x7f5eb8054560 msgr2=0x7f5eb80340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd264078850 msgr2=0x7fd26407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd264078850 0x7fd26407ad10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001d40 tx=0x7fd27c009040 comp rx=0 tx=0).stop 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 msgr2=0x7fd28414f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.426+0000 7fd28a000640 1 --2- 172.21.15.31:0/2738766043 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2840ca300 0x7fd28414f1f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd27400c970 tx=0x7fd27400ce30 comp rx=0 tx=0).stop 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.427+0000 7fd283e00640 1 -- 172.21.15.31:0/2738766043 reap_dead start 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.427+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 shutdown_connections 2024-09-13T08:37:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.427+0000 7f5ec6a00640 1 -- 172.21.15.31:0/1842033568 wait complete. 2024-09-13T08:37:21.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.428+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 shutdown_connections 2024-09-13T08:37:21.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.428+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 >> 172.21.15.31:0/2738766043 conn(0x7fd2840b18f0 msgr2=0x7fd2840ae470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.442 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:21.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T08:37:21.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.431+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 shutdown_connections 2024-09-13T08:37:21.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.431+0000 7fd28a000640 1 -- 172.21.15.31:0/2738766043 wait complete. 2024-09-13T08:37:21.444 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:21.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T08:37:21.450 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:21.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T08:37:21.456 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-13T08:37:21.456 INFO:teuthology.orchestra.run.smithi031.stdout:60129542150 2024-09-13T08:37:21.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T08:37:21.457 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:21.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T08:37:21.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T08:37:21.460 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:21.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T08:37:21.464 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:21.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T08:37:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 -- 172.21.15.31:0/900998458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd5806f280 msgr2=0x7fdd5815a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 --2- 172.21.15.31:0/900998458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd5806f280 0x7fdd5815a420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000afd0 tx=0x7fdd5002b3b0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 -- 172.21.15.31:0/900998458 shutdown_connections 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 --2- 172.21.15.31:0/900998458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd5806f280 0x7fdd5815a420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 --2- 172.21.15.31:0/900998458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd580b1eb0 0x7fdd58154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.736+0000 7fdd5d200640 1 -- 172.21.15.31:0/900998458 >> 172.21.15.31:0/900998458 conn(0x7fdd580cc7a0 msgr2=0x7fdd580ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.737+0000 7fdd5d200640 1 -- 172.21.15.31:0/900998458 shutdown_connections 2024-09-13T08:37:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.737+0000 7fdd5d200640 1 -- 172.21.15.31:0/900998458 wait complete. 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 -- 172.21.15.31:0/2369766688 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0036510 msgr2=0x7f0af0009a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 --2- 172.21.15.31:0/2369766688 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0036510 0x7f0af0009a40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0af806b950 tx=0x7f0af80a67c0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 -- 172.21.15.31:0/2369766688 shutdown_connections 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 --2- 172.21.15.31:0/2369766688 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0036510 0x7f0af0009a40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 --2- 172.21.15.31:0/2369766688 >> v2:172.21.15.31:3300/0 conn(0x7f0af0005aa0 0x7f0af0035fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.738+0000 7f0af7e00640 1 -- 172.21.15.31:0/2369766688 >> 172.21.15.31:0/2369766688 conn(0x7f0af0054550 msgr2=0x7f0af0054930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.739+0000 7fdd5d200640 1 Processor -- start 2024-09-13T08:37:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.740+0000 7fdd5d200640 1 -- start start 2024-09-13T08:37:21.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.739+0000 7f0af7e00640 1 -- 172.21.15.31:0/2369766688 shutdown_connections 2024-09-13T08:37:21.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.740+0000 7f0af7e00640 1 -- 172.21.15.31:0/2369766688 wait complete. 2024-09-13T08:37:21.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.742+0000 7fdd5d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.742+0000 7fdd5d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 0x7fdd58152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.742+0000 7fdd5d200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.742+0000 7f0af7e00640 1 Processor -- start 2024-09-13T08:37:21.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.742+0000 7f0af7e00640 1 -- start start 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.743+0000 7fdd57400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.743+0000 7fdd57400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60848/0 (socket says 172.21.15.31:60848) 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.743+0000 7fdd5d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5815aeb0 con 0x7fdd580b1eb0 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.743+0000 7fdd5d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5815ad30 con 0x7fdd5806f280 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.743+0000 7fdd5d200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdd580cd8f0 con 0x7fdd581541b0 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48130/0 (socket says 172.21.15.31:48130) 2024-09-13T08:37:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 -- 172.21.15.31:0/3480397084 learned_addr learned my addr 172.21.15.31:0/3480397084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd56a00640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 0x7fdd58152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 -- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 msgr2=0x7fdd580c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57e00640 1 -- 172.21.15.31:0/3480397084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 msgr2=0x7fdd58152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.744+0000 7fdd57400640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.745+0000 7fdd57e00640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 0x7fdd58152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.745+0000 7fdd57e00640 1 -- 172.21.15.31:0/3480397084 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd58153d40 con 0x7fdd581541b0 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 0x7f0af0021d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af0036510 0x7f0af004f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0009f80 con 0x7f0af001c880 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0af000bc00 con 0x7f0af0036510 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7f0af7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0af000a100 con 0x7f0af0005aa0 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.745+0000 7fdd56a00640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 0x7fdd58152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.745+0000 7fdd57e00640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd500076a0 tx=0x7fdd5000a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50033800 con 0x7fdd581541b0 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd50040070 con 0x7fdd581541b0 2024-09-13T08:37:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.746+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50033d30 con 0x7fdd581541b0 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7f0af6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7f0af6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34618/0 (socket says 172.21.15.31:34618) 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7f0af6000640 1 -- 172.21.15.31:0/1173558619 learned_addr learned my addr 172.21.15.31:0/1173558619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7f0af7400640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af0036510 0x7f0af004f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7f0af6a00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 0x7f0af0021d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.747+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd580b1c40 con 0x7fdd581541b0 2024-09-13T08:37:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5815c320 con 0x7fdd581541b0 2024-09-13T08:37:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6000640 1 -- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 msgr2=0x7f0af0021d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6000640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 0x7f0af0021d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f106aa00640 1 -- 172.21.15.31:0/2062050168 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1064161560 msgr2=0x7f106406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f106aa00640 1 --2- 172.21.15.31:0/2062050168 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1064161560 0x7f106406e660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f105c005fd0 tx=0x7f105c02a980 comp rx=0 tx=0).stop 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7fbf38a00640 1 -- 172.21.15.31:0/3861410165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c0e2bd0 msgr2=0x7fbf2c006790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7fbf38a00640 1 --2- 172.21.15.31:0/3861410165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c006790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0077c0 tx=0x7fbf340a5dd0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6000640 1 -- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af0036510 msgr2=0x7f0af004f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6000640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af0036510 0x7f0af004f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6000640 1 -- 172.21.15.31:0/1173558619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af001ed80 con 0x7f0af001c880 2024-09-13T08:37:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.748+0000 7f0af6a00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 0x7f0af0021d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 -- 172.21.15.31:0/3861410165 shutdown_connections 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 --2- 172.21.15.31:0/3861410165 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbf2c03a990 0x7fbf2c0e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 --2- 172.21.15.31:0/3861410165 >> v2:172.21.15.31:3300/0 conn(0x7fbf2c037a00 0x7fbf2c03a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 --2- 172.21.15.31:0/3861410165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c006790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 -- 172.21.15.31:0/3861410165 >> 172.21.15.31:0/3861410165 conn(0x7fbf2c055590 msgr2=0x7fbf2c055970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 -- 172.21.15.31:0/3861410165 shutdown_connections 2024-09-13T08:37:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7f106aa00640 1 -- 172.21.15.31:0/2062050168 shutdown_connections 2024-09-13T08:37:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7f106aa00640 1 --2- 172.21.15.31:0/2062050168 >> v2:172.21.15.31:3300/0 conn(0x7f106406f8f0 0x7f1064072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7f106aa00640 1 --2- 172.21.15.31:0/2062050168 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1064161560 0x7f106406e660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7f106aa00640 1 -- 172.21.15.31:0/2062050168 >> 172.21.15.31:0/2062050168 conn(0x7f10640b18f0 msgr2=0x7f10640b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.749+0000 7fbf38a00640 1 -- 172.21.15.31:0/3861410165 wait complete. 2024-09-13T08:37:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f0af6000640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400b450 tx=0x7f0ae400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f106aa00640 1 -- 172.21.15.31:0/2062050168 shutdown_connections 2024-09-13T08:37:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f106aa00640 1 -- 172.21.15.31:0/2062050168 wait complete. 2024-09-13T08:37:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5806a170 con 0x7fdd581541b0 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4015020 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ae40119a0 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4011df0 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.750+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af0048c90 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.751+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af001f060 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.752+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af00078a0 con 0x7f0af001c880 2024-09-13T08:37:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.752+0000 7fbf38a00640 1 Processor -- start 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 -- 172.21.15.31:0/2172347393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 msgr2=0x7fcdbc0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 --2- 172.21.15.31:0/2172347393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc0b5640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0005fd0 tx=0x7fcdb002aa10 comp rx=0 tx=0).stop 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 -- 172.21.15.31:0/2172347393 shutdown_connections 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 --2- 172.21.15.31:0/2172347393 >> v2:172.21.15.31:3300/0 conn(0x7fcdbc0d4c20 0x7fcdbc0b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 --2- 172.21.15.31:0/2172347393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc0b5640 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 -- 172.21.15.31:0/2172347393 >> 172.21.15.31:0/2172347393 conn(0x7fcdbc0d4100 msgr2=0x7fcdbc0d4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 -- 172.21.15.31:0/2172347393 shutdown_connections 2024-09-13T08:37:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fcdc4400640 1 -- 172.21.15.31:0/2172347393 wait complete. 2024-09-13T08:37:21.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.753+0000 7fbf38a00640 1 -- start start 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 0x7fbf2c04c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbf2c03a990 0x7fbf2c04ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c0e6c60 con 0x7fbf2c0e2bd0 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c0e6ae0 con 0x7fbf2c037a00 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf38a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbf2c0e6de0 con 0x7fbf2c03a990 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fbf33200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf33200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34624/0 (socket says 172.21.15.31:34624) 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf33200640 1 -- 172.21.15.31:0/1208526842 learned_addr learned my addr 172.21.15.31:0/1208526842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 0x7fbf2c04c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 -- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbf2c03a990 msgr2=0x7fbf2c04ea50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbf2c03a990 0x7fbf2c04ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 -- 172.21.15.31:0/1208526842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 msgr2=0x7fbf2c04dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf32800640 1 -- 172.21.15.31:0/1208526842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2c04ff10 con 0x7fbf2c037a00 2024-09-13T08:37:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fbf33200640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:37:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.754+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fdd5003a070 con 0x7fdd581541b0 2024-09-13T08:37:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7fbf32800640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 0x7fbf2c04c200 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf34066670 tx=0x7fbf340ae9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7fdd4f400640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 0x7fdd3807ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fdd500ba8b0 con 0x7fdd581541b0 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.752+0000 7f106aa00640 1 Processor -- start 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.755+0000 7f106aa00640 1 -- start start 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7f106aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f106406ec90 0x7f10640cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7f106aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 0x7f1064202090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7f106aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.756+0000 7fcdc4400640 1 Processor -- start 2024-09-13T08:37:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fdd57400640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 0x7fdd3807ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fcdc4400640 1 -- start start 2024-09-13T08:37:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0ae4018020 con 0x7f0af001c880 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340b3050 con 0x7fbf2c037a00 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf340bf070 con 0x7fbf2c037a00 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340ba980 con 0x7fbf2c037a00 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2c0501f0 con 0x7fbf2c037a00 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7f1068a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.757+0000 7f1068a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60872/0 (socket says 172.21.15.31:60872) 2024-09-13T08:37:21.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2c04c740 con 0x7fbf2c037a00 2024-09-13T08:37:21.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f0aeea00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 0x7f0ad807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0ae409aba0 con 0x7f0af001c880 2024-09-13T08:37:21.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7fdd57400640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 0x7fdd3807ade0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd48007fd0 tx=0x7fdd48009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.759+0000 7f0af6a00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 0x7f0ad807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.759+0000 7fcdc4400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 0x7fcdbc0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.759+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2c0e72b0 con 0x7fbf2c037a00 2024-09-13T08:37:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f1068a00640 1 -- 172.21.15.31:0/2720931865 learned_addr learned my addr 172.21.15.31:0/2720931865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1064073890 con 0x7f106406f8f0 2024-09-13T08:37:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1064073710 con 0x7f1064161560 2024-09-13T08:37:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.758+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1064073a10 con 0x7f106406ec90 2024-09-13T08:37:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.760+0000 7fcdc4400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.760+0000 7f1063400640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 0x7f1064202090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.760+0000 7fcdc4400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc1fcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063e00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f106406ec90 0x7f10640cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdd5806a170 con 0x7fdd581541b0 2024-09-13T08:37:21.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063400640 1 -- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f106406ec90 msgr2=0x7f10640cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f0af6a00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 0x7f0ad807ad10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8001350 tx=0x7f0ae8009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 0x7fcdbc0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 0x7fcdbc0ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60876/0 (socket says 172.21.15.31:60876) 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc2400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc1fcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc2400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc1fcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48206/0 (socket says 172.21.15.31:48206) 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34652/0 (socket says 172.21.15.31:34652) 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7fcdc1a00640 1 -- 172.21.15.31:0/3163727971 learned_addr learned my addr 172.21.15.31:0/3163727971 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063400640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f106406ec90 0x7f10640cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063400640 1 -- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 msgr2=0x7f10640ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063400640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.761+0000 7f1063400640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106415bcf0 con 0x7f106406f8f0 2024-09-13T08:37:21.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.762+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc0cfb90 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.762+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc161bd0 con 0x7fcdbc0d0840 2024-09-13T08:37:21.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.762+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcdbc0cfd10 con 0x7fcdbc1615e0 2024-09-13T08:37:21.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.762+0000 7f1068a00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:21.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7f1063400640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 0x7f1064202090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f105400c440 tx=0x7f105400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7fcdc1000640 1 -- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 msgr2=0x7fcdbc1fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7fcdc1000640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc1fcc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7fcdc1000640 1 -- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 msgr2=0x7fcdbc0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7fcdc1000640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 0x7fcdbc0ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.763+0000 7fcdc1000640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc0c8610 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.764+0000 7fcdc1000640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e9c0 tx=0x7fcda800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.762+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fbf340bab20 con 0x7fbf2c037a00 2024-09-13T08:37:21.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.765+0000 7fbf2aa00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 0x7fbf1407ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.765+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fbf34136bd0 con 0x7fbf2c037a00 2024-09-13T08:37:21.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.765+0000 7fc09f200640 1 -- 172.21.15.31:0/2061946610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 msgr2=0x7fc0980ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.765+0000 7fc09f200640 1 --2- 172.21.15.31:0/2061946610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980ce050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc09806d790 tx=0x7fc08802a580 comp rx=0 tx=0).stop 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.765+0000 7fbf31e00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 0x7fbf1407ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 -- 172.21.15.31:0/2061946610 shutdown_connections 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 --2- 172.21.15.31:0/2061946610 >> v2:172.21.15.31:3300/0 conn(0x7fc098071950 0x7fc09815d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 --2- 172.21.15.31:0/2061946610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980ce050 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 -- 172.21.15.31:0/2061946610 >> 172.21.15.31:0/2061946610 conn(0x7fc09806d240 msgr2=0x7fc0980688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 -- 172.21.15.31:0/2061946610 shutdown_connections 2024-09-13T08:37:21.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fc09f200640 1 -- 172.21.15.31:0/2061946610 wait complete. 2024-09-13T08:37:21.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.766+0000 7fbf31e00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 0x7fbf1407ad60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c052290 tx=0x7fbf2000b2f0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054017070 con 0x7f106406f8f0 2024-09-13T08:37:21.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10540035f0 con 0x7f106406f8f0 2024-09-13T08:37:21.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054003a40 con 0x7f106406f8f0 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda800cd80 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdbc0c8890 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdbc0c8e50 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.768+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcda8010520 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.768+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda800cd80 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.767+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106415bf90 con 0x7f106406f8f0 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.768+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106415c270 con 0x7f106406f8f0 2024-09-13T08:37:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fc09f200640 1 Processor -- start 2024-09-13T08:37:21.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fa4bdc00640 1 -- 172.21.15.31:0/3895418551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa49c0051c0 msgr2=0x7fa49c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/3895418551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa49c0051c0 0x7fa49c00b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0001d90 tx=0x7fa49c005830 comp rx=0 tx=0).stop 2024-09-13T08:37:21.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fa4b7400640 1 -- 172.21.15.31:0/3895418551 reap_dead start 2024-09-13T08:37:21.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fc09f200640 1 -- start start 2024-09-13T08:37:21.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.769+0000 7fa4bdc00640 1 -- 172.21.15.31:0/3895418551 shutdown_connections 2024-09-13T08:37:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0af00078a0 con 0x7f0af001c880 2024-09-13T08:37:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7fa4bdc00640 1 -- 172.21.15.31:0/3895418551 >> 172.21.15.31:0/3895418551 conn(0x7fa4b80d4100 msgr2=0x7fa4b80d4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7fa4bdc00640 1 -- 172.21.15.31:0/3895418551 shutdown_connections 2024-09-13T08:37:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7fa4bdc00640 1 -- 172.21.15.31:0/3895418551 wait complete. 2024-09-13T08:37:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd98005290 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.770+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103c005290 con 0x7f106406f8f0 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 0x7fc0980c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 0x7fc098147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815de50 con 0x7fc098156680 2024-09-13T08:37:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815dcd0 con 0x7fc0981562a0 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fc09f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc09815dfd0 con 0x7fc098071950 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.772+0000 7fc097e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 0x7fc0980c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.772+0000 7fc097e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 0x7fc0980c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48218/0 (socket says 172.21.15.31:48218) 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.772+0000 7fc097e00640 1 -- 172.21.15.31:0/138243389 learned_addr learned my addr 172.21.15.31:0/138243389 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.772+0000 7fc09d200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 0x7fc098147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.771+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbf340ff3b0 con 0x7fbf2c037a00 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.772+0000 7fc097400640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 -- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 msgr2=0x7fc0980c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 0x7fc0980c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 -- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 msgr2=0x7fc0980c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 -- 172.21.15.31:0/138243389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0981489a0 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.773+0000 7fc09d200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 0x7fc098147b30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090014070 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc090007930 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090007d80 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0980c8470 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0980c8a30 con 0x7fc098156680 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.774+0000 7fa4bdc00640 1 Processor -- start 2024-09-13T08:37:21.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.775+0000 7fa4bdc00640 1 -- start start 2024-09-13T08:37:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.775+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fcda801f020 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4b81fcf20 0x7fa4b80c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4b80c6880 0x7fa4b80c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.777+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f105401a070 con 0x7f106406f8f0 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b80b59d0 con 0x7fa4b81f8f80 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b80b5850 con 0x7fa4b81fcf20 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.776+0000 7fa4bdc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b8161bb0 con 0x7fa4b80c6880 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.777+0000 7fa4b6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.777+0000 7fa4b6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34688/0 (socket says 172.21.15.31:34688) 2024-09-13T08:37:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.777+0000 7fa4b6000640 1 -- 172.21.15.31:0/575572644 learned_addr learned my addr 172.21.15.31:0/575572644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.777+0000 7fa4b7400640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4b80c6880 0x7fa4b80c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6a00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4b81fcf20 0x7fa4b80c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6000640 1 -- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4b80c6880 msgr2=0x7fa4b80c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6000640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4b80c6880 0x7fa4b80c2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6000640 1 -- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4b81fcf20 msgr2=0x7fa4b80c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6000640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4b81fcf20 0x7fa4b80c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fa4b6000640 1 -- 172.21.15.31:0/575572644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b81f9460 con 0x7fa4b81f8f80 2024-09-13T08:37:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7f105be00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 0x7f104407ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f1054099b60 con 0x7f106406f8f0 2024-09-13T08:37:21.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.779+0000 7fa4b6000640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4001ce0 tx=0x7fa4a400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.779+0000 7fcdb9600640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 0x7fcd9c07af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.778+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc09806fc20 con 0x7fc098156680 2024-09-13T08:37:21.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.779+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fcda809ab10 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.780+0000 7f1063e00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 0x7f104407ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.781+0000 7f1063e00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 0x7f104407ae10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f105c035000 tx=0x7f105c004ab0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.781+0000 7fcdc1a00640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 0x7fcd9c07af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.782+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1054062270 con 0x7f106406f8f0 2024-09-13T08:37:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.783+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4015020 con 0x7fa4b81f8f80 2024-09-13T08:37:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.782+0000 7fcdc1a00640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 0x7fcd9c07af40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc073880 tx=0x7fcdb0004ab0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.783+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b81f9740 con 0x7fa4b81f8f80 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.783+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b81fd300 con 0x7fa4b81f8f80 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.787+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b80cfea0 con 0x7fa4b81f8f80 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.785+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fc090017070 con 0x7fc098156680 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.787+0000 7fc08fe00640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 0x7fc07807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.788+0000 7fc097e00640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 0x7fc07807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.789+0000 7fc097e00640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 0x7fc07807ad10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc088004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.791+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcdbc0c8610 con 0x7fcdbc0d4c20 2024-09-13T08:37:21.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.792+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fc09005f520 con 0x7fc098156680 2024-09-13T08:37:21.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.792+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4a40035f0 con 0x7fa4b81f8f80 2024-09-13T08:37:21.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.792+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4003a00 con 0x7fa4b81f8f80 2024-09-13T08:37:21.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.807+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc09806fc20 con 0x7fc098156680 2024-09-13T08:37:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.809+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fa4a4018020 con 0x7fa4b81f8f80 2024-09-13T08:37:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.811+0000 7fa4aea00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 0x7fa49807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.812+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fa4a409b6b0 con 0x7fa4b81f8f80 2024-09-13T08:37:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.812+0000 7fa4b6a00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 0x7fa49807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.814+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa4a401f020 con 0x7fa4b81f8f80 2024-09-13T08:37:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.815+0000 7fa4b6a00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 0x7fa49807ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80d1570 tx=0x7fa4a8005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.823+0000 7f00a3400640 1 -- 172.21.15.31:0/593952515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0b6f50 msgr2=0x7f009c0cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.823+0000 7f00a3400640 1 --2- 172.21.15.31:0/593952515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0b6f50 0x7f009c0cf480 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f009c074c20 tx=0x7f008c02a460 comp rx=0 tx=0).stop 2024-09-13T08:37:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.824+0000 7f00a3400640 1 -- 172.21.15.31:0/593952515 shutdown_connections 2024-09-13T08:37:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.824+0000 7f00a3400640 1 --2- 172.21.15.31:0/593952515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0b6f50 0x7f009c0cf480 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.824+0000 7f00a3400640 1 --2- 172.21.15.31:0/593952515 >> v2:172.21.15.31:3300/0 conn(0x7f009c0d4c20 0x7f009c0d0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.824+0000 7f00a3400640 1 --2- 172.21.15.31:0/593952515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 0x7f009c0b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.824+0000 7f00a3400640 1 -- 172.21.15.31:0/593952515 >> 172.21.15.31:0/593952515 conn(0x7f009c0d4100 msgr2=0x7f009c0d4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.826+0000 7f00a3400640 1 -- 172.21.15.31:0/593952515 shutdown_connections 2024-09-13T08:37:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.826+0000 7f00a3400640 1 -- 172.21.15.31:0/593952515 wait complete. 2024-09-13T08:37:21.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.829+0000 7f00a3400640 1 Processor -- start 2024-09-13T08:37:21.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.829+0000 7f00a3400640 1 -- start start 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 0x7f009c148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0d4c20 0x7f009c1fc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0cfb40 con 0x7f009c0b6f50 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0cf9c0 con 0x7f009c0d4c20 2024-09-13T08:37:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a3400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f009c0cfcc0 con 0x7f009c0d0840 2024-09-13T08:37:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34700/0 (socket says 172.21.15.31:34700) 2024-09-13T08:37:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.831+0000 7f00a0a00640 1 -- 172.21.15.31:0/4195999790 learned_addr learned my addr 172.21.15.31:0/4195999790 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f009be00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 0x7f009c148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a1400640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0d4c20 0x7f009c1fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a0a00640 1 -- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 msgr2=0x7f009c148e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a0a00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 0x7f009c148e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a0a00640 1 -- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0d4c20 msgr2=0x7f009c1fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a0a00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0d4c20 0x7f009c1fc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.832+0000 7f00a0a00640 1 -- 172.21.15.31:0/4195999790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c1f9ec0 con 0x7f009c0b6f50 2024-09-13T08:37:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f00a0a00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f009400eb50 tx=0x7f009400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094010630 con 0x7f009c0b6f50 2024-09-13T08:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00940107d0 con 0x7f009c0b6f50 2024-09-13T08:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094010bc0 con 0x7f009c0b6f50 2024-09-13T08:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f00a3400640 1 -- 172.21.15.31:0/4195999790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c0cc350 con 0x7f009c0b6f50 2024-09-13T08:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.833+0000 7f00a3400640 1 -- 172.21.15.31:0/4195999790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c1f9be0 con 0x7f009c0b6f50 2024-09-13T08:37:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.837+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0094021440 con 0x7f009c0b6f50 2024-09-13T08:37:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.838+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009c0b5620 con 0x7f009c0b6f50 2024-09-13T08:37:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.839+0000 7f0098c00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 0x7f007807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.843+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f009409b8c0 con 0x7f009c0b6f50 2024-09-13T08:37:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.843+0000 7f009be00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 0x7f007807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.844+0000 7f009be00640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 0x7f007807ad10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00840095f0 tx=0x7f0084005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:21.851+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0094063fc0 con 0x7f009c0b6f50 2024-09-13T08:37:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.178+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdd580ca710 con 0x7fdd581541b0 2024-09-13T08:37:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.179+0000 7fdd4f400640 1 -- 172.21.15.31:0/3480397084 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd50082940 con 0x7fdd581541b0 2024-09-13T08:37:22.180 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.183+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 msgr2=0x7fdd3807ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.183+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 0x7fdd3807ade0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd48007fd0 tx=0x7fdd48009040 comp rx=0 tx=0).stop 2024-09-13T08:37:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.183+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 msgr2=0x7fdd58153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.183+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd500076a0 tx=0x7fdd5000a980 comp rx=0 tx=0).stop 2024-09-13T08:37:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 shutdown_connections 2024-09-13T08:37:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd38078920 0x7fdd3807ade0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd581541b0 0x7fdd58153180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd580b1eb0 0x7fdd58152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 --2- 172.21.15.31:0/3480397084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5806f280 0x7fdd580c2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 >> 172.21.15.31:0/3480397084 conn(0x7fdd580cc7a0 msgr2=0x7fdd5814d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.184+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 shutdown_connections 2024-09-13T08:37:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.185+0000 7fdd5d200640 1 -- 172.21.15.31:0/3480397084 wait complete. 2024-09-13T08:37:22.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.194+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbf2c00f2e0 con 0x7fbf2c037a00 2024-09-13T08:37:22.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.195+0000 7fbf2aa00640 1 -- 172.21.15.31:0/1208526842 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf340fecb0 con 0x7fbf2c037a00 2024-09-13T08:37:22.196 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.199+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 msgr2=0x7fbf1407ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.199+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 0x7fbf1407ad60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c052290 tx=0x7fbf2000b2f0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.199+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 msgr2=0x7fbf2c04c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.199+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 0x7fbf2c04c200 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf34066670 tx=0x7fbf340ae9e0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 shutdown_connections 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c0e2bd0 0x7fbf2c04dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbf140788a0 0x7fbf1407ad60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbf2c03a990 0x7fbf2c04ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 --2- 172.21.15.31:0/1208526842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf2c037a00 0x7fbf2c04c200 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 >> 172.21.15.31:0/1208526842 conn(0x7fbf2c055590 msgr2=0x7fbf2c056d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 shutdown_connections 2024-09-13T08:37:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.200+0000 7fbf38a00640 1 -- 172.21.15.31:0/1208526842 wait complete. 2024-09-13T08:37:22.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.2 2024-09-13T08:37:22.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.215+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0af001c350 con 0x7f0af001c880 2024-09-13T08:37:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.216+0000 7f0aeea00640 1 -- 172.21.15.31:0/1173558619 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0af001c350 con 0x7f0af001c880 2024-09-13T08:37:22.220 INFO:teuthology.orchestra.run.smithi031.stdout:64424509445 2024-09-13T08:37:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.220+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 msgr2=0x7f0ad807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.220+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 0x7f0ad807ad10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8001350 tx=0x7f0ae8009040 comp rx=0 tx=0).stop 2024-09-13T08:37:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.220+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 msgr2=0x7f0af004ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.220+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400b450 tx=0x7f0ae400b910 comp rx=0 tx=0).stop 2024-09-13T08:37:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 shutdown_connections 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af0036510 0x7f0af004f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0af001c880 0x7f0af004ab30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0ad8078850 0x7f0ad807ad10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 --2- 172.21.15.31:0/1173558619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0af0005aa0 0x7f0af0021d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 >> 172.21.15.31:0/1173558619 conn(0x7f0af0054550 msgr2=0x7f0af0035800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.221+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 shutdown_connections 2024-09-13T08:37:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.222+0000 7f0af7e00640 1 -- 172.21.15.31:0/1173558619 wait complete. 2024-09-13T08:37:22.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.3 2024-09-13T08:37:22.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.238+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa4b80b8ad0 con 0x7fa4b81f8f80 2024-09-13T08:37:22.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.238+0000 7fa4aea00640 1 -- 172.21.15.31:0/575572644 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa4a4063d00 con 0x7fa4b81f8f80 2024-09-13T08:37:22.239 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 msgr2=0x7fa49807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 0x7fa49807ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80d1570 tx=0x7fa4a8005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 msgr2=0x7fa4b80c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4001ce0 tx=0x7fa4a400eaf0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 shutdown_connections 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa498078880 0x7fa49807ad40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4b80c6880 0x7fa4b80c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b81f8f80 0x7fa4b80c4db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 --2- 172.21.15.31:0/575572644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4b81fcf20 0x7fa4b80c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.241+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 >> 172.21.15.31:0/575572644 conn(0x7fa4b80d4100 msgr2=0x7fa4b80b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.242+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 shutdown_connections 2024-09-13T08:37:22.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.242+0000 7fa4bdc00640 1 -- 172.21.15.31:0/575572644 wait complete. 2024-09-13T08:37:22.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.246+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc09815d580 con 0x7fc098156680 2024-09-13T08:37:22.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.0 2024-09-13T08:37:22.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.247+0000 7fc08fe00640 1 -- 172.21.15.31:0/138243389 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc0900632b0 con 0x7fc098156680 2024-09-13T08:37:22.248 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.249+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 msgr2=0x7fc07807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.249+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 0x7fc07807ad10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc088004730 comp rx=0 tx=0).stop 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 msgr2=0x7fc098147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 0x7fc098147b30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09000cf10 comp rx=0 tx=0).stop 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 shutdown_connections 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc098156680 0x7fc098147b30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0981562a0 0x7fc0980c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc078078850 0x7fc07807ad10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 --2- 172.21.15.31:0/138243389 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc098071950 0x7fc0980c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 >> 172.21.15.31:0/138243389 conn(0x7fc09806d240 msgr2=0x7fc0980b6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 shutdown_connections 2024-09-13T08:37:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.250+0000 7fc09f200640 1 -- 172.21.15.31:0/138243389 wait complete. 2024-09-13T08:37:22.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.254+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f103c005580 con 0x7f106406f8f0 2024-09-13T08:37:22.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.254+0000 7f105be00640 1 -- 172.21.15.31:0/2720931865 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1054061b70 con 0x7f106406f8f0 2024-09-13T08:37:22.255 INFO:teuthology.orchestra.run.smithi031.stdout:60129542149 2024-09-13T08:37:22.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.257+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcd98005580 con 0x7fcdbc0d4c20 2024-09-13T08:37:22.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.257+0000 7fcdb9600640 1 -- 172.21.15.31:0/3163727971 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcda8063170 con 0x7fcdbc0d4c20 2024-09-13T08:37:22.258 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:22.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.258+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 msgr2=0x7f104407ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.258+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 0x7f104407ae10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f105c035000 tx=0x7f105c004ab0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.258+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 msgr2=0x7f1064202090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.258+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 0x7f1064202090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f105400c440 tx=0x7f105400c840 comp rx=0 tx=0).stop 2024-09-13T08:37:22.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 msgr2=0x7fcd9c07af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 0x7fcd9c07af40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc073880 tx=0x7fcdb0004ab0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 msgr2=0x7fcdbc1fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e9c0 tx=0x7fcda800ee80 comp rx=0 tx=0).stop 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 shutdown_connections 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1064161560 0x7f10640ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f106406f8f0 0x7f1064202090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1044078950 0x7f104407ae10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 --2- 172.21.15.31:0/2720931865 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f106406ec90 0x7f10640cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 >> 172.21.15.31:0/2720931865 conn(0x7f10640b18f0 msgr2=0x7f106406f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 shutdown_connections 2024-09-13T08:37:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.260+0000 7f106aa00640 1 -- 172.21.15.31:0/2720931865 wait complete. 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 shutdown_connections 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd9c078a80 0x7fcd9c07af40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcdbc1615e0 0x7fcdbc1fcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdbc0d4c20 0x7fcdbc1fc6c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 --2- 172.21.15.31:0/3163727971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdbc0d0840 0x7fcdbc0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 >> 172.21.15.31:0/3163727971 conn(0x7fcdbc0d4100 msgr2=0x7fcdbc0b7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 shutdown_connections 2024-09-13T08:37:22.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.261+0000 7fcdc4400640 1 -- 172.21.15.31:0/3163727971 wait complete. 2024-09-13T08:37:22.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.264+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f009c0d19e0 con 0x7f009c0b6f50 2024-09-13T08:37:22.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.265+0000 7f0098c00640 1 -- 172.21.15.31:0/4195999790 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f00940638c0 con 0x7f009c0b6f50 2024-09-13T08:37:22.265 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:22.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.6 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.267+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 msgr2=0x7f007807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.267+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 0x7f007807ad10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00840095f0 tx=0x7f0084005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.267+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 msgr2=0x7f009c09dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.267+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f009400eb50 tx=0x7f009400c6a0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 shutdown_connections 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0d4c20 0x7f009c1fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0078078850 0x7f007807ad10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c0d0840 0x7f009c148e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 --2- 172.21.15.31:0/4195999790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0b6f50 0x7f009c09dd80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 >> 172.21.15.31:0/4195999790 conn(0x7f009c0d4100 msgr2=0x7f009c15ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 shutdown_connections 2024-09-13T08:37:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:22.268+0000 7f0092000640 1 -- 172.21.15.31:0/4195999790 wait complete. 2024-09-13T08:37:22.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.7 2024-09-13T08:37:22.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.1 2024-09-13T08:37:22.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-09-13T08:37:22.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.5 2024-09-13T08:37:23.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T08:37:23.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T08:37:23.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T08:37:23.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T08:37:23.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T08:37:23.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T08:37:23.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T08:37:23.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T08:37:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.439+0000 7f60e8e00640 1 -- 172.21.15.31:0/1724242849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 msgr2=0x7f60e408ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.439+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1724242849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e408ba60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60d400dc90 tx=0x7f60d402e450 comp rx=0 tx=0).stop 2024-09-13T08:37:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.440+0000 7f60e8e00640 1 -- 172.21.15.31:0/1724242849 shutdown_connections 2024-09-13T08:37:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.440+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1724242849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e4072730 0x7f60e4072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.440+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1724242849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e408ba60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.440+0000 7f60e8e00640 1 -- 172.21.15.31:0/1724242849 >> 172.21.15.31:0/1724242849 conn(0x7f60e40b18f0 msgr2=0x7f60e40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.441+0000 7f60e8e00640 1 -- 172.21.15.31:0/1724242849 shutdown_connections 2024-09-13T08:37:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.441+0000 7f60e8e00640 1 -- 172.21.15.31:0/1724242849 wait complete. 2024-09-13T08:37:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.444+0000 7f60e8e00640 1 Processor -- start 2024-09-13T08:37:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.444+0000 7f60e8e00640 1 -- start start 2024-09-13T08:37:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.446+0000 7f60e8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 0x7f60e408a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.447+0000 7f60e8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e409c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.447+0000 7f60e8e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 0x7f60e409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f60e406e9f0 con 0x7f60e40b49c0 2024-09-13T08:37:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60e406e870 con 0x7f60e4072730 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e8e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f60e406eb70 con 0x7f60e416aba0 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 0x7f60e408a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 0x7f60e408a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60932/0 (socket says 172.21.15.31:60932) 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e1e00640 1 -- 172.21.15.31:0/1991780557 learned_addr learned my addr 172.21.15.31:0/1991780557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.448+0000 7f60e2800640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 0x7f60e409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.449+0000 7f60e2800640 1 -- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 msgr2=0x7f60e408a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.449+0000 7f60e1400640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e409c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.449+0000 7f60e2800640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 0x7f60e408a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.449+0000 7f60e2800640 1 -- 172.21.15.31:0/1991780557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 msgr2=0x7f60e409c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.450+0000 7f60e2800640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e409c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.450+0000 7f60e2800640 1 -- 172.21.15.31:0/1991780557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e409d330 con 0x7f60e416aba0 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.450+0000 7f60e1400640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e409c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:23.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.450+0000 7f60e2800640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 0x7f60e409aa10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00be30 tx=0x7f60dc00bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.451+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc019070 con 0x7f60e416aba0 2024-09-13T08:37:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.451+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60dc014440 con 0x7f60e416aba0 2024-09-13T08:37:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.451+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc014870 con 0x7f60e416aba0 2024-09-13T08:37:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.451+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e409d610 con 0x7f60e416aba0 2024-09-13T08:37:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.452+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e408af20 con 0x7f60e416aba0 2024-09-13T08:37:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.454+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e4073ad0 con 0x7f60e416aba0 2024-09-13T08:37:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.456+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f60dc01c070 con 0x7f60e416aba0 2024-09-13T08:37:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.458+0000 7f60da000640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 0x7f60c407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.458+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f60dc09a750 con 0x7f60e416aba0 2024-09-13T08:37:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.459+0000 7f60e1e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 0x7f60c407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.460+0000 7f60e1e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 0x7f60c407ad10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f60d400dc90 tx=0x7f60d4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.463+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f60e4073ad0 con 0x7f60e416aba0 2024-09-13T08:37:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 -- 172.21.15.31:0/1808572273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d74049de0 msgr2=0x7f4d740333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1808572273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d74049de0 0x7f4d740333e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0097b0 tx=0x7f4d6c02a390 comp rx=0 tx=0).stop 2024-09-13T08:37:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 -- 172.21.15.31:0/1808572273 shutdown_connections 2024-09-13T08:37:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1808572273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d74049de0 0x7f4d740333e0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0097b0 tx=0x7f4d6c02a390 comp rx=0 tx=0).stop 2024-09-13T08:37:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 -- 172.21.15.31:0/1808572273 >> 172.21.15.31:0/1808572273 conn(0x7f4d7401f9a0 msgr2=0x7f4d74056700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 -- 172.21.15.31:0/1808572273 shutdown_connections 2024-09-13T08:37:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.517+0000 7f4d80e00640 1 -- 172.21.15.31:0/1808572273 wait complete. 2024-09-13T08:37:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.519+0000 7f6f01000640 1 -- 172.21.15.31:0/1291891496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc1618b0 msgr2=0x7f6efc161d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.519+0000 7f6f01000640 1 --2- 172.21.15.31:0/1291891496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc1618b0 0x7f6efc161d20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6efc075310 tx=0x7f6ee802a560 comp rx=0 tx=0).stop 2024-09-13T08:37:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 -- 172.21.15.31:0/1291891496 shutdown_connections 2024-09-13T08:37:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f4d80e00640 1 Processor -- start 2024-09-13T08:37:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 --2- 172.21.15.31:0/1291891496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc1618b0 0x7f6efc161d20 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6efc075310 tx=0x7f6ee802a560 comp rx=0 tx=0).stop 2024-09-13T08:37:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 --2- 172.21.15.31:0/1291891496 >> v2:172.21.15.45:3300/0 conn(0x7f6efc0cc040 0x7f6efc0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 -- 172.21.15.31:0/1291891496 >> 172.21.15.31:0/1291891496 conn(0x7f6efc0cfd20 msgr2=0x7f6efc0d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f4d80e00640 1 -- start start 2024-09-13T08:37:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 -- 172.21.15.31:0/1291891496 shutdown_connections 2024-09-13T08:37:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.520+0000 7f6f01000640 1 -- 172.21.15.31:0/1291891496 wait complete. 2024-09-13T08:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.523+0000 7f4d80e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 0x7f4d74039530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.523+0000 7f6f01000640 1 Processor -- start 2024-09-13T08:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.523+0000 7f4d80e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 0x7f4d7403b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.523+0000 7f6f01000640 1 -- start start 2024-09-13T08:37:23.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d80e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 0x7f4d74052a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d80e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d74054b60 con 0x7f4d74033920 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d80e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d740549e0 con 0x7f4d740536a0 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d7a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 0x7f4d74052a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d7a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 0x7f4d74052a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60952/0 (socket says 172.21.15.31:60952) 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.524+0000 7f4d7a800640 1 -- 172.21.15.31:0/1019397789 learned_addr learned my addr 172.21.15.31:0/1019397789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.525+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4d74054ce0 con 0x7f4d740e6e30 2024-09-13T08:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.525+0000 7f4d79400640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 0x7f4d7403b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.525+0000 7f4d79e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 0x7f4d74039530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f6f01000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 0x7f6efc15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f4d79e00640 1 -- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 msgr2=0x7f4d7403b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f4d79e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 0x7f4d7403b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f4d79e00640 1 -- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 msgr2=0x7f4d74052a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f4d79e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 0x7f4d74052a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.526+0000 7f4d79e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d741813e0 con 0x7f4d74033920 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f4d79400640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 0x7f4d7403b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6f01000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 0x7f6efc2050d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6f01000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6f01000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1625a0 con 0x7f6efc0cc040 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6f01000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc0af2b0 con 0x7f6efc15e2c0 2024-09-13T08:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6f01000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc162720 con 0x7f6efc205610 2024-09-13T08:37:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f4d79e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 0x7f4d74039530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c500 tx=0x7f4d6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68017070 con 0x7f4d74033920 2024-09-13T08:37:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d680035f0 con 0x7f4d74033920 2024-09-13T08:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68003a80 con 0x7f4d74033920 2024-09-13T08:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48260/0 (socket says 172.21.15.31:48260) 2024-09-13T08:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.527+0000 7f6efbe00640 1 -- 172.21.15.31:0/688264557 learned_addr learned my addr 172.21.15.31:0/688264557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f6efbe00640 1 -- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 msgr2=0x7f6efc2050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f6efb400640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 0x7f6efc15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f6efaa00640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 0x7f6efc2050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f6efbe00640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 0x7f6efc2050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.528+0000 7f6efbe00640 1 -- 172.21.15.31:0/688264557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 msgr2=0x7f6efc15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.529+0000 7f6efbe00640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 0x7f6efc15b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.529+0000 7f6efbe00640 1 -- 172.21.15.31:0/688264557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc0c4230 con 0x7f6efc205610 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.529+0000 7f6efaa00640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 0x7f6efc2050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.529+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d741816c0 con 0x7f4d74033920 2024-09-13T08:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.529+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d74181c80 con 0x7f4d74033920 2024-09-13T08:37:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.531+0000 7f6efb400640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 0x7f6efc15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.531+0000 7f6efbe00640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c6c0 tx=0x7f6ef400cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.532+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4017070 con 0x7f6efc205610 2024-09-13T08:37:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.531+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d740551c0 con 0x7f4d74033920 2024-09-13T08:37:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.532+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6efc0c4510 con 0x7f6efc205610 2024-09-13T08:37:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.533+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6efc204bb0 con 0x7f6efc205610 2024-09-13T08:37:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.532+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ef4003c60 con 0x7f6efc205610 2024-09-13T08:37:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.533+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4017070 con 0x7f6efc205610 2024-09-13T08:37:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.537+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed4005290 con 0x7f6efc205610 2024-09-13T08:37:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.538+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f6ef401a070 con 0x7f6efc205610 2024-09-13T08:37:23.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.535+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4d6801a070 con 0x7f4d74033920 2024-09-13T08:37:23.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.540+0000 7f6ef3400640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 0x7f6edc07af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.540+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f6ef4099970 con 0x7f6efc205610 2024-09-13T08:37:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.541+0000 7f6efb400640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 0x7f6edc07af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.540+0000 7f4d72000640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 0x7f4d5c07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.543+0000 7f6efb400640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 0x7f6edc07af10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00a8b0 tx=0x7f6eec008040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.542+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f4d68099bb0 con 0x7f4d74033920 2024-09-13T08:37:23.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.544+0000 7f4d79400640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 0x7f4d5c07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.544+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6ef40618f0 con 0x7f6efc205610 2024-09-13T08:37:23.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.545+0000 7f4d79400640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 0x7f4d5c07ad10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c034000 tx=0x7f4d6c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.547+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4d68061bc0 con 0x7f4d74033920 2024-09-13T08:37:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.603+0000 7f3965800640 1 -- 172.21.15.31:0/1256892323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396015e160 msgr2=0x7f39600d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f3965800640 1 --2- 172.21.15.31:0/1256892323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396015e160 0x7f39600d0790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3950009900 tx=0x7f395002a560 comp rx=0 tx=0).stop 2024-09-13T08:37:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f96cd000640 1 -- 172.21.15.31:0/3109181847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96b00039b0 msgr2=0x7f96b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f96c6000640 1 -- 172.21.15.31:0/3109181847 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0094f0 con 0x7f96b00039b0 2024-09-13T08:37:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f96cd000640 1 --2- 172.21.15.31:0/3109181847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96b00039b0 0x7f96b0003dd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009920 tx=0x7f96b0005830 comp rx=0 tx=0).stop 2024-09-13T08:37:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f3965800640 1 -- 172.21.15.31:0/1256892323 shutdown_connections 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f3965800640 1 --2- 172.21.15.31:0/1256892323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39600d0cd0 0x7f39600d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f3965800640 1 --2- 172.21.15.31:0/1256892323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396015e160 0x7f39600d0790 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.604+0000 7f3965800640 1 -- 172.21.15.31:0/1256892323 >> 172.21.15.31:0/1256892323 conn(0x7f39600d3f50 msgr2=0x7f39600d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f3965800640 1 -- 172.21.15.31:0/1256892323 shutdown_connections 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f96cd000640 1 -- 172.21.15.31:0/3109181847 shutdown_connections 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f96cd000640 1 --2- 172.21.15.31:0/3109181847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96b00039b0 0x7f96b0003dd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f3965800640 1 -- 172.21.15.31:0/1256892323 wait complete. 2024-09-13T08:37:23.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f96cd000640 1 -- 172.21.15.31:0/3109181847 >> 172.21.15.31:0/3109181847 conn(0x7f96c0054560 msgr2=0x7f96c0054940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f96cd000640 1 -- 172.21.15.31:0/3109181847 shutdown_connections 2024-09-13T08:37:23.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.605+0000 7f96cd000640 1 -- 172.21.15.31:0/3109181847 wait complete. 2024-09-13T08:37:23.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 -- 172.21.15.31:0/3942747541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf34162260 msgr2=0x7fdf340cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3942747541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf34162260 0x7fdf340cdb80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c005fd0 tx=0x7fdf2c02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 -- 172.21.15.31:0/3942747541 shutdown_connections 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3942747541 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf340ce0c0 0x7fdf34166040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3942747541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf34162260 0x7fdf340cdb80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.606+0000 7fdf3b200640 1 -- 172.21.15.31:0/3942747541 >> 172.21.15.31:0/3942747541 conn(0x7fdf340b18f0 msgr2=0x7fdf340b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.607+0000 7fdf3b200640 1 -- 172.21.15.31:0/3942747541 shutdown_connections 2024-09-13T08:37:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.607+0000 7fdf3b200640 1 -- 172.21.15.31:0/3942747541 wait complete. 2024-09-13T08:37:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.608+0000 7f96cd000640 1 Processor -- start 2024-09-13T08:37:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.608+0000 7f3965800640 1 Processor -- start 2024-09-13T08:37:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.609+0000 7f96cd000640 1 -- start start 2024-09-13T08:37:23.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.608+0000 7f3965800640 1 -- start start 2024-09-13T08:37:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.610+0000 7fdf3b200640 1 Processor -- start 2024-09-13T08:37:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.610+0000 7fdf3b200640 1 -- start start 2024-09-13T08:37:23.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f96cd000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f96cd000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96c00dd2e0 0x7f96c004eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f96cd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 0x7f39600a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39600d0cd0 0x7f3960153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39600cfe00 con 0x7f39600c8340 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39600cfc80 con 0x7f396015e160 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.611+0000 7f3965800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39600c8f00 con 0x7f39600d0cd0 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f3964e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f3964e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60986/0 (socket says 172.21.15.31:60986) 2024-09-13T08:37:23.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f3964e00640 1 -- 172.21.15.31:0/897783051 learned_addr learned my addr 172.21.15.31:0/897783051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34796/0 (socket says 172.21.15.31:34796) 2024-09-13T08:37:23.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96c7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96c7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:60984/0 (socket says 172.21.15.31:60984) 2024-09-13T08:37:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96c7e00640 1 -- 172.21.15.31:0/377309336 learned_addr learned my addr 172.21.15.31:0/377309336 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96c0036160 con 0x7f96c004f430 2024-09-13T08:37:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96c0035fe0 con 0x7f96b00039b0 2024-09-13T08:37:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7f395fe00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 0x7f39600a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f3964e00640 1 -- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39600d0cd0 msgr2=0x7f3960153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f3964e00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39600d0cd0 0x7f3960153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f3964e00640 1 -- 172.21.15.31:0/897783051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 msgr2=0x7f39600a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f3964e00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 0x7f39600a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f3964e00640 1 -- 172.21.15.31:0/897783051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3960153bf0 con 0x7f396015e160 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f96c00362e0 con 0x7f96c00dd2e0 2024-09-13T08:37:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f96c6a00640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96c00dd2e0 0x7f96c004eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.613+0000 7f395fe00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 0x7f39600a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f3964e00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f395800c780 tx=0x7f395800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 0x7fdf34157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf34162260 0x7fdf34074270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf34166580 con 0x7fdf340ccc20 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf34162900 con 0x7fdf340ce0c0 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.612+0000 7fdf3b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdf34166700 con 0x7fdf34162260 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7fdf33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7fdf33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34822/0 (socket says 172.21.15.31:34822) 2024-09-13T08:37:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7fdf33e00640 1 -- 172.21.15.31:0/3508755684 learned_addr learned my addr 172.21.15.31:0/3508755684 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f96c7400640 1 -- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96c00dd2e0 msgr2=0x7f96c004eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f96c7400640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96c00dd2e0 0x7f96c004eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f96c7400640 1 -- 172.21.15.31:0/377309336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 msgr2=0x7f96c0175010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f96c7400640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf39200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf34162260 0x7fdf34074270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7fdf33400640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 0x7fdf34157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33e00640 1 -- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf34162260 msgr2=0x7fdf34074270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33e00640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf34162260 0x7fdf34074270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33e00640 1 -- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 msgr2=0x7fdf34157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33e00640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 0x7fdf34157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33e00640 1 -- 172.21.15.31:0/3508755684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf341f94f0 con 0x7fdf340ccc20 2024-09-13T08:37:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.614+0000 7f96c7400640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c0175df0 con 0x7f96b00039b0 2024-09-13T08:37:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7f96c7e00640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.615+0000 7fdf33400640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 0x7fdf34157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7fdf33e00640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009cd0 tx=0x7fdf2c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c031990 con 0x7fdf340ccc20 2024-09-13T08:37:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7f96c7400640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96b000c200 tx=0x7f96bc033dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf2c031c20 con 0x7fdf340ccc20 2024-09-13T08:37:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c0330d0 con 0x7fdf340ccc20 2024-09-13T08:37:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.616+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958017070 con 0x7f396015e160 2024-09-13T08:37:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf341f97d0 con 0x7fdf340ccc20 2024-09-13T08:37:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc005bd0 con 0x7f96b00039b0 2024-09-13T08:37:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96bc0332e0 con 0x7f96b00039b0 2024-09-13T08:37:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf341f9d90 con 0x7fdf340ccc20 2024-09-13T08:37:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c001e6d0 con 0x7f96b00039b0 2024-09-13T08:37:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.617+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c00dd760 con 0x7f96b00039b0 2024-09-13T08:37:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf34166040 con 0x7fdf340ccc20 2024-09-13T08:37:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.618+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3958005070 con 0x7f396015e160 2024-09-13T08:37:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.618+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39600cd450 con 0x7f396015e160 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.618+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39600cda10 con 0x7f396015e160 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958005500 con 0x7f396015e160 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.618+0000 7fe6fd600640 1 -- 172.21.15.31:0/757840319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f81615e0 msgr2=0x7fe6f80b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.618+0000 7fe6fd600640 1 --2- 172.21.15.31:0/757840319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f81615e0 0x7fe6f80b5640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe6f806d290 tx=0x7fe6e002a570 comp rx=0 tx=0).stop 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 -- 172.21.15.31:0/757840319 shutdown_connections 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 --2- 172.21.15.31:0/757840319 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f80d0840 0x7fe6f80d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 --2- 172.21.15.31:0/757840319 >> v2:172.21.15.31:3300/0 conn(0x7fe6f80d4c20 0x7fe6f80b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 --2- 172.21.15.31:0/757840319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f81615e0 0x7fe6f80b5640 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 -- 172.21.15.31:0/757840319 >> 172.21.15.31:0/757840319 conn(0x7fe6f80d4100 msgr2=0x7fe6f80d4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7fe6fd600640 1 -- 172.21.15.31:0/757840319 shutdown_connections 2024-09-13T08:37:23.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.619+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0317b0 con 0x7f96b00039b0 2024-09-13T08:37:23.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.621+0000 7fe6fd600640 1 -- 172.21.15.31:0/757840319 wait complete. 2024-09-13T08:37:23.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.622+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fdf2c033270 con 0x7fdf340ccc20 2024-09-13T08:37:23.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.622+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3938005290 con 0x7f396015e160 2024-09-13T08:37:23.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.623+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f395801a070 con 0x7f396015e160 2024-09-13T08:37:23.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.622+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96a4005290 con 0x7f96b00039b0 2024-09-13T08:37:23.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.624+0000 7fe6fd600640 1 Processor -- start 2024-09-13T08:37:23.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.625+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f96bc031990 con 0x7f96b00039b0 2024-09-13T08:37:23.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.625+0000 7f3957e00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 0x7f394007ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.625+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f3958099f60 con 0x7f396015e160 2024-09-13T08:37:23.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7f395fe00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 0x7f394007ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.624+0000 7fe6fd600640 1 -- start start 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 0x7fe6f80a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 0x7fe6f81514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f81615e0 0x7fe6f80ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f8161b40 con 0x7fe6f80d0840 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f81619c0 con 0x7fe6f80d4c20 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6fd600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f80cf480 con 0x7fe6f81615e0 2024-09-13T08:37:23.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7f96bb400640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 0x7f96a807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 0x7fe6f81514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fdf2be00640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 0x7fdf1407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fdf2c0bbc90 con 0x7fdf340ccc20 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 0x7fe6f80a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 0x7fe6f80a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34844/0 (socket says 172.21.15.31:34844) 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.626+0000 7fe6f6a00640 1 -- 172.21.15.31:0/1332341422 learned_addr learned my addr 172.21.15.31:0/1332341422 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6f7400640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f81615e0 0x7fe6f80ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f96bc0bace0 con 0x7f96b00039b0 2024-09-13T08:37:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7f395fe00640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 0x7f394007ade0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3950033000 tx=0x7f3950004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7f96c6a00640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 0x7f96a807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6efe00640 1 -- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f81615e0 msgr2=0x7fe6f80ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6efe00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f81615e0 0x7fe6f80ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6efe00640 1 -- 172.21.15.31:0/1332341422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 msgr2=0x7fe6f80a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6efe00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 0x7fe6f80a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.627+0000 7fe6efe00640 1 -- 172.21.15.31:0/1332341422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f809f3b0 con 0x7fe6f80d4c20 2024-09-13T08:37:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.628+0000 7fdf33400640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 0x7fdf1407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.628+0000 7fe6efe00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 0x7fe6f81514e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800d6d0 tx=0x7fe6e800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.628+0000 7f96c6a00640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 0x7f96a807ad40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96c0049430 tx=0x7f96b4006c70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.629+0000 7fdf33400640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 0x7fdf1407ad10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf20001220 tx=0x7fdf20009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.629+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f39580625b0 con 0x7f396015e160 2024-09-13T08:37:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.630+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdf2c03f030 con 0x7fdf340ccc20 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 -- 172.21.15.31:0/106798729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc161560 msgr2=0x7f0efc06e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 --2- 172.21.15.31:0/106798729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc161560 0x7f0efc06e660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0efc074fd0 tx=0x7f0ef402a980 comp rx=0 tx=0).stop 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 -- 172.21.15.31:0/106798729 shutdown_connections 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 --2- 172.21.15.31:0/106798729 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0efc06f720 0x7f0efc161b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 --2- 172.21.15.31:0/106798729 >> v2:172.21.15.45:3300/0 conn(0x7f0efc06ec90 0x7f0efc06f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 --2- 172.21.15.31:0/106798729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc161560 0x7f0efc06e660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 -- 172.21.15.31:0/106798729 >> 172.21.15.31:0/106798729 conn(0x7f0efc0b18f0 msgr2=0x7f0efc0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 -- 172.21.15.31:0/106798729 shutdown_connections 2024-09-13T08:37:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.631+0000 7f0f01e00640 1 -- 172.21.15.31:0/106798729 wait complete. 2024-09-13T08:37:23.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.634+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f96bc03f030 con 0x7f96b00039b0 2024-09-13T08:37:23.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.634+0000 7f0f01e00640 1 Processor -- start 2024-09-13T08:37:23.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.634+0000 7f0f01e00640 1 -- start start 2024-09-13T08:37:23.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0efc06f720 0x7f0efc150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc161560 0x7f0efc0ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc162410 con 0x7f0efc06ec90 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc162290 con 0x7f0efc161560 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0f01e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc162590 con 0x7f0efc06f720 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.636+0000 7f0efaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34848/0 (socket says 172.21.15.31:34848) 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 -- 172.21.15.31:0/3179969202 learned_addr learned my addr 172.21.15.31:0/3179969202 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.633+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8017070 con 0x7fe6f80d4c20 2024-09-13T08:37:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6e8005310 con 0x7fe6f80d4c20 2024-09-13T08:37:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8009820 con 0x7fe6f80d4c20 2024-09-13T08:37:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efb400640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc161560 0x7f0efc0ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 -- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0efc06f720 msgr2=0x7f0efc150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0efc06f720 0x7f0efc150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 -- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc161560 msgr2=0x7f0efc0ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc161560 0x7f0efc0ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.637+0000 7f0efaa00640 1 -- 172.21.15.31:0/3179969202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc15b320 con 0x7f0efc06ec90 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7f0efaa00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000f80 tx=0x7f0ef4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f809f690 con 0x7fe6f80d4c20 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4034040 con 0x7f0efc06ec90 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ef4004700 con 0x7f0efc06ec90 2024-09-13T08:37:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40456e0 con 0x7f0efc06ec90 2024-09-13T08:37:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.638+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f80a1080 con 0x7fe6f80d4c20 2024-09-13T08:37:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.639+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc1fc530 con 0x7f0efc06ec90 2024-09-13T08:37:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.639+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc15b010 con 0x7f0efc06ec90 2024-09-13T08:37:23.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.641+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc0cfa80 con 0x7f0efc06ec90 2024-09-13T08:37:23.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.642+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f80d3520 con 0x7fe6f80d4c20 2024-09-13T08:37:23.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.644+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0ef4037020 con 0x7f0efc06ec90 2024-09-13T08:37:23.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.645+0000 7f0ef3400640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 0x7f0edc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.645+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0ef40bc210 con 0x7f0efc06ec90 2024-09-13T08:37:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.646+0000 7f0efa000640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 0x7f0edc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.645+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe6e8009bd0 con 0x7fe6f80d4c20 2024-09-13T08:37:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.647+0000 7f0efa000640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 0x7f0edc07ad10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0efc160d00 tx=0x7f0ee8005bf0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.647+0000 7fe6eea00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 0x7fe6d407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.647+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe6e809ac60 con 0x7fe6f80d4c20 2024-09-13T08:37:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.647+0000 7fe6f6a00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 0x7fe6d407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.648+0000 7fe6f6a00640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 0x7fe6d407ad10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009230 tx=0x7fe6e0004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.651+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe6f80d3520 con 0x7fe6f80d4c20 2024-09-13T08:37:23.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.659+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0efc0cfa80 con 0x7f0efc06ec90 2024-09-13T08:37:23.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.900+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f60e4072ba0 con 0x7f60e416aba0 2024-09-13T08:37:23.902 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.901+0000 7f60da000640 1 -- 172.21.15.31:0/1991780557 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f60dc062760 con 0x7f60e416aba0 2024-09-13T08:37:23.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 msgr2=0x7f60c407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 0x7f60c407ad10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f60d400dc90 tx=0x7f60d4004730 comp rx=0 tx=0).stop 2024-09-13T08:37:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 msgr2=0x7f60e409aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 0x7f60e409aa10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00be30 tx=0x7f60dc00bf20 comp rx=0 tx=0).stop 2024-09-13T08:37:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 shutdown_connections 2024-09-13T08:37:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f60c4078850 0x7f60c407ad10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60e416aba0 0x7f60e409aa10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60e40b49c0 0x7f60e409c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 --2- 172.21.15.31:0/1991780557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60e4072730 0x7f60e408a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.905+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 >> 172.21.15.31:0/1991780557 conn(0x7f60e40b18f0 msgr2=0x7f60e40cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.906+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 shutdown_connections 2024-09-13T08:37:23.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.906+0000 7f60e8e00640 1 -- 172.21.15.31:0/1991780557 wait complete. 2024-09-13T08:37:23.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.2 2024-09-13T08:37:23.934 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:23.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.969+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6ed4005580 con 0x7f6efc205610 2024-09-13T08:37:23.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.970+0000 7f6ef3400640 1 -- 172.21.15.31:0/688264557 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6ef4066320 con 0x7f6efc205610 2024-09-13T08:37:23.972 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-13T08:37:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.973+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 msgr2=0x7f6edc07af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.973+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 0x7f6edc07af10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00a8b0 tx=0x7f6eec008040 comp rx=0 tx=0).stop 2024-09-13T08:37:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.973+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 msgr2=0x7f6efc0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.973+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c6c0 tx=0x7f6ef400cb80 comp rx=0 tx=0).stop 2024-09-13T08:37:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 shutdown_connections 2024-09-13T08:37:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6edc078a50 0x7f6edc07af10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6efc205610 0x7f6efc0b40b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6efc15e2c0 0x7f6efc2050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 --2- 172.21.15.31:0/688264557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6efc0cc040 0x7f6efc15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:23.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 >> 172.21.15.31:0/688264557 conn(0x7f6efc0cfd20 msgr2=0x7f6efc0d0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:23.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 shutdown_connections 2024-09-13T08:37:23.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.974+0000 7f6f01000640 1 -- 172.21.15.31:0/688264557 wait complete. 2024-09-13T08:37:23.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.993+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4d7400caf0 con 0x7f4d74033920 2024-09-13T08:37:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.994+0000 7f4d72000640 1 -- 172.21.15.31:0/1019397789 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4d680665f0 con 0x7f4d74033920 2024-09-13T08:37:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 msgr2=0x7f4d5c07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 0x7f4d5c07ad10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c034000 tx=0x7f4d6c004730 comp rx=0 tx=0).stop 2024-09-13T08:37:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3938005580 con 0x7f396015e160 2024-09-13T08:37:24.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2024-09-13T08:37:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 msgr2=0x7f4d74039530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 0x7f4d74039530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c500 tx=0x7f4d6800c9c0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 shutdown_connections 2024-09-13T08:37:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d740536a0 0x7f4d74052a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4d5c078850 0x7f4d5c07ad10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4d740e6e30 0x7f4d7403b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 --2- 172.21.15.31:0/1019397789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d74033920 0x7f4d74039530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.998+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 >> 172.21.15.31:0/1019397789 conn(0x7f4d7401f9a0 msgr2=0x7f4d740e4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.999+0000 7f3957e00640 1 -- 172.21.15.31:0/897783051 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3958061eb0 con 0x7f396015e160 2024-09-13T08:37:24.001 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.001 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-13T08:37:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.999+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 shutdown_connections 2024-09-13T08:37:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:23.999+0000 7f4d80e00640 1 -- 172.21.15.31:0/1019397789 wait complete. 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 msgr2=0x7f394007ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 0x7f394007ade0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3950033000 tx=0x7f3950004730 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 msgr2=0x7f39601536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f395800c780 tx=0x7f395800cc40 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 shutdown_connections 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396015e160 0x7f39601536b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3940078920 0x7f394007ade0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39600d0cd0 0x7f3960153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 --2- 172.21.15.31:0/897783051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39600c8340 0x7f39600a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.002+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 >> 172.21.15.31:0/897783051 conn(0x7f39600d3f50 msgr2=0x7f39600cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.003+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 shutdown_connections 2024-09-13T08:37:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.003+0000 7f3965800640 1 -- 172.21.15.31:0/897783051 wait complete. 2024-09-13T08:37:24.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.1 2024-09-13T08:37:24.020 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.3 2024-09-13T08:37:24.025 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.038+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdf34165cc0 con 0x7fdf340ccc20 2024-09-13T08:37:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.039+0000 7fdf2be00640 1 -- 172.21.15.31:0/3508755684 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdf2c0842f0 con 0x7fdf340ccc20 2024-09-13T08:37:24.039 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 msgr2=0x7fdf1407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 0x7fdf1407ad10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf20001220 tx=0x7fdf20009040 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 msgr2=0x7fdf3409bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009cd0 tx=0x7fdf2c0316f0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 shutdown_connections 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf14078850 0x7fdf1407ad10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf34162260 0x7fdf34074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf340ce0c0 0x7fdf34157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 --2- 172.21.15.31:0/3508755684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf340ccc20 0x7fdf3409bef0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 >> 172.21.15.31:0/3508755684 conn(0x7fdf340b18f0 msgr2=0x7fdf340cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 shutdown_connections 2024-09-13T08:37:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.041+0000 7fdf3b200640 1 -- 172.21.15.31:0/3508755684 wait complete. 2024-09-13T08:37:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.044+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe6f808e0c0 con 0x7fe6f80d4c20 2024-09-13T08:37:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.044+0000 7fe6eea00640 1 -- 172.21.15.31:0/1332341422 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe6e8062be0 con 0x7fe6f80d4c20 2024-09-13T08:37:24.045 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 msgr2=0x7fe6d407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 0x7fe6d407ad10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009230 tx=0x7fe6e0004730 comp rx=0 tx=0).stop 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 msgr2=0x7fe6f81514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 0x7fe6f81514e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800d6d0 tx=0x7fe6e800db90 comp rx=0 tx=0).stop 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 shutdown_connections 2024-09-13T08:37:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6d4078850 0x7fe6d407ad10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6f81615e0 0x7fe6f80ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6f80d4c20 0x7fe6f81514e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 --2- 172.21.15.31:0/1332341422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f80d0840 0x7fe6f80a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 >> 172.21.15.31:0/1332341422 conn(0x7fe6f80d4100 msgr2=0x7fe6f80b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 shutdown_connections 2024-09-13T08:37:24.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.049+0000 7fe6fd600640 1 -- 172.21.15.31:0/1332341422 wait complete. 2024-09-13T08:37:24.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.7 2024-09-13T08:37:24.064 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2024-09-13T08:37:24.069 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.095+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f96a4005580 con 0x7f96b00039b0 2024-09-13T08:37:24.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.095+0000 7f96bb400640 1 -- 172.21.15.31:0/377309336 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96bc082cf0 con 0x7f96b00039b0 2024-09-13T08:37:24.096 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 msgr2=0x7f96a807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 0x7f96a807ad40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96c0049430 tx=0x7f96b4006c70 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 msgr2=0x7f96c001f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96b000c200 tx=0x7f96bc033dc0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 shutdown_connections 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96c004f430 0x7f96c0175010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f96a8078880 0x7f96a807ad40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96c00dd2e0 0x7f96c004eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 --2- 172.21.15.31:0/377309336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00039b0 0x7f96c001f1b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 >> 172.21.15.31:0/377309336 conn(0x7f96c0054560 msgr2=0x7f96c0053340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 shutdown_connections 2024-09-13T08:37:24.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.099+0000 7f96cd000640 1 -- 172.21.15.31:0/377309336 wait complete. 2024-09-13T08:37:24.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.107+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0efc161b40 con 0x7f0efc06ec90 2024-09-13T08:37:24.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.107+0000 7f0ef3400640 1 -- 172.21.15.31:0/3179969202 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ef40847f0 con 0x7f0efc06ec90 2024-09-13T08:37:24.108 INFO:teuthology.orchestra.run.smithi031.stdout:60129542151 2024-09-13T08:37:24.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 msgr2=0x7f0edc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 0x7f0edc07ad10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0efc160d00 tx=0x7f0ee8005bf0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 msgr2=0x7f0efc15aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000f80 tx=0x7f0ef4003f90 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 shutdown_connections 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc161560 0x7f0efc0ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0edc078850 0x7f0edc07ad10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0efc06f720 0x7f0efc150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 --2- 172.21.15.31:0/3179969202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc06ec90 0x7f0efc15aad0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.109+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 >> 172.21.15.31:0/3179969202 conn(0x7f0efc0b18f0 msgr2=0x7f0efc0cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.110+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 shutdown_connections 2024-09-13T08:37:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.110+0000 7f0f01e00640 1 -- 172.21.15.31:0/3179969202 wait complete. 2024-09-13T08:37:24.117 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.6 2024-09-13T08:37:24.117 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.4 2024-09-13T08:37:24.128 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:24.128 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:37:24.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 -- 172.21.15.31:0/4023460229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd280ce3c0 msgr2=0x7fcd280ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 --2- 172.21.15.31:0/4023460229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd280ce3c0 0x7fcd280ce850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd280cd9e0 tx=0x7fcd2002aa10 comp rx=0 tx=0).stop 2024-09-13T08:37:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 -- 172.21.15.31:0/4023460229 shutdown_connections 2024-09-13T08:37:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 --2- 172.21.15.31:0/4023460229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd280ce3c0 0x7fcd280ce850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 --2- 172.21.15.31:0/4023460229 >> v2:172.21.15.45:3300/0 conn(0x7fcd2815f9e0 0x7fcd2815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 --2- 172.21.15.31:0/4023460229 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd2815bd70 0x7fcd280cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 -- 172.21.15.31:0/4023460229 >> 172.21.15.31:0/4023460229 conn(0x7fcd280cc910 msgr2=0x7fcd280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 -- 172.21.15.31:0/4023460229 shutdown_connections 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.326+0000 7fcd2f800640 1 -- 172.21.15.31:0/4023460229 wait complete. 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.328+0000 7fcd2f800640 1 Processor -- start 2024-09-13T08:37:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.328+0000 7fcd2f800640 1 -- start start 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd280ce3c0 0x7fcd2815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd2815f9e0 0x7fcd280b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcd280cb9d0 con 0x7fcd2815bd70 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcd280cb850 con 0x7fcd2815f9e0 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2f800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcd280cbc50 con 0x7fcd280ce3c0 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34860/0 (socket says 172.21.15.31:34860) 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 -- 172.21.15.31:0/2970420685 learned_addr learned my addr 172.21.15.31:0/2970420685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2d800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd2815f9e0 0x7fcd280b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd2ce00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd280ce3c0 0x7fcd2815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 -- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd280ce3c0 msgr2=0x7fcd2815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd280ce3c0 0x7fcd2815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 -- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd2815f9e0 msgr2=0x7fcd280b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd2815f9e0 0x7fcd280b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.329+0000 7fcd1fe00640 1 -- 172.21.15.31:0/2970420685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2809d500 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd1fe00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000d6d0 tx=0x7fcd1000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10017070 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd10004310 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10009a80 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2809da80 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.330+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd280b3e20 con 0x7fcd2815bd70 2024-09-13T08:37:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.331+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd28072aa0 con 0x7fcd2815bd70 2024-09-13T08:37:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.332+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fcd100035f0 con 0x7fcd2815bd70 2024-09-13T08:37:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.333+0000 7fcd25600640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 0x7fcd0407ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.333+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fcd1009abe0 con 0x7fcd2815bd70 2024-09-13T08:37:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.333+0000 7fcd2ce00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 0x7fcd0407ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.334+0000 7fcd2ce00640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 0x7fcd0407ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcd18005fd0 tx=0x7fcd18005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:24.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.337+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcd100631b0 con 0x7fcd2815bd70 2024-09-13T08:37:24.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.529+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd280ccf00 con 0x7fcd04078880 2024-09-13T08:37:24.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.534+0000 7fcd25600640 1 -- 172.21.15.31:0/2970420685 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251822 (secure 0 0 0) 0x7fcd280ccf00 con 0x7fcd04078880 2024-09-13T08:37:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:24.539 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:24.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 msgr2=0x7fcd0407ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 0x7fcd0407ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcd18005fd0 tx=0x7fcd18005850 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 msgr2=0x7fcd2809b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000d6d0 tx=0x7fcd1000db90 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 shutdown_connections 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd2815f9e0 0x7fcd280b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd2815bd70 0x7fcd2809b800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcd04078880 0x7fcd0407ad40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 --2- 172.21.15.31:0/2970420685 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcd280ce3c0 0x7fcd2815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 >> 172.21.15.31:0/2970420685 conn(0x7fcd280cc910 msgr2=0x7fcd280ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 shutdown_connections 2024-09-13T08:37:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.538+0000 7fcd2f800640 1 -- 172.21.15.31:0/2970420685 wait complete. 2024-09-13T08:37:24.555 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-13T08:37:23.040197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235936,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738784,"statfs":{"total":773094113280,"available":772852514816,"internally_reserved":0,"allocated":3121152,"data_stored":1833196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003379"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021976+0000","last_change":"2024-09-13T08:37:10.038242+0000","last_active":"2024-09-13T08:37:11.021976+0000","last_peered":"2024-09-13T08:37:11.021976+0000","last_clean":"2024-09-13T08:37:11.021976+0000","last_became_active":"2024-09-13T08:37:10.037984+0000","last_became_peered":"2024-09-13T08:37:10.037984+0000","last_unstale":"2024-09-13T08:37:11.021976+0000","last_undegraded":"2024-09-13T08:37:11.021976+0000","last_fullsized":"2024-09-13T08:37:11.021976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:05.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328871+0000","last_change":"2024-09-13T08:37:10.043402+0000","last_active":"2024-09-13T08:37:11.328871+0000","last_peered":"2024-09-13T08:37:11.328871+0000","last_clean":"2024-09-13T08:37:11.328871+0000","last_became_active":"2024-09-13T08:37:10.041028+0000","last_became_peered":"2024-09-13T08:37:10.041028+0000","last_unstale":"2024-09-13T08:37:11.328871+0000","last_undegraded":"2024-09-13T08:37:11.328871+0000","last_fullsized":"2024-09-13T08:37:11.328871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328967+0000","last_change":"2024-09-13T08:37:10.038931+0000","last_active":"2024-09-13T08:37:11.328967+0000","last_peered":"2024-09-13T08:37:11.328967+0000","last_clean":"2024-09-13T08:37:11.328967+0000","last_became_active":"2024-09-13T08:37:10.038676+0000","last_became_peered":"2024-09-13T08:37:10.038676+0000","last_unstale":"2024-09-13T08:37:11.328967+0000","last_undegraded":"2024-09-13T08:37:11.328967+0000","last_fullsized":"2024-09-13T08:37:11.328967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015339+0000","last_change":"2024-09-13T08:37:10.018106+0000","last_active":"2024-09-13T08:37:11.015339+0000","last_peered":"2024-09-13T08:37:11.015339+0000","last_clean":"2024-09-13T08:37:11.015339+0000","last_became_active":"2024-09-13T08:37:10.017812+0000","last_became_peered":"2024-09-13T08:37:10.017812+0000","last_unstale":"2024-09-13T08:37:11.015339+0000","last_undegraded":"2024-09-13T08:37:11.015339+0000","last_fullsized":"2024-09-13T08:37:11.015339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021892+0000","last_change":"2024-09-13T08:37:10.035463+0000","last_active":"2024-09-13T08:37:11.021892+0000","last_peered":"2024-09-13T08:37:11.021892+0000","last_clean":"2024-09-13T08:37:11.021892+0000","last_became_active":"2024-09-13T08:37:10.032852+0000","last_became_peered":"2024-09-13T08:37:10.032852+0000","last_unstale":"2024-09-13T08:37:11.021892+0000","last_undegraded":"2024-09-13T08:37:11.021892+0000","last_fullsized":"2024-09-13T08:37:11.021892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:34.731391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015062+0000","last_change":"2024-09-13T08:37:10.030116+0000","last_active":"2024-09-13T08:37:11.015062+0000","last_peered":"2024-09-13T08:37:11.015062+0000","last_clean":"2024-09-13T08:37:11.015062+0000","last_became_active":"2024-09-13T08:37:10.029773+0000","last_became_peered":"2024-09-13T08:37:10.029773+0000","last_unstale":"2024-09-13T08:37:11.015062+0000","last_undegraded":"2024-09-13T08:37:11.015062+0000","last_fullsized":"2024-09-13T08:37:11.015062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021640+0000","last_change":"2024-09-13T08:37:10.025604+0000","last_active":"2024-09-13T08:37:11.021640+0000","last_peered":"2024-09-13T08:37:11.021640+0000","last_clean":"2024-09-13T08:37:11.021640+0000","last_became_active":"2024-09-13T08:37:10.020828+0000","last_became_peered":"2024-09-13T08:37:10.020828+0000","last_unstale":"2024-09-13T08:37:11.021640+0000","last_undegraded":"2024-09-13T08:37:11.021640+0000","last_fullsized":"2024-09-13T08:37:11.021640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328358+0000","last_change":"2024-09-13T08:37:10.041296+0000","last_active":"2024-09-13T08:37:11.328358+0000","last_peered":"2024-09-13T08:37:11.328358+0000","last_clean":"2024-09-13T08:37:11.328358+0000","last_became_active":"2024-09-13T08:37:10.035151+0000","last_became_peered":"2024-09-13T08:37:10.035151+0000","last_unstale":"2024-09-13T08:37:11.328358+0000","last_undegraded":"2024-09-13T08:37:11.328358+0000","last_fullsized":"2024-09-13T08:37:11.328358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:28.830741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723967+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:12.723967+0000","last_peered":"2024-09-13T08:37:12.723967+0000","last_clean":"2024-09-13T08:37:12.723967+0000","last_became_active":"2024-09-13T08:37:10.026743+0000","last_became_peered":"2024-09-13T08:37:10.026743+0000","last_unstale":"2024-09-13T08:37:12.723967+0000","last_undegraded":"2024-09-13T08:37:12.723967+0000","last_fullsized":"2024-09-13T08:37:12.723967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022008+0000","last_change":"2024-09-13T08:37:10.020548+0000","last_active":"2024-09-13T08:37:11.022008+0000","last_peered":"2024-09-13T08:37:11.022008+0000","last_clean":"2024-09-13T08:37:11.022008+0000","last_became_active":"2024-09-13T08:37:10.020144+0000","last_became_peered":"2024-09-13T08:37:10.020144+0000","last_unstale":"2024-09-13T08:37:11.022008+0000","last_undegraded":"2024-09-13T08:37:11.022008+0000","last_fullsized":"2024-09-13T08:37:11.022008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:13.408879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022140+0000","last_change":"2024-09-13T08:37:10.038191+0000","last_active":"2024-09-13T08:37:11.022140+0000","last_peered":"2024-09-13T08:37:11.022140+0000","last_clean":"2024-09-13T08:37:11.022140+0000","last_became_active":"2024-09-13T08:37:10.037935+0000","last_became_peered":"2024-09-13T08:37:10.037935+0000","last_unstale":"2024-09-13T08:37:11.022140+0000","last_undegraded":"2024-09-13T08:37:11.022140+0000","last_fullsized":"2024-09-13T08:37:11.022140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021867+0000","last_change":"2024-09-13T08:37:10.025910+0000","last_active":"2024-09-13T08:37:11.021867+0000","last_peered":"2024-09-13T08:37:11.021867+0000","last_clean":"2024-09-13T08:37:11.021867+0000","last_became_active":"2024-09-13T08:37:10.023349+0000","last_became_peered":"2024-09-13T08:37:10.023349+0000","last_unstale":"2024-09-13T08:37:11.021867+0000","last_undegraded":"2024-09-13T08:37:11.021867+0000","last_fullsized":"2024-09-13T08:37:11.021867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:02.005454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863687+0000","last_change":"2024-09-13T08:37:10.033914+0000","last_active":"2024-09-13T08:37:12.863687+0000","last_peered":"2024-09-13T08:37:12.863687+0000","last_clean":"2024-09-13T08:37:12.863687+0000","last_became_active":"2024-09-13T08:37:10.030947+0000","last_became_peered":"2024-09-13T08:37:10.030947+0000","last_unstale":"2024-09-13T08:37:12.863687+0000","last_undegraded":"2024-09-13T08:37:12.863687+0000","last_fullsized":"2024-09-13T08:37:12.863687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:49.714886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863461+0000","last_change":"2024-09-13T08:37:10.024803+0000","last_active":"2024-09-13T08:37:12.863461+0000","last_peered":"2024-09-13T08:37:12.863461+0000","last_clean":"2024-09-13T08:37:12.863461+0000","last_became_active":"2024-09-13T08:37:10.024512+0000","last_became_peered":"2024-09-13T08:37:10.024512+0000","last_unstale":"2024-09-13T08:37:12.863461+0000","last_undegraded":"2024-09-13T08:37:12.863461+0000","last_fullsized":"2024-09-13T08:37:12.863461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328365+0000","last_change":"2024-09-13T08:37:10.035650+0000","last_active":"2024-09-13T08:37:11.328365+0000","last_peered":"2024-09-13T08:37:11.328365+0000","last_clean":"2024-09-13T08:37:11.328365+0000","last_became_active":"2024-09-13T08:37:10.035331+0000","last_became_peered":"2024-09-13T08:37:10.035331+0000","last_unstale":"2024-09-13T08:37:11.328365+0000","last_undegraded":"2024-09-13T08:37:11.328365+0000","last_fullsized":"2024-09-13T08:37:11.328365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.703160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.022368+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.022095+0000","last_became_peered":"2024-09-13T08:37:10.022095+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596810+0000","last_change":"2024-09-13T08:37:10.032355+0000","last_active":"2024-09-13T08:37:13.596810+0000","last_peered":"2024-09-13T08:37:13.596810+0000","last_clean":"2024-09-13T08:37:13.596810+0000","last_became_active":"2024-09-13T08:37:10.032098+0000","last_became_peered":"2024-09-13T08:37:10.032098+0000","last_unstale":"2024-09-13T08:37:13.596810+0000","last_undegraded":"2024-09-13T08:37:13.596810+0000","last_fullsized":"2024-09-13T08:37:13.596810+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:32:27.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017318+0000","last_change":"2024-09-13T08:37:10.022000+0000","last_active":"2024-09-13T08:37:11.017318+0000","last_peered":"2024-09-13T08:37:11.017318+0000","last_clean":"2024-09-13T08:37:11.017318+0000","last_became_active":"2024-09-13T08:37:10.021622+0000","last_became_peered":"2024-09-13T08:37:10.021622+0000","last_unstale":"2024-09-13T08:37:11.017318+0000","last_undegraded":"2024-09-13T08:37:11.017318+0000","last_fullsized":"2024-09-13T08:37:11.017318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596836+0000","last_change":"2024-09-13T08:37:10.026234+0000","last_active":"2024-09-13T08:37:13.596836+0000","last_peered":"2024-09-13T08:37:13.596836+0000","last_clean":"2024-09-13T08:37:13.596836+0000","last_became_active":"2024-09-13T08:37:10.025970+0000","last_became_peered":"2024-09-13T08:37:10.025970+0000","last_unstale":"2024-09-13T08:37:13.596836+0000","last_undegraded":"2024-09-13T08:37:13.596836+0000","last_fullsized":"2024-09-13T08:37:13.596836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:52.574964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021932+0000","last_change":"2024-09-13T08:37:10.028101+0000","last_active":"2024-09-13T08:37:11.021932+0000","last_peered":"2024-09-13T08:37:11.021932+0000","last_clean":"2024-09-13T08:37:11.021932+0000","last_became_active":"2024-09-13T08:37:10.021666+0000","last_became_peered":"2024-09-13T08:37:10.021666+0000","last_unstale":"2024-09-13T08:37:11.021932+0000","last_undegraded":"2024-09-13T08:37:11.021932+0000","last_fullsized":"2024-09-13T08:37:11.021932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:43.682175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723677+0000","last_change":"2024-09-13T08:37:10.020945+0000","last_active":"2024-09-13T08:37:12.723677+0000","last_peered":"2024-09-13T08:37:12.723677+0000","last_clean":"2024-09-13T08:37:12.723677+0000","last_became_active":"2024-09-13T08:37:10.020675+0000","last_became_peered":"2024-09-13T08:37:10.020675+0000","last_unstale":"2024-09-13T08:37:12.723677+0000","last_undegraded":"2024-09-13T08:37:12.723677+0000","last_fullsized":"2024-09-13T08:37:12.723677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:41.501115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328354+0000","last_change":"2024-09-13T08:37:10.036002+0000","last_active":"2024-09-13T08:37:11.328354+0000","last_peered":"2024-09-13T08:37:11.328354+0000","last_clean":"2024-09-13T08:37:11.328354+0000","last_became_active":"2024-09-13T08:37:10.035684+0000","last_became_peered":"2024-09-13T08:37:10.035684+0000","last_unstale":"2024-09-13T08:37:11.328354+0000","last_undegraded":"2024-09-13T08:37:11.328354+0000","last_fullsized":"2024-09-13T08:37:11.328354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.032065+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.031802+0000","last_became_peered":"2024-09-13T08:37:10.031802+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328937+0000","last_change":"2024-09-13T08:37:10.036040+0000","last_active":"2024-09-13T08:37:11.328937+0000","last_peered":"2024-09-13T08:37:11.328937+0000","last_clean":"2024-09-13T08:37:11.328937+0000","last_became_active":"2024-09-13T08:37:10.035698+0000","last_became_peered":"2024-09-13T08:37:10.035698+0000","last_unstale":"2024-09-13T08:37:11.328937+0000","last_undegraded":"2024-09-13T08:37:11.328937+0000","last_fullsized":"2024-09-13T08:37:11.328937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015904+0000","last_change":"2024-09-13T08:37:10.023312+0000","last_active":"2024-09-13T08:37:11.015904+0000","last_peered":"2024-09-13T08:37:11.015904+0000","last_clean":"2024-09-13T08:37:11.015904+0000","last_became_active":"2024-09-13T08:37:10.022193+0000","last_became_peered":"2024-09-13T08:37:10.022193+0000","last_unstale":"2024-09-13T08:37:11.015904+0000","last_undegraded":"2024-09-13T08:37:11.015904+0000","last_fullsized":"2024-09-13T08:37:11.015904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863802+0000","last_change":"2024-09-13T08:37:10.028354+0000","last_active":"2024-09-13T08:37:12.863802+0000","last_peered":"2024-09-13T08:37:12.863802+0000","last_clean":"2024-09-13T08:37:12.863802+0000","last_became_active":"2024-09-13T08:37:10.027989+0000","last_became_peered":"2024-09-13T08:37:10.027989+0000","last_unstale":"2024-09-13T08:37:12.863802+0000","last_undegraded":"2024-09-13T08:37:12.863802+0000","last_fullsized":"2024-09-13T08:37:12.863802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863758+0000","last_change":"2024-09-13T08:37:10.039029+0000","last_active":"2024-09-13T08:37:12.863758+0000","last_peered":"2024-09-13T08:37:12.863758+0000","last_clean":"2024-09-13T08:37:12.863758+0000","last_became_active":"2024-09-13T08:37:10.038689+0000","last_became_peered":"2024-09-13T08:37:10.038689+0000","last_unstale":"2024-09-13T08:37:12.863758+0000","last_undegraded":"2024-09-13T08:37:12.863758+0000","last_fullsized":"2024-09-13T08:37:12.863758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328458+0000","last_change":"2024-09-13T08:37:10.035713+0000","last_active":"2024-09-13T08:37:11.328458+0000","last_peered":"2024-09-13T08:37:11.328458+0000","last_clean":"2024-09-13T08:37:11.328458+0000","last_became_active":"2024-09-13T08:37:10.035292+0000","last_became_peered":"2024-09-13T08:37:10.035292+0000","last_unstale":"2024-09-13T08:37:11.328458+0000","last_undegraded":"2024-09-13T08:37:11.328458+0000","last_fullsized":"2024-09-13T08:37:11.328458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596170+0000","last_change":"2024-09-13T08:37:10.029548+0000","last_active":"2024-09-13T08:37:13.596170+0000","last_peered":"2024-09-13T08:37:13.596170+0000","last_clean":"2024-09-13T08:37:13.596170+0000","last_became_active":"2024-09-13T08:37:10.029292+0000","last_became_peered":"2024-09-13T08:37:10.029292+0000","last_unstale":"2024-09-13T08:37:13.596170+0000","last_undegraded":"2024-09-13T08:37:13.596170+0000","last_fullsized":"2024-09-13T08:37:13.596170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016762+0000","last_change":"2024-09-13T08:37:10.012936+0000","last_active":"2024-09-13T08:37:11.016762+0000","last_peered":"2024-09-13T08:37:11.016762+0000","last_clean":"2024-09-13T08:37:11.016762+0000","last_became_active":"2024-09-13T08:37:10.012582+0000","last_became_peered":"2024-09-13T08:37:10.012582+0000","last_unstale":"2024-09-13T08:37:11.016762+0000","last_undegraded":"2024-09-13T08:37:11.016762+0000","last_fullsized":"2024-09-13T08:37:11.016762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015064+0000","last_change":"2024-09-13T08:37:10.017195+0000","last_active":"2024-09-13T08:37:11.015064+0000","last_peered":"2024-09-13T08:37:11.015064+0000","last_clean":"2024-09-13T08:37:11.015064+0000","last_became_active":"2024-09-13T08:37:10.016877+0000","last_became_peered":"2024-09-13T08:37:10.016877+0000","last_unstale":"2024-09-13T08:37:11.015064+0000","last_undegraded":"2024-09-13T08:37:11.015064+0000","last_fullsized":"2024-09-13T08:37:11.015064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.474080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723910+0000","last_change":"2024-09-13T08:37:10.023947+0000","last_active":"2024-09-13T08:37:12.723910+0000","last_peered":"2024-09-13T08:37:12.723910+0000","last_clean":"2024-09-13T08:37:12.723910+0000","last_became_active":"2024-09-13T08:37:10.023659+0000","last_became_peered":"2024-09-13T08:37:10.023659+0000","last_unstale":"2024-09-13T08:37:12.723910+0000","last_undegraded":"2024-09-13T08:37:12.723910+0000","last_fullsized":"2024-09-13T08:37:12.723910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328520+0000","last_change":"2024-09-13T08:37:09.014843+0000","last_active":"2024-09-13T08:37:11.328520+0000","last_peered":"2024-09-13T08:37:11.328520+0000","last_clean":"2024-09-13T08:37:11.328520+0000","last_became_active":"2024-09-13T08:37:09.014454+0000","last_became_peered":"2024-09-13T08:37:09.014454+0000","last_unstale":"2024-09-13T08:37:11.328520+0000","last_undegraded":"2024-09-13T08:37:11.328520+0000","last_fullsized":"2024-09-13T08:37:11.328520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596441+0000","last_change":"2024-09-13T08:37:09.024672+0000","last_active":"2024-09-13T08:37:13.596441+0000","last_peered":"2024-09-13T08:37:13.596441+0000","last_clean":"2024-09-13T08:37:13.596441+0000","last_became_active":"2024-09-13T08:37:09.024363+0000","last_became_peered":"2024-09-13T08:37:09.024363+0000","last_unstale":"2024-09-13T08:37:13.596441+0000","last_undegraded":"2024-09-13T08:37:13.596441+0000","last_fullsized":"2024-09-13T08:37:13.596441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:55.612953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863222+0000","last_change":"2024-09-13T08:37:09.014986+0000","last_active":"2024-09-13T08:37:12.863222+0000","last_peered":"2024-09-13T08:37:12.863222+0000","last_clean":"2024-09-13T08:37:12.863222+0000","last_became_active":"2024-09-13T08:37:09.014658+0000","last_became_peered":"2024-09-13T08:37:09.014658+0000","last_unstale":"2024-09-13T08:37:12.863222+0000","last_undegraded":"2024-09-13T08:37:12.863222+0000","last_fullsized":"2024-09-13T08:37:12.863222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328628+0000","last_change":"2024-09-13T08:37:10.030653+0000","last_active":"2024-09-13T08:37:11.328628+0000","last_peered":"2024-09-13T08:37:11.328628+0000","last_clean":"2024-09-13T08:37:11.328628+0000","last_became_active":"2024-09-13T08:37:10.030383+0000","last_became_peered":"2024-09-13T08:37:10.030383+0000","last_unstale":"2024-09-13T08:37:11.328628+0000","last_undegraded":"2024-09-13T08:37:11.328628+0000","last_fullsized":"2024-09-13T08:37:11.328628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596255+0000","last_change":"2024-09-13T08:37:09.023571+0000","last_active":"2024-09-13T08:37:13.596255+0000","last_peered":"2024-09-13T08:37:13.596255+0000","last_clean":"2024-09-13T08:37:13.596255+0000","last_became_active":"2024-09-13T08:37:09.023048+0000","last_became_peered":"2024-09-13T08:37:09.023048+0000","last_unstale":"2024-09-13T08:37:13.596255+0000","last_undegraded":"2024-09-13T08:37:13.596255+0000","last_fullsized":"2024-09-13T08:37:13.596255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:14.567695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723890+0000","last_change":"2024-09-13T08:37:09.010512+0000","last_active":"2024-09-13T08:37:12.723890+0000","last_peered":"2024-09-13T08:37:12.723890+0000","last_clean":"2024-09-13T08:37:12.723890+0000","last_became_active":"2024-09-13T08:37:09.010229+0000","last_became_peered":"2024-09-13T08:37:09.010229+0000","last_unstale":"2024-09-13T08:37:12.723890+0000","last_undegraded":"2024-09-13T08:37:12.723890+0000","last_fullsized":"2024-09-13T08:37:12.723890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015969+0000","last_change":"2024-09-13T08:37:10.020061+0000","last_active":"2024-09-13T08:37:11.015969+0000","last_peered":"2024-09-13T08:37:11.015969+0000","last_clean":"2024-09-13T08:37:11.015969+0000","last_became_active":"2024-09-13T08:37:10.019663+0000","last_became_peered":"2024-09-13T08:37:10.019663+0000","last_unstale":"2024-09-13T08:37:11.015969+0000","last_undegraded":"2024-09-13T08:37:11.015969+0000","last_fullsized":"2024-09-13T08:37:11.015969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:48.021040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021832+0000","last_change":"2024-09-13T08:37:09.011936+0000","last_active":"2024-09-13T08:37:11.021832+0000","last_peered":"2024-09-13T08:37:11.021832+0000","last_clean":"2024-09-13T08:37:11.021832+0000","last_became_active":"2024-09-13T08:37:09.011670+0000","last_became_peered":"2024-09-13T08:37:09.011670+0000","last_unstale":"2024-09-13T08:37:11.021832+0000","last_undegraded":"2024-09-13T08:37:11.021832+0000","last_fullsized":"2024-09-13T08:37:11.021832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:31.341815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329184+0000","last_change":"2024-09-13T08:37:10.027709+0000","last_active":"2024-09-13T08:37:11.329184+0000","last_peered":"2024-09-13T08:37:11.329184+0000","last_clean":"2024-09-13T08:37:11.329184+0000","last_became_active":"2024-09-13T08:37:10.027410+0000","last_became_peered":"2024-09-13T08:37:10.027410+0000","last_unstale":"2024-09-13T08:37:11.329184+0000","last_undegraded":"2024-09-13T08:37:11.329184+0000","last_fullsized":"2024-09-13T08:37:11.329184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:25.100540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017898+0000","last_change":"2024-09-13T08:37:09.003083+0000","last_active":"2024-09-13T08:37:11.017898+0000","last_peered":"2024-09-13T08:37:11.017898+0000","last_clean":"2024-09-13T08:37:11.017898+0000","last_became_active":"2024-09-13T08:37:09.002697+0000","last_became_peered":"2024-09-13T08:37:09.002697+0000","last_unstale":"2024-09-13T08:37:11.017898+0000","last_undegraded":"2024-09-13T08:37:11.017898+0000","last_fullsized":"2024-09-13T08:37:11.017898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:56.231941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596339+0000","last_change":"2024-09-13T08:37:10.021360+0000","last_active":"2024-09-13T08:37:13.596339+0000","last_peered":"2024-09-13T08:37:13.596339+0000","last_clean":"2024-09-13T08:37:13.596339+0000","last_became_active":"2024-09-13T08:37:10.021080+0000","last_became_peered":"2024-09-13T08:37:10.021080+0000","last_unstale":"2024-09-13T08:37:13.596339+0000","last_undegraded":"2024-09-13T08:37:13.596339+0000","last_fullsized":"2024-09-13T08:37:13.596339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:32.591794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015415+0000","last_change":"2024-09-13T08:37:09.014879+0000","last_active":"2024-09-13T08:37:11.015415+0000","last_peered":"2024-09-13T08:37:11.015415+0000","last_clean":"2024-09-13T08:37:11.015415+0000","last_became_active":"2024-09-13T08:37:09.014572+0000","last_became_peered":"2024-09-13T08:37:09.014572+0000","last_unstale":"2024-09-13T08:37:11.015415+0000","last_undegraded":"2024-09-13T08:37:11.015415+0000","last_fullsized":"2024-09-13T08:37:11.015415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:14.033544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022331+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:11.022331+0000","last_peered":"2024-09-13T08:37:11.022331+0000","last_clean":"2024-09-13T08:37:11.022331+0000","last_became_active":"2024-09-13T08:37:10.026781+0000","last_became_peered":"2024-09-13T08:37:10.026781+0000","last_unstale":"2024-09-13T08:37:11.022331+0000","last_undegraded":"2024-09-13T08:37:11.022331+0000","last_fullsized":"2024-09-13T08:37:11.022331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017361+0000","last_change":"2024-09-13T08:37:09.015199+0000","last_active":"2024-09-13T08:37:11.017361+0000","last_peered":"2024-09-13T08:37:11.017361+0000","last_clean":"2024-09-13T08:37:11.017361+0000","last_became_active":"2024-09-13T08:37:09.014848+0000","last_became_peered":"2024-09-13T08:37:09.014848+0000","last_unstale":"2024-09-13T08:37:11.017361+0000","last_undegraded":"2024-09-13T08:37:11.017361+0000","last_fullsized":"2024-09-13T08:37:11.017361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:01.967164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863695+0000","last_change":"2024-09-13T08:37:10.027034+0000","last_active":"2024-09-13T08:37:12.863695+0000","last_peered":"2024-09-13T08:37:12.863695+0000","last_clean":"2024-09-13T08:37:12.863695+0000","last_became_active":"2024-09-13T08:37:10.026723+0000","last_became_peered":"2024-09-13T08:37:10.026723+0000","last_unstale":"2024-09-13T08:37:12.863695+0000","last_undegraded":"2024-09-13T08:37:12.863695+0000","last_fullsized":"2024-09-13T08:37:12.863695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596545+0000","last_change":"2024-09-13T08:37:09.021683+0000","last_active":"2024-09-13T08:37:13.596545+0000","last_peered":"2024-09-13T08:37:13.596545+0000","last_clean":"2024-09-13T08:37:13.596545+0000","last_became_active":"2024-09-13T08:37:09.019630+0000","last_became_peered":"2024-09-13T08:37:09.019630+0000","last_unstale":"2024-09-13T08:37:13.596545+0000","last_undegraded":"2024-09-13T08:37:13.596545+0000","last_fullsized":"2024-09-13T08:37:13.596545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:47.015443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596308+0000","last_change":"2024-09-13T08:37:10.032446+0000","last_active":"2024-09-13T08:37:13.596308+0000","last_peered":"2024-09-13T08:37:13.596308+0000","last_clean":"2024-09-13T08:37:13.596308+0000","last_became_active":"2024-09-13T08:37:10.031448+0000","last_became_peered":"2024-09-13T08:37:10.031448+0000","last_unstale":"2024-09-13T08:37:13.596308+0000","last_undegraded":"2024-09-13T08:37:13.596308+0000","last_fullsized":"2024-09-13T08:37:13.596308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:38:36.201616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021502+0000","last_change":"2024-09-13T08:37:09.025565+0000","last_active":"2024-09-13T08:37:11.021502+0000","last_peered":"2024-09-13T08:37:11.021502+0000","last_clean":"2024-09-13T08:37:11.021502+0000","last_became_active":"2024-09-13T08:37:09.025242+0000","last_became_peered":"2024-09-13T08:37:09.025242+0000","last_unstale":"2024-09-13T08:37:11.021502+0000","last_undegraded":"2024-09-13T08:37:11.021502+0000","last_fullsized":"2024-09-13T08:37:11.021502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:08.638798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723679+0000","last_change":"2024-09-13T08:37:10.030817+0000","last_active":"2024-09-13T08:37:12.723679+0000","last_peered":"2024-09-13T08:37:12.723679+0000","last_clean":"2024-09-13T08:37:12.723679+0000","last_became_active":"2024-09-13T08:37:10.030554+0000","last_became_peered":"2024-09-13T08:37:10.030554+0000","last_unstale":"2024-09-13T08:37:12.723679+0000","last_undegraded":"2024-09-13T08:37:12.723679+0000","last_fullsized":"2024-09-13T08:37:12.723679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596103+0000","last_change":"2024-09-13T08:37:10.023766+0000","last_active":"2024-09-13T08:37:13.596103+0000","last_peered":"2024-09-13T08:37:13.596103+0000","last_clean":"2024-09-13T08:37:13.596103+0000","last_became_active":"2024-09-13T08:37:10.022889+0000","last_became_peered":"2024-09-13T08:37:10.022889+0000","last_unstale":"2024-09-13T08:37:13.596103+0000","last_undegraded":"2024-09-13T08:37:13.596103+0000","last_fullsized":"2024-09-13T08:37:13.596103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022117+0000","last_change":"2024-09-13T08:37:09.023974+0000","last_active":"2024-09-13T08:37:11.022117+0000","last_peered":"2024-09-13T08:37:11.022117+0000","last_clean":"2024-09-13T08:37:11.022117+0000","last_became_active":"2024-09-13T08:37:09.023471+0000","last_became_peered":"2024-09-13T08:37:09.023471+0000","last_unstale":"2024-09-13T08:37:11.022117+0000","last_undegraded":"2024-09-13T08:37:11.022117+0000","last_fullsized":"2024-09-13T08:37:11.022117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:38.190525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016062+0000","last_change":"2024-09-13T08:37:10.023008+0000","last_active":"2024-09-13T08:37:11.016062+0000","last_peered":"2024-09-13T08:37:11.016062+0000","last_clean":"2024-09-13T08:37:11.016062+0000","last_became_active":"2024-09-13T08:37:10.021695+0000","last_became_peered":"2024-09-13T08:37:10.021695+0000","last_unstale":"2024-09-13T08:37:11.016062+0000","last_undegraded":"2024-09-13T08:37:11.016062+0000","last_fullsized":"2024-09-13T08:37:11.016062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:43.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022182+0000","last_change":"2024-09-13T08:37:10.031929+0000","last_active":"2024-09-13T08:37:11.022182+0000","last_peered":"2024-09-13T08:37:11.022182+0000","last_clean":"2024-09-13T08:37:11.022182+0000","last_became_active":"2024-09-13T08:37:10.031598+0000","last_became_peered":"2024-09-13T08:37:10.031598+0000","last_unstale":"2024-09-13T08:37:11.022182+0000","last_undegraded":"2024-09-13T08:37:11.022182+0000","last_fullsized":"2024-09-13T08:37:11.022182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328623+0000","last_change":"2024-09-13T08:37:09.011613+0000","last_active":"2024-09-13T08:37:11.328623+0000","last_peered":"2024-09-13T08:37:11.328623+0000","last_clean":"2024-09-13T08:37:11.328623+0000","last_became_active":"2024-09-13T08:37:09.011071+0000","last_became_peered":"2024-09-13T08:37:09.011071+0000","last_unstale":"2024-09-13T08:37:11.328623+0000","last_undegraded":"2024-09-13T08:37:11.328623+0000","last_fullsized":"2024-09-13T08:37:11.328623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022072+0000","last_change":"2024-09-13T08:37:10.028420+0000","last_active":"2024-09-13T08:37:11.022072+0000","last_peered":"2024-09-13T08:37:11.022072+0000","last_clean":"2024-09-13T08:37:11.022072+0000","last_became_active":"2024-09-13T08:37:10.028166+0000","last_became_peered":"2024-09-13T08:37:10.028166+0000","last_unstale":"2024-09-13T08:37:11.022072+0000","last_undegraded":"2024-09-13T08:37:11.022072+0000","last_fullsized":"2024-09-13T08:37:11.022072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:58.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863837+0000","last_change":"2024-09-13T08:37:09.015072+0000","last_active":"2024-09-13T08:37:12.863837+0000","last_peered":"2024-09-13T08:37:12.863837+0000","last_clean":"2024-09-13T08:37:12.863837+0000","last_became_active":"2024-09-13T08:37:09.014597+0000","last_became_peered":"2024-09-13T08:37:09.014597+0000","last_unstale":"2024-09-13T08:37:12.863837+0000","last_undegraded":"2024-09-13T08:37:12.863837+0000","last_fullsized":"2024-09-13T08:37:12.863837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596652+0000","last_change":"2024-09-13T08:37:10.018219+0000","last_active":"2024-09-13T08:37:13.596652+0000","last_peered":"2024-09-13T08:37:13.596652+0000","last_clean":"2024-09-13T08:37:13.596652+0000","last_became_active":"2024-09-13T08:37:10.017946+0000","last_became_peered":"2024-09-13T08:37:10.017946+0000","last_unstale":"2024-09-13T08:37:13.596652+0000","last_undegraded":"2024-09-13T08:37:13.596652+0000","last_fullsized":"2024-09-13T08:37:13.596652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:12.941476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723548+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:12.723548+0000","last_peered":"2024-09-13T08:37:12.723548+0000","last_clean":"2024-09-13T08:37:12.723548+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:12.723548+0000","last_undegraded":"2024-09-13T08:37:12.723548+0000","last_fullsized":"2024-09-13T08:37:12.723548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596411+0000","last_change":"2024-09-13T08:37:09.021195+0000","last_active":"2024-09-13T08:37:13.596411+0000","last_peered":"2024-09-13T08:37:13.596411+0000","last_clean":"2024-09-13T08:37:13.596411+0000","last_became_active":"2024-09-13T08:37:09.019692+0000","last_became_peered":"2024-09-13T08:37:09.019692+0000","last_unstale":"2024-09-13T08:37:13.596411+0000","last_undegraded":"2024-09-13T08:37:13.596411+0000","last_fullsized":"2024-09-13T08:37:13.596411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.430584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596362+0000","last_change":"2024-09-13T08:37:09.019756+0000","last_active":"2024-09-13T08:37:13.596362+0000","last_peered":"2024-09-13T08:37:13.596362+0000","last_clean":"2024-09-13T08:37:13.596362+0000","last_became_active":"2024-09-13T08:37:09.019375+0000","last_became_peered":"2024-09-13T08:37:09.019375+0000","last_unstale":"2024-09-13T08:37:13.596362+0000","last_undegraded":"2024-09-13T08:37:13.596362+0000","last_fullsized":"2024-09-13T08:37:13.596362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:40.495518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863575+0000","last_change":"2024-09-13T08:37:10.020491+0000","last_active":"2024-09-13T08:37:12.863575+0000","last_peered":"2024-09-13T08:37:12.863575+0000","last_clean":"2024-09-13T08:37:12.863575+0000","last_became_active":"2024-09-13T08:37:10.020271+0000","last_became_peered":"2024-09-13T08:37:10.020271+0000","last_unstale":"2024-09-13T08:37:12.863575+0000","last_undegraded":"2024-09-13T08:37:12.863575+0000","last_fullsized":"2024-09-13T08:37:12.863575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:29.393985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016620+0000","last_change":"2024-09-13T08:37:09.015889+0000","last_active":"2024-09-13T08:37:11.016620+0000","last_peered":"2024-09-13T08:37:11.016620+0000","last_clean":"2024-09-13T08:37:11.016620+0000","last_became_active":"2024-09-13T08:37:09.015587+0000","last_became_peered":"2024-09-13T08:37:09.015587+0000","last_unstale":"2024-09-13T08:37:11.016620+0000","last_undegraded":"2024-09-13T08:37:11.016620+0000","last_fullsized":"2024-09-13T08:37:11.016620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328457+0000","last_change":"2024-09-13T08:37:10.036912+0000","last_active":"2024-09-13T08:37:11.328457+0000","last_peered":"2024-09-13T08:37:11.328457+0000","last_clean":"2024-09-13T08:37:11.328457+0000","last_became_active":"2024-09-13T08:37:10.036665+0000","last_became_peered":"2024-09-13T08:37:10.036665+0000","last_unstale":"2024-09-13T08:37:11.328457+0000","last_undegraded":"2024-09-13T08:37:11.328457+0000","last_fullsized":"2024-09-13T08:37:11.328457+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596672+0000","last_change":"2024-09-13T08:37:09.023866+0000","last_active":"2024-09-13T08:37:13.596672+0000","last_peered":"2024-09-13T08:37:13.596672+0000","last_clean":"2024-09-13T08:37:13.596672+0000","last_became_active":"2024-09-13T08:37:09.021825+0000","last_became_peered":"2024-09-13T08:37:09.021825+0000","last_unstale":"2024-09-13T08:37:13.596672+0000","last_undegraded":"2024-09-13T08:37:13.596672+0000","last_fullsized":"2024-09-13T08:37:13.596672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:42.779836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022032+0000","last_change":"2024-09-13T08:37:10.027315+0000","last_active":"2024-09-13T08:37:11.022032+0000","last_peered":"2024-09-13T08:37:11.022032+0000","last_clean":"2024-09-13T08:37:11.022032+0000","last_became_active":"2024-09-13T08:37:10.027018+0000","last_became_peered":"2024-09-13T08:37:10.027018+0000","last_unstale":"2024-09-13T08:37:11.022032+0000","last_undegraded":"2024-09-13T08:37:11.022032+0000","last_fullsized":"2024-09-13T08:37:11.022032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328769+0000","last_change":"2024-09-13T08:37:09.018684+0000","last_active":"2024-09-13T08:37:11.328769+0000","last_peered":"2024-09-13T08:37:11.328769+0000","last_clean":"2024-09-13T08:37:11.328769+0000","last_became_active":"2024-09-13T08:37:09.016570+0000","last_became_peered":"2024-09-13T08:37:09.016570+0000","last_unstale":"2024-09-13T08:37:11.328769+0000","last_undegraded":"2024-09-13T08:37:11.328769+0000","last_fullsized":"2024-09-13T08:37:11.328769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015606+0000","last_change":"2024-09-13T08:37:10.018230+0000","last_active":"2024-09-13T08:37:11.015606+0000","last_peered":"2024-09-13T08:37:11.015606+0000","last_clean":"2024-09-13T08:37:11.015606+0000","last_became_active":"2024-09-13T08:37:10.017427+0000","last_became_peered":"2024-09-13T08:37:10.017427+0000","last_unstale":"2024-09-13T08:37:11.015606+0000","last_undegraded":"2024-09-13T08:37:11.015606+0000","last_fullsized":"2024-09-13T08:37:11.015606+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596769+0000","last_change":"2024-09-13T08:37:09.018807+0000","last_active":"2024-09-13T08:37:13.596769+0000","last_peered":"2024-09-13T08:37:13.596769+0000","last_clean":"2024-09-13T08:37:13.596769+0000","last_became_active":"2024-09-13T08:37:09.018298+0000","last_became_peered":"2024-09-13T08:37:09.018298+0000","last_unstale":"2024-09-13T08:37:13.596769+0000","last_undegraded":"2024-09-13T08:37:13.596769+0000","last_fullsized":"2024-09-13T08:37:13.596769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.295316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863880+0000","last_change":"2024-09-13T08:37:10.028730+0000","last_active":"2024-09-13T08:37:12.863880+0000","last_peered":"2024-09-13T08:37:12.863880+0000","last_clean":"2024-09-13T08:37:12.863880+0000","last_became_active":"2024-09-13T08:37:10.028481+0000","last_became_peered":"2024-09-13T08:37:10.028481+0000","last_unstale":"2024-09-13T08:37:12.863880+0000","last_undegraded":"2024-09-13T08:37:12.863880+0000","last_fullsized":"2024-09-13T08:37:12.863880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596783+0000","last_change":"2024-09-13T08:37:09.024256+0000","last_active":"2024-09-13T08:37:13.596783+0000","last_peered":"2024-09-13T08:37:13.596783+0000","last_clean":"2024-09-13T08:37:13.596783+0000","last_became_active":"2024-09-13T08:37:09.023967+0000","last_became_peered":"2024-09-13T08:37:09.023967+0000","last_unstale":"2024-09-13T08:37:13.596783+0000","last_undegraded":"2024-09-13T08:37:13.596783+0000","last_fullsized":"2024-09-13T08:37:13.596783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:41.577735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021357+0000","last_change":"2024-09-13T08:37:09.011069+0000","last_active":"2024-09-13T08:37:11.021357+0000","last_peered":"2024-09-13T08:37:11.021357+0000","last_clean":"2024-09-13T08:37:11.021357+0000","last_became_active":"2024-09-13T08:37:09.010687+0000","last_became_peered":"2024-09-13T08:37:09.010687+0000","last_unstale":"2024-09-13T08:37:11.021357+0000","last_undegraded":"2024-09-13T08:37:11.021357+0000","last_fullsized":"2024-09-13T08:37:11.021357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596198+0000","last_change":"2024-09-13T08:37:10.026288+0000","last_active":"2024-09-13T08:37:13.596198+0000","last_peered":"2024-09-13T08:37:13.596198+0000","last_clean":"2024-09-13T08:37:13.596198+0000","last_became_active":"2024-09-13T08:37:10.026029+0000","last_became_peered":"2024-09-13T08:37:10.026029+0000","last_unstale":"2024-09-13T08:37:13.596198+0000","last_undegraded":"2024-09-13T08:37:13.596198+0000","last_fullsized":"2024-09-13T08:37:13.596198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863577+0000","last_change":"2024-09-13T08:37:09.019714+0000","last_active":"2024-09-13T08:37:12.863577+0000","last_peered":"2024-09-13T08:37:12.863577+0000","last_clean":"2024-09-13T08:37:12.863577+0000","last_became_active":"2024-09-13T08:37:09.018957+0000","last_became_peered":"2024-09-13T08:37:09.018957+0000","last_unstale":"2024-09-13T08:37:12.863577+0000","last_undegraded":"2024-09-13T08:37:12.863577+0000","last_fullsized":"2024-09-13T08:37:12.863577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017312+0000","last_change":"2024-09-13T08:37:10.026722+0000","last_active":"2024-09-13T08:37:11.017312+0000","last_peered":"2024-09-13T08:37:11.017312+0000","last_clean":"2024-09-13T08:37:11.017312+0000","last_became_active":"2024-09-13T08:37:10.025267+0000","last_became_peered":"2024-09-13T08:37:10.025267+0000","last_unstale":"2024-09-13T08:37:11.017312+0000","last_undegraded":"2024-09-13T08:37:11.017312+0000","last_fullsized":"2024-09-13T08:37:11.017312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596295+0000","last_change":"2024-09-13T08:37:09.022982+0000","last_active":"2024-09-13T08:37:13.596295+0000","last_peered":"2024-09-13T08:37:13.596295+0000","last_clean":"2024-09-13T08:37:13.596295+0000","last_became_active":"2024-09-13T08:37:09.021746+0000","last_became_peered":"2024-09-13T08:37:09.021746+0000","last_unstale":"2024-09-13T08:37:13.596295+0000","last_undegraded":"2024-09-13T08:37:13.596295+0000","last_fullsized":"2024-09-13T08:37:13.596295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.468482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596487+0000","last_change":"2024-09-13T08:37:10.016999+0000","last_active":"2024-09-13T08:37:13.596487+0000","last_peered":"2024-09-13T08:37:13.596487+0000","last_clean":"2024-09-13T08:37:13.596487+0000","last_became_active":"2024-09-13T08:37:10.016738+0000","last_became_peered":"2024-09-13T08:37:10.016738+0000","last_unstale":"2024-09-13T08:37:13.596487+0000","last_undegraded":"2024-09-13T08:37:13.596487+0000","last_fullsized":"2024-09-13T08:37:13.596487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:16:16.734699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596582+0000","last_change":"2024-09-13T08:37:10.026898+0000","last_active":"2024-09-13T08:37:13.596582+0000","last_peered":"2024-09-13T08:37:13.596582+0000","last_clean":"2024-09-13T08:37:13.596582+0000","last_became_active":"2024-09-13T08:37:10.026602+0000","last_became_peered":"2024-09-13T08:37:10.026602+0000","last_unstale":"2024-09-13T08:37:13.596582+0000","last_undegraded":"2024-09-13T08:37:13.596582+0000","last_fullsized":"2024-09-13T08:37:13.596582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:42.330227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022006+0000","last_change":"2024-09-13T08:37:09.011138+0000","last_active":"2024-09-13T08:37:11.022006+0000","last_peered":"2024-09-13T08:37:11.022006+0000","last_clean":"2024-09-13T08:37:11.022006+0000","last_became_active":"2024-09-13T08:37:09.010863+0000","last_became_peered":"2024-09-13T08:37:09.010863+0000","last_unstale":"2024-09-13T08:37:11.022006+0000","last_undegraded":"2024-09-13T08:37:11.022006+0000","last_fullsized":"2024-09-13T08:37:11.022006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:14.227052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723804+0000","last_change":"2024-09-13T08:37:10.029092+0000","last_active":"2024-09-13T08:37:12.723804+0000","last_peered":"2024-09-13T08:37:12.723804+0000","last_clean":"2024-09-13T08:37:12.723804+0000","last_became_active":"2024-09-13T08:37:10.028796+0000","last_became_peered":"2024-09-13T08:37:10.028796+0000","last_unstale":"2024-09-13T08:37:12.723804+0000","last_undegraded":"2024-09-13T08:37:12.723804+0000","last_fullsized":"2024-09-13T08:37:12.723804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022178+0000","last_change":"2024-09-13T08:37:09.013579+0000","last_active":"2024-09-13T08:37:11.022178+0000","last_peered":"2024-09-13T08:37:11.022178+0000","last_clean":"2024-09-13T08:37:11.022178+0000","last_became_active":"2024-09-13T08:37:09.013327+0000","last_became_peered":"2024-09-13T08:37:09.013327+0000","last_unstale":"2024-09-13T08:37:11.022178+0000","last_undegraded":"2024-09-13T08:37:11.022178+0000","last_fullsized":"2024-09-13T08:37:11.022178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:34.207907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328992+0000","last_change":"2024-09-13T08:37:10.032358+0000","last_active":"2024-09-13T08:37:11.328992+0000","last_peered":"2024-09-13T08:37:11.328992+0000","last_clean":"2024-09-13T08:37:11.328992+0000","last_became_active":"2024-09-13T08:37:10.032047+0000","last_became_peered":"2024-09-13T08:37:10.032047+0000","last_unstale":"2024-09-13T08:37:11.328992+0000","last_undegraded":"2024-09-13T08:37:11.328992+0000","last_fullsized":"2024-09-13T08:37:11.328992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:19.283020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329066+0000","last_change":"2024-09-13T08:37:09.015906+0000","last_active":"2024-09-13T08:37:11.329066+0000","last_peered":"2024-09-13T08:37:11.329066+0000","last_clean":"2024-09-13T08:37:11.329066+0000","last_became_active":"2024-09-13T08:37:09.015500+0000","last_became_peered":"2024-09-13T08:37:09.015500+0000","last_unstale":"2024-09-13T08:37:11.329066+0000","last_undegraded":"2024-09-13T08:37:11.329066+0000","last_fullsized":"2024-09-13T08:37:11.329066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:51.846602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329023+0000","last_change":"2024-09-13T08:37:10.043965+0000","last_active":"2024-09-13T08:37:11.329023+0000","last_peered":"2024-09-13T08:37:11.329023+0000","last_clean":"2024-09-13T08:37:11.329023+0000","last_became_active":"2024-09-13T08:37:10.041509+0000","last_became_peered":"2024-09-13T08:37:10.041509+0000","last_unstale":"2024-09-13T08:37:11.329023+0000","last_undegraded":"2024-09-13T08:37:11.329023+0000","last_fullsized":"2024-09-13T08:37:11.329023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":241664,"data_stored":81079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":80955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":835584,"data_stored":671415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":81537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":245760,"data_stored":82567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":671393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":81011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:24.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.755+0000 7fcf06e00640 1 -- 172.21.15.31:0/2542897552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 msgr2=0x7fcf000d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.755+0000 7fcf06e00640 1 --2- 172.21.15.31:0/2542897552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf000d3bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fceec009830 tx=0x7fceec02a510 comp rx=0 tx=0).stop 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 -- 172.21.15.31:0/2542897552 shutdown_connections 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 --2- 172.21.15.31:0/2542897552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 0x7fcf000b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 --2- 172.21.15.31:0/2542897552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf000d3bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 -- 172.21.15.31:0/2542897552 >> 172.21.15.31:0/2542897552 conn(0x7fcf0008d8b0 msgr2=0x7fcf0009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 -- 172.21.15.31:0/2542897552 shutdown_connections 2024-09-13T08:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.756+0000 7fcf06e00640 1 -- 172.21.15.31:0/2542897552 wait complete. 2024-09-13T08:37:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.757+0000 7fcf06e00640 1 Processor -- start 2024-09-13T08:37:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.757+0000 7fcf06e00640 1 -- start start 2024-09-13T08:37:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 0x7fcf0006e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 0x7fcf0015cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf000b1900 con 0x7fcf000b47f0 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcf000b1780 con 0x7fcf0015dfa0 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.758+0000 7fcf06e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcf000b1a80 con 0x7fcf000d4110 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34864/0 (socket says 172.21.15.31:34864) 2024-09-13T08:37:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 -- 172.21.15.31:0/1579846717 learned_addr learned my addr 172.21.15.31:0/1579846717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05000640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 0x7fcf0006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf06400640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 0x7fcf0015cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 -- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 msgr2=0x7fcf0006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 0x7fcf0006e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 -- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 msgr2=0x7fcf0015cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 0x7fcf0015cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 -- 172.21.15.31:0/1579846717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf001e8820 con 0x7fcf000b47f0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf06400640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 0x7fcf0015cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.759+0000 7fcf05a00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcef400cbc0 tx=0x7fcef400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.760+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4016020 con 0x7fcf000b47f0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.760+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcef400ad90 con 0x7fcf000b47f0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.760+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf001e9140 con 0x7fcf000b47f0 2024-09-13T08:37:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.760+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4012680 con 0x7fcf000b47f0 2024-09-13T08:37:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.760+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf001e99e0 con 0x7fcf000b47f0 2024-09-13T08:37:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.761+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf000b1f30 con 0x7fcf000b47f0 2024-09-13T08:37:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.762+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fcef4019020 con 0x7fcf000b47f0 2024-09-13T08:37:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.763+0000 7fcefd600640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 0x7fcee007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.763+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fcef409acd0 con 0x7fcf000b47f0 2024-09-13T08:37:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.763+0000 7fcf05000640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 0x7fcee007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.764+0000 7fcf05000640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 0x7fcee007ad10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fceec034000 tx=0x7fceec004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.767+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcef4020030 con 0x7fcf000b47f0 2024-09-13T08:37:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.959+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcf000d4cf0 con 0x7fcee0078850 2024-09-13T08:37:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.964+0000 7fcefd600640 1 -- 172.21.15.31:0/1579846717 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251822 (secure 0 0 0) 0x7fcf000d4cf0 con 0x7fcee0078850 2024-09-13T08:37:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 msgr2=0x7fcee007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 0x7fcee007ad10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fceec034000 tx=0x7fceec004730 comp rx=0 tx=0).stop 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 msgr2=0x7fcf001e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcef400cbc0 tx=0x7fcef400a060 comp rx=0 tx=0).stop 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 shutdown_connections 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf0015dfa0 0x7fcf0015cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcee0078850 0x7fcee007ad10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf000d4110 0x7fcf0006e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 --2- 172.21.15.31:0/1579846717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf000b47f0 0x7fcf001e8a60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 >> 172.21.15.31:0/1579846717 conn(0x7fcf0008d8b0 msgr2=0x7fcf000d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 shutdown_connections 2024-09-13T08:37:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:24.968+0000 7fcf06e00640 1 -- 172.21.15.31:0/1579846717 wait complete. 2024-09-13T08:37:24.985 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-13T08:37:23.040197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235936,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738784,"statfs":{"total":773094113280,"available":772852514816,"internally_reserved":0,"allocated":3121152,"data_stored":1833196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003379"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021976+0000","last_change":"2024-09-13T08:37:10.038242+0000","last_active":"2024-09-13T08:37:11.021976+0000","last_peered":"2024-09-13T08:37:11.021976+0000","last_clean":"2024-09-13T08:37:11.021976+0000","last_became_active":"2024-09-13T08:37:10.037984+0000","last_became_peered":"2024-09-13T08:37:10.037984+0000","last_unstale":"2024-09-13T08:37:11.021976+0000","last_undegraded":"2024-09-13T08:37:11.021976+0000","last_fullsized":"2024-09-13T08:37:11.021976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:05.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328871+0000","last_change":"2024-09-13T08:37:10.043402+0000","last_active":"2024-09-13T08:37:11.328871+0000","last_peered":"2024-09-13T08:37:11.328871+0000","last_clean":"2024-09-13T08:37:11.328871+0000","last_became_active":"2024-09-13T08:37:10.041028+0000","last_became_peered":"2024-09-13T08:37:10.041028+0000","last_unstale":"2024-09-13T08:37:11.328871+0000","last_undegraded":"2024-09-13T08:37:11.328871+0000","last_fullsized":"2024-09-13T08:37:11.328871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:32.216178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328967+0000","last_change":"2024-09-13T08:37:10.038931+0000","last_active":"2024-09-13T08:37:11.328967+0000","last_peered":"2024-09-13T08:37:11.328967+0000","last_clean":"2024-09-13T08:37:11.328967+0000","last_became_active":"2024-09-13T08:37:10.038676+0000","last_became_peered":"2024-09-13T08:37:10.038676+0000","last_unstale":"2024-09-13T08:37:11.328967+0000","last_undegraded":"2024-09-13T08:37:11.328967+0000","last_fullsized":"2024-09-13T08:37:11.328967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.737921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015339+0000","last_change":"2024-09-13T08:37:10.018106+0000","last_active":"2024-09-13T08:37:11.015339+0000","last_peered":"2024-09-13T08:37:11.015339+0000","last_clean":"2024-09-13T08:37:11.015339+0000","last_became_active":"2024-09-13T08:37:10.017812+0000","last_became_peered":"2024-09-13T08:37:10.017812+0000","last_unstale":"2024-09-13T08:37:11.015339+0000","last_undegraded":"2024-09-13T08:37:11.015339+0000","last_fullsized":"2024-09-13T08:37:11.015339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021892+0000","last_change":"2024-09-13T08:37:10.035463+0000","last_active":"2024-09-13T08:37:11.021892+0000","last_peered":"2024-09-13T08:37:11.021892+0000","last_clean":"2024-09-13T08:37:11.021892+0000","last_became_active":"2024-09-13T08:37:10.032852+0000","last_became_peered":"2024-09-13T08:37:10.032852+0000","last_unstale":"2024-09-13T08:37:11.021892+0000","last_undegraded":"2024-09-13T08:37:11.021892+0000","last_fullsized":"2024-09-13T08:37:11.021892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:34.731391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015062+0000","last_change":"2024-09-13T08:37:10.030116+0000","last_active":"2024-09-13T08:37:11.015062+0000","last_peered":"2024-09-13T08:37:11.015062+0000","last_clean":"2024-09-13T08:37:11.015062+0000","last_became_active":"2024-09-13T08:37:10.029773+0000","last_became_peered":"2024-09-13T08:37:10.029773+0000","last_unstale":"2024-09-13T08:37:11.015062+0000","last_undegraded":"2024-09-13T08:37:11.015062+0000","last_fullsized":"2024-09-13T08:37:11.015062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021640+0000","last_change":"2024-09-13T08:37:10.025604+0000","last_active":"2024-09-13T08:37:11.021640+0000","last_peered":"2024-09-13T08:37:11.021640+0000","last_clean":"2024-09-13T08:37:11.021640+0000","last_became_active":"2024-09-13T08:37:10.020828+0000","last_became_peered":"2024-09-13T08:37:10.020828+0000","last_unstale":"2024-09-13T08:37:11.021640+0000","last_undegraded":"2024-09-13T08:37:11.021640+0000","last_fullsized":"2024-09-13T08:37:11.021640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328358+0000","last_change":"2024-09-13T08:37:10.041296+0000","last_active":"2024-09-13T08:37:11.328358+0000","last_peered":"2024-09-13T08:37:11.328358+0000","last_clean":"2024-09-13T08:37:11.328358+0000","last_became_active":"2024-09-13T08:37:10.035151+0000","last_became_peered":"2024-09-13T08:37:10.035151+0000","last_unstale":"2024-09-13T08:37:11.328358+0000","last_undegraded":"2024-09-13T08:37:11.328358+0000","last_fullsized":"2024-09-13T08:37:11.328358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:28.830741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723967+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:12.723967+0000","last_peered":"2024-09-13T08:37:12.723967+0000","last_clean":"2024-09-13T08:37:12.723967+0000","last_became_active":"2024-09-13T08:37:10.026743+0000","last_became_peered":"2024-09-13T08:37:10.026743+0000","last_unstale":"2024-09-13T08:37:12.723967+0000","last_undegraded":"2024-09-13T08:37:12.723967+0000","last_fullsized":"2024-09-13T08:37:12.723967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022008+0000","last_change":"2024-09-13T08:37:10.020548+0000","last_active":"2024-09-13T08:37:11.022008+0000","last_peered":"2024-09-13T08:37:11.022008+0000","last_clean":"2024-09-13T08:37:11.022008+0000","last_became_active":"2024-09-13T08:37:10.020144+0000","last_became_peered":"2024-09-13T08:37:10.020144+0000","last_unstale":"2024-09-13T08:37:11.022008+0000","last_undegraded":"2024-09-13T08:37:11.022008+0000","last_fullsized":"2024-09-13T08:37:11.022008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:13.408879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022140+0000","last_change":"2024-09-13T08:37:10.038191+0000","last_active":"2024-09-13T08:37:11.022140+0000","last_peered":"2024-09-13T08:37:11.022140+0000","last_clean":"2024-09-13T08:37:11.022140+0000","last_became_active":"2024-09-13T08:37:10.037935+0000","last_became_peered":"2024-09-13T08:37:10.037935+0000","last_unstale":"2024-09-13T08:37:11.022140+0000","last_undegraded":"2024-09-13T08:37:11.022140+0000","last_fullsized":"2024-09-13T08:37:11.022140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021867+0000","last_change":"2024-09-13T08:37:10.025910+0000","last_active":"2024-09-13T08:37:11.021867+0000","last_peered":"2024-09-13T08:37:11.021867+0000","last_clean":"2024-09-13T08:37:11.021867+0000","last_became_active":"2024-09-13T08:37:10.023349+0000","last_became_peered":"2024-09-13T08:37:10.023349+0000","last_unstale":"2024-09-13T08:37:11.021867+0000","last_undegraded":"2024-09-13T08:37:11.021867+0000","last_fullsized":"2024-09-13T08:37:11.021867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:02.005454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863687+0000","last_change":"2024-09-13T08:37:10.033914+0000","last_active":"2024-09-13T08:37:12.863687+0000","last_peered":"2024-09-13T08:37:12.863687+0000","last_clean":"2024-09-13T08:37:12.863687+0000","last_became_active":"2024-09-13T08:37:10.030947+0000","last_became_peered":"2024-09-13T08:37:10.030947+0000","last_unstale":"2024-09-13T08:37:12.863687+0000","last_undegraded":"2024-09-13T08:37:12.863687+0000","last_fullsized":"2024-09-13T08:37:12.863687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:49.714886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863461+0000","last_change":"2024-09-13T08:37:10.024803+0000","last_active":"2024-09-13T08:37:12.863461+0000","last_peered":"2024-09-13T08:37:12.863461+0000","last_clean":"2024-09-13T08:37:12.863461+0000","last_became_active":"2024-09-13T08:37:10.024512+0000","last_became_peered":"2024-09-13T08:37:10.024512+0000","last_unstale":"2024-09-13T08:37:12.863461+0000","last_undegraded":"2024-09-13T08:37:12.863461+0000","last_fullsized":"2024-09-13T08:37:12.863461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328365+0000","last_change":"2024-09-13T08:37:10.035650+0000","last_active":"2024-09-13T08:37:11.328365+0000","last_peered":"2024-09-13T08:37:11.328365+0000","last_clean":"2024-09-13T08:37:11.328365+0000","last_became_active":"2024-09-13T08:37:10.035331+0000","last_became_peered":"2024-09-13T08:37:10.035331+0000","last_unstale":"2024-09-13T08:37:11.328365+0000","last_undegraded":"2024-09-13T08:37:11.328365+0000","last_fullsized":"2024-09-13T08:37:11.328365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.703160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.022368+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.022095+0000","last_became_peered":"2024-09-13T08:37:10.022095+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596810+0000","last_change":"2024-09-13T08:37:10.032355+0000","last_active":"2024-09-13T08:37:13.596810+0000","last_peered":"2024-09-13T08:37:13.596810+0000","last_clean":"2024-09-13T08:37:13.596810+0000","last_became_active":"2024-09-13T08:37:10.032098+0000","last_became_peered":"2024-09-13T08:37:10.032098+0000","last_unstale":"2024-09-13T08:37:13.596810+0000","last_undegraded":"2024-09-13T08:37:13.596810+0000","last_fullsized":"2024-09-13T08:37:13.596810+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:32:27.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017318+0000","last_change":"2024-09-13T08:37:10.022000+0000","last_active":"2024-09-13T08:37:11.017318+0000","last_peered":"2024-09-13T08:37:11.017318+0000","last_clean":"2024-09-13T08:37:11.017318+0000","last_became_active":"2024-09-13T08:37:10.021622+0000","last_became_peered":"2024-09-13T08:37:10.021622+0000","last_unstale":"2024-09-13T08:37:11.017318+0000","last_undegraded":"2024-09-13T08:37:11.017318+0000","last_fullsized":"2024-09-13T08:37:11.017318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:39.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596836+0000","last_change":"2024-09-13T08:37:10.026234+0000","last_active":"2024-09-13T08:37:13.596836+0000","last_peered":"2024-09-13T08:37:13.596836+0000","last_clean":"2024-09-13T08:37:13.596836+0000","last_became_active":"2024-09-13T08:37:10.025970+0000","last_became_peered":"2024-09-13T08:37:10.025970+0000","last_unstale":"2024-09-13T08:37:13.596836+0000","last_undegraded":"2024-09-13T08:37:13.596836+0000","last_fullsized":"2024-09-13T08:37:13.596836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:52.574964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021932+0000","last_change":"2024-09-13T08:37:10.028101+0000","last_active":"2024-09-13T08:37:11.021932+0000","last_peered":"2024-09-13T08:37:11.021932+0000","last_clean":"2024-09-13T08:37:11.021932+0000","last_became_active":"2024-09-13T08:37:10.021666+0000","last_became_peered":"2024-09-13T08:37:10.021666+0000","last_unstale":"2024-09-13T08:37:11.021932+0000","last_undegraded":"2024-09-13T08:37:11.021932+0000","last_fullsized":"2024-09-13T08:37:11.021932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:43.682175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016924+0000","last_change":"2024-09-13T08:37:10.021357+0000","last_active":"2024-09-13T08:37:11.016924+0000","last_peered":"2024-09-13T08:37:11.016924+0000","last_clean":"2024-09-13T08:37:11.016924+0000","last_became_active":"2024-09-13T08:37:10.020990+0000","last_became_peered":"2024-09-13T08:37:10.020990+0000","last_unstale":"2024-09-13T08:37:11.016924+0000","last_undegraded":"2024-09-13T08:37:11.016924+0000","last_fullsized":"2024-09-13T08:37:11.016924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:47.043003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723677+0000","last_change":"2024-09-13T08:37:10.020945+0000","last_active":"2024-09-13T08:37:12.723677+0000","last_peered":"2024-09-13T08:37:12.723677+0000","last_clean":"2024-09-13T08:37:12.723677+0000","last_became_active":"2024-09-13T08:37:10.020675+0000","last_became_peered":"2024-09-13T08:37:10.020675+0000","last_unstale":"2024-09-13T08:37:12.723677+0000","last_undegraded":"2024-09-13T08:37:12.723677+0000","last_fullsized":"2024-09-13T08:37:12.723677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:41.501115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328354+0000","last_change":"2024-09-13T08:37:10.036002+0000","last_active":"2024-09-13T08:37:11.328354+0000","last_peered":"2024-09-13T08:37:11.328354+0000","last_clean":"2024-09-13T08:37:11.328354+0000","last_became_active":"2024-09-13T08:37:10.035684+0000","last_became_peered":"2024-09-13T08:37:10.035684+0000","last_unstale":"2024-09-13T08:37:11.328354+0000","last_undegraded":"2024-09-13T08:37:11.328354+0000","last_fullsized":"2024-09-13T08:37:11.328354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:02.312081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596071+0000","last_change":"2024-09-13T08:37:10.032065+0000","last_active":"2024-09-13T08:37:13.596071+0000","last_peered":"2024-09-13T08:37:13.596071+0000","last_clean":"2024-09-13T08:37:13.596071+0000","last_became_active":"2024-09-13T08:37:10.031802+0000","last_became_peered":"2024-09-13T08:37:10.031802+0000","last_unstale":"2024-09-13T08:37:13.596071+0000","last_undegraded":"2024-09-13T08:37:13.596071+0000","last_fullsized":"2024-09-13T08:37:13.596071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328937+0000","last_change":"2024-09-13T08:37:10.036040+0000","last_active":"2024-09-13T08:37:11.328937+0000","last_peered":"2024-09-13T08:37:11.328937+0000","last_clean":"2024-09-13T08:37:11.328937+0000","last_became_active":"2024-09-13T08:37:10.035698+0000","last_became_peered":"2024-09-13T08:37:10.035698+0000","last_unstale":"2024-09-13T08:37:11.328937+0000","last_undegraded":"2024-09-13T08:37:11.328937+0000","last_fullsized":"2024-09-13T08:37:11.328937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:58.093384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015904+0000","last_change":"2024-09-13T08:37:10.023312+0000","last_active":"2024-09-13T08:37:11.015904+0000","last_peered":"2024-09-13T08:37:11.015904+0000","last_clean":"2024-09-13T08:37:11.015904+0000","last_became_active":"2024-09-13T08:37:10.022193+0000","last_became_peered":"2024-09-13T08:37:10.022193+0000","last_unstale":"2024-09-13T08:37:11.015904+0000","last_undegraded":"2024-09-13T08:37:11.015904+0000","last_fullsized":"2024-09-13T08:37:11.015904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863802+0000","last_change":"2024-09-13T08:37:10.028354+0000","last_active":"2024-09-13T08:37:12.863802+0000","last_peered":"2024-09-13T08:37:12.863802+0000","last_clean":"2024-09-13T08:37:12.863802+0000","last_became_active":"2024-09-13T08:37:10.027989+0000","last_became_peered":"2024-09-13T08:37:10.027989+0000","last_unstale":"2024-09-13T08:37:12.863802+0000","last_undegraded":"2024-09-13T08:37:12.863802+0000","last_fullsized":"2024-09-13T08:37:12.863802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:54:38.823311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863758+0000","last_change":"2024-09-13T08:37:10.039029+0000","last_active":"2024-09-13T08:37:12.863758+0000","last_peered":"2024-09-13T08:37:12.863758+0000","last_clean":"2024-09-13T08:37:12.863758+0000","last_became_active":"2024-09-13T08:37:10.038689+0000","last_became_peered":"2024-09-13T08:37:10.038689+0000","last_unstale":"2024-09-13T08:37:12.863758+0000","last_undegraded":"2024-09-13T08:37:12.863758+0000","last_fullsized":"2024-09-13T08:37:12.863758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:21.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328458+0000","last_change":"2024-09-13T08:37:10.035713+0000","last_active":"2024-09-13T08:37:11.328458+0000","last_peered":"2024-09-13T08:37:11.328458+0000","last_clean":"2024-09-13T08:37:11.328458+0000","last_became_active":"2024-09-13T08:37:10.035292+0000","last_became_peered":"2024-09-13T08:37:10.035292+0000","last_unstale":"2024-09-13T08:37:11.328458+0000","last_undegraded":"2024-09-13T08:37:11.328458+0000","last_fullsized":"2024-09-13T08:37:11.328458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:29.489758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596170+0000","last_change":"2024-09-13T08:37:10.029548+0000","last_active":"2024-09-13T08:37:13.596170+0000","last_peered":"2024-09-13T08:37:13.596170+0000","last_clean":"2024-09-13T08:37:13.596170+0000","last_became_active":"2024-09-13T08:37:10.029292+0000","last_became_peered":"2024-09-13T08:37:10.029292+0000","last_unstale":"2024-09-13T08:37:13.596170+0000","last_undegraded":"2024-09-13T08:37:13.596170+0000","last_fullsized":"2024-09-13T08:37:13.596170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016762+0000","last_change":"2024-09-13T08:37:10.012936+0000","last_active":"2024-09-13T08:37:11.016762+0000","last_peered":"2024-09-13T08:37:11.016762+0000","last_clean":"2024-09-13T08:37:11.016762+0000","last_became_active":"2024-09-13T08:37:10.012582+0000","last_became_peered":"2024-09-13T08:37:10.012582+0000","last_unstale":"2024-09-13T08:37:11.016762+0000","last_undegraded":"2024-09-13T08:37:11.016762+0000","last_fullsized":"2024-09-13T08:37:11.016762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015064+0000","last_change":"2024-09-13T08:37:10.017195+0000","last_active":"2024-09-13T08:37:11.015064+0000","last_peered":"2024-09-13T08:37:11.015064+0000","last_clean":"2024-09-13T08:37:11.015064+0000","last_became_active":"2024-09-13T08:37:10.016877+0000","last_became_peered":"2024-09-13T08:37:10.016877+0000","last_unstale":"2024-09-13T08:37:11.015064+0000","last_undegraded":"2024-09-13T08:37:11.015064+0000","last_fullsized":"2024-09-13T08:37:11.015064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.474080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723910+0000","last_change":"2024-09-13T08:37:10.023947+0000","last_active":"2024-09-13T08:37:12.723910+0000","last_peered":"2024-09-13T08:37:12.723910+0000","last_clean":"2024-09-13T08:37:12.723910+0000","last_became_active":"2024-09-13T08:37:10.023659+0000","last_became_peered":"2024-09-13T08:37:10.023659+0000","last_unstale":"2024-09-13T08:37:12.723910+0000","last_undegraded":"2024-09-13T08:37:12.723910+0000","last_fullsized":"2024-09-13T08:37:12.723910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:42.583333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328520+0000","last_change":"2024-09-13T08:37:09.014843+0000","last_active":"2024-09-13T08:37:11.328520+0000","last_peered":"2024-09-13T08:37:11.328520+0000","last_clean":"2024-09-13T08:37:11.328520+0000","last_became_active":"2024-09-13T08:37:09.014454+0000","last_became_peered":"2024-09-13T08:37:09.014454+0000","last_unstale":"2024-09-13T08:37:11.328520+0000","last_undegraded":"2024-09-13T08:37:11.328520+0000","last_fullsized":"2024-09-13T08:37:11.328520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016454+0000","last_change":"2024-09-13T08:37:10.032852+0000","last_active":"2024-09-13T08:37:11.016454+0000","last_peered":"2024-09-13T08:37:11.016454+0000","last_clean":"2024-09-13T08:37:11.016454+0000","last_became_active":"2024-09-13T08:37:10.032407+0000","last_became_peered":"2024-09-13T08:37:10.032407+0000","last_unstale":"2024-09-13T08:37:11.016454+0000","last_undegraded":"2024-09-13T08:37:11.016454+0000","last_fullsized":"2024-09-13T08:37:11.016454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:01.091860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596441+0000","last_change":"2024-09-13T08:37:09.024672+0000","last_active":"2024-09-13T08:37:13.596441+0000","last_peered":"2024-09-13T08:37:13.596441+0000","last_clean":"2024-09-13T08:37:13.596441+0000","last_became_active":"2024-09-13T08:37:09.024363+0000","last_became_peered":"2024-09-13T08:37:09.024363+0000","last_unstale":"2024-09-13T08:37:13.596441+0000","last_undegraded":"2024-09-13T08:37:13.596441+0000","last_fullsized":"2024-09-13T08:37:13.596441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:55.612953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863222+0000","last_change":"2024-09-13T08:37:09.014986+0000","last_active":"2024-09-13T08:37:12.863222+0000","last_peered":"2024-09-13T08:37:12.863222+0000","last_clean":"2024-09-13T08:37:12.863222+0000","last_became_active":"2024-09-13T08:37:09.014658+0000","last_became_peered":"2024-09-13T08:37:09.014658+0000","last_unstale":"2024-09-13T08:37:12.863222+0000","last_undegraded":"2024-09-13T08:37:12.863222+0000","last_fullsized":"2024-09-13T08:37:12.863222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328628+0000","last_change":"2024-09-13T08:37:10.030653+0000","last_active":"2024-09-13T08:37:11.328628+0000","last_peered":"2024-09-13T08:37:11.328628+0000","last_clean":"2024-09-13T08:37:11.328628+0000","last_became_active":"2024-09-13T08:37:10.030383+0000","last_became_peered":"2024-09-13T08:37:10.030383+0000","last_unstale":"2024-09-13T08:37:11.328628+0000","last_undegraded":"2024-09-13T08:37:11.328628+0000","last_fullsized":"2024-09-13T08:37:11.328628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:20:47.231393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596255+0000","last_change":"2024-09-13T08:37:09.023571+0000","last_active":"2024-09-13T08:37:13.596255+0000","last_peered":"2024-09-13T08:37:13.596255+0000","last_clean":"2024-09-13T08:37:13.596255+0000","last_became_active":"2024-09-13T08:37:09.023048+0000","last_became_peered":"2024-09-13T08:37:09.023048+0000","last_unstale":"2024-09-13T08:37:13.596255+0000","last_undegraded":"2024-09-13T08:37:13.596255+0000","last_fullsized":"2024-09-13T08:37:13.596255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:14.567695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016336+0000","last_change":"2024-09-13T08:37:10.010601+0000","last_active":"2024-09-13T08:37:11.016336+0000","last_peered":"2024-09-13T08:37:11.016336+0000","last_clean":"2024-09-13T08:37:11.016336+0000","last_became_active":"2024-09-13T08:37:10.010323+0000","last_became_peered":"2024-09-13T08:37:10.010323+0000","last_unstale":"2024-09-13T08:37:11.016336+0000","last_undegraded":"2024-09-13T08:37:11.016336+0000","last_fullsized":"2024-09-13T08:37:11.016336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:39:25.479292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723890+0000","last_change":"2024-09-13T08:37:09.010512+0000","last_active":"2024-09-13T08:37:12.723890+0000","last_peered":"2024-09-13T08:37:12.723890+0000","last_clean":"2024-09-13T08:37:12.723890+0000","last_became_active":"2024-09-13T08:37:09.010229+0000","last_became_peered":"2024-09-13T08:37:09.010229+0000","last_unstale":"2024-09-13T08:37:12.723890+0000","last_undegraded":"2024-09-13T08:37:12.723890+0000","last_fullsized":"2024-09-13T08:37:12.723890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:27:27.593797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015969+0000","last_change":"2024-09-13T08:37:10.020061+0000","last_active":"2024-09-13T08:37:11.015969+0000","last_peered":"2024-09-13T08:37:11.015969+0000","last_clean":"2024-09-13T08:37:11.015969+0000","last_became_active":"2024-09-13T08:37:10.019663+0000","last_became_peered":"2024-09-13T08:37:10.019663+0000","last_unstale":"2024-09-13T08:37:11.015969+0000","last_undegraded":"2024-09-13T08:37:11.015969+0000","last_fullsized":"2024-09-13T08:37:11.015969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:48.021040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021832+0000","last_change":"2024-09-13T08:37:09.011936+0000","last_active":"2024-09-13T08:37:11.021832+0000","last_peered":"2024-09-13T08:37:11.021832+0000","last_clean":"2024-09-13T08:37:11.021832+0000","last_became_active":"2024-09-13T08:37:09.011670+0000","last_became_peered":"2024-09-13T08:37:09.011670+0000","last_unstale":"2024-09-13T08:37:11.021832+0000","last_undegraded":"2024-09-13T08:37:11.021832+0000","last_fullsized":"2024-09-13T08:37:11.021832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:31.341815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329184+0000","last_change":"2024-09-13T08:37:10.027709+0000","last_active":"2024-09-13T08:37:11.329184+0000","last_peered":"2024-09-13T08:37:11.329184+0000","last_clean":"2024-09-13T08:37:11.329184+0000","last_became_active":"2024-09-13T08:37:10.027410+0000","last_became_peered":"2024-09-13T08:37:10.027410+0000","last_unstale":"2024-09-13T08:37:11.329184+0000","last_undegraded":"2024-09-13T08:37:11.329184+0000","last_fullsized":"2024-09-13T08:37:11.329184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:25.100540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017898+0000","last_change":"2024-09-13T08:37:09.003083+0000","last_active":"2024-09-13T08:37:11.017898+0000","last_peered":"2024-09-13T08:37:11.017898+0000","last_clean":"2024-09-13T08:37:11.017898+0000","last_became_active":"2024-09-13T08:37:09.002697+0000","last_became_peered":"2024-09-13T08:37:09.002697+0000","last_unstale":"2024-09-13T08:37:11.017898+0000","last_undegraded":"2024-09-13T08:37:11.017898+0000","last_fullsized":"2024-09-13T08:37:11.017898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:56.231941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596339+0000","last_change":"2024-09-13T08:37:10.021360+0000","last_active":"2024-09-13T08:37:13.596339+0000","last_peered":"2024-09-13T08:37:13.596339+0000","last_clean":"2024-09-13T08:37:13.596339+0000","last_became_active":"2024-09-13T08:37:10.021080+0000","last_became_peered":"2024-09-13T08:37:10.021080+0000","last_unstale":"2024-09-13T08:37:13.596339+0000","last_undegraded":"2024-09-13T08:37:13.596339+0000","last_fullsized":"2024-09-13T08:37:13.596339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:32.591794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015415+0000","last_change":"2024-09-13T08:37:09.014879+0000","last_active":"2024-09-13T08:37:11.015415+0000","last_peered":"2024-09-13T08:37:11.015415+0000","last_clean":"2024-09-13T08:37:11.015415+0000","last_became_active":"2024-09-13T08:37:09.014572+0000","last_became_peered":"2024-09-13T08:37:09.014572+0000","last_unstale":"2024-09-13T08:37:11.015415+0000","last_undegraded":"2024-09-13T08:37:11.015415+0000","last_fullsized":"2024-09-13T08:37:11.015415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:14.033544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022331+0000","last_change":"2024-09-13T08:37:10.027041+0000","last_active":"2024-09-13T08:37:11.022331+0000","last_peered":"2024-09-13T08:37:11.022331+0000","last_clean":"2024-09-13T08:37:11.022331+0000","last_became_active":"2024-09-13T08:37:10.026781+0000","last_became_peered":"2024-09-13T08:37:10.026781+0000","last_unstale":"2024-09-13T08:37:11.022331+0000","last_undegraded":"2024-09-13T08:37:11.022331+0000","last_fullsized":"2024-09-13T08:37:11.022331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017361+0000","last_change":"2024-09-13T08:37:09.015199+0000","last_active":"2024-09-13T08:37:11.017361+0000","last_peered":"2024-09-13T08:37:11.017361+0000","last_clean":"2024-09-13T08:37:11.017361+0000","last_became_active":"2024-09-13T08:37:09.014848+0000","last_became_peered":"2024-09-13T08:37:09.014848+0000","last_unstale":"2024-09-13T08:37:11.017361+0000","last_undegraded":"2024-09-13T08:37:11.017361+0000","last_fullsized":"2024-09-13T08:37:11.017361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:01.967164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863695+0000","last_change":"2024-09-13T08:37:10.027034+0000","last_active":"2024-09-13T08:37:12.863695+0000","last_peered":"2024-09-13T08:37:12.863695+0000","last_clean":"2024-09-13T08:37:12.863695+0000","last_became_active":"2024-09-13T08:37:10.026723+0000","last_became_peered":"2024-09-13T08:37:10.026723+0000","last_unstale":"2024-09-13T08:37:12.863695+0000","last_undegraded":"2024-09-13T08:37:12.863695+0000","last_fullsized":"2024-09-13T08:37:12.863695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:56.618550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596545+0000","last_change":"2024-09-13T08:37:09.021683+0000","last_active":"2024-09-13T08:37:13.596545+0000","last_peered":"2024-09-13T08:37:13.596545+0000","last_clean":"2024-09-13T08:37:13.596545+0000","last_became_active":"2024-09-13T08:37:09.019630+0000","last_became_peered":"2024-09-13T08:37:09.019630+0000","last_unstale":"2024-09-13T08:37:13.596545+0000","last_undegraded":"2024-09-13T08:37:13.596545+0000","last_fullsized":"2024-09-13T08:37:13.596545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:47.015443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596308+0000","last_change":"2024-09-13T08:37:10.032446+0000","last_active":"2024-09-13T08:37:13.596308+0000","last_peered":"2024-09-13T08:37:13.596308+0000","last_clean":"2024-09-13T08:37:13.596308+0000","last_became_active":"2024-09-13T08:37:10.031448+0000","last_became_peered":"2024-09-13T08:37:10.031448+0000","last_unstale":"2024-09-13T08:37:13.596308+0000","last_undegraded":"2024-09-13T08:37:13.596308+0000","last_fullsized":"2024-09-13T08:37:13.596308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:38:36.201616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021502+0000","last_change":"2024-09-13T08:37:09.025565+0000","last_active":"2024-09-13T08:37:11.021502+0000","last_peered":"2024-09-13T08:37:11.021502+0000","last_clean":"2024-09-13T08:37:11.021502+0000","last_became_active":"2024-09-13T08:37:09.025242+0000","last_became_peered":"2024-09-13T08:37:09.025242+0000","last_unstale":"2024-09-13T08:37:11.021502+0000","last_undegraded":"2024-09-13T08:37:11.021502+0000","last_fullsized":"2024-09-13T08:37:11.021502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:08.638798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723679+0000","last_change":"2024-09-13T08:37:10.030817+0000","last_active":"2024-09-13T08:37:12.723679+0000","last_peered":"2024-09-13T08:37:12.723679+0000","last_clean":"2024-09-13T08:37:12.723679+0000","last_became_active":"2024-09-13T08:37:10.030554+0000","last_became_peered":"2024-09-13T08:37:10.030554+0000","last_unstale":"2024-09-13T08:37:12.723679+0000","last_undegraded":"2024-09-13T08:37:12.723679+0000","last_fullsized":"2024-09-13T08:37:12.723679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:15.232649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016219+0000","last_change":"2024-09-13T08:37:08.998468+0000","last_active":"2024-09-13T08:37:11.016219+0000","last_peered":"2024-09-13T08:37:11.016219+0000","last_clean":"2024-09-13T08:37:11.016219+0000","last_became_active":"2024-09-13T08:37:08.998166+0000","last_became_peered":"2024-09-13T08:37:08.998166+0000","last_unstale":"2024-09-13T08:37:11.016219+0000","last_undegraded":"2024-09-13T08:37:11.016219+0000","last_fullsized":"2024-09-13T08:37:11.016219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:42.430033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596103+0000","last_change":"2024-09-13T08:37:10.023766+0000","last_active":"2024-09-13T08:37:13.596103+0000","last_peered":"2024-09-13T08:37:13.596103+0000","last_clean":"2024-09-13T08:37:13.596103+0000","last_became_active":"2024-09-13T08:37:10.022889+0000","last_became_peered":"2024-09-13T08:37:10.022889+0000","last_unstale":"2024-09-13T08:37:13.596103+0000","last_undegraded":"2024-09-13T08:37:13.596103+0000","last_fullsized":"2024-09-13T08:37:13.596103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:37:08.617480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022117+0000","last_change":"2024-09-13T08:37:09.023974+0000","last_active":"2024-09-13T08:37:11.022117+0000","last_peered":"2024-09-13T08:37:11.022117+0000","last_clean":"2024-09-13T08:37:11.022117+0000","last_became_active":"2024-09-13T08:37:09.023471+0000","last_became_peered":"2024-09-13T08:37:09.023471+0000","last_unstale":"2024-09-13T08:37:11.022117+0000","last_undegraded":"2024-09-13T08:37:11.022117+0000","last_fullsized":"2024-09-13T08:37:11.022117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:16:38.190525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016062+0000","last_change":"2024-09-13T08:37:10.023008+0000","last_active":"2024-09-13T08:37:11.016062+0000","last_peered":"2024-09-13T08:37:11.016062+0000","last_clean":"2024-09-13T08:37:11.016062+0000","last_became_active":"2024-09-13T08:37:10.021695+0000","last_became_peered":"2024-09-13T08:37:10.021695+0000","last_unstale":"2024-09-13T08:37:11.016062+0000","last_undegraded":"2024-09-13T08:37:11.016062+0000","last_fullsized":"2024-09-13T08:37:11.016062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:43.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016583+0000","last_change":"2024-09-13T08:37:09.020522+0000","last_active":"2024-09-13T08:37:11.016583+0000","last_peered":"2024-09-13T08:37:11.016583+0000","last_clean":"2024-09-13T08:37:11.016583+0000","last_became_active":"2024-09-13T08:37:09.020245+0000","last_became_peered":"2024-09-13T08:37:09.020245+0000","last_unstale":"2024-09-13T08:37:11.016583+0000","last_undegraded":"2024-09-13T08:37:11.016583+0000","last_fullsized":"2024-09-13T08:37:11.016583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:48:10.286016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022182+0000","last_change":"2024-09-13T08:37:10.031929+0000","last_active":"2024-09-13T08:37:11.022182+0000","last_peered":"2024-09-13T08:37:11.022182+0000","last_clean":"2024-09-13T08:37:11.022182+0000","last_became_active":"2024-09-13T08:37:10.031598+0000","last_became_peered":"2024-09-13T08:37:10.031598+0000","last_unstale":"2024-09-13T08:37:11.022182+0000","last_undegraded":"2024-09-13T08:37:11.022182+0000","last_fullsized":"2024-09-13T08:37:11.022182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:38.056766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328623+0000","last_change":"2024-09-13T08:37:09.011613+0000","last_active":"2024-09-13T08:37:11.328623+0000","last_peered":"2024-09-13T08:37:11.328623+0000","last_clean":"2024-09-13T08:37:11.328623+0000","last_became_active":"2024-09-13T08:37:09.011071+0000","last_became_peered":"2024-09-13T08:37:09.011071+0000","last_unstale":"2024-09-13T08:37:11.328623+0000","last_undegraded":"2024-09-13T08:37:11.328623+0000","last_fullsized":"2024-09-13T08:37:11.328623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022072+0000","last_change":"2024-09-13T08:37:10.028420+0000","last_active":"2024-09-13T08:37:11.022072+0000","last_peered":"2024-09-13T08:37:11.022072+0000","last_clean":"2024-09-13T08:37:11.022072+0000","last_became_active":"2024-09-13T08:37:10.028166+0000","last_became_peered":"2024-09-13T08:37:10.028166+0000","last_unstale":"2024-09-13T08:37:11.022072+0000","last_undegraded":"2024-09-13T08:37:11.022072+0000","last_fullsized":"2024-09-13T08:37:11.022072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:58.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863837+0000","last_change":"2024-09-13T08:37:09.015072+0000","last_active":"2024-09-13T08:37:12.863837+0000","last_peered":"2024-09-13T08:37:12.863837+0000","last_clean":"2024-09-13T08:37:12.863837+0000","last_became_active":"2024-09-13T08:37:09.014597+0000","last_became_peered":"2024-09-13T08:37:09.014597+0000","last_unstale":"2024-09-13T08:37:12.863837+0000","last_undegraded":"2024-09-13T08:37:12.863837+0000","last_fullsized":"2024-09-13T08:37:12.863837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596652+0000","last_change":"2024-09-13T08:37:10.018219+0000","last_active":"2024-09-13T08:37:13.596652+0000","last_peered":"2024-09-13T08:37:13.596652+0000","last_clean":"2024-09-13T08:37:13.596652+0000","last_became_active":"2024-09-13T08:37:10.017946+0000","last_became_peered":"2024-09-13T08:37:10.017946+0000","last_unstale":"2024-09-13T08:37:13.596652+0000","last_undegraded":"2024-09-13T08:37:13.596652+0000","last_fullsized":"2024-09-13T08:37:13.596652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:12.941476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723548+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:37:12.723548+0000","last_peered":"2024-09-13T08:37:12.723548+0000","last_clean":"2024-09-13T08:37:12.723548+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:37:12.723548+0000","last_undegraded":"2024-09-13T08:37:12.723548+0000","last_fullsized":"2024-09-13T08:37:12.723548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596411+0000","last_change":"2024-09-13T08:37:09.021195+0000","last_active":"2024-09-13T08:37:13.596411+0000","last_peered":"2024-09-13T08:37:13.596411+0000","last_clean":"2024-09-13T08:37:13.596411+0000","last_became_active":"2024-09-13T08:37:09.019692+0000","last_became_peered":"2024-09-13T08:37:09.019692+0000","last_unstale":"2024-09-13T08:37:13.596411+0000","last_undegraded":"2024-09-13T08:37:13.596411+0000","last_fullsized":"2024-09-13T08:37:13.596411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.430584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016371+0000","last_change":"2024-09-13T08:37:10.020656+0000","last_active":"2024-09-13T08:37:11.016371+0000","last_peered":"2024-09-13T08:37:11.016371+0000","last_clean":"2024-09-13T08:37:11.016371+0000","last_became_active":"2024-09-13T08:37:10.020273+0000","last_became_peered":"2024-09-13T08:37:10.020273+0000","last_unstale":"2024-09-13T08:37:11.016371+0000","last_undegraded":"2024-09-13T08:37:11.016371+0000","last_fullsized":"2024-09-13T08:37:11.016371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:51.457651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596362+0000","last_change":"2024-09-13T08:37:09.019756+0000","last_active":"2024-09-13T08:37:13.596362+0000","last_peered":"2024-09-13T08:37:13.596362+0000","last_clean":"2024-09-13T08:37:13.596362+0000","last_became_active":"2024-09-13T08:37:09.019375+0000","last_became_peered":"2024-09-13T08:37:09.019375+0000","last_unstale":"2024-09-13T08:37:13.596362+0000","last_undegraded":"2024-09-13T08:37:13.596362+0000","last_fullsized":"2024-09-13T08:37:13.596362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:40.495518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863575+0000","last_change":"2024-09-13T08:37:10.020491+0000","last_active":"2024-09-13T08:37:12.863575+0000","last_peered":"2024-09-13T08:37:12.863575+0000","last_clean":"2024-09-13T08:37:12.863575+0000","last_became_active":"2024-09-13T08:37:10.020271+0000","last_became_peered":"2024-09-13T08:37:10.020271+0000","last_unstale":"2024-09-13T08:37:12.863575+0000","last_undegraded":"2024-09-13T08:37:12.863575+0000","last_fullsized":"2024-09-13T08:37:12.863575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:29.393985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016620+0000","last_change":"2024-09-13T08:37:09.015889+0000","last_active":"2024-09-13T08:37:11.016620+0000","last_peered":"2024-09-13T08:37:11.016620+0000","last_clean":"2024-09-13T08:37:11.016620+0000","last_became_active":"2024-09-13T08:37:09.015587+0000","last_became_peered":"2024-09-13T08:37:09.015587+0000","last_unstale":"2024-09-13T08:37:11.016620+0000","last_undegraded":"2024-09-13T08:37:11.016620+0000","last_fullsized":"2024-09-13T08:37:11.016620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328457+0000","last_change":"2024-09-13T08:37:10.036912+0000","last_active":"2024-09-13T08:37:11.328457+0000","last_peered":"2024-09-13T08:37:11.328457+0000","last_clean":"2024-09-13T08:37:11.328457+0000","last_became_active":"2024-09-13T08:37:10.036665+0000","last_became_peered":"2024-09-13T08:37:10.036665+0000","last_unstale":"2024-09-13T08:37:11.328457+0000","last_undegraded":"2024-09-13T08:37:11.328457+0000","last_fullsized":"2024-09-13T08:37:11.328457+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:40.984142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596672+0000","last_change":"2024-09-13T08:37:09.023866+0000","last_active":"2024-09-13T08:37:13.596672+0000","last_peered":"2024-09-13T08:37:13.596672+0000","last_clean":"2024-09-13T08:37:13.596672+0000","last_became_active":"2024-09-13T08:37:09.021825+0000","last_became_peered":"2024-09-13T08:37:09.021825+0000","last_unstale":"2024-09-13T08:37:13.596672+0000","last_undegraded":"2024-09-13T08:37:13.596672+0000","last_fullsized":"2024-09-13T08:37:13.596672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:42.779836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022032+0000","last_change":"2024-09-13T08:37:10.027315+0000","last_active":"2024-09-13T08:37:11.022032+0000","last_peered":"2024-09-13T08:37:11.022032+0000","last_clean":"2024-09-13T08:37:11.022032+0000","last_became_active":"2024-09-13T08:37:10.027018+0000","last_became_peered":"2024-09-13T08:37:10.027018+0000","last_unstale":"2024-09-13T08:37:11.022032+0000","last_undegraded":"2024-09-13T08:37:11.022032+0000","last_fullsized":"2024-09-13T08:37:11.022032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:44.371052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328769+0000","last_change":"2024-09-13T08:37:09.018684+0000","last_active":"2024-09-13T08:37:11.328769+0000","last_peered":"2024-09-13T08:37:11.328769+0000","last_clean":"2024-09-13T08:37:11.328769+0000","last_became_active":"2024-09-13T08:37:09.016570+0000","last_became_peered":"2024-09-13T08:37:09.016570+0000","last_unstale":"2024-09-13T08:37:11.328769+0000","last_undegraded":"2024-09-13T08:37:11.328769+0000","last_fullsized":"2024-09-13T08:37:11.328769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:00.098137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.015606+0000","last_change":"2024-09-13T08:37:10.018230+0000","last_active":"2024-09-13T08:37:11.015606+0000","last_peered":"2024-09-13T08:37:11.015606+0000","last_clean":"2024-09-13T08:37:11.015606+0000","last_became_active":"2024-09-13T08:37:10.017427+0000","last_became_peered":"2024-09-13T08:37:10.017427+0000","last_unstale":"2024-09-13T08:37:11.015606+0000","last_undegraded":"2024-09-13T08:37:11.015606+0000","last_fullsized":"2024-09-13T08:37:11.015606+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:24.300914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596769+0000","last_change":"2024-09-13T08:37:09.018807+0000","last_active":"2024-09-13T08:37:13.596769+0000","last_peered":"2024-09-13T08:37:13.596769+0000","last_clean":"2024-09-13T08:37:13.596769+0000","last_became_active":"2024-09-13T08:37:09.018298+0000","last_became_peered":"2024-09-13T08:37:09.018298+0000","last_unstale":"2024-09-13T08:37:13.596769+0000","last_undegraded":"2024-09-13T08:37:13.596769+0000","last_fullsized":"2024-09-13T08:37:13.596769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.295316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863880+0000","last_change":"2024-09-13T08:37:10.028730+0000","last_active":"2024-09-13T08:37:12.863880+0000","last_peered":"2024-09-13T08:37:12.863880+0000","last_clean":"2024-09-13T08:37:12.863880+0000","last_became_active":"2024-09-13T08:37:10.028481+0000","last_became_peered":"2024-09-13T08:37:10.028481+0000","last_unstale":"2024-09-13T08:37:12.863880+0000","last_undegraded":"2024-09-13T08:37:12.863880+0000","last_fullsized":"2024-09-13T08:37:12.863880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:55.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.020773+0000","last_change":"2024-09-13T08:37:09.014011+0000","last_active":"2024-09-13T08:37:11.020773+0000","last_peered":"2024-09-13T08:37:11.020773+0000","last_clean":"2024-09-13T08:37:11.020773+0000","last_became_active":"2024-09-13T08:37:09.013557+0000","last_became_peered":"2024-09-13T08:37:09.013557+0000","last_unstale":"2024-09-13T08:37:11.020773+0000","last_undegraded":"2024-09-13T08:37:11.020773+0000","last_fullsized":"2024-09-13T08:37:11.020773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:08.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017084+0000","last_change":"2024-09-13T08:37:10.023150+0000","last_active":"2024-09-13T08:37:11.017084+0000","last_peered":"2024-09-13T08:37:11.017084+0000","last_clean":"2024-09-13T08:37:11.017084+0000","last_became_active":"2024-09-13T08:37:10.020537+0000","last_became_peered":"2024-09-13T08:37:10.020537+0000","last_unstale":"2024-09-13T08:37:11.017084+0000","last_undegraded":"2024-09-13T08:37:11.017084+0000","last_fullsized":"2024-09-13T08:37:11.017084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:11.212577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596783+0000","last_change":"2024-09-13T08:37:09.024256+0000","last_active":"2024-09-13T08:37:13.596783+0000","last_peered":"2024-09-13T08:37:13.596783+0000","last_clean":"2024-09-13T08:37:13.596783+0000","last_became_active":"2024-09-13T08:37:09.023967+0000","last_became_peered":"2024-09-13T08:37:09.023967+0000","last_unstale":"2024-09-13T08:37:13.596783+0000","last_undegraded":"2024-09-13T08:37:13.596783+0000","last_fullsized":"2024-09-13T08:37:13.596783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:41.577735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016440+0000","last_change":"2024-09-13T08:37:10.029427+0000","last_active":"2024-09-13T08:37:11.016440+0000","last_peered":"2024-09-13T08:37:11.016440+0000","last_clean":"2024-09-13T08:37:11.016440+0000","last_became_active":"2024-09-13T08:37:10.029154+0000","last_became_peered":"2024-09-13T08:37:10.029154+0000","last_unstale":"2024-09-13T08:37:11.016440+0000","last_undegraded":"2024-09-13T08:37:11.016440+0000","last_fullsized":"2024-09-13T08:37:11.016440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:46.110619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.021357+0000","last_change":"2024-09-13T08:37:09.011069+0000","last_active":"2024-09-13T08:37:11.021357+0000","last_peered":"2024-09-13T08:37:11.021357+0000","last_clean":"2024-09-13T08:37:11.021357+0000","last_became_active":"2024-09-13T08:37:09.010687+0000","last_became_peered":"2024-09-13T08:37:09.010687+0000","last_unstale":"2024-09-13T08:37:11.021357+0000","last_undegraded":"2024-09-13T08:37:11.021357+0000","last_fullsized":"2024-09-13T08:37:11.021357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:09:52.672909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596198+0000","last_change":"2024-09-13T08:37:10.026288+0000","last_active":"2024-09-13T08:37:13.596198+0000","last_peered":"2024-09-13T08:37:13.596198+0000","last_clean":"2024-09-13T08:37:13.596198+0000","last_became_active":"2024-09-13T08:37:10.026029+0000","last_became_peered":"2024-09-13T08:37:10.026029+0000","last_unstale":"2024-09-13T08:37:13.596198+0000","last_undegraded":"2024-09-13T08:37:13.596198+0000","last_fullsized":"2024-09-13T08:37:13.596198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:38.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017168+0000","last_change":"2024-09-13T08:37:09.010927+0000","last_active":"2024-09-13T08:37:11.017168+0000","last_peered":"2024-09-13T08:37:11.017168+0000","last_clean":"2024-09-13T08:37:11.017168+0000","last_became_active":"2024-09-13T08:37:09.010405+0000","last_became_peered":"2024-09-13T08:37:09.010405+0000","last_unstale":"2024-09-13T08:37:11.017168+0000","last_undegraded":"2024-09-13T08:37:11.017168+0000","last_fullsized":"2024-09-13T08:37:11.017168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:53.930944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016937+0000","last_change":"2024-09-13T08:37:10.026070+0000","last_active":"2024-09-13T08:37:11.016937+0000","last_peered":"2024-09-13T08:37:11.016937+0000","last_clean":"2024-09-13T08:37:11.016937+0000","last_became_active":"2024-09-13T08:37:10.025659+0000","last_became_peered":"2024-09-13T08:37:10.025659+0000","last_unstale":"2024-09-13T08:37:11.016937+0000","last_undegraded":"2024-09-13T08:37:11.016937+0000","last_fullsized":"2024-09-13T08:37:11.016937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:13.518772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.863577+0000","last_change":"2024-09-13T08:37:09.019714+0000","last_active":"2024-09-13T08:37:12.863577+0000","last_peered":"2024-09-13T08:37:12.863577+0000","last_clean":"2024-09-13T08:37:12.863577+0000","last_became_active":"2024-09-13T08:37:09.018957+0000","last_became_peered":"2024-09-13T08:37:09.018957+0000","last_unstale":"2024-09-13T08:37:12.863577+0000","last_undegraded":"2024-09-13T08:37:12.863577+0000","last_fullsized":"2024-09-13T08:37:12.863577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:13.856443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.017312+0000","last_change":"2024-09-13T08:37:10.026722+0000","last_active":"2024-09-13T08:37:11.017312+0000","last_peered":"2024-09-13T08:37:11.017312+0000","last_clean":"2024-09-13T08:37:11.017312+0000","last_became_active":"2024-09-13T08:37:10.025267+0000","last_became_peered":"2024-09-13T08:37:10.025267+0000","last_unstale":"2024-09-13T08:37:11.017312+0000","last_undegraded":"2024-09-13T08:37:11.017312+0000","last_fullsized":"2024-09-13T08:37:11.017312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596295+0000","last_change":"2024-09-13T08:37:09.022982+0000","last_active":"2024-09-13T08:37:13.596295+0000","last_peered":"2024-09-13T08:37:13.596295+0000","last_clean":"2024-09-13T08:37:13.596295+0000","last_became_active":"2024-09-13T08:37:09.021746+0000","last_became_peered":"2024-09-13T08:37:09.021746+0000","last_unstale":"2024-09-13T08:37:13.596295+0000","last_undegraded":"2024-09-13T08:37:13.596295+0000","last_fullsized":"2024-09-13T08:37:13.596295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.468482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596487+0000","last_change":"2024-09-13T08:37:10.016999+0000","last_active":"2024-09-13T08:37:13.596487+0000","last_peered":"2024-09-13T08:37:13.596487+0000","last_clean":"2024-09-13T08:37:13.596487+0000","last_became_active":"2024-09-13T08:37:10.016738+0000","last_became_peered":"2024-09-13T08:37:10.016738+0000","last_unstale":"2024-09-13T08:37:13.596487+0000","last_undegraded":"2024-09-13T08:37:13.596487+0000","last_fullsized":"2024-09-13T08:37:13.596487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:16:16.734699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:13.596582+0000","last_change":"2024-09-13T08:37:10.026898+0000","last_active":"2024-09-13T08:37:13.596582+0000","last_peered":"2024-09-13T08:37:13.596582+0000","last_clean":"2024-09-13T08:37:13.596582+0000","last_became_active":"2024-09-13T08:37:10.026602+0000","last_became_peered":"2024-09-13T08:37:10.026602+0000","last_unstale":"2024-09-13T08:37:13.596582+0000","last_undegraded":"2024-09-13T08:37:13.596582+0000","last_fullsized":"2024-09-13T08:37:13.596582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:42.330227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022006+0000","last_change":"2024-09-13T08:37:09.011138+0000","last_active":"2024-09-13T08:37:11.022006+0000","last_peered":"2024-09-13T08:37:11.022006+0000","last_clean":"2024-09-13T08:37:11.022006+0000","last_became_active":"2024-09-13T08:37:09.010863+0000","last_became_peered":"2024-09-13T08:37:09.010863+0000","last_unstale":"2024-09-13T08:37:11.022006+0000","last_undegraded":"2024-09-13T08:37:11.022006+0000","last_fullsized":"2024-09-13T08:37:11.022006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:14.227052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:12.723804+0000","last_change":"2024-09-13T08:37:10.029092+0000","last_active":"2024-09-13T08:37:12.723804+0000","last_peered":"2024-09-13T08:37:12.723804+0000","last_clean":"2024-09-13T08:37:12.723804+0000","last_became_active":"2024-09-13T08:37:10.028796+0000","last_became_peered":"2024-09-13T08:37:10.028796+0000","last_unstale":"2024-09-13T08:37:12.723804+0000","last_undegraded":"2024-09-13T08:37:12.723804+0000","last_fullsized":"2024-09-13T08:37:12.723804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:05.436181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.022178+0000","last_change":"2024-09-13T08:37:09.013579+0000","last_active":"2024-09-13T08:37:11.022178+0000","last_peered":"2024-09-13T08:37:11.022178+0000","last_clean":"2024-09-13T08:37:11.022178+0000","last_became_active":"2024-09-13T08:37:09.013327+0000","last_became_peered":"2024-09-13T08:37:09.013327+0000","last_unstale":"2024-09-13T08:37:11.022178+0000","last_undegraded":"2024-09-13T08:37:11.022178+0000","last_fullsized":"2024-09-13T08:37:11.022178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:34.207907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.328992+0000","last_change":"2024-09-13T08:37:10.032358+0000","last_active":"2024-09-13T08:37:11.328992+0000","last_peered":"2024-09-13T08:37:11.328992+0000","last_clean":"2024-09-13T08:37:11.328992+0000","last_became_active":"2024-09-13T08:37:10.032047+0000","last_became_peered":"2024-09-13T08:37:10.032047+0000","last_unstale":"2024-09-13T08:37:11.328992+0000","last_undegraded":"2024-09-13T08:37:11.328992+0000","last_fullsized":"2024-09-13T08:37:11.328992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:19.283020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329066+0000","last_change":"2024-09-13T08:37:09.015906+0000","last_active":"2024-09-13T08:37:11.329066+0000","last_peered":"2024-09-13T08:37:11.329066+0000","last_clean":"2024-09-13T08:37:11.329066+0000","last_became_active":"2024-09-13T08:37:09.015500+0000","last_became_peered":"2024-09-13T08:37:09.015500+0000","last_unstale":"2024-09-13T08:37:11.329066+0000","last_undegraded":"2024-09-13T08:37:11.329066+0000","last_fullsized":"2024-09-13T08:37:11.329066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:51.846602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.329023+0000","last_change":"2024-09-13T08:37:10.043965+0000","last_active":"2024-09-13T08:37:11.329023+0000","last_peered":"2024-09-13T08:37:11.329023+0000","last_clean":"2024-09-13T08:37:11.329023+0000","last_became_active":"2024-09-13T08:37:10.041509+0000","last_became_peered":"2024-09-13T08:37:10.041509+0000","last_unstale":"2024-09-13T08:37:11.329023+0000","last_undegraded":"2024-09-13T08:37:11.329023+0000","last_fullsized":"2024-09-13T08:37:11.329023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:08.985399+0000","last_clean_scrub_stamp":"2024-09-13T08:37:08.985399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:33.817466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-13T08:37:11.016520+0000","last_change":"2024-09-13T08:37:09.024589+0000","last_active":"2024-09-13T08:37:11.016520+0000","last_peered":"2024-09-13T08:37:11.016520+0000","last_clean":"2024-09-13T08:37:11.016520+0000","last_became_active":"2024-09-13T08:37:09.024297+0000","last_became_peered":"2024-09-13T08:37:09.024297+0000","last_unstale":"2024-09-13T08:37:11.016520+0000","last_undegraded":"2024-09-13T08:37:11.016520+0000","last_fullsized":"2024-09-13T08:37:11.016520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:07.979801+0000","last_clean_scrub_stamp":"2024-09-13T08:37:07.979801+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:03.816662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":241664,"data_stored":81079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":80955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":835584,"data_stored":671415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":81537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":245760,"data_stored":82567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":671393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":81011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":83239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:37:24.990 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:37:24.990 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T08:37:24.991 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T08:37:24.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T08:37:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.184+0000 7f3b3ce00640 1 -- 172.21.15.31:0/1022259745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b3815dfc0 msgr2=0x7f3b380cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.184+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/1022259745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b3815dfc0 0x7f3b380cc300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009830 tx=0x7f3b2802a5c0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.184+0000 7f3b3ce00640 1 -- 172.21.15.31:0/1022259745 shutdown_connections 2024-09-13T08:37:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.184+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/1022259745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b3815dfc0 0x7f3b380cc300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.185+0000 7f3b3ce00640 1 -- 172.21.15.31:0/1022259745 >> 172.21.15.31:0/1022259745 conn(0x7f3b380cc910 msgr2=0x7f3b380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.185+0000 7f3b3ce00640 1 -- 172.21.15.31:0/1022259745 shutdown_connections 2024-09-13T08:37:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.185+0000 7f3b3ce00640 1 -- 172.21.15.31:0/1022259745 wait complete. 2024-09-13T08:37:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.186+0000 7f3b3ce00640 1 Processor -- start 2024-09-13T08:37:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.186+0000 7f3b3ce00640 1 -- start start 2024-09-13T08:37:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3b380b1910 0x7f3b3809d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b380ca5c0 0x7f3b3815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380b1d60 con 0x7f3b3815dfc0 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380740a0 con 0x7f3b380ca5c0 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b3ce00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3b380b1ee0 con 0x7f3b380b1910 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b35e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3b380b1910 0x7f3b3809d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42974/0 (socket says 172.21.15.31:42974) 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b36800640 1 -- 172.21.15.31:0/3026369253 learned_addr learned my addr 172.21.15.31:0/3026369253 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.187+0000 7f3b2d400640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b380ca5c0 0x7f3b3815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 -- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3b380b1910 msgr2=0x7f3b3809d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3b380b1910 0x7f3b3809d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 -- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b380ca5c0 msgr2=0x7f3b3815c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b380ca5c0 0x7f3b3815c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 -- 172.21.15.31:0/3026369253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b38072950 con 0x7f3b3815dfc0 2024-09-13T08:37:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b36800640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c7a0 tx=0x7f3b3000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30003b70 con 0x7f3b3815dfc0 2024-09-13T08:37:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b30003d10 con 0x7f3b3815dfc0 2024-09-13T08:37:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b38072fc0 con 0x7f3b3815dfc0 2024-09-13T08:37:25.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.188+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000aab0 con 0x7f3b3815dfc0 2024-09-13T08:37:25.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.189+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b38072c90 con 0x7f3b3815dfc0 2024-09-13T08:37:25.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.190+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b380cc300 con 0x7f3b3815dfc0 2024-09-13T08:37:25.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.191+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f3b3000ac50 con 0x7f3b3815dfc0 2024-09-13T08:37:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.192+0000 7f3b2ea00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 0x7f3b1407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.192+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f3b3009a6e0 con 0x7f3b3815dfc0 2024-09-13T08:37:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.192+0000 7f3b35e00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 0x7f3b1407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.193+0000 7f3b35e00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 0x7f3b1407ad10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b28034000 tx=0x7f3b28004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.196+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3b30062660 con 0x7f3b3815dfc0 2024-09-13T08:37:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.432+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3b38073190 con 0x7f3b3815dfc0 2024-09-13T08:37:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.433+0000 7f3b2ea00640 1 -- 172.21.15.31:0/3026369253 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3b30067090 con 0x7f3b3815dfc0 2024-09-13T08:37:25.434 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:25.434 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 msgr2=0x7f3b1407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 0x7f3b1407ad10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b28034000 tx=0x7f3b28004730 comp rx=0 tx=0).stop 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 msgr2=0x7f3b381f8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c7a0 tx=0x7f3b3000cc60 comp rx=0 tx=0).stop 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 shutdown_connections 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b3815dfc0 0x7f3b381f8b00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b380ca5c0 0x7f3b3815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3b14078850 0x7f3b1407ad10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 --2- 172.21.15.31:0/3026369253 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3b380b1910 0x7f3b3809d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 >> 172.21.15.31:0/3026369253 conn(0x7f3b380cc910 msgr2=0x7f3b38073750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 shutdown_connections 2024-09-13T08:37:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.435+0000 7f3b3ce00640 1 -- 172.21.15.31:0/3026369253 wait complete. 2024-09-13T08:37:25.451 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T08:37:25.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 -- 172.21.15.31:0/371063780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 msgr2=0x7ff8140ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 --2- 172.21.15.31:0/371063780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140ce3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff8140ca470 tx=0x7ff7fc02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 -- 172.21.15.31:0/371063780 shutdown_connections 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 --2- 172.21.15.31:0/371063780 >> v2:172.21.15.45:3300/0 conn(0x7ff8140ce900 0x7ff8140cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 --2- 172.21.15.31:0/371063780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140ce3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 -- 172.21.15.31:0/371063780 >> 172.21.15.31:0/371063780 conn(0x7ff8140cc910 msgr2=0x7ff8140cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 -- 172.21.15.31:0/371063780 shutdown_connections 2024-09-13T08:37:25.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.611+0000 7ff81b600640 1 -- 172.21.15.31:0/371063780 wait complete. 2024-09-13T08:37:25.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.613+0000 7ff81b600640 1 Processor -- start 2024-09-13T08:37:25.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.613+0000 7ff81b600640 1 -- start start 2024-09-13T08:37:25.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 0x7ff8140b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff81415bd70 0x7ff8140b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141600f0 con 0x7ff8140cf850 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8140cb850 con 0x7ff8140ce900 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.614+0000 7ff81b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff8140b5bc0 con 0x7ff81415bd70 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff818c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 0x7ff8140b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff818c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 0x7ff8140b5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:33476/0 (socket says 172.21.15.31:33476) 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff818c00640 1 -- 172.21.15.31:0/3286930493 learned_addr learned my addr 172.21.15.31:0/3286930493 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff819600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff81415bd70 0x7ff8140b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 -- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff81415bd70 msgr2=0x7ff8140b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff81415bd70 0x7ff8140b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 -- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 msgr2=0x7ff8140b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 0x7ff8140b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 -- 172.21.15.31:0/3286930493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8140b4d90 con 0x7ff8140cf850 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.615+0000 7ff80be00640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140b4850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff8140b26b0 tx=0x7ff7fc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.616+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc03f070 con 0x7ff8140cf850 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.616+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7fc0047a0 con 0x7ff8140cf850 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.616+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc035030 con 0x7ff8140cf850 2024-09-13T08:37:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.616+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8141f4cd0 con 0x7ff8140cf850 2024-09-13T08:37:25.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.616+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141f5290 con 0x7ff8140cf850 2024-09-13T08:37:25.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.617+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8140ce3c0 con 0x7ff8140cf850 2024-09-13T08:37:25.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.618+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7ff7fc042070 con 0x7ff8140cf850 2024-09-13T08:37:25.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.619+0000 7ff811600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 0x7ff7f007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:25.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.619+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7ff7fc0bb6a0 con 0x7ff8140cf850 2024-09-13T08:37:25.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.619+0000 7ff818c00640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 0x7ff7f007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:25.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.620+0000 7ff818c00640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 0x7ff7f007ad40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8040017e0 tx=0x7ff804009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:25.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.623+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff7fc0836e0 con 0x7ff8140cf850 2024-09-13T08:37:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.857+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8140b0010 con 0x7ff8140cf850 2024-09-13T08:37:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.858+0000 7ff811600640 1 -- 172.21.15.31:0/3286930493 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7ff7fc088110 con 0x7ff8140cf850 2024-09-13T08:37:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:25.859 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:25.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 msgr2=0x7ff7f007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 0x7ff7f007ad40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8040017e0 tx=0x7ff804009040 comp rx=0 tx=0).stop 2024-09-13T08:37:25.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 msgr2=0x7ff8140b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:25.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140b4850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff8140b26b0 tx=0x7ff7fc005e80 comp rx=0 tx=0).stop 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 shutdown_connections 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff7f0078880 0x7ff7f007ad40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff81415bd70 0x7ff8140b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8140cf850 0x7ff8140b4850 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 --2- 172.21.15.31:0/3286930493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8140ce900 0x7ff8140b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 >> 172.21.15.31:0/3286930493 conn(0x7ff8140cc910 msgr2=0x7ff8140acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 shutdown_connections 2024-09-13T08:37:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:25.860+0000 7ff81b600640 1 -- 172.21.15.31:0/3286930493 wait complete. 2024-09-13T08:37:25.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 -- 172.21.15.31:0/4234427875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0cf2d0 msgr2=0x7fe68c0b6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 --2- 172.21.15.31:0/4234427875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c0b6c10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0d0350 tx=0x7fe67402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 -- 172.21.15.31:0/4234427875 shutdown_connections 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 --2- 172.21.15.31:0/4234427875 >> v2:172.21.15.45:3300/0 conn(0x7fe68c0d44f0 0x7fe68c0b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 --2- 172.21.15.31:0/4234427875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c0b6c10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 -- 172.21.15.31:0/4234427875 >> 172.21.15.31:0/4234427875 conn(0x7fe68c09d1c0 msgr2=0x7fe68c0d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 -- 172.21.15.31:0/4234427875 shutdown_connections 2024-09-13T08:37:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.039+0000 7fe691400640 1 -- 172.21.15.31:0/4234427875 wait complete. 2024-09-13T08:37:26.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.040+0000 7fe691400640 1 Processor -- start 2024-09-13T08:37:26.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.041+0000 7fe691400640 1 -- start start 2024-09-13T08:37:26.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.041+0000 7fe691400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c15e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.041+0000 7fe691400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe691400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe68c0d44f0 0x7fe68c15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe691400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1628f0 con 0x7fe68c0d20c0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe691400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c162770 con 0x7fe68c0cf2d0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe691400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe68c162a70 con 0x7fe68c0d44f0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43008/0 (socket says 172.21.15.31:43008) 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 -- 172.21.15.31:0/3656652784 learned_addr learned my addr 172.21.15.31:0/3656652784 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe68be00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c15e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe690a00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe68c0d44f0 0x7fe68c15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 -- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe68c0d44f0 msgr2=0x7fe68c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe68c0d44f0 0x7fe68c15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 -- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c0cf2d0 msgr2=0x7fe68c15e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c15e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.042+0000 7fe683e00640 1 -- 172.21.15.31:0/3656652784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c15c3e0 con 0x7fe68c0d20c0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe683e00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe674009780 tx=0x7fe674032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674040070 con 0x7fe68c0d20c0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe674005110 con 0x7fe68c0d20c0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c0c7ad0 con 0x7fe68c0d20c0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674005540 con 0x7fe68c0d20c0 2024-09-13T08:37:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.043+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c0c8090 con 0x7fe68c0d20c0 2024-09-13T08:37:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.044+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c162060 con 0x7fe68c0d20c0 2024-09-13T08:37:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.045+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe674039040 con 0x7fe68c0d20c0 2024-09-13T08:37:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.046+0000 7fe688c00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 0x7fe66807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.046+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe6740bb670 con 0x7fe68c0d20c0 2024-09-13T08:37:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.047+0000 7fe68be00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 0x7fe66807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.047+0000 7fe68be00640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 0x7fe66807ad10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0017c0 tx=0x7fe67c007e90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.050+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe674083720 con 0x7fe68c0d20c0 2024-09-13T08:37:26.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.284+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe68c08df70 con 0x7fe68c0d20c0 2024-09-13T08:37:26.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.284+0000 7fe688c00640 1 -- 172.21.15.31:0/3656652784 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7fe674088150 con 0x7fe68c0d20c0 2024-09-13T08:37:26.285 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:26.285 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:26.286 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 msgr2=0x7fe66807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 0x7fe66807ad10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0017c0 tx=0x7fe67c007e90 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 msgr2=0x7fe68c0c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe674009780 tx=0x7fe674032960 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 shutdown_connections 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe668078850 0x7fe66807ad10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe68c0d44f0 0x7fe68c15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe68c0d20c0 0x7fe68c0c85b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 --2- 172.21.15.31:0/3656652784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe68c0cf2d0 0x7fe68c15e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 >> 172.21.15.31:0/3656652784 conn(0x7fe68c09d1c0 msgr2=0x7fe68c06ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 shutdown_connections 2024-09-13T08:37:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.287+0000 7fe691400640 1 -- 172.21.15.31:0/3656652784 wait complete. 2024-09-13T08:37:26.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:26.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 -- 172.21.15.31:0/1098156230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015bd70 msgr2=0x7fea700cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 --2- 172.21.15.31:0/1098156230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015bd70 0x7fea700cf850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fea60005fd0 tx=0x7fea6002a670 comp rx=0 tx=0).stop 2024-09-13T08:37:26.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 -- 172.21.15.31:0/1098156230 shutdown_connections 2024-09-13T08:37:26.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 --2- 172.21.15.31:0/1098156230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015bd70 0x7fea700cf850 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 -- 172.21.15.31:0/1098156230 >> 172.21.15.31:0/1098156230 conn(0x7fea700cc910 msgr2=0x7fea700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 -- 172.21.15.31:0/1098156230 shutdown_connections 2024-09-13T08:37:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.461+0000 7fea77200640 1 -- 172.21.15.31:0/1098156230 wait complete. 2024-09-13T08:37:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.463+0000 7fea77200640 1 Processor -- start 2024-09-13T08:37:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.463+0000 7fea77200640 1 -- start start 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7015bd70 0x7fea700b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 0x7fea701f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fea700cec50 con 0x7fea7015f9e0 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea700cead0 con 0x7fea7015bd70 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea77200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fea700cb850 con 0x7fea7015fe20 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43026/0 (socket says 172.21.15.31:43026) 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 -- 172.21.15.31:0/979254874 learned_addr learned my addr 172.21.15.31:0/979254874 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea6fe00640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7015bd70 0x7fea700b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea75200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 0x7fea701f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 -- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 msgr2=0x7fea701f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 0x7fea701f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 -- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7015bd70 msgr2=0x7fea700b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7015bd70 0x7fea700b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea67400640 1 -- 172.21.15.31:0/979254874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea700cb5f0 con 0x7fea7015f9e0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.464+0000 7fea75200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 0x7fea701f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea67400640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea5800c450 tx=0x7fea5800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580052d0 con 0x7fea7015f9e0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea58005470 con 0x7fea7015f9e0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58012870 con 0x7fea7015f9e0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea700c3340 con 0x7fea7015f9e0 2024-09-13T08:37:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.465+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7006fe50 con 0x7fea7015f9e0 2024-09-13T08:37:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.466+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea701f5160 con 0x7fea7015f9e0 2024-09-13T08:37:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.467+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fea580035f0 con 0x7fea7015f9e0 2024-09-13T08:37:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.468+0000 7fea6cc00640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 0x7fea4c07ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.468+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fea5809a570 con 0x7fea7015f9e0 2024-09-13T08:37:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.468+0000 7fea6fe00640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 0x7fea4c07ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.469+0000 7fea6fe00640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 0x7fea4c07ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea60033000 tx=0x7fea60004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.472+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fea700c3340 con 0x7fea7015f9e0 2024-09-13T08:37:26.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.668+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea700cf850 con 0x7fea7015f9e0 2024-09-13T08:37:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.669+0000 7fea6cc00640 1 -- 172.21.15.31:0/979254874 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7fea58098e20 con 0x7fea7015f9e0 2024-09-13T08:37:26.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:26.670 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.671+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 msgr2=0x7fea4c07ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.671+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 0x7fea4c07ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea60033000 tx=0x7fea60004730 comp rx=0 tx=0).stop 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.671+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 msgr2=0x7fea700c2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.671+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea5800c450 tx=0x7fea5800c910 comp rx=0 tx=0).stop 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 shutdown_connections 2024-09-13T08:37:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fea4c078880 0x7fea4c07ad40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea7015fe20 0x7fea701f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7015f9e0 0x7fea700c2e00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 --2- 172.21.15.31:0/979254874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7015bd70 0x7fea700b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 >> 172.21.15.31:0/979254874 conn(0x7fea700cc910 msgr2=0x7fea7015e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 shutdown_connections 2024-09-13T08:37:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.672+0000 7fea77200640 1 -- 172.21.15.31:0/979254874 wait complete. 2024-09-13T08:37:26.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 -- 172.21.15.31:0/1853213968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 msgr2=0x7fe45c0d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 --2- 172.21.15.31:0/1853213968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0d2500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009830 tx=0x7fe44c02a510 comp rx=0 tx=0).stop 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 -- 172.21.15.31:0/1853213968 shutdown_connections 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 --2- 172.21.15.31:0/1853213968 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0d42e0 0x7fe45c0b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 --2- 172.21.15.31:0/1853213968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0d2500 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.847+0000 7fe461000640 1 -- 172.21.15.31:0/1853213968 >> 172.21.15.31:0/1853213968 conn(0x7fe45c08d8b0 msgr2=0x7fe45c09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.848+0000 7fe461000640 1 -- 172.21.15.31:0/1853213968 shutdown_connections 2024-09-13T08:37:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.848+0000 7fe461000640 1 -- 172.21.15.31:0/1853213968 wait complete. 2024-09-13T08:37:26.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.849+0000 7fe461000640 1 Processor -- start 2024-09-13T08:37:26.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.849+0000 7fe461000640 1 -- start start 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0b47f0 0x7fe45c162320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0d42e0 0x7fe45c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c0b1910 con 0x7fe45c15dfa0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c0b1790 con 0x7fe45c0d42e0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe461000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe45c0b1a90 con 0x7fe45c0b47f0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe45be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe45be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43046/0 (socket says 172.21.15.31:43046) 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe45be00640 1 -- 172.21.15.31:0/2261077304 learned_addr learned my addr 172.21.15.31:0/2261077304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.850+0000 7fe45b400640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0b47f0 0x7fe45c162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45aa00640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0d42e0 0x7fe45c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 -- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0b47f0 msgr2=0x7fe45c162320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0b47f0 0x7fe45c162320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 -- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0d42e0 msgr2=0x7fe45c15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:26.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0d42e0 0x7fe45c15d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:26.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 -- 172.21.15.31:0/2261077304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c15f6c0 con 0x7fe45c15dfa0 2024-09-13T08:37:26.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe45be00640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe45400cce0 tx=0x7fe45400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45400f450 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe45400f700 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45c15f930 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.851+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45400fb50 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.852+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45c15fef0 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.853+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45c0d4cf0 con 0x7fe45c15dfa0 2024-09-13T08:37:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.854+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe45400fcf0 con 0x7fe45c15dfa0 2024-09-13T08:37:26.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.854+0000 7fe453400640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 0x7fe43c07ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:26.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.855+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe45409b460 con 0x7fe45c15dfa0 2024-09-13T08:37:26.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.855+0000 7fe45b400640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 0x7fe43c07ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:26.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.855+0000 7fe45b400640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 0x7fe43c07ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe44c034000 tx=0x7fe44c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:26.859+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe454014030 con 0x7fe45c15dfa0 2024-09-13T08:37:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.092+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe45c076040 con 0x7fe45c15dfa0 2024-09-13T08:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.092+0000 7fe453400640 1 -- 172.21.15.31:0/2261077304 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7fe454063350 con 0x7fe45c15dfa0 2024-09-13T08:37:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:27.094 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 msgr2=0x7fe43c07ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 0x7fe43c07ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe44c034000 tx=0x7fe44c004730 comp rx=0 tx=0).stop 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 msgr2=0x7fe45c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe45400cce0 tx=0x7fe45400a620 comp rx=0 tx=0).stop 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 shutdown_connections 2024-09-13T08:37:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe45c15dfa0 0x7fe45c0b8af0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45c0d42e0 0x7fe45c15d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe43c078880 0x7fe43c07ad40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.094+0000 7fe461000640 1 --2- 172.21.15.31:0/2261077304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe45c0b47f0 0x7fe45c162320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.095+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 >> 172.21.15.31:0/2261077304 conn(0x7fe45c08d8b0 msgr2=0x7fe45c0b79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.095+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 shutdown_connections 2024-09-13T08:37:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.095+0000 7fe461000640 1 -- 172.21.15.31:0/2261077304 wait complete. 2024-09-13T08:37:27.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:37:09.446277+0000', 'modified': '2024-09-13T08:37:12.461695+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4366}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4366': {'gid': 4366, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6837/2663745938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6835', 'nonce': 2663745938}, {'type': 'v1', 'addr': '172.21.15.45:6837', 'nonce': 2663745938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4366, 'qdb_cluster': [4366]} 2024-09-13T08:37:27.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:37:27.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T08:37:27.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.676+0000 7fb0ed200640 1 -- 172.21.15.31:0/403979255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0cc003a50 msgr2=0x7fb0cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.676+0000 7fb0ed200640 1 --2- 172.21.15.31:0/403979255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0cc003a50 0x7fb0cc003e70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009b50 tx=0x7fb0cc005830 comp rx=0 tx=0).stop 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.676+0000 7fb0ed200640 1 -- 172.21.15.31:0/403979255 shutdown_connections 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.676+0000 7fb0ed200640 1 --2- 172.21.15.31:0/403979255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0cc003a50 0x7fb0cc003e70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.676+0000 7fb0ed200640 1 -- 172.21.15.31:0/403979255 >> 172.21.15.31:0/403979255 conn(0x7fb0e80b2000 msgr2=0x7fb0e80b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.677+0000 7fb0ed200640 1 -- 172.21.15.31:0/403979255 shutdown_connections 2024-09-13T08:37:27.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.677+0000 7fb0ed200640 1 -- 172.21.15.31:0/403979255 wait complete. 2024-09-13T08:37:27.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.678+0000 7fb0ed200640 1 Processor -- start 2024-09-13T08:37:27.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.678+0000 7fb0ed200640 1 -- start start 2024-09-13T08:37:27.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0cc003a50 0x7fb0e81e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb0e81f8050 0x7fb0e81f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e815e6c0 con 0x7fb0e81f5b70 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e8068f40 con 0x7fb0cc003a50 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.679+0000 7fb0ed200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e815e840 con 0x7fb0e81f8050 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0cc003a50 0x7fb0e81e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43066/0 (socket says 172.21.15.31:43066) 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e6a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb0e81f8050 0x7fb0e81f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 -- 172.21.15.31:0/754764054 learned_addr learned my addr 172.21.15.31:0/754764054 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 -- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb0e81f8050 msgr2=0x7fb0e81f8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb0e81f8050 0x7fb0e81f8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 -- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0cc003a50 msgr2=0x7fb0e81e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0cc003a50 0x7fb0e81e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 -- 172.21.15.31:0/754764054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e81e3940 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.680+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80cf890 tx=0x7fb0e000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.681+0000 7fb0df400640 1 -- 172.21.15.31:0/754764054 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000ca10 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.681+0000 7fb0df400640 1 -- 172.21.15.31:0/754764054 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0e0017070 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.681+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e81e3c20 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.681+0000 7fb0df400640 1 -- 172.21.15.31:0/754764054 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0011460 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.681+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e81e41e0 con 0x7fb0e81f5b70 2024-09-13T08:37:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.683+0000 7fb0df400640 1 -- 172.21.15.31:0/754764054 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fb0e0011600 con 0x7fb0e81f5b70 2024-09-13T08:37:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.685+0000 7fb0df400640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 0x7fb0c807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.685+0000 7fb0df400640 1 -- 172.21.15.31:0/754764054 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fb0e009ac20 con 0x7fb0e81f5b70 2024-09-13T08:37:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.685+0000 7fb0f1861740 1 Processor -- start 2024-09-13T08:37:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.685+0000 7fb0f1861740 1 -- start start 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.685+0000 7fb0e7400640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 0x7fb0c807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c882e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c881ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56363c7c0430 con 0x56363c83dc10 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56363c7b05d0 con 0x56363c83e190 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x56363c79fa00 con 0x56363c882aa0 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e7400640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 0x7fb0c807ad40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8035000 tx=0x7fb0d80046c0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43070/0 (socket says 172.21.15.31:43070) 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 -- 172.21.15.31:0/111870178 learned_addr learned my addr 172.21.15.31:0/111870178 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e7400640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c882e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c881ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 -- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 msgr2=0x56363c882e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c882e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 -- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 msgr2=0x56363c881ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c881ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e6a00640 1 -- 172.21.15.31:0/111870178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56363c768520 con 0x56363c83dc10 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.686+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c881ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0e815ec00 tx=0x7fb0d4006670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826e330c17fc788 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0d3e00640 1 -- 172.21.15.31:0/111870178 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011770 con 0x56363c83dc10 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0d3e00640 1 -- 172.21.15.31:0/111870178 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0d4016070 con 0x56363c83dc10 2024-09-13T08:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0d3e00640 1 -- 172.21.15.31:0/111870178 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011c60 con 0x56363c83dc10 2024-09-13T08:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0f1861740 1 -- 172.21.15.31:0/111870178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 msgr2=0x56363c881580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.687+0000 7fb0f1861740 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0e815ec00 tx=0x7fb0d4006670 comp rx=0 tx=0).stop 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 -- 172.21.15.31:0/111870178 shutdown_connections 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c881ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83dc10 0x56363c881580 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 --2- 172.21.15.31:0/111870178 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c882e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 -- 172.21.15.31:0/111870178 >> 172.21.15.31:0/111870178 conn(0x56363c8391e0 msgr2=0x56363c83cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 -- 172.21.15.31:0/111870178 shutdown_connections 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 -- 172.21.15.31:0/111870178 wait complete. 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.688+0000 7fb0f1861740 1 -- start start 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.689+0000 7fb0f1861740 1 Processor -- start 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.689+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c851930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.689+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c8544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0f1861740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56363c746330 con 0x56363c8549f0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56363c7a3810 con 0x56363c83e190 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x56363c510620 con 0x56363c882aa0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0f1861740 10 client.0 _ll_register_callbacks cb 0x56363c74b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43080/0 (socket says 172.21.15.31:43080) 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 -- 172.21.15.31:0/2868291907 learned_addr learned my addr 172.21.15.31:0/2868291907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7400640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c851930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c8544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 -- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 msgr2=0x56363c8544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c8544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 -- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 msgr2=0x56363c851930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c851930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.690+0000 7fb0e7e00640 1 -- 172.21.15.31:0/2868291907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56363c501850 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0098ff0 tx=0x7fb0e00625b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84cac30824961c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0062000 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fb0e005e4f0 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d4016070 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fb0e007c030 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 10 client.4677 fetch_fsmap learned FSMap version 8 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x56363c5786b0 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0d4016070 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fb0e005e6f0 con 0x56363c8549f0 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 10 client.4677 fetch_fsmap finished waiting for FSMap version 8 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 10 client.4677 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T08:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 20 client.4677 populate_metadata read hostname 'smithi031' 2024-09-13T08:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c8524c0 0x56363c881600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 4 client.4677 mds_command: new command op to 4366 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.691+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 --> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56363c7b2520 con 0x56363c8524c0 2024-09-13T08:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.692+0000 7fb0e7400640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c8524c0 0x56363c881600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.692+0000 7fb0e7400640 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c8524c0 0x56363c881600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd60ed38976c310c server_cookie=fefe6831e4df8c26 in_seq=0 out_seq=0 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.692+0000 7fb0d0c00640 10 client.4677 ms_handle_connect on v2:172.21.15.45:6835/2663745938 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.693+0000 7fb0d0c00640 1 -- 172.21.15.31:0/2868291907 <== mds.0 v2:172.21.15.45:6835/2663745938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56363c7b2520 con 0x56363c8524c0 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.693+0000 7fb0d0c00640 10 client.4677 handle_command_reply: tid=0 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 client.4677 shutdown 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 msgr2=0x56363c83c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0098ff0 tx=0x7fb0e00625b0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 shutdown_connections 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c8524c0 0x56363c881600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c83c860 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c8544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 --2- 172.21.15.31:0/2868291907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c851930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0d0c00640 0 client.4677 ms_handle_reset on v2:172.21.15.45:6835/2663745938 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 >> 172.21.15.31:0/2868291907 conn(0x56363c83dc10 msgr2=0x56363c87a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 shutdown_connections 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 1 -- 172.21.15.31:0/2868291907 wait complete. 2024-09-13T08:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.694+0000 7fb0f1861740 20 client.4677 trim_cache size 0 max 16384 2024-09-13T08:37:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.713+0000 7fb0f1861740 1 Processor -- start 2024-09-13T08:37:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.713+0000 7fb0f1861740 1 -- start start 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 0x56363c837420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 0x56363c87a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e8200770 con 0x56363c83e190 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81f5370 con 0x56363c882aa0 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e82003e0 con 0x56363c8549f0 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43092/0 (socket says 172.21.15.31:43092) 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 -- 172.21.15.31:0/4208352347 learned_addr learned my addr 172.21.15.31:0/4208352347 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 0x56363c87a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 -- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 msgr2=0x56363c837420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 0x56363c837420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 0x56363c837420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 -- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 msgr2=0x56363c87a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 0x56363c87a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7400640 1 -- 172.21.15.31:0/4208352347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56363c8638f0 con 0x56363c83e190 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.714+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 0x56363c87a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 0x56363c837420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0e7400640 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009b50 tx=0x7fb0d803b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad3343c54ca37c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0d3400640 1 -- 172.21.15.31:0/4208352347 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8056070 con 0x56363c83e190 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0d3400640 1 -- 172.21.15.31:0/4208352347 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0d8042040 con 0x56363c83e190 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0d3400640 1 -- 172.21.15.31:0/4208352347 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8038040 con 0x56363c83e190 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0f1861740 1 -- 172.21.15.31:0/4208352347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 msgr2=0x56363c836ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.715+0000 7fb0f1861740 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009b50 tx=0x7fb0d803b000 comp rx=0 tx=0).stop 2024-09-13T08:37:27.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 -- 172.21.15.31:0/4208352347 shutdown_connections 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c882aa0 0x56363c87a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c8549f0 0x56363c837420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 --2- 172.21.15.31:0/4208352347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c83e190 0x56363c836ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 -- 172.21.15.31:0/4208352347 >> 172.21.15.31:0/4208352347 conn(0x56363c83dc10 msgr2=0x56363c837d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 -- 172.21.15.31:0/4208352347 shutdown_connections 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 -- 172.21.15.31:0/4208352347 wait complete. 2024-09-13T08:37:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 -- start start 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.716+0000 7fb0f1861740 1 Processor -- start 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c905bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c881d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56363c852340 con 0x56363c8549f0 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56363c8530c0 con 0x56363c83e190 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x56363c852990 con 0x56363c882aa0 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.717+0000 7fb0f1861740 10 client.0 _ll_register_callbacks cb 0x7fb0e0062990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:37:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43102/0 (socket says 172.21.15.31:43102) 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 -- 172.21.15.31:0/3945227256 learned_addr learned my addr 172.21.15.31:0/3945227256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e7400640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c905bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c881d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 -- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 msgr2=0x56363c881d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c881d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 -- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 msgr2=0x56363c905bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c905bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 -- 172.21.15.31:0/3945227256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56363c88ffb0 con 0x56363c8549f0 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e7e00640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c881d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:27.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0e6a00640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4004e60 tx=0x7fb0d4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aefda772620babd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011c00 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.718+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fb0d4011da0 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d8042040 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fb0d4012440 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 10 client.4683 fetch_fsmap learned FSMap version 8 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x56363c881700 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0d8042040 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fb0d4033020 con 0x56363c8549f0 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 10 client.4683 fetch_fsmap finished waiting for FSMap version 8 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 10 client.4683 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 20 client.4683 populate_metadata read hostname 'smithi031' 2024-09-13T08:37:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c838040 0x56363c8532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 4 client.4683 mds_command: new command op to 4366 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 --> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] -- command(tid 0: {"prefix": "status"}) -- 0x56363c7adfd0 con 0x56363c838040 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.719+0000 7fb0e7400640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c838040 0x56363c8532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0e7400640 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c838040 0x56363c8532d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ce2626a31e1523 server_cookie=e404d73ee38eef97 in_seq=0 out_seq=0 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0d0c00640 10 client.4683 ms_handle_connect on v2:172.21.15.45:6835/2663745938 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0d0c00640 1 -- 172.21.15.31:0/3945227256 <== mds.0 v2:172.21.15.45:6835/2663745938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x56363c7adfd0 con 0x56363c838040 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0d0c00640 10 client.4683 handle_command_reply: tid=0 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0f1861740 1 client.4683 shutdown 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 msgr2=0x56363c8f8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.720+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4004e60 tx=0x7fb0d4006000 comp rx=0 tx=0).stop 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 shutdown_connections 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x56363c882aa0 0x56363c881d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:6835/2663745938,v1:172.21.15.45:6837/2663745938] conn(0x56363c838040 0x56363c8532d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56363c8549f0 0x56363c8f8d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 --2- 172.21.15.31:0/3945227256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56363c83e190 0x56363c905bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0d0c00640 0 client.4683 ms_handle_reset on v2:172.21.15.45:6835/2663745938 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 >> 172.21.15.31:0/3945227256 conn(0x56363c83dc10 msgr2=0x56363c8516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 shutdown_connections 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 1 -- 172.21.15.31:0/3945227256 wait complete. 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0f1861740 20 client.4683 trim_cache size 0 max 16384 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:37:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4366, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 17.712192206000001, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 8, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 23, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 23, 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 23.741793156 2024-09-13T08:37:27.728 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:37:27.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 msgr2=0x7fb0c807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 0x7fb0c807ad40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8035000 tx=0x7fb0d80046c0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 msgr2=0x7fb0e81f5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:27.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80cf890 tx=0x7fb0e000eb60 comp rx=0 tx=0).stop 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 shutdown_connections 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e81f5b70 0x7fb0e81f5ff0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb0c8078880 0x7fb0c807ad40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb0e81f8050 0x7fb0e81f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 --2- 172.21.15.31:0/754764054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0cc003a50 0x7fb0e81e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.721+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 >> 172.21.15.31:0/754764054 conn(0x7fb0e80b2000 msgr2=0x7fb0e806ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.722+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 shutdown_connections 2024-09-13T08:37:27.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:27.722+0000 7fb0ed200640 1 -- 172.21.15.31:0/754764054 wait complete. 2024-09-13T08:37:27.737 INFO:teuthology.run_tasks:Running task kclient... 2024-09-13T08:37:27.766 INFO:tasks.kclient:Mounting kernel clients... 2024-09-13T08:37:27.766 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-13T08:37:27.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:27.766 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:37:27.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:27.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-13T08:37:27.902 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-09-13T08:37:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:37:27.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:27.906 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:37:27.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:27.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-13T08:37:28.040 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-09-13T08:37:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:37:28.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:28.043 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:37:28.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:28.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-13T08:37:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-09-13T08:37:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:37:28.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:28.190 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:37:28.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:28.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-13T08:37:28.326 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-09-13T08:37:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:37:28.329 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-09-13T08:37:28.329 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T08:37:28.329 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:37:28.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:37:28.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 -- 172.21.15.31:0/3153528796 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 msgr2=0x7f0b1415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/3153528796 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b1415fe20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0afc005fd0 tx=0x7f0afc02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 -- 172.21.15.31:0/3153528796 shutdown_connections 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/3153528796 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b1415fe20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/3153528796 >> v2:172.21.15.45:3300/0 conn(0x7f0b1415bd70 0x7f0b140cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 -- 172.21.15.31:0/3153528796 >> 172.21.15.31:0/3153528796 conn(0x7f0b140cc910 msgr2=0x7f0b140cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 -- 172.21.15.31:0/3153528796 shutdown_connections 2024-09-13T08:37:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.489+0000 7f0b1bc00640 1 -- 172.21.15.31:0/3153528796 wait complete. 2024-09-13T08:37:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.491+0000 7f0b1bc00640 1 Processor -- start 2024-09-13T08:37:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.491+0000 7f0b1bc00640 1 -- start start 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b140ce3c0 0x7f0b140c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b14156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b140cb9d0 con 0x7f0b1415bd70 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b140cb850 con 0x7f0b140ce3c0 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b1bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0b140cc360 con 0x7f0b1415f9e0 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43108/0 (socket says 172.21.15.31:43108) 2024-09-13T08:37:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b0be00640 1 -- 172.21.15.31:0/852211687 learned_addr learned my addr 172.21.15.31:0/852211687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:28.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b19200640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b140ce3c0 0x7f0b140c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:28.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.492+0000 7f0b19c00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b14156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 -- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 msgr2=0x7f0b14156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b14156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 -- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b140ce3c0 msgr2=0x7f0b140c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b140ce3c0 0x7f0b140c4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 -- 172.21.15.31:0/852211687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b141f5c00 con 0x7f0b1415bd70 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b0be00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0afc034040 tx=0x7f0afc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc03e030 con 0x7f0b1415bd70 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0afc003ae0 con 0x7f0b1415bd70 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b141f5ea0 con 0x7f0b1415bd70 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc035040 con 0x7f0b1415bd70 2024-09-13T08:37:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.493+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b141f6460 con 0x7f0b1415bd70 2024-09-13T08:37:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.494+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b140cf850 con 0x7f0b1415bd70 2024-09-13T08:37:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.496+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f0afc041030 con 0x7f0b1415bd70 2024-09-13T08:37:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.496+0000 7f0b11600640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 0x7f0af007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:28.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.497+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f0afc0bb140 con 0x7f0b1415bd70 2024-09-13T08:37:28.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.497+0000 7f0b19200640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 0x7f0af007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:28.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.498+0000 7f0b19200640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 0x7f0af007ad10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009c50 tx=0x7f0b04005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:28.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.501+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0afc083040 con 0x7f0b1415bd70 2024-09-13T08:37:28.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.735+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0b14072810 con 0x7f0b1415bd70 2024-09-13T08:37:28.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.735+0000 7f0b11600640 1 -- 172.21.15.31:0/852211687 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f0afc087a70 con 0x7f0b1415bd70 2024-09-13T08:37:28.736 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 msgr2=0x7f0af007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 0x7f0af007ad10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009c50 tx=0x7f0b04005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 msgr2=0x7f0b140b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0afc034040 tx=0x7f0afc0035f0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 shutdown_connections 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0af0078850 0x7f0af007ad10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b1415f9e0 0x7f0b14156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1415bd70 0x7f0b140b45a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 --2- 172.21.15.31:0/852211687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b140ce3c0 0x7f0b140c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 >> 172.21.15.31:0/852211687 conn(0x7f0b140cc910 msgr2=0x7f0b140ad3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 shutdown_connections 2024-09-13T08:37:28.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28.738+0000 7f0b1bc00640 1 -- 172.21.15.31:0/852211687 wait complete. 2024-09-13T08:37:28.754 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:37:28.755 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:37:28.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:28.755 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:37:28.791 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:37:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42683sec preferred_lft 42683sec 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:37:28.793 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T08:37:28.794 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:28.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:29.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:29.203 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:37:29.259 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:37:29.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:29.262 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:37:29.319 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:37:29.319 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:37:29.319 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:37:29.319 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:37:29.319 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:37:29.320 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:29.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:29.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:29.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:29.930 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:37:29.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:29.967 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-09-13T08:37:30.043 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:30.043 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:30.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:37:30.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-13T08:37:30.043 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:30.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:30.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:30.222 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-13T08:37:30.222 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:30.222 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-13T08:37:30.223 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:30.470 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:30.470 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:30.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:37:30.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T08:37:30.470 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:30.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:30.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:30.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:30.638 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:37:30.673 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:37:30.674 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:30.674 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:37:30.749 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:37:30.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:30.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:30.890 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:30.891 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:31.513 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:31.513 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:31.513 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:37:31.513 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:37:31.514 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:37:31.515 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:37:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:37:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:37:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:37:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:37:31.519 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:37:31.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:31.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:37:31.576 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:31.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:31.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:31.678 INFO:teuthology.orchestra.run.smithi031.stdout:client4660 2024-09-13T08:37:31.678 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:31.678 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:31.714 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:31.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:31.853 INFO:teuthology.orchestra.run.smithi031.stdout:client4660 2024-09-13T08:37:31.853 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:31.951 INFO:teuthology.orchestra.run.smithi031.stdout:client4660 2024-09-13T08:37:31.951 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0251b593-a02a-47fa-b938-c065b7284822.client4660/status') 2024-09-13T08:37:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:32.059 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.4660 (3)172.21.15.31:0/413681840 2024-09-13T08:37:32.059 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-09-13T08:37:32.059 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-09-13T08:37:32.059 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-09-13T08:37:32.059 INFO:teuthology.orchestra.run.smithi031.stderr:69 bytes copied, 5.6532e-05 s, 1.2 MB/s 2024-09-13T08:37:32.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:32.069 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-09-13T08:37:32.069 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T08:37:32.069 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T08:37:32.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 -- 172.21.15.31:0/2091095083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00b52d0 msgr2=0x7f4ee0152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/2091095083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00b52d0 0x7f4ee0152680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8005fd0 tx=0x7f4ed802aaa0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 -- 172.21.15.31:0/2091095083 shutdown_connections 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/2091095083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00b52d0 0x7f4ee0152680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/2091095083 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00cc720 0x7f4ee006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 -- 172.21.15.31:0/2091095083 >> 172.21.15.31:0/2091095083 conn(0x7f4ee006cb00 msgr2=0x7f4ee006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 -- 172.21.15.31:0/2091095083 shutdown_connections 2024-09-13T08:37:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.229+0000 7f4ee4c00640 1 -- 172.21.15.31:0/2091095083 wait complete. 2024-09-13T08:37:32.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.230+0000 7f4ee4c00640 1 Processor -- start 2024-09-13T08:37:32.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.230+0000 7f4ee4c00640 1 -- start start 2024-09-13T08:37:32.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 0x7f4ee00c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:32.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 0x7f4ee015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00ccb00 0x7f4ee015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee0152bc0 con 0x7f4ee00cc720 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee00704f0 con 0x7f4ee00b52d0 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.231+0000 7f4ee4c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee0152d40 con 0x7f4ee00ccb00 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 0x7f4ee015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 0x7f4ee00c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 0x7f4ee00c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:33620/0 (socket says 172.21.15.31:33620) 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edf400640 1 -- 172.21.15.31:0/1425604413 learned_addr learned my addr 172.21.15.31:0/1425604413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edfe00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00ccb00 0x7f4ee015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 -- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00ccb00 msgr2=0x7f4ee015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00ccb00 0x7f4ee015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 -- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 msgr2=0x7f4ee00c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 0x7f4ee00c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 -- 172.21.15.31:0/1425604413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee015ed90 con 0x7f4ee00cc720 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.232+0000 7f4edea00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 0x7f4ee015dca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00c3c0 tx=0x7f4ecc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.233+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc017070 con 0x7f4ee00cc720 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.233+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ecc0053e0 con 0x7f4ee00cc720 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.233+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc003850 con 0x7f4ee00cc720 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.233+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee0143ff0 con 0x7f4ee00cc720 2024-09-13T08:37:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.233+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee015eab0 con 0x7f4ee00cc720 2024-09-13T08:37:32.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.234+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee0153230 con 0x7f4ee00cc720 2024-09-13T08:37:32.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.235+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4ecc01a070 con 0x7f4ee00cc720 2024-09-13T08:37:32.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.236+0000 7f4ed7400640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 0x7f4ec007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:32.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.237+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f4ecc099b00 con 0x7f4ee00cc720 2024-09-13T08:37:32.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.237+0000 7f4edf400640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 0x7f4ec007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:32.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.238+0000 7f4edf400640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 0x7f4ec007ad10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000bca0 tx=0x7f4ed000b410 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:32.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.241+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4ecc061b10 con 0x7f4ee00cc720 2024-09-13T08:37:32.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.478+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4ee00b3eb0 con 0x7f4ee00cc720 2024-09-13T08:37:32.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.478+0000 7f4ed7400640 1 -- 172.21.15.31:0/1425604413 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f4ecc066540 con 0x7f4ee00cc720 2024-09-13T08:37:32.479 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:37:32.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.480+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 msgr2=0x7f4ec007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:32.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.480+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 0x7f4ec007ad10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000bca0 tx=0x7f4ed000b410 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.480+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 msgr2=0x7f4ee015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.480+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 0x7f4ee015dca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00c3c0 tx=0x7f4ecc00c880 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 shutdown_connections 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ec0078850 0x7f4ec007ad10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ee00ccb00 0x7f4ee015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ee00cc720 0x7f4ee015dca0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 --2- 172.21.15.31:0/1425604413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00b52d0 0x7f4ee00c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 >> 172.21.15.31:0/1425604413 conn(0x7f4ee006cb00 msgr2=0x7f4ee006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 shutdown_connections 2024-09-13T08:37:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32.481+0000 7f4ee4c00640 1 -- 172.21.15.31:0/1425604413 wait complete. 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:37:32.497 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-13T08:37:32.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:32.498 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:32.534 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42679sec preferred_lft 42679sec 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:32.535 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:32.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:32.536 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:37:32.627 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:37:32.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:32.630 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:37:32.666 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:37:32.667 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:37:32.667 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:37:32.667 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:37:32.667 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:37:32.667 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:32.667 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:32.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:37:32.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:37:32.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:37:32.668 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:32.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:32.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:32.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:32.866 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:37:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:37:32.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:32.904 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-09-13T08:37:32.978 INFO:teuthology.orchestra.run.smithi031.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-13T08:37:32.978 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:32.979 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:32.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:37:32.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-13T08:37:32.979 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:33.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:33.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:33.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:33.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T08:37:33.212 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 02:fd:12:26:ba:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::fd:12ff:fe26:ba09/64 scope link 2024-09-13T08:37:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:33.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 02:fd:12:26:ba:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::fd:12ff:fe26:ba09/64 scope link 2024-09-13T08:37:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:33.274 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-13T08:37:33.274 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:33.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:33.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:33.493 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:33.493 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:37:33.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.1 up 2024-09-13T08:37:33.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T08:37:33.493 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:37:33.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:33.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:33.659 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:33.659 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-13T08:37:33.695 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-13T08:37:33.695 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:33.696 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-13T08:37:33.770 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-09-13T08:37:33.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:33.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:33.899 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:33.899 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:33.927 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:33.927 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:33.928 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:37:33.928 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 2@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:37:33.928 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:37:33.929 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-09-13T08:37:33.930 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:37:33.930 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:37:33.930 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:37:33.930 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:37:33.932 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:37:33.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:33.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-13T08:37:33.988 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:34.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:34.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:34.082 INFO:teuthology.orchestra.run.smithi031.stdout:client4695 2024-09-13T08:37:34.082 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:34.082 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:34.118 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:34.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:34.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:34.248 INFO:teuthology.orchestra.run.smithi031.stdout:client4695 2024-09-13T08:37:34.248 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:34.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:34.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:34.338 INFO:teuthology.orchestra.run.smithi031.stdout:client4695 2024-09-13T08:37:34.339 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0251b593-a02a-47fa-b938-c065b7284822.client4695/status') 2024-09-13T08:37:34.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:34.439 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.4695 (3)192.168.0.2:0/1188873960 2024-09-13T08:37:34.439 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-09-13T08:37:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-09-13T08:37:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-09-13T08:37:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:69 bytes copied, 5.5565e-05 s, 1.2 MB/s 2024-09-13T08:37:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:34.450 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-09-13T08:37:34.450 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T08:37:34.450 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:37:34.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:37:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 -- 172.21.15.31:0/964294398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 msgr2=0x7f53e00cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 --2- 172.21.15.31:0/964294398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e00cbc50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53e009e090 tx=0x7f53d802a530 comp rx=0 tx=0).stop 2024-09-13T08:37:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 -- 172.21.15.31:0/964294398 shutdown_connections 2024-09-13T08:37:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 --2- 172.21.15.31:0/964294398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e00cbc50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 --2- 172.21.15.31:0/964294398 >> v2:172.21.15.45:3300/0 conn(0x7f53e015bd70 0x7f53e00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 -- 172.21.15.31:0/964294398 >> 172.21.15.31:0/964294398 conn(0x7f53e00cc910 msgr2=0x7f53e00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 -- 172.21.15.31:0/964294398 shutdown_connections 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.610+0000 7f53e8600640 1 -- 172.21.15.31:0/964294398 wait complete. 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.612+0000 7f53e8600640 1 Processor -- start 2024-09-13T08:37:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.612+0000 7f53e8600640 1 -- start start 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53e015bd70 0x7f53e006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 0x7f53e006fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53e00cec50 con 0x7f53e015fe20 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53e00cead0 con 0x7f53e015f9e0 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e8600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f53e00cb850 con 0x7f53e015bd70 2024-09-13T08:37:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43142/0 (socket says 172.21.15.31:43142) 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 -- 172.21.15.31:0/53342721 learned_addr learned my addr 172.21.15.31:0/53342721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e5200640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 0x7f53e006fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e5c00640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53e015bd70 0x7f53e006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 -- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53e015bd70 msgr2=0x7f53e006ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53e015bd70 0x7f53e006ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 -- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 msgr2=0x7f53e006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 0x7f53e006fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e6600640 1 -- 172.21.15.31:0/53342721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e01f7da0 con 0x7f53e015fe20 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.613+0000 7f53e5200640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 0x7f53e006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53e6600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009920 tx=0x7f53d8033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8031b50 con 0x7f53e015fe20 2024-09-13T08:37:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53d8031e00 con 0x7f53e015fe20 2024-09-13T08:37:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8005280 con 0x7f53e015fe20 2024-09-13T08:37:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e01f8040 con 0x7f53e015fe20 2024-09-13T08:37:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.614+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e01f8600 con 0x7f53e015fe20 2024-09-13T08:37:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.615+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e00cf850 con 0x7f53e015fe20 2024-09-13T08:37:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.616+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f53d8005420 con 0x7f53e015fe20 2024-09-13T08:37:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.617+0000 7f53dd600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 0x7f53c007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.617+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f53d80bba30 con 0x7f53e015fe20 2024-09-13T08:37:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.618+0000 7f53e5c00640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 0x7f53c007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.618+0000 7f53e5c00640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 0x7f53c007ad10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f53d4005fd0 tx=0x7f53d4009500 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:34.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.622+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f53d803f030 con 0x7f53e015fe20 2024-09-13T08:37:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.853+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f53e00b2d60 con 0x7f53e015fe20 2024-09-13T08:37:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.853+0000 7f53dd600640 1 -- 172.21.15.31:0/53342721 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f53d8084000 con 0x7f53e015fe20 2024-09-13T08:37:34.855 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:37:34.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 msgr2=0x7f53c007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 0x7f53c007ad10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f53d4005fd0 tx=0x7f53d4009500 comp rx=0 tx=0).stop 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 msgr2=0x7f53e009cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009920 tx=0x7f53d8033ba0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 shutdown_connections 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e015fe20 0x7f53e009cbe0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53e015f9e0 0x7f53e006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53c0078850 0x7f53c007ad10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 --2- 172.21.15.31:0/53342721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53e015bd70 0x7f53e006ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 >> 172.21.15.31:0/53342721 conn(0x7f53e00cc910 msgr2=0x7f53e00b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 shutdown_connections 2024-09-13T08:37:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:34.856+0000 7f53e8600640 1 -- 172.21.15.31:0/53342721 wait complete. 2024-09-13T08:37:34.872 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T08:37:34.872 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T08:37:34.873 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:37:34.874 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:37:34.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:34.874 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:37:34.913 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:37:34.914 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42681sec preferred_lft 42681sec 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:34.915 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:37:34.915 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T08:37:34.916 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:35.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:35.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:35.349 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:37:35.406 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:37:35.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:35.410 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:37:35.462 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:37:35.462 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:37:35.463 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:37:35.463 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:37:35.463 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:37:35.463 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:35.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:35.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:37:35.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:37:36.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:36.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:36.029 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:37:36.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:36.066 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-09-13T08:37:36.142 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:36.143 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:36.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:37:36.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-13T08:37:36.143 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:36.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:36.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:36.324 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-13T08:37:36.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-13T08:37:36.325 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:36.571 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:36.571 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:36.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:37:36.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T08:37:36.571 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:36.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:36.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:36.739 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:37:36.773 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:37:36.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:36.774 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:37:36.851 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:37:36.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:36.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:36.997 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:36.997 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:37.656 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:37.656 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:37.656 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:37:37.656 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:37:37.656 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:37:37.659 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:37:37.659 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:37:37.659 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:37:37.659 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:37:37.659 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:37:37.664 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:37:37.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:37.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:37:37.720 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:37.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:37.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:37.813 INFO:teuthology.orchestra.run.smithi045.stdout:client4571 2024-09-13T08:37:37.813 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:37.813 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:37.849 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:37.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:37.978 INFO:teuthology.orchestra.run.smithi045.stdout:client4571 2024-09-13T08:37:37.979 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:38.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:38.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:38.068 INFO:teuthology.orchestra.run.smithi045.stdout:client4571 2024-09-13T08:37:38.069 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0251b593-a02a-47fa-b938-c065b7284822.client4571/status') 2024-09-13T08:37:38.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:38.170 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4571 (3)192.168.0.1:0/3234463430 2024-09-13T08:37:38.170 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2024-09-13T08:37:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2024-09-13T08:37:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2024-09-13T08:37:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:69 bytes copied, 5.5819e-05 s, 1.2 MB/s 2024-09-13T08:37:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:38.181 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-09-13T08:37:38.181 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T08:37:38.181 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T08:37:38.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:37:38.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 -- 172.21.15.31:0/3328506945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015bd70 msgr2=0x7f86700cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:38.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 --2- 172.21.15.31:0/3328506945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015bd70 0x7f86700cf850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8660005fd0 tx=0x7f866002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 -- 172.21.15.31:0/3328506945 shutdown_connections 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 --2- 172.21.15.31:0/3328506945 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015f9e0 0x7f867015fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 --2- 172.21.15.31:0/3328506945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015bd70 0x7f86700cf850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 -- 172.21.15.31:0/3328506945 >> 172.21.15.31:0/3328506945 conn(0x7f86700cc910 msgr2=0x7f86700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 -- 172.21.15.31:0/3328506945 shutdown_connections 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.352+0000 7f8677a00640 1 -- 172.21.15.31:0/3328506945 wait complete. 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.354+0000 7f8677a00640 1 Processor -- start 2024-09-13T08:37:38.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.354+0000 7f8677a00640 1 -- start start 2024-09-13T08:37:38.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86700ce3c0 0x7f86701565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:38.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015bd70 0x7f8670155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86700cb9d0 con 0x7f867015f9e0 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f86700cb850 con 0x7f86700ce3c0 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8677a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f86700cc360 con 0x7f867015bd70 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8675a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8675000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86700ce3c0 0x7f86701565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8675a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57434/0 (socket says 172.21.15.31:57434) 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8667e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015bd70 0x7f8670155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.355+0000 7f8675a00640 1 -- 172.21.15.31:0/582077400 learned_addr learned my addr 172.21.15.31:0/582077400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 -- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015bd70 msgr2=0x7f8670155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015bd70 0x7f8670155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 -- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86700ce3c0 msgr2=0x7f86701565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86700ce3c0 0x7f86701565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 -- 172.21.15.31:0/582077400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867009d1f0 con 0x7f867015f9e0 2024-09-13T08:37:38.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8675a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f866800c7a0 tx=0x7f866800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:38.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8668003b70 con 0x7f867015f9e0 2024-09-13T08:37:38.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8668003d30 con 0x7f867015f9e0 2024-09-13T08:37:38.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866800aab0 con 0x7f867015f9e0 2024-09-13T08:37:38.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86700c6420 con 0x7f867015f9e0 2024-09-13T08:37:38.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.356+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867009c710 con 0x7f867015f9e0 2024-09-13T08:37:38.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.357+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86700cf850 con 0x7f867015f9e0 2024-09-13T08:37:38.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.358+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f866800ac50 con 0x7f867015f9e0 2024-09-13T08:37:38.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.359+0000 7f866d600640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 0x7f864c07ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:38.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.360+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f866809a660 con 0x7f867015f9e0 2024-09-13T08:37:38.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.360+0000 7f8675000640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 0x7f864c07ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:38.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.360+0000 7f8675000640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 0x7f864c07ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8660033000 tx=0x7f8660004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:38.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.363+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8668062670 con 0x7f867015f9e0 2024-09-13T08:37:38.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.595+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f86700cce70 con 0x7f867015f9e0 2024-09-13T08:37:38.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.596+0000 7f866d600640 1 -- 172.21.15.31:0/582077400 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f86680670a0 con 0x7f867015f9e0 2024-09-13T08:37:38.597 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:37:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 msgr2=0x7f864c07ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 0x7f864c07ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8660033000 tx=0x7f8660004730 comp rx=0 tx=0).stop 2024-09-13T08:37:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 msgr2=0x7f86700b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f866800c7a0 tx=0x7f866800cc60 comp rx=0 tx=0).stop 2024-09-13T08:37:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 shutdown_connections 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f867015f9e0 0x7f86700b2cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f864c078880 0x7f864c07ad40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f867015bd70 0x7f8670155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 --2- 172.21.15.31:0/582077400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86700ce3c0 0x7f86701565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 >> 172.21.15.31:0/582077400 conn(0x7f86700cc910 msgr2=0x7f86700ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 shutdown_connections 2024-09-13T08:37:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:38.598+0000 7f8677a00640 1 -- 172.21.15.31:0/582077400 wait complete. 2024-09-13T08:37:38.614 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T08:37:38.614 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:37:38.614 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:37:38.614 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-13T08:37:38.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T08:37:38.615 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T08:37:38.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T08:37:38.615 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:37:38.615 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-13T08:37:38.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:38.615 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:37:38.651 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:38.651 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:38.651 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:38.651 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.651 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:37:38.652 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42678sec preferred_lft 42678sec 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:37:38.653 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:38.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:38.654 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:37:38.746 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:37:38.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:38.749 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:37:38.785 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:37:38.840 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:37:38.840 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:37:38.840 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:37:38.840 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:37:38.840 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:38.840 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:38.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:37:38.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:37:38.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:37:38.841 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:38.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:38.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:38.998 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:37:39.034 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:37:39.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:39.034 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-09-13T08:37:39.108 INFO:teuthology.orchestra.run.smithi045.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-13T08:37:39.109 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:39.109 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:39.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:37:39.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-13T08:37:39.110 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:39.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:39.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:39.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 12:99:34:43:88:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T08:37:39.345 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.346 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::1099:34ff:fe43:88a6/64 scope link 2024-09-13T08:37:39.346 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:39.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T08:37:39.403 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:37:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:37:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:37:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 12:99:34:43:88:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::1099:34ff:fe43:88a6/64 scope link 2024-09-13T08:37:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:37:39.406 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-13T08:37:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-13T08:37:39.407 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:39.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:39.625 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:37:39.625 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:37:39.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.1 up 2024-09-13T08:37:39.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T08:37:39.625 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:37:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:39.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:39.792 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:39.792 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-13T08:37:39.827 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-13T08:37:39.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:37:39.828 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-13T08:37:39.902 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-09-13T08:37:39.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:39.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:40.031 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:40.031 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:40.058 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:37:40.058 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:37:40.058 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:37:40.058 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 3@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:37:40.058 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:37:40.060 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-09-13T08:37:40.060 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:37:40.060 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:37:40.060 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:37:40.060 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:37:40.063 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:37:40.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:40.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-13T08:37:40.120 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:40.210 INFO:teuthology.orchestra.run.smithi045.stdout:client4678 2024-09-13T08:37:40.211 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:40.211 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:40.246 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:40.375 INFO:teuthology.orchestra.run.smithi045.stdout:client4678 2024-09-13T08:37:40.375 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:37:40.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:37:40.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:40.464 INFO:teuthology.orchestra.run.smithi045.stdout:client4678 2024-09-13T08:37:40.464 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0251b593-a02a-47fa-b938-c065b7284822.client4678/status') 2024-09-13T08:37:40.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:37:40.563 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4678 (3)192.168.0.2:0/3352210229 2024-09-13T08:37:40.564 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2024-09-13T08:37:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2024-09-13T08:37:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2024-09-13T08:37:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:69 bytes copied, 7.5586e-05 s, 913 kB/s 2024-09-13T08:37:40.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:37:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:37:40.574 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-13T08:37:40.668 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate binary .meta file on legacy subvol root doesn't break the system 2024-09-13T08:37:40.669 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:37:40.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:40.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 -- 172.21.15.31:0/1114228698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46d4003a00 msgr2=0x7f46d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:40.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 --2- 172.21.15.31:0/1114228698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46d4003a00 0x7f46d4003e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c2c0 tx=0x7f46d4005830 comp rx=0 tx=0).stop 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f2400640 1 -- 172.21.15.31:0/1114228698 reap_dead start 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 -- 172.21.15.31:0/1114228698 shutdown_connections 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 -- 172.21.15.31:0/1114228698 >> 172.21.15.31:0/1114228698 conn(0x7f46ec0cc910 msgr2=0x7f46ec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 -- 172.21.15.31:0/1114228698 shutdown_connections 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.841+0000 7f46f4400640 1 -- 172.21.15.31:0/1114228698 wait complete. 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.843+0000 7f46f4400640 1 Processor -- start 2024-09-13T08:37:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.843+0000 7f46f4400640 1 -- start start 2024-09-13T08:37:40.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4003a00 0x7f46ec150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:40.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:40.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 0x7f46ec1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:40.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1600f0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec0cb850 con 0x7f46d4003a00 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f4400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f46ec0c7fb0 con 0x7f46ec1f6700 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57460/0 (socket says 172.21.15.31:57460) 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f1000640 1 -- 172.21.15.31:0/3855139352 learned_addr learned my addr 172.21.15.31:0/3855139352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.844+0000 7f46f1a00640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4003a00 0x7f46ec150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f2400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 0x7f46ec1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 -- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 msgr2=0x7f46ec1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 0x7f46ec1503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:40.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 -- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4003a00 msgr2=0x7f46ec150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4003a00 0x7f46ec150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 -- 172.21.15.31:0/3855139352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ec09d4f0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f2400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 0x7f46ec1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f1000640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46d80118a0 tx=0x7f46d8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d800fb50 con 0x7f46ec0c3b60 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46d800fcf0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8019670 con 0x7f46ec0c3b60 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ec09d7a0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.845+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ec1f6fe0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.846+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ec14abf0 con 0x7f46ec0c3b60 2024-09-13T08:37:40.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.847+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f46d8019810 con 0x7f46ec0c3b60 2024-09-13T08:37:40.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.848+0000 7f46e9600640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 0x7f46cc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:40.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.848+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f46d809b350 con 0x7f46ec0c3b60 2024-09-13T08:37:40.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.849+0000 7f46f1a00640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 0x7f46cc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:40.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.850+0000 7f46f1a00640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 0x7f46cc07ad10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46e0033000 tx=0x7f46ec0cdea0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:40.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:40.853+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f46ec14abf0 con 0x7f46ec0c3b60 2024-09-13T08:37:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.051+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46ec0cc580 con 0x7f46ec0c3b60 2024-09-13T08:37:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.052+0000 7f46e9600640 1 -- 172.21.15.31:0/3855139352 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7f46d80639e0 con 0x7f46ec0c3b60 2024-09-13T08:37:41.053 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:41.053 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 msgr2=0x7f46cc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 0x7f46cc07ad10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46e0033000 tx=0x7f46ec0cdea0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 msgr2=0x7f46ec1f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46d80118a0 tx=0x7f46d8011d60 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 shutdown_connections 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f46cc078850 0x7f46cc07ad10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46ec1f6700 0x7f46ec1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ec0c3b60 0x7f46ec1f61c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 --2- 172.21.15.31:0/3855139352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4003a00 0x7f46ec150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.054+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 >> 172.21.15.31:0/3855139352 conn(0x7f46ec0cc910 msgr2=0x7f46ec15e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.055+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 shutdown_connections 2024-09-13T08:37:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.055+0000 7f46f4400640 1 -- 172.21.15.31:0/3855139352 wait complete. 2024-09-13T08:37:41.071 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T08:37:07.473068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T08:37:08.401008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:37:41.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:37:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 -- 172.21.15.31:0/4129809421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404015f3e0 msgr2=0x7f40400b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 --2- 172.21.15.31:0/4129809421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404015f3e0 0x7f40400b4aa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4040073540 tx=0x7f403802a820 comp rx=0 tx=0).stop 2024-09-13T08:37:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 -- 172.21.15.31:0/4129809421 shutdown_connections 2024-09-13T08:37:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 --2- 172.21.15.31:0/4129809421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404015f3e0 0x7f40400b4aa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 --2- 172.21.15.31:0/4129809421 >> v2:172.21.15.45:3300/0 conn(0x7f40400ce450 0x7f40400d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 -- 172.21.15.31:0/4129809421 >> 172.21.15.31:0/4129809421 conn(0x7f40400d1b70 msgr2=0x7f40400d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 -- 172.21.15.31:0/4129809421 shutdown_connections 2024-09-13T08:37:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.235+0000 7f4045c00640 1 -- 172.21.15.31:0/4129809421 wait complete. 2024-09-13T08:37:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.237+0000 7f4045c00640 1 Processor -- start 2024-09-13T08:37:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.237+0000 7f4045c00640 1 -- start start 2024-09-13T08:37:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 0x7f40401f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f404015f3e0 0x7f404009efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40400b5160 con 0x7f40400ce830 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40400b4fe0 con 0x7f40400ce450 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f40400b52e0 con 0x7f404015f3e0 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57476/0 (socket says 172.21.15.31:57476) 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 -- 172.21.15.31:0/292846630 learned_addr learned my addr 172.21.15.31:0/292846630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403fe00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 0x7f40401f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f4045200640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f404015f3e0 0x7f404009efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 -- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f404015f3e0 msgr2=0x7f404009efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f404015f3e0 0x7f404009efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 -- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 msgr2=0x7f40401f4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 0x7f40401f4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403f400640 1 -- 172.21.15.31:0/292846630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40400caa20 con 0x7f40400ce830 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.238+0000 7f403fe00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 0x7f40401f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f403f400640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c490 tx=0x7f402c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c016020 con 0x7f40400ce830 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f402c012940 con 0x7f40400ce830 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40400cad00 con 0x7f40400ce830 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c012d90 con 0x7f40400ce830 2024-09-13T08:37:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.239+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40400cb2c0 con 0x7f40400ce830 2024-09-13T08:37:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.240+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40400d2690 con 0x7f40400ce830 2024-09-13T08:37:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.241+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f402c019020 con 0x7f40400ce830 2024-09-13T08:37:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.242+0000 7f4037e00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 0x7f402007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.242+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f402c09a660 con 0x7f40400ce830 2024-09-13T08:37:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.243+0000 7f403fe00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 0x7f402007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.243+0000 7f403fe00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 0x7f402007ad10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4030009c20 tx=0x7f4030005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.247+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f402c020030 con 0x7f40400ce830 2024-09-13T08:37:41.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.439+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f404006ebe0 con 0x7f40400ce830 2024-09-13T08:37:41.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.439+0000 7f4037e00640 1 -- 172.21.15.31:0/292846630 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f402c0690a0 con 0x7f40400ce830 2024-09-13T08:37:41.441 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:37:41.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 msgr2=0x7f402007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 0x7f402007ad10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4030009c20 tx=0x7f4030005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:41.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 msgr2=0x7f40400cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c490 tx=0x7f402c00c950 comp rx=0 tx=0).stop 2024-09-13T08:37:41.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 shutdown_connections 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4020078850 0x7f402007ad10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f404015f3e0 0x7f404009efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40400ce830 0x7f40400cb7e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 --2- 172.21.15.31:0/292846630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40400ce450 0x7f40401f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 >> 172.21.15.31:0/292846630 conn(0x7f40400d1b70 msgr2=0x7f4040071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 shutdown_connections 2024-09-13T08:37:41.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.442+0000 7f4045c00640 1 -- 172.21.15.31:0/292846630 wait complete. 2024-09-13T08:37:41.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T08:37:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 -- 172.21.15.31:0/3682936192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7815bfc0 msgr2=0x7fbd78072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 --2- 172.21.15.31:0/3682936192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7815bfc0 0x7fbd78072560 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd68009830 tx=0x7fbd6802a510 comp rx=0 tx=0).stop 2024-09-13T08:37:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 -- 172.21.15.31:0/3682936192 shutdown_connections 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 --2- 172.21.15.31:0/3682936192 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7806fa60 0x7fbd780cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 --2- 172.21.15.31:0/3682936192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7815bfc0 0x7fbd78072560 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 -- 172.21.15.31:0/3682936192 >> 172.21.15.31:0/3682936192 conn(0x7fbd780cfbb0 msgr2=0x7fbd780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 -- 172.21.15.31:0/3682936192 shutdown_connections 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.621+0000 7fbd7d200640 1 -- 172.21.15.31:0/3682936192 wait complete. 2024-09-13T08:37:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.623+0000 7fbd7d200640 1 Processor -- start 2024-09-13T08:37:41.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.623+0000 7fbd7d200640 1 -- start start 2024-09-13T08:37:41.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7815bfc0 0x7fbd780b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd780c8160 0x7fbd780ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780b3270 con 0x7fbd7806fa60 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780b30f0 con 0x7fbd780c8160 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd7d200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbd780b33f0 con 0x7fbd7815bfc0 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57488/0 (socket says 172.21.15.31:57488) 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 -- 172.21.15.31:0/1435056406 learned_addr learned my addr 172.21.15.31:0/1435056406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd6d400640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7815bfc0 0x7fbd780b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd76800640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd780c8160 0x7fbd780ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 -- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7815bfc0 msgr2=0x7fbd780b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7815bfc0 0x7fbd780b5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 -- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd780c8160 msgr2=0x7fbd780ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.624+0000 7fbd75e00640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd780c8160 0x7fbd780ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd75e00640 1 -- 172.21.15.31:0/1435056406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd781f8770 con 0x7fbd7806fa60 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd75e00640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd68000c00 tx=0x7fbd68033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68005c90 con 0x7fbd7806fa60 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd68031e50 con 0x7fbd7806fa60 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68033310 con 0x7fbd7806fa60 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd781f8a50 con 0x7fbd7806fa60 2024-09-13T08:37:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.625+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd781f9010 con 0x7fbd7806fa60 2024-09-13T08:37:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.626+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd780cbf20 con 0x7fbd7806fa60 2024-09-13T08:37:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.627+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fbd680334b0 con 0x7fbd7806fa60 2024-09-13T08:37:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.628+0000 7fbd6ea00640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 0x7fbd5407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.628+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fbd680bbd10 con 0x7fbd7806fa60 2024-09-13T08:37:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.629+0000 7fbd6d400640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 0x7fbd5407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:41.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.629+0000 7fbd6d400640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 0x7fbd5407ad10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005fd0 tx=0x7fbd60005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.633+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbd680c1020 con 0x7fbd7806fa60 2024-09-13T08:37:41.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.826+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbd780cb310 con 0x7fbd7806fa60 2024-09-13T08:37:41.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.826+0000 7fbd6ea00640 1 -- 172.21.15.31:0/1435056406 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fbd6803f030 con 0x7fbd7806fa60 2024-09-13T08:37:41.828 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-09-13T08:37:41.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 msgr2=0x7fbd5407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 0x7fbd5407ad10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005fd0 tx=0x7fbd60005850 comp rx=0 tx=0).stop 2024-09-13T08:37:41.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 msgr2=0x7fbd7815d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:41.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd68000c00 tx=0x7fbd68033c90 comp rx=0 tx=0).stop 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 shutdown_connections 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd780c8160 0x7fbd780ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd54078850 0x7fbd5407ad10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd7815bfc0 0x7fbd780b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 --2- 172.21.15.31:0/1435056406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd7806fa60 0x7fbd7815d540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 >> 172.21.15.31:0/1435056406 conn(0x7fbd780cfbb0 msgr2=0x7fbd781589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 shutdown_connections 2024-09-13T08:37:41.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:41.829+0000 7fbd7d200640 1 -- 172.21.15.31:0/1435056406 wait complete. 2024-09-13T08:37:41.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T08:37:42.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 -- 172.21.15.31:0/1928532391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac072560 msgr2=0x7fb2ac0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/1928532391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac072560 0x7fb2ac0cb740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb294009830 tx=0x7fb29402a5c0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 -- 172.21.15.31:0/1928532391 shutdown_connections 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/1928532391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac072560 0x7fb2ac0cb740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 -- 172.21.15.31:0/1928532391 >> 172.21.15.31:0/1928532391 conn(0x7fb2ac0cfbb0 msgr2=0x7fb2ac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 -- 172.21.15.31:0/1928532391 shutdown_connections 2024-09-13T08:37:42.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.009+0000 7fb2b2a00640 1 -- 172.21.15.31:0/1928532391 wait complete. 2024-09-13T08:37:42.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.011+0000 7fb2b2a00640 1 Processor -- start 2024-09-13T08:37:42.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.011+0000 7fb2b2a00640 1 -- start start 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 0x7fb2ac0c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 0x7fb2ac151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac06e740 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac06e5c0 con 0x7fb2ac072560 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b2a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ac0b30f0 con 0x7fb2ac15bfc0 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57508/0 (socket says 172.21.15.31:57508) 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2a3400640 1 -- 172.21.15.31:0/2954670230 learned_addr learned my addr 172.21.15.31:0/2954670230 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2abe00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 0x7fb2ac0c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.012+0000 7fb2b0a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 0x7fb2ac151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 -- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 msgr2=0x7fb2ac151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 0x7fb2ac151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 -- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 msgr2=0x7fb2ac0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 0x7fb2ac0c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 -- 172.21.15.31:0/2954670230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac0c3c20 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2b0a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 0x7fb2ac151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2abe00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 0x7fb2ac0c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a3400640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb294000c00 tx=0x7fb294032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294040070 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb294003af0 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294045460 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.013+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac0b2500 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.014+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac0c56e0 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.014+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ac0cb740 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.016+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fb294039040 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.017+0000 7fb2a8c00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 0x7fb28807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.017+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fb2940bbf00 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.017+0000 7fb2abe00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 0x7fb28807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.018+0000 7fb2abe00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 0x7fb28807ad10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb29c007fd0 tx=0x7fb29c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:42.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.022+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb294084600 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.219+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2ac0cb310 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.219+0000 7fb2a8c00640 1 -- 172.21.15.31:0/2954670230 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fb294083f00 con 0x7fb2ac0cbc80 2024-09-13T08:37:42.220 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-09-13T08:37:42.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 msgr2=0x7fb28807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 0x7fb28807ad10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb29c007fd0 tx=0x7fb29c009040 comp rx=0 tx=0).stop 2024-09-13T08:37:42.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 msgr2=0x7fb2ac0b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb294000c00 tx=0x7fb294032d50 comp rx=0 tx=0).stop 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 shutdown_connections 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078850 0x7fb28807ad10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bfc0 0x7fb2ac151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0cbc80 0x7fb2ac0b1fc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 --2- 172.21.15.31:0/2954670230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac072560 0x7fb2ac0c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 >> 172.21.15.31:0/2954670230 conn(0x7fb2ac0cfbb0 msgr2=0x7fb2ac1532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 shutdown_connections 2024-09-13T08:37:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.222+0000 7fb2b2a00640 1 -- 172.21.15.31:0/2954670230 wait complete. 2024-09-13T08:37:42.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.399+0000 7fe8fcc00640 1 -- 172.21.15.31:0/501717969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8dc004d40 msgr2=0x7fe8dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.399+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/501717969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8dc004d40 0x7fe8dc0051c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0009f00 tx=0x7fe8dc005830 comp rx=0 tx=0).stop 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.400+0000 7fe8f6800640 1 -- 172.21.15.31:0/501717969 reap_dead start 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.400+0000 7fe8fcc00640 1 -- 172.21.15.31:0/501717969 shutdown_connections 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.400+0000 7fe8fcc00640 1 -- 172.21.15.31:0/501717969 >> 172.21.15.31:0/501717969 conn(0x7fe8f80cc910 msgr2=0x7fe8f80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.400+0000 7fe8fcc00640 1 -- 172.21.15.31:0/501717969 shutdown_connections 2024-09-13T08:37:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.400+0000 7fe8fcc00640 1 -- 172.21.15.31:0/501717969 wait complete. 2024-09-13T08:37:42.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.401+0000 7fe8fcc00640 1 Processor -- start 2024-09-13T08:37:42.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.401+0000 7fe8fcc00640 1 -- start start 2024-09-13T08:37:42.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8dc004d40 0x7fe8f814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8f80c6f60 0x7fe8f814c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f80cc480 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f80cc300 con 0x7fe8dc004d40 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.402+0000 7fe8fcc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f80cc600 con 0x7fe8f80c6f60 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57522/0 (socket says 172.21.15.31:57522) 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 -- 172.21.15.31:0/1802288721 learned_addr learned my addr 172.21.15.31:0/1802288721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f5e00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8dc004d40 0x7fe8f814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8efe00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8f80c6f60 0x7fe8f814c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 -- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8f80c6f60 msgr2=0x7fe8f814c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8f80c6f60 0x7fe8f814c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 -- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8dc004d40 msgr2=0x7fe8f814cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8dc004d40 0x7fe8f814cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 -- 172.21.15.31:0/1802288721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f80c75f0 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8f6800640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e380 tx=0x7fe8f0033d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0042070 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.403+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8f0003c60 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.404+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0038040 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.404+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f80adc30 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.404+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f80ae1f0 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.405+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f80b0b80 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.406+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe8f0004290 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.407+0000 7fe8ee000640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 0x7fe8d407ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:42.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.407+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe8f00bac90 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.407+0000 7fe8f5e00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 0x7fe8d407ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:42.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.408+0000 7fe8f5e00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 0x7fe8d407ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8001a30 tx=0x7fe8e80011f0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.411+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe8f80b0b80 con 0x7fe8f80c9ce0 2024-09-13T08:37:42.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:42.677+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fe8f80b1da0 con 0x7fe8f80c9ce0 2024-09-13T08:37:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.133+0000 7fe8ee000640 1 -- 172.21.15.31:0/1802288721 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v57) ==== 161+0+0 (secure 0 0 0) 0x7fe8f0082c10 con 0x7fe8f80c9ce0 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 msgr2=0x7fe8d407ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 0x7fe8d407ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8001a30 tx=0x7fe8e80011f0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 msgr2=0x7fe8f80c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e380 tx=0x7fe8f0033d70 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 shutdown_connections 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8f80c9ce0 0x7fe8f80c7ca0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe8d4078880 0x7fe8d407ad40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8f80c6f60 0x7fe8f814c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 --2- 172.21.15.31:0/1802288721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8dc004d40 0x7fe8f814cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 >> 172.21.15.31:0/1802288721 conn(0x7fe8f80cc910 msgr2=0x7fe8f80b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 shutdown_connections 2024-09-13T08:37:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.136+0000 7fe8fcc00640 1 -- 172.21.15.31:0/1802288721 wait complete. 2024-09-13T08:37:43.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:37:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 -- 172.21.15.31:0/1424695032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 msgr2=0x7fab4c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 --2- 172.21.15.31:0/1424695032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab4c00b9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009f90 tx=0x7fab4c005830 comp rx=0 tx=0).stop 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab69e00640 1 -- 172.21.15.31:0/1424695032 reap_dead start 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 -- 172.21.15.31:0/1424695032 shutdown_connections 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 -- 172.21.15.31:0/1424695032 >> 172.21.15.31:0/1424695032 conn(0x7fab640cc910 msgr2=0x7fab640cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 -- 172.21.15.31:0/1424695032 shutdown_connections 2024-09-13T08:37:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.314+0000 7fab6be00640 1 -- 172.21.15.31:0/1424695032 wait complete. 2024-09-13T08:37:43.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.316+0000 7fab6be00640 1 Processor -- start 2024-09-13T08:37:43.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.316+0000 7fab6be00640 1 -- start start 2024-09-13T08:37:43.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 0x7fab6414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 0x7fab6409ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab640cead0 con 0x7fab4c004e60 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab640b19c0 con 0x7fab6414e640 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab6be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab640cec50 con 0x7fab6406ed60 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab69400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab69400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57534/0 (socket says 172.21.15.31:57534) 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab69400640 1 -- 172.21.15.31:0/2422297028 learned_addr learned my addr 172.21.15.31:0/2422297028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab69e00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 0x7fab6409ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.317+0000 7fab68a00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 0x7fab6414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 -- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 msgr2=0x7fab6409ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 0x7fab6409ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 -- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 msgr2=0x7fab6414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 0x7fab6414dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 -- 172.21.15.31:0/2422297028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab640c7000 con 0x7fab4c004e60 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab68a00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 0x7fab6414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69e00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 0x7fab6409ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab69400640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab5800b560 tx=0x7fab5800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5800f850 con 0x7fab4c004e60 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab5800fae0 con 0x7fab4c004e60 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab58017400 con 0x7fab4c004e60 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab640c72e0 con 0x7fab4c004e60 2024-09-13T08:37:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.318+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab640c8fd0 con 0x7fab4c004e60 2024-09-13T08:37:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.319+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab640c91d0 con 0x7fab4c004e60 2024-09-13T08:37:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.321+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fab580175a0 con 0x7fab4c004e60 2024-09-13T08:37:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.321+0000 7fab61600640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 0x7fab4407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.322+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fab5809a9f0 con 0x7fab4c004e60 2024-09-13T08:37:43.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.322+0000 7fab68a00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 0x7fab4407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.323+0000 7fab68a00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 0x7fab4407ad10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fab6409d960 tx=0x7fab50009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.326+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab640c91d0 con 0x7fab4c004e60 2024-09-13T08:37:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.525+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fab640c8ac0 con 0x7fab4c004e60 2024-09-13T08:37:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.525+0000 7fab61600640 1 -- 172.21.15.31:0/2422297028 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7fab58013030 con 0x7fab4c004e60 2024-09-13T08:37:43.526 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:37:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:37:43.528 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:37:43.529 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:37:43.529 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:37:43.529 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:37:43.529 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 msgr2=0x7fab4407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 0x7fab4407ad10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fab6409d960 tx=0x7fab50009040 comp rx=0 tx=0).stop 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 msgr2=0x7fab640b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab5800b560 tx=0x7fab5800ba20 comp rx=0 tx=0).stop 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 shutdown_connections 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fab44078850 0x7fab4407ad10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab6406ed60 0x7fab6409ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6414e640 0x7fab6414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 --2- 172.21.15.31:0/2422297028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c004e60 0x7fab640b2fd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 >> 172.21.15.31:0/2422297028 conn(0x7fab640cc910 msgr2=0x7fab640cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 shutdown_connections 2024-09-13T08:37:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.528+0000 7fab6be00640 1 -- 172.21.15.31:0/2422297028 wait complete. 2024-09-13T08:37:43.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 -- 172.21.15.31:0/3355843190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 msgr2=0x7fecec0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 --2- 172.21.15.31:0/3355843190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fecec0051f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fed00001d60 tx=0x7fecec005830 comp rx=0 tx=0).stop 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0ce00640 1 -- 172.21.15.31:0/3355843190 reap_dead start 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 -- 172.21.15.31:0/3355843190 shutdown_connections 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 -- 172.21.15.31:0/3355843190 >> 172.21.15.31:0/3355843190 conn(0x7fed080d3e90 msgr2=0x7fed080d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 -- 172.21.15.31:0/3355843190 shutdown_connections 2024-09-13T08:37:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.709+0000 7fed0d800640 1 -- 172.21.15.31:0/3355843190 wait complete. 2024-09-13T08:37:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.711+0000 7fed0d800640 1 Processor -- start 2024-09-13T08:37:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.711+0000 7fed0d800640 1 -- start start 2024-09-13T08:37:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 0x7fed0815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed081f4db0 0x7fed080b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed080d2620 con 0x7fecec004d90 2024-09-13T08:37:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed080d24a0 con 0x7fed081f4db0 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fed080d27a0 con 0x7fed081f4950 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57552/0 (socket says 172.21.15.31:57552) 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 -- 172.21.15.31:0/1655957997 learned_addr learned my addr 172.21.15.31:0/1655957997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fecffe00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 0x7fed0815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 -- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 msgr2=0x7fed0815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed0ce00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed081f4db0 0x7fed080b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 0x7fed0815a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 -- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed081f4db0 msgr2=0x7fed080b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed081f4db0 0x7fed080b7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fed07e00640 1 -- 172.21.15.31:0/1655957997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0815c130 con 0x7fecec004d90 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.712+0000 7fecffe00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 0x7fed0815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed07e00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fecf80066b0 tx=0x7fecf80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8015070 con 0x7fecec004d90 2024-09-13T08:37:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecf8011900 con 0x7fecec004d90 2024-09-13T08:37:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8011d30 con 0x7fecec004d90 2024-09-13T08:37:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0815c3b0 con 0x7fecec004d90 2024-09-13T08:37:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.713+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0815c970 con 0x7fecec004d90 2024-09-13T08:37:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.714+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0806e8e0 con 0x7fecec004d90 2024-09-13T08:37:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.715+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fecf800f430 con 0x7fecec004d90 2024-09-13T08:37:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.717+0000 7fed04c00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 0x7fece407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.717+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fecf809b640 con 0x7fecec004d90 2024-09-13T08:37:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.717+0000 7fecffe00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 0x7fece407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.718+0000 7fecffe00640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 0x7fece407ad10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fed0815bf20 tx=0x7fecf0005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.721+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fed0806e8e0 con 0x7fecec004d90 2024-09-13T08:37:43.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.918+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fed080b1910 con 0x7fecec004d90 2024-09-13T08:37:43.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.921+0000 7fed04c00640 1 -- 172.21.15.31:0/1655957997 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7fed0815c130 con 0x7fecec004d90 2024-09-13T08:37:43.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 msgr2=0x7fece407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 0x7fece407ad10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fed0815bf20 tx=0x7fecf0005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:43.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 msgr2=0x7fed0815d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:43.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fecf80066b0 tx=0x7fecf80039c0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 shutdown_connections 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed081f4db0 0x7fed080b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fece4078850 0x7fece407ad10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed081f4950 0x7fed0815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 --2- 172.21.15.31:0/1655957997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecec004d90 0x7fed0815d450 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 >> 172.21.15.31:0/1655957997 conn(0x7fed080d3e90 msgr2=0x7fed0815daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 shutdown_connections 2024-09-13T08:37:43.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:43.924+0000 7fed0d800640 1 -- 172.21.15.31:0/1655957997 wait complete. 2024-09-13T08:37:43.944 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:37:43.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:37:44.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:37:44.132 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:37:44.132 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:44.132 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:44.168 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:44.168 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:44.243 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:44.243 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:44.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:37:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 -- 172.21.15.31:0/1216031335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 msgr2=0x7f7008004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 --2- 172.21.15.31:0/1216031335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7008004c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f701000c610 tx=0x7f7008005830 comp rx=0 tx=0).stop 2024-09-13T08:37:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 -- 172.21.15.31:0/1216031335 shutdown_connections 2024-09-13T08:37:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 --2- 172.21.15.31:0/1216031335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7008004c80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 -- 172.21.15.31:0/1216031335 >> 172.21.15.31:0/1216031335 conn(0x7f70240b18f0 msgr2=0x7f70240b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:44.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 -- 172.21.15.31:0/1216031335 shutdown_connections 2024-09-13T08:37:44.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.519+0000 7f702a000640 1 -- 172.21.15.31:0/1216031335 wait complete. 2024-09-13T08:37:44.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.521+0000 7f702a000640 1 Processor -- start 2024-09-13T08:37:44.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.521+0000 7f702a000640 1 -- start start 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70241f36f0 0x7f702409d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f702413d1d0 0x7f702413d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f702415bf40 con 0x7f7008004800 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f702415bdc0 con 0x7f702413d1d0 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f702a000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f702415c0c0 con 0x7f70241f36f0 2024-09-13T08:37:44.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57568/0 (socket says 172.21.15.31:57568) 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 -- 172.21.15.31:0/3992305678 learned_addr learned my addr 172.21.15.31:0/3992305678 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7022a00640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70241f36f0 0x7f702409d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023e00640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f702413d1d0 0x7f702413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 -- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70241f36f0 msgr2=0x7f702409d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70241f36f0 0x7f702409d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 -- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f702413d1d0 msgr2=0x7f702413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f702413d1d0 0x7f702413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.522+0000 7f7023400640 1 -- 172.21.15.31:0/3992305678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702413c590 con 0x7f7008004800 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f7023400640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7014011bd0 tx=0x7f701400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701401e070 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f701400fd70 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702413b7d0 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014019670 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.523+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f702413bd90 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.524+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f702415f0f0 con 0x7f7008004800 2024-09-13T08:37:44.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.525+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f7014021070 con 0x7f7008004800 2024-09-13T08:37:44.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.526+0000 7f701b400640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 0x7f700407ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:44.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.526+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f701409b1d0 con 0x7f7008004800 2024-09-13T08:37:44.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.527+0000 7f7022a00640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 0x7f700407ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.528+0000 7f7022a00640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 0x7f700407ad40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7010033000 tx=0x7f7010001bc0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:44.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.530+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f70140631e0 con 0x7f7008004800 2024-09-13T08:37:44.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.737+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f702409e0b0 con 0x7f7008004800 2024-09-13T08:37:44.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.737+0000 7f701b400640 1 -- 172.21.15.31:0/3992305678 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f7014067c10 con 0x7f7008004800 2024-09-13T08:37:44.738 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:37:44.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 msgr2=0x7f700407ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 0x7f700407ad40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7010033000 tx=0x7f7010001bc0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 msgr2=0x7f7024158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7014011bd0 tx=0x7f701400f3c0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 shutdown_connections 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f702413d1d0 0x7f702413d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7004078880 0x7f700407ad40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70241f36f0 0x7f702409d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 --2- 172.21.15.31:0/3992305678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7008004800 0x7f7024158790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.739+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 >> 172.21.15.31:0/3992305678 conn(0x7f70240b18f0 msgr2=0x7f70241428f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.740+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 shutdown_connections 2024-09-13T08:37:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:44.740+0000 7f702a000640 1 -- 172.21.15.31:0/3992305678 wait complete. 2024-09-13T08:37:44.756 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:37:44.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:44.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-13T08:37:44.817 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:37:44.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:44.817 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:37:44.862 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:44.862 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:44.902 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:44.902 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:44.980 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:44.980 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:45.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.226+0000 7fd2b9400640 1 -- 172.21.15.31:0/1252592975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 msgr2=0x7fd2b40cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.226+0000 7fd2b9400640 1 --2- 172.21.15.31:0/1252592975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b40cee40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009910 tx=0x7fd2a402a4d0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 -- 172.21.15.31:0/1252592975 shutdown_connections 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 --2- 172.21.15.31:0/1252592975 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 --2- 172.21.15.31:0/1252592975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b40cee40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 -- 172.21.15.31:0/1252592975 >> 172.21.15.31:0/1252592975 conn(0x7fd2b40cc7a0 msgr2=0x7fd2b40ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 -- 172.21.15.31:0/1252592975 shutdown_connections 2024-09-13T08:37:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.227+0000 7fd2b9400640 1 -- 172.21.15.31:0/1252592975 wait complete. 2024-09-13T08:37:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.228+0000 7fd2b9400640 1 Processor -- start 2024-09-13T08:37:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 -- start start 2024-09-13T08:37:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 0x7fd2b4156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b40b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b4070040 con 0x7fd2b415bb70 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b406fec0 con 0x7fd2b406e670 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.229+0000 7fd2b9400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b40701c0 con 0x7fd2b415f820 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57766/0 (socket says 172.21.15.31:57766) 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 -- 172.21.15.31:0/2405246904 learned_addr learned my addr 172.21.15.31:0/2405246904 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2b3e00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 0x7fd2b4156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2b8a00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b40b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 -- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 msgr2=0x7fd2b40b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b40b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 -- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 msgr2=0x7fd2b4156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 0x7fd2b4156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 -- 172.21.15.31:0/2405246904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b40cbb70 con 0x7fd2b415bb70 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2b3e00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 0x7fd2b4156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2b8a00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b40b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.230+0000 7fd2abe00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c430 tx=0x7fd29c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.231+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c011790 con 0x7fd2b415bb70 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.231+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd29c017070 con 0x7fd2b415bb70 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.231+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c011d00 con 0x7fd2b415bb70 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.231+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b41f8a80 con 0x7fd2b415bb70 2024-09-13T08:37:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.231+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b40c8070 con 0x7fd2b415bb70 2024-09-13T08:37:45.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.232+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b4070750 con 0x7fd2b415bb70 2024-09-13T08:37:45.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.233+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd29c01c460 con 0x7fd2b415bb70 2024-09-13T08:37:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.234+0000 7fd2b0c00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 0x7fd29007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.234+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fd29c09a810 con 0x7fd2b415bb70 2024-09-13T08:37:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.235+0000 7fd2b3e00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 0x7fd29007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.236+0000 7fd2b3e00640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 0x7fd29007ad10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4034000 tx=0x7fd2a4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.238+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd29c062e70 con 0x7fd2b415bb70 2024-09-13T08:37:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.449+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd2b41e8050 con 0x7fd2b415bb70 2024-09-13T08:37:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.449+0000 7fd2b0c00640 1 -- 172.21.15.31:0/2405246904 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fd29c062770 con 0x7fd2b415bb70 2024-09-13T08:37:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 msgr2=0x7fd29007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 0x7fd29007ad10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4034000 tx=0x7fd2a4004730 comp rx=0 tx=0).stop 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 msgr2=0x7fd2b41f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c430 tx=0x7fd29c00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 shutdown_connections 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd290078850 0x7fd29007ad10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd2b415f820 0x7fd2b40b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2b415bb70 0x7fd2b41f8030 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 --2- 172.21.15.31:0/2405246904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2b406e670 0x7fd2b4156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 >> 172.21.15.31:0/2405246904 conn(0x7fd2b40cc7a0 msgr2=0x7fd2b40cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 shutdown_connections 2024-09-13T08:37:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.452+0000 7fd2b9400640 1 -- 172.21.15.31:0/2405246904 wait complete. 2024-09-13T08:37:45.468 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:37:45.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:45.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-09-13T08:37:45.538 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:37:45.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:45.538 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:37:45.585 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:45.585 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:45.623 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:45.623 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:45.700 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:37:45.700 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:45.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:37:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.978+0000 7fae58400640 1 -- 172.21.15.31:0/832630331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 msgr2=0x7fae5015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.978+0000 7fae58400640 1 --2- 172.21.15.31:0/832630331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae5015d580 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fae5006d790 tx=0x7fae4802a8c0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 -- 172.21.15.31:0/832630331 shutdown_connections 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 --2- 172.21.15.31:0/832630331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae5015d580 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 --2- 172.21.15.31:0/832630331 >> v2:172.21.15.45:3300/0 conn(0x7fae501562a0 0x7fae500ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 -- 172.21.15.31:0/832630331 >> 172.21.15.31:0/832630331 conn(0x7fae5006d240 msgr2=0x7fae500688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 -- 172.21.15.31:0/832630331 shutdown_connections 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.979+0000 7fae58400640 1 -- 172.21.15.31:0/832630331 wait complete. 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.980+0000 7fae58400640 1 Processor -- start 2024-09-13T08:37:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.981+0000 7fae58400640 1 -- start start 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.981+0000 7fae58400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.981+0000 7fae58400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae501562a0 0x7fae500b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.981+0000 7fae58400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae50156680 0x7fae500c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae58400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fae5015de50 con 0x7fae50071950 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae58400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae5015dcd0 con 0x7fae50156680 2024-09-13T08:37:45.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae58400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fae5015dfd0 con 0x7fae501562a0 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57770/0 (socket says 172.21.15.31:57770) 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 -- 172.21.15.31:0/3750763913 learned_addr learned my addr 172.21.15.31:0/3750763913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 -- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae501562a0 msgr2=0x7fae500b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae501562a0 0x7fae500b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 -- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae50156680 msgr2=0x7fae500c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae50156680 0x7fae500c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 -- 172.21.15.31:0/3750763913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae500c7500 con 0x7fae50071950 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.982+0000 7fae55a00640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fae4400d950 tx=0x7fae4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.983+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44017070 con 0x7fae50071950 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.983+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae44009d20 con 0x7fae50071950 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.983+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae500c7220 con 0x7fae50071950 2024-09-13T08:37:45.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.983+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae440106c0 con 0x7fae50071950 2024-09-13T08:37:45.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.983+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae500d0140 con 0x7fae50071950 2024-09-13T08:37:45.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.984+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5015d580 con 0x7fae50071950 2024-09-13T08:37:45.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.985+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fae44010860 con 0x7fae50071950 2024-09-13T08:37:45.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.987+0000 7fae4d600640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 0x7fae3007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:45.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.987+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fae4409ad40 con 0x7fae50071950 2024-09-13T08:37:45.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.987+0000 7fae55000640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 0x7fae3007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:45.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.988+0000 7fae55000640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 0x7fae3007ad10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae500b50b0 tx=0x7fae3c007a30 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:45.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:45.991+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fae440633a0 con 0x7fae50071950 2024-09-13T08:37:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.199+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fae50160540 con 0x7fae50071950 2024-09-13T08:37:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.199+0000 7fae4d600640 1 -- 172.21.15.31:0/3750763913 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fae44062ca0 con 0x7fae50071950 2024-09-13T08:37:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 msgr2=0x7fae3007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 0x7fae3007ad10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae500b50b0 tx=0x7fae3c007a30 comp rx=0 tx=0).stop 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 msgr2=0x7fae500d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fae4400d950 tx=0x7fae4400de10 comp rx=0 tx=0).stop 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 shutdown_connections 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae50156680 0x7fae500c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae30078850 0x7fae3007ad10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae501562a0 0x7fae500b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 --2- 172.21.15.31:0/3750763913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae50071950 0x7fae500d0940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 >> 172.21.15.31:0/3750763913 conn(0x7fae5006d240 msgr2=0x7fae500b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:46.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 shutdown_connections 2024-09-13T08:37:46.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.202+0000 7fae58400640 1 -- 172.21.15.31:0/3750763913 wait complete. 2024-09-13T08:37:46.219 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-09-13T08:37:46.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:46.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-09-13T08:37:46.280 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:37:46.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:46.280 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T08:37:46.316 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:46.316 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:46.354 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:46.354 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:46.431 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:37:46.431 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:37:46.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 -- 172.21.15.31:0/327518676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00ced20 msgr2=0x7f6fe00ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/327518676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00ced20 0x7f6fe00ce290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8005fd0 tx=0x7f6fc802a540 comp rx=0 tx=0).stop 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 -- 172.21.15.31:0/327518676 shutdown_connections 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/327518676 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe0073860 0x7f6fe0073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/327518676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00ced20 0x7f6fe00ce290 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.669+0000 7f6fe6c00640 1 -- 172.21.15.31:0/327518676 >> 172.21.15.31:0/327518676 conn(0x7f6fe00b18f0 msgr2=0x7f6fe00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.670+0000 7f6fe6c00640 1 -- 172.21.15.31:0/327518676 shutdown_connections 2024-09-13T08:37:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.670+0000 7f6fe6c00640 1 -- 172.21.15.31:0/327518676 wait complete. 2024-09-13T08:37:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.671+0000 7f6fe6c00640 1 Processor -- start 2024-09-13T08:37:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.671+0000 7f6fe6c00640 1 -- start start 2024-09-13T08:37:46.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe0073860 0x7f6fe009d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe00ced20 0x7f6fe01f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00733e0 con 0x7f6fe015be30 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00ce9e0 con 0x7f6fe0073860 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe6c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe0074230 con 0x7f6fe00ced20 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe0073860 0x7f6fe009d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.672+0000 7f6fe4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57776/0 (socket says 172.21.15.31:57776) 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 -- 172.21.15.31:0/2183578355 learned_addr learned my addr 172.21.15.31:0/2183578355 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:46.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fd7400640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe00ced20 0x7f6fe01f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 -- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe00ced20 msgr2=0x7f6fe01f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe00ced20 0x7f6fe01f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 -- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe0073860 msgr2=0x7f6fe009d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe0073860 0x7f6fe009d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 -- 172.21.15.31:0/2183578355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe01f3570 con 0x7f6fe015be30 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe4c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800eaa0 tx=0x7f6fd800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800cdb0 con 0x7f6fe015be30 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fd8004c10 con 0x7f6fe015be30 2024-09-13T08:37:46.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8005020 con 0x7f6fe015be30 2024-09-13T08:37:46.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.673+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe01f3850 con 0x7f6fe015be30 2024-09-13T08:37:46.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.674+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe0152200 con 0x7f6fe015be30 2024-09-13T08:37:46.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.675+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe006ab90 con 0x7f6fe015be30 2024-09-13T08:37:46.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.676+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f6fd80051c0 con 0x7f6fe015be30 2024-09-13T08:37:46.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.677+0000 7f6fdcc00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 0x7f6fbc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:46.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.677+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f6fd809b330 con 0x7f6fe015be30 2024-09-13T08:37:46.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.677+0000 7f6fdfe00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 0x7f6fbc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:46.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.678+0000 7f6fdfe00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 0x7f6fbc07ad10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd0009060 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:46.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.681+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6fe006ab90 con 0x7f6fe015be30 2024-09-13T08:37:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.888+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6fe0074720 con 0x7f6fe015be30 2024-09-13T08:37:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.889+0000 7f6fdcc00640 1 -- 172.21.15.31:0/2183578355 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f6fd8016030 con 0x7f6fe015be30 2024-09-13T08:37:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 msgr2=0x7f6fbc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 0x7f6fbc07ad10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd0009060 comp rx=0 tx=0).stop 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 msgr2=0x7f6fe00ae5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800eaa0 tx=0x7f6fd800ef60 comp rx=0 tx=0).stop 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 shutdown_connections 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe015be30 0x7f6fe00ae5a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6fbc078850 0x7f6fbc07ad10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe00ced20 0x7f6fe01f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 --2- 172.21.15.31:0/2183578355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe0073860 0x7f6fe009d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 >> 172.21.15.31:0/2183578355 conn(0x7f6fe00b18f0 msgr2=0x7f6fe00afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 shutdown_connections 2024-09-13T08:37:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:46.891+0000 7f6fe6c00640 1 -- 172.21.15.31:0/2183578355 wait complete. 2024-09-13T08:37:46.909 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-09-13T08:37:46.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:46.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-09-13T08:37:46.971 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:37:46.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:37:46.972 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T08:37:47.009 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:37:47.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:47.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.170+0000 7fd10f200640 1 -- 172.21.15.31:0/528332275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f4003aa0 msgr2=0x7fd0f4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.170+0000 7fd10f200640 1 --2- 172.21.15.31:0/528332275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f4003aa0 0x7fd0f4003ec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd100006f60 tx=0x7fd0f4005830 comp rx=0 tx=0).stop 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.170+0000 7fd10f200640 1 -- 172.21.15.31:0/528332275 shutdown_connections 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.170+0000 7fd10f200640 1 --2- 172.21.15.31:0/528332275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f4003aa0 0x7fd0f4003ec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.170+0000 7fd10f200640 1 -- 172.21.15.31:0/528332275 >> 172.21.15.31:0/528332275 conn(0x7fd11009d1c0 msgr2=0x7fd1100d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.171+0000 7fd10f200640 1 -- 172.21.15.31:0/528332275 shutdown_connections 2024-09-13T08:37:47.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.171+0000 7fd10f200640 1 -- 172.21.15.31:0/528332275 wait complete. 2024-09-13T08:37:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.172+0000 7fd10f200640 1 Processor -- start 2024-09-13T08:37:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.172+0000 7fd10f200640 1 -- start start 2024-09-13T08:37:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 0x7fd1101f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd11015de50 0x7fd1100c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1101621e0 con 0x7fd11015b5e0 2024-09-13T08:37:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd110162060 con 0x7fd11015de50 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd10f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd110162360 con 0x7fd0f4003aa0 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd107e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd107e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57792/0 (socket says 172.21.15.31:57792) 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.173+0000 7fd107e00640 1 -- 172.21.15.31:0/2264884952 learned_addr learned my addr 172.21.15.31:0/2264884952 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd10de00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 0x7fd1101f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd10e800640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd11015de50 0x7fd1100c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 -- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 msgr2=0x7fd1101f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 0x7fd1101f4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 -- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd11015de50 msgr2=0x7fd1100c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd11015de50 0x7fd1100c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 -- 172.21.15.31:0/2264884952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1101f4e80 con 0x7fd11015b5e0 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd10de00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 0x7fd1101f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd107e00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8004d60 tx=0x7fd0f8001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.174+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8015020 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.175+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f8004e50 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.175+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8011b80 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.175+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1100c7880 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.175+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1100c7e40 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.176+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1101628c0 con 0x7fd11015b5e0 2024-09-13T08:37:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.177+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd0f8018020 con 0x7fd11015b5e0 2024-09-13T08:37:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.178+0000 7fd106000640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 0x7fd0ec07ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.178+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fd0f809b120 con 0x7fd11015b5e0 2024-09-13T08:37:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.178+0000 7fd10de00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 0x7fd0ec07ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.179+0000 7fd10de00640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 0x7fd0ec07ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd100033000 tx=0x7fd1100cf9a0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:47.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.182+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd0f801f020 con 0x7fd11015b5e0 2024-09-13T08:37:47.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.415+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1100d40c0 con 0x7fd11015b5e0 2024-09-13T08:37:47.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.416+0000 7fd106000640 1 -- 172.21.15.31:0/2264884952 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7fd0f8063780 con 0x7fd11015b5e0 2024-09-13T08:37:47.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:47.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:47.417 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.418+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 msgr2=0x7fd0ec07ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.418+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 0x7fd0ec07ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd100033000 tx=0x7fd1100cf9a0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.418+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 msgr2=0x7fd11015d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.418+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8004d60 tx=0x7fd0f8001630 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 shutdown_connections 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd11015de50 0x7fd1100c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd11015b5e0 0x7fd11015d910 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd0ec078880 0x7fd0ec07ad40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 --2- 172.21.15.31:0/2264884952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0f4003aa0 0x7fd1101f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 >> 172.21.15.31:0/2264884952 conn(0x7fd11009d1c0 msgr2=0x7fd1100d1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 shutdown_connections 2024-09-13T08:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.419+0000 7fd10f200640 1 -- 172.21.15.31:0/2264884952 wait complete. 2024-09-13T08:37:47.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:47.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 -- 172.21.15.31:0/3263823061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09c0003fd0 msgr2=0x7f09c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 --2- 172.21.15.31:0/3263823061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09c0003fd0 0x7f09c0004800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009d20 tx=0x7f09c0005830 comp rx=0 tx=0).stop 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e0e00640 1 -- 172.21.15.31:0/3263823061 reap_dead start 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 -- 172.21.15.31:0/3263823061 shutdown_connections 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 -- 172.21.15.31:0/3263823061 >> 172.21.15.31:0/3263823061 conn(0x7f09dc0cc910 msgr2=0x7f09dc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 -- 172.21.15.31:0/3263823061 shutdown_connections 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.595+0000 7f09e2e00640 1 -- 172.21.15.31:0/3263823061 wait complete. 2024-09-13T08:37:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.597+0000 7f09e2e00640 1 Processor -- start 2024-09-13T08:37:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.597+0000 7f09e2e00640 1 -- start start 2024-09-13T08:37:47.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09c0003fd0 0x7f09dc0b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09dc0c9640 0x7f09dc0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc0cb9d0 con 0x7f09dc159460 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc0cb850 con 0x7f09c0003fd0 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e2e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc0cc360 con 0x7f09dc0c9640 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57820/0 (socket says 172.21.15.31:57820) 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09d3400640 1 -- 172.21.15.31:0/82530733 learned_addr learned my addr 172.21.15.31:0/82530733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09dbe00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09c0003fd0 0x7f09dc0b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.598+0000 7f09e0e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09dc0c9640 0x7f09dc0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 -- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09dc0c9640 msgr2=0x7f09dc0b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09dc0c9640 0x7f09dc0b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 -- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09c0003fd0 msgr2=0x7f09dc0b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09c0003fd0 0x7f09dc0b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 -- 172.21.15.31:0/82530733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc06e8d0 con 0x7f09dc159460 2024-09-13T08:37:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d3400640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09c4035040 tx=0x7f09c40098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4042020 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09c40049b0 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc06ebb0 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4004dc0 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.599+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc06f170 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.600+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc15fe20 con 0x7f09dc159460 2024-09-13T08:37:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.602+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f09c4045020 con 0x7f09dc159460 2024-09-13T08:37:47.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.602+0000 7f09d8c00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 0x7f09b807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.603+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7f09c40bb970 con 0x7f09dc159460 2024-09-13T08:37:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.603+0000 7f09dbe00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 0x7f09b807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.604+0000 7f09dbe00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 0x7f09b807ad40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009b70 tx=0x7f09cc005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:47.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.607+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f09c404c070 con 0x7f09dc159460 2024-09-13T08:37:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.841+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09dc0729c0 con 0x7f09dc159460 2024-09-13T08:37:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.842+0000 7f09d8c00640 1 -- 172.21.15.31:0/82530733 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7f09c4083f40 con 0x7f09dc159460 2024-09-13T08:37:47.843 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:47.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:47.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 msgr2=0x7f09b807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 0x7f09b807ad40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009b70 tx=0x7f09cc005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 msgr2=0x7f09dc06f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09c4035040 tx=0x7f09c40098f0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 shutdown_connections 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f09b8078880 0x7f09b807ad40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09dc0c9640 0x7f09dc0b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09dc159460 0x7f09dc06f970 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 --2- 172.21.15.31:0/82530733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09c0003fd0 0x7f09dc0b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.844+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 >> 172.21.15.31:0/82530733 conn(0x7f09dc0cc910 msgr2=0x7f09dc0acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.845+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 shutdown_connections 2024-09-13T08:37:47.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:47.845+0000 7f09e2e00640 1 -- 172.21.15.31:0/82530733 wait complete. 2024-09-13T08:37:47.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 -- 172.21.15.31:0/3731027890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfe40051c0 msgr2=0x7fbfe400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 --2- 172.21.15.31:0/3731027890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfe40051c0 0x7fbfe400b9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc0000cece0 tx=0x7fbfe4005830 comp rx=0 tx=0).stop 2024-09-13T08:37:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fbfff400640 1 -- 172.21.15.31:0/3731027890 reap_dead start 2024-09-13T08:37:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 -- 172.21.15.31:0/3731027890 shutdown_connections 2024-09-13T08:37:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 -- 172.21.15.31:0/3731027890 >> 172.21.15.31:0/3731027890 conn(0x7fc0000cc910 msgr2=0x7fc0000cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:48.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 -- 172.21.15.31:0/3731027890 shutdown_connections 2024-09-13T08:37:48.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.021+0000 7fc005c00640 1 -- 172.21.15.31:0/3731027890 wait complete. 2024-09-13T08:37:48.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.023+0000 7fc005c00640 1 Processor -- start 2024-09-13T08:37:48.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.023+0000 7fc005c00640 1 -- start start 2024-09-13T08:37:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 0x7fc0001f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0001563a0 0x7fc00015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc0001600f0 con 0x7fc00009d720 2024-09-13T08:37:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0000cb850 con 0x7fc0001563a0 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fc005c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc0001f35a0 con 0x7fbfe40051c0 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57832/0 (socket says 172.21.15.31:57832) 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 -- 172.21.15.31:0/1450172146 learned_addr learned my addr 172.21.15.31:0/1450172146 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbfff400640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0001563a0 0x7fc00015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffea00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 0x7fc0001f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 -- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 msgr2=0x7fc0001f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 0x7fc0001f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 -- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0001563a0 msgr2=0x7fc00015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0001563a0 0x7fc00015e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffe000640 1 -- 172.21.15.31:0/1450172146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0000b4a40 con 0x7fc00009d720 2024-09-13T08:37:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.024+0000 7fbffea00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 0x7fc0001f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fbffe000640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfec011ae0 tx=0x7fbfec009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec00fcb0 con 0x7fc00009d720 2024-09-13T08:37:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfec00fe50 con 0x7fc00009d720 2024-09-13T08:37:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0000b4d20 con 0x7fc00009d720 2024-09-13T08:37:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec004280 con 0x7fc00009d720 2024-09-13T08:37:48.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.025+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0000b52e0 con 0x7fc00009d720 2024-09-13T08:37:48.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.026+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0000ce3c0 con 0x7fc00009d720 2024-09-13T08:37:48.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.027+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fbfec0035f0 con 0x7fc00009d720 2024-09-13T08:37:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.028+0000 7fbff6a00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 0x7fbfe007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.028+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fbfec09b430 con 0x7fc00009d720 2024-09-13T08:37:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.029+0000 7fbffea00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 0x7fbfe007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.030+0000 7fbffea00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 0x7fbfe007ad10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000fb0 tx=0x7fbff0006720 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.033+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbfec063a90 con 0x7fc00009d720 2024-09-13T08:37:48.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.230+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0000cab20 con 0x7fc00009d720 2024-09-13T08:37:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.232+0000 7fbff6a00640 1 -- 172.21.15.31:0/1450172146 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16621 (secure 0 0 0) 0x7fbfec063390 con 0x7fc00009d720 2024-09-13T08:37:48.233 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:48.234 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:11.007120+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 msgr2=0x7fbfe007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 0x7fbfe007ad10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000fb0 tx=0x7fbff0006720 comp rx=0 tx=0).stop 2024-09-13T08:37:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 msgr2=0x7fc0001f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfec011ae0 tx=0x7fbfec009f10 comp rx=0 tx=0).stop 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 shutdown_connections 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0001563a0 0x7fc00015e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc00009d720 0x7fc0001f2e60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbfe0078850 0x7fbfe007ad10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 --2- 172.21.15.31:0/1450172146 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbfe40051c0 0x7fc0001f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 >> 172.21.15.31:0/1450172146 conn(0x7fc0000cc910 msgr2=0x7fc0000cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.234+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 shutdown_connections 2024-09-13T08:37:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.235+0000 7fc005c00640 1 -- 172.21.15.31:0/1450172146 wait complete. 2024-09-13T08:37:48.251 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:37:48.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:48.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 -- 172.21.15.31:0/1351113098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe760004800 msgr2=0x7fe760004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 --2- 172.21.15.31:0/1351113098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe760004800 0x7fe760004c80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe764006ef0 tx=0x7fe760005830 comp rx=0 tx=0).stop 2024-09-13T08:37:48.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 -- 172.21.15.31:0/1351113098 shutdown_connections 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 --2- 172.21.15.31:0/1351113098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe760004800 0x7fe760004c80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 -- 172.21.15.31:0/1351113098 >> 172.21.15.31:0/1351113098 conn(0x7fe7780cda20 msgr2=0x7fe7780cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 -- 172.21.15.31:0/1351113098 shutdown_connections 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.412+0000 7fe77f800640 1 -- 172.21.15.31:0/1351113098 wait complete. 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.413+0000 7fe77f800640 1 Processor -- start 2024-09-13T08:37:48.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 -- start start 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe760004800 0x7fe7781eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 0x7fe7780b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe77806cc80 con 0x7fe7780b1f90 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe77806cb00 con 0x7fe7780c9030 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.414+0000 7fe77f800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe77806ce00 con 0x7fe760004800 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57836/0 (socket says 172.21.15.31:57836) 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 -- 172.21.15.31:0/4108888525 learned_addr learned my addr 172.21.15.31:0/4108888525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77ce00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe760004800 0x7fe7781eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe777e00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 0x7fe7780b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 -- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe760004800 msgr2=0x7fe7781eec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe760004800 0x7fe7781eec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 -- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 msgr2=0x7fe7780b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 0x7fe7780b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 -- 172.21.15.31:0/4108888525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7780c98c0 con 0x7fe7780b1f90 2024-09-13T08:37:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe777e00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 0x7fe7780b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.415+0000 7fe77d800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe770005fd0 tx=0x7fe77000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.416+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770003d80 con 0x7fe7780b1f90 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.416+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe770004b20 con 0x7fe7780b1f90 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.416+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7780c9ba0 con 0x7fe7780b1f90 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.416+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770004030 con 0x7fe7780b1f90 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.416+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe778070480 con 0x7fe7780b1f90 2024-09-13T08:37:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.417+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77815c150 con 0x7fe7780b1f90 2024-09-13T08:37:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.418+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe7700041d0 con 0x7fe7780b1f90 2024-09-13T08:37:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.419+0000 7fe774c00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 0x7fe75807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.419+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7fe77009b730 con 0x7fe7780b1f90 2024-09-13T08:37:48.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.419+0000 7fe77ce00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 0x7fe75807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.420+0000 7fe77ce00640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 0x7fe75807ad10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe77806fea0 tx=0x7fe76c006bf0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.423+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe770063d90 con 0x7fe7780b1f90 2024-09-13T08:37:48.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.654+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7780b6f20 con 0x7fe7780b1f90 2024-09-13T08:37:48.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.655+0000 7fe774c00640 1 -- 172.21.15.31:0/4108888525 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7fe770063690 con 0x7fe7780b1f90 2024-09-13T08:37:48.656 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:48.656 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:48.656 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:48.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 msgr2=0x7fe75807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 0x7fe75807ad10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe77806fea0 tx=0x7fe76c006bf0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 msgr2=0x7fe778070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe770005fd0 tx=0x7fe77000f790 comp rx=0 tx=0).stop 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 shutdown_connections 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7780b1f90 0x7fe778070c80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7780c9030 0x7fe7780b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe758078850 0x7fe75807ad10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 --2- 172.21.15.31:0/4108888525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe760004800 0x7fe7781eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 >> 172.21.15.31:0/4108888525 conn(0x7fe7780cda20 msgr2=0x7fe7780cf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 shutdown_connections 2024-09-13T08:37:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.657+0000 7fe77f800640 1 -- 172.21.15.31:0/4108888525 wait complete. 2024-09-13T08:37:48.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 -- 172.21.15.31:0/2829807337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 msgr2=0x7ff09815a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2829807337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff09815a010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff090005fd0 tx=0x7ff09002b980 comp rx=0 tx=0).stop 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 -- 172.21.15.31:0/2829807337 shutdown_connections 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2829807337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff09815a010 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2829807337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 0x7ff0980ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 -- 172.21.15.31:0/2829807337 >> 172.21.15.31:0/2829807337 conn(0x7ff0980cc910 msgr2=0x7ff0980cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 -- 172.21.15.31:0/2829807337 shutdown_connections 2024-09-13T08:37:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.833+0000 7ff0a0400640 1 -- 172.21.15.31:0/2829807337 wait complete. 2024-09-13T08:37:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.835+0000 7ff0a0400640 1 Processor -- start 2024-09-13T08:37:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.835+0000 7ff0a0400640 1 -- start start 2024-09-13T08:37:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff098070860 0x7ff0981f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 0x7ff09815d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff09806ef90 con 0x7ff0980cc270 2024-09-13T08:37:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff09806ee10 con 0x7ff098070860 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff0a0400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff09806f110 con 0x7ff0980aef70 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff09e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff09e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57850/0 (socket says 172.21.15.31:57850) 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff09da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff098070860 0x7ff0981f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.836+0000 7ff09e400640 1 -- 172.21.15.31:0/2341019265 learned_addr learned my addr 172.21.15.31:0/2341019265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09d000640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 0x7ff09815d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 -- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 msgr2=0x7ff09815d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 0x7ff09815d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 -- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff098070860 msgr2=0x7ff0981f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff098070860 0x7ff0981f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 -- 172.21.15.31:0/2341019265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0981f4e20 con 0x7ff0980cc270 2024-09-13T08:37:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff09e400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff090006010 tx=0x7ff090003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090035050 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff090004540 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.837+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090045480 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.838+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0981f9360 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.838+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0981f50a0 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.839+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff098074680 con 0x7ff0980cc270 2024-09-13T08:37:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.840+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7ff090038020 con 0x7ff0980cc270 2024-09-13T08:37:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.841+0000 7ff095600640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 0x7ff07807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.841+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5709+0+0 (secure 0 0 0) 0x7ff0900bbd30 con 0x7ff0980cc270 2024-09-13T08:37:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.841+0000 7ff09da00640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 0x7ff07807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.842+0000 7ff09da00640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 0x7ff07807ad10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff0981f5f90 tx=0x7ff08c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:48.845+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff098074680 con 0x7ff0980cc270 2024-09-13T08:37:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.078+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0980b5e10 con 0x7ff0980cc270 2024-09-13T08:37:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.079+0000 7ff095600640 1 -- 172.21.15.31:0/2341019265 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5208 (secure 0 0 0) 0x7ff0981f9360 con 0x7ff0980cc270 2024-09-13T08:37:49.080 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:49.080 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"btime":"2024-09-13T08:37:13:018595+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/825324451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":825324451},{"type":"v1","addr":"172.21.15.31:6835","nonce":825324451}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4334,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1115573594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1115573594},{"type":"v1","addr":"172.21.15.45:6836","nonce":1115573594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/41829158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":41829158},{"type":"v1","addr":"172.21.15.31:6834","nonce":41829158}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:37:09.446277+0000","modified":"2024-09-13T08:37:12.461695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4366":{"gid":4366,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6837/2663745938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":2663745938},{"type":"v1","addr":"172.21.15.45:6837","nonce":2663745938}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4366,"qdb_cluster":[4366]},"id":1}]} 2024-09-13T08:37:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-09-13T08:37:49.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 msgr2=0x7ff07807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 0x7ff07807ad10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff0981f5f90 tx=0x7ff08c005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 msgr2=0x7ff0980b0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff090006010 tx=0x7ff090003f10 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 shutdown_connections 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0980cc270 0x7ff0980b0110 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff078078850 0x7ff07807ad10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0980aef70 0x7ff09815d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 --2- 172.21.15.31:0/2341019265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff098070860 0x7ff0981f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 >> 172.21.15.31:0/2341019265 conn(0x7ff0980cc910 msgr2=0x7ff09806e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 shutdown_connections 2024-09-13T08:37:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.081+0000 7ff0a0400640 1 -- 172.21.15.31:0/2341019265 wait complete. 2024-09-13T08:37:49.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.258+0000 7f418ec00640 1 -- 172.21.15.31:0/4101527681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 msgr2=0x7f41880cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.258+0000 7f418ec00640 1 --2- 172.21.15.31:0/4101527681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f41880cc830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4180005fd0 tx=0x7f418002aa00 comp rx=0 tx=0).stop 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 -- 172.21.15.31:0/4101527681 shutdown_connections 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 --2- 172.21.15.31:0/4101527681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f41880cc830 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 --2- 172.21.15.31:0/4101527681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880cf2f0 0x7f418815fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 -- 172.21.15.31:0/4101527681 >> 172.21.15.31:0/4101527681 conn(0x7f418806cb00 msgr2=0x7f418806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:49.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 -- 172.21.15.31:0/4101527681 shutdown_connections 2024-09-13T08:37:49.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.259+0000 7f418ec00640 1 -- 172.21.15.31:0/4101527681 wait complete. 2024-09-13T08:37:49.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.260+0000 7f418ec00640 1 Processor -- start 2024-09-13T08:37:49.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.260+0000 7f418ec00640 1 -- start start 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880caa00 0x7f41880b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 0x7f41880c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41880ce3c0 con 0x7f41880cc3e0 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41880ce240 con 0x7f41880cf2f0 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.261+0000 7f418ec00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f41880ce540 con 0x7f41880caa00 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57874/0 (socket says 172.21.15.31:57874) 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 -- 172.21.15.31:0/2969200695 learned_addr learned my addr 172.21.15.31:0/2969200695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418d800640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880caa00 0x7f41880b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418e200640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 0x7f41880c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 -- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880caa00 msgr2=0x7f41880b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880caa00 0x7f41880b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 -- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 msgr2=0x7f41880c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 0x7f41880c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 -- 172.21.15.31:0/2969200695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41880b30c0 con 0x7f41880cc3e0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418e200640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 0x7f41880c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f418ce00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f417400d700 tx=0x7f417400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174017070 con 0x7f41880cc3e0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4174005310 con 0x7f41880cc3e0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.262+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174009820 con 0x7f41880cc3e0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.263+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41880b33a0 con 0x7f41880cc3e0 2024-09-13T08:37:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.263+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41880b3960 con 0x7f41880cc3e0 2024-09-13T08:37:49.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.264+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41880cc830 con 0x7f41880cc3e0 2024-09-13T08:37:49.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.265+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f41740099c0 con 0x7f41880cc3e0 2024-09-13T08:37:49.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.266+0000 7f4185600640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 0x7f416807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.266+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5738+0+0 (secure 0 0 0) 0x7f417409a9a0 con 0x7f41880cc3e0 2024-09-13T08:37:49.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.266+0000 7f418d800640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 0x7f416807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.267+0000 7f418d800640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 0x7f416807ad10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f417c004f60 tx=0x7f417c007e30 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:49.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.270+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4174062990 con 0x7f41880cc3e0 2024-09-13T08:37:49.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.469+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41880707a0 con 0x7f41880cc3e0 2024-09-13T08:37:49.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.470+0000 7f4185600640 1 -- 172.21.15.31:0/2969200695 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16670 (secure 0 0 0) 0x7f41740673c0 con 0x7f41880cc3e0 2024-09-13T08:37:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:49.471 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:37:49.149419+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:07.473068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:08.401008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2a","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 msgr2=0x7f416807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 0x7f416807ad10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f417c004f60 tx=0x7f417c007e30 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 msgr2=0x7f418814cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f417400d700 tx=0x7f417400dbc0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 shutdown_connections 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41880cf2f0 0x7f41880c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41880cc3e0 0x7f418814cf80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4168078850 0x7f416807ad10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 --2- 172.21.15.31:0/2969200695 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41880caa00 0x7f41880b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 >> 172.21.15.31:0/2969200695 conn(0x7f418806cb00 msgr2=0x7f41880cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 shutdown_connections 2024-09-13T08:37:49.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.473+0000 7f418ec00640 1 -- 172.21.15.31:0/2969200695 wait complete. 2024-09-13T08:37:49.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.652+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3937023286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 msgr2=0x7fb7e0166a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.652+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3937023286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e0166a70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8005fd0 tx=0x7fb7d802a840 comp rx=0 tx=0).stop 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.653+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3937023286 shutdown_connections 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.653+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3937023286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e0166a70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.653+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3937023286 >> 172.21.15.31:0/3937023286 conn(0x7fb7e00d0f70 msgr2=0x7fb7e00d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.653+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3937023286 shutdown_connections 2024-09-13T08:37:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.653+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3937023286 wait complete. 2024-09-13T08:37:49.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.654+0000 7fb7e5c00640 1 Processor -- start 2024-09-13T08:37:49.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.654+0000 7fb7e5c00640 1 -- start start 2024-09-13T08:37:49.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e00b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7e00d05c0 0x7fb7e00b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0167130 con 0x7fb7e01419d0 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0166fb0 con 0x7fb7e006d210 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.655+0000 7fb7e5c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e01672b0 con 0x7fb7e00d05c0 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57886/0 (socket says 172.21.15.31:57886) 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 -- 172.21.15.31:0/3044473169 learned_addr learned my addr 172.21.15.31:0/3044473169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7dea00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e00b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7d6000640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7e00d05c0 0x7fb7e00b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 -- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7e00d05c0 msgr2=0x7fb7e00b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7e00d05c0 0x7fb7e00b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 -- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 msgr2=0x7fb7e00b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e00b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 -- 172.21.15.31:0/3044473169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e00b4530 con 0x7fb7e01419d0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.656+0000 7fb7df400640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8005fd0 tx=0x7fb7d8032cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.657+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8040070 con 0x7fb7e01419d0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.657+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7d8004560 con 0x7fb7e01419d0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.657+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e0139a00 con 0x7fb7e01419d0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.657+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d803c420 con 0x7fb7e01419d0 2024-09-13T08:37:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.657+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e0139fc0 con 0x7fb7e01419d0 2024-09-13T08:37:49.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.658+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e006cb00 con 0x7fb7e01419d0 2024-09-13T08:37:49.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.659+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fb7d8043070 con 0x7fb7e01419d0 2024-09-13T08:37:49.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.660+0000 7fb7d7400640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 0x7fb7bc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:49.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.660+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5738+0+0 (secure 0 0 0) 0x7fb7d80bbab0 con 0x7fb7e01419d0 2024-09-13T08:37:49.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.660+0000 7fb7dea00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 0x7fb7bc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:49.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.661+0000 7fb7dea00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 0x7fb7bc07ad10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0001220 tx=0x7fb7d0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:49.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.664+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb7d80840f0 con 0x7fb7e01419d0 2024-09-13T08:37:49.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.898+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb7e0164fb0 con 0x7fb7e01419d0 2024-09-13T08:37:49.899 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:37:49.899+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:37:49.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.901+0000 7fb7d7400640 1 -- 172.21.15.31:0/3044473169 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fb7d80839f0 con 0x7fb7e01419d0 2024-09-13T08:37:49.902 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:37:49.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 msgr2=0x7fb7bc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 0x7fb7bc07ad10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0001220 tx=0x7fb7d0009040 comp rx=0 tx=0).stop 2024-09-13T08:37:49.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 msgr2=0x7fb7e0151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:49.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8005fd0 tx=0x7fb7d8032cf0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 shutdown_connections 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e01419d0 0x7fb7e0151b00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb7bc078850 0x7fb7bc07ad10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7e00d05c0 0x7fb7e00b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 --2- 172.21.15.31:0/3044473169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7e006d210 0x7fb7e00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 >> 172.21.15.31:0/3044473169 conn(0x7fb7e00d0f70 msgr2=0x7fb7e00c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 shutdown_connections 2024-09-13T08:37:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:49.904+0000 7fb7e5c00640 1 -- 172.21.15.31:0/3044473169 wait complete. 2024-09-13T08:37:49.918 INFO:tasks.ceph.mds.b.smithi045.stderr: -7272> 2024-09-13T08:37:14.978+0000 7f59cda00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:37:49.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:37:49.972 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:37:50.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.088+0000 7f11f7200640 1 -- 172.21.15.31:0/2121398704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11dc003c10 msgr2=0x7f11dc004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:50.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.088+0000 7f11f7200640 1 --2- 172.21.15.31:0/2121398704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11dc003c10 0x7f11dc004010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11e8001830 tx=0x7f11dc005830 comp rx=0 tx=0).stop 2024-09-13T08:37:50.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.089+0000 7f11f6800640 1 -- 172.21.15.31:0/2121398704 reap_dead start 2024-09-13T08:37:50.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.089+0000 7f11f7200640 1 -- 172.21.15.31:0/2121398704 shutdown_connections 2024-09-13T08:37:50.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.089+0000 7f11f7200640 1 -- 172.21.15.31:0/2121398704 >> 172.21.15.31:0/2121398704 conn(0x7f11f80d3e20 msgr2=0x7f11f80d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:50.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.089+0000 7f11f7200640 1 -- 172.21.15.31:0/2121398704 shutdown_connections 2024-09-13T08:37:50.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.089+0000 7f11f7200640 1 -- 172.21.15.31:0/2121398704 wait complete. 2024-09-13T08:37:50.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.090+0000 7f11f7200640 1 Processor -- start 2024-09-13T08:37:50.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.090+0000 7f11f7200640 1 -- start start 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc003c10 0x7f11f80b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f11f80c9fa0 0x7f11f81f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f11f80cf3d0 con 0x7f11f80b8a70 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11f80cf250 con 0x7f11dc003c10 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.091+0000 7f11f7200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f11f80b4710 con 0x7f11f80c9fa0 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11ed400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f11f80c9fa0 0x7f11f81f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc003c10 0x7f11f80b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57908/0 (socket says 172.21.15.31:57908) 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 -- 172.21.15.31:0/1993696726 learned_addr learned my addr 172.21.15.31:0/1993696726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 -- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f11f80c9fa0 msgr2=0x7f11f81f8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f11f80c9fa0 0x7f11f81f8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 -- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc003c10 msgr2=0x7f11f80b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc003c10 0x7f11f80b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 -- 172.21.15.31:0/1993696726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f80b6c50 con 0x7f11f80b8a70 2024-09-13T08:37:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.092+0000 7f11f6800640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f11f815e4c0 tx=0x7f11f0006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.093+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f000fa80 con 0x7f11f80b8a70 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.093+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11f000fc20 con 0x7f11f80b8a70 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.093+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0011430 con 0x7f11f80b8a70 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.093+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f80b6ef0 con 0x7f11f80b8a70 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.093+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f80b74b0 con 0x7f11f80b8a70 2024-09-13T08:37:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.094+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f80b4000 con 0x7f11f80b8a70 2024-09-13T08:37:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.095+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f11f00115d0 con 0x7f11f80b8a70 2024-09-13T08:37:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.096+0000 7f11eea00640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 0x7f11d407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.096+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5824+0+0 (secure 0 0 0) 0x7f11f009b270 con 0x7f11f80b8a70 2024-09-13T08:37:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.097+0000 7f11f5e00640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 0x7f11d407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.097+0000 7f11f5e00640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 0x7f11d407ad10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f11e8001830 tx=0x7f11e8001360 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.100+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f11f00637f0 con 0x7f11f80b8a70 2024-09-13T08:37:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.335+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f11f80b2120 con 0x7f11f80b8a70 2024-09-13T08:37:50.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.909+0000 7f11eea00640 1 -- 172.21.15.31:0/1993696726 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f11f00630f0 con 0x7f11f80b8a70 2024-09-13T08:37:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 msgr2=0x7f11d407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 0x7f11d407ad10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f11e8001830 tx=0x7f11e8001360 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 msgr2=0x7f11f814fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f11f815e4c0 tx=0x7f11f0006660 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 shutdown_connections 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11f80b8a70 0x7f11f814fd70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f11d4078850 0x7f11d407ad10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f11f80c9fa0 0x7f11f81f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 --2- 172.21.15.31:0/1993696726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11dc003c10 0x7f11f80b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 >> 172.21.15.31:0/1993696726 conn(0x7f11f80d3e20 msgr2=0x7f11f80b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.912+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 shutdown_connections 2024-09-13T08:37:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:50.913+0000 7f11f7200640 1 -- 172.21.15.31:0/1993696726 wait complete. 2024-09-13T08:37:50.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.099+0000 7fa9a7e00640 1 -- 172.21.15.31:0/3023809527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 msgr2=0x7fa988004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.099+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/3023809527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa988004c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c310 tx=0x7fa988005830 comp rx=0 tx=0).stop 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.100+0000 7fa9a7e00640 1 -- 172.21.15.31:0/3023809527 shutdown_connections 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.100+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/3023809527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa988004c80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.100+0000 7fa9a7e00640 1 -- 172.21.15.31:0/3023809527 >> 172.21.15.31:0/3023809527 conn(0x7fa9a00d0440 msgr2=0x7fa9a00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.100+0000 7fa9a7e00640 1 -- 172.21.15.31:0/3023809527 shutdown_connections 2024-09-13T08:37:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.100+0000 7fa9a7e00640 1 -- 172.21.15.31:0/3023809527 wait complete. 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.101+0000 7fa9a7e00640 1 Processor -- start 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.101+0000 7fa9a7e00640 1 -- start start 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 0x7fa9a0157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9a00af120 0x7fa9a0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0159c80 con 0x7fa988004800 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a015c420 con 0x7fa9a00b1540 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.102+0000 7fa9a7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a0159e00 con 0x7fa9a00af120 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57928/0 (socket says 172.21.15.31:57928) 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 -- 172.21.15.31:0/1274719414 learned_addr learned my addr 172.21.15.31:0/1274719414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9a00af120 0x7fa9a0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a4a00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 0x7fa9a0157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 -- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9a00af120 msgr2=0x7fa9a0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9a00af120 0x7fa9a0157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 -- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 msgr2=0x7fa9a0157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 0x7fa9a0157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 -- 172.21.15.31:0/1274719414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a00c65a0 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a4a00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 0x7fa9a0157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa9a5400640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa994011a10 tx=0x7fa994011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.103+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99401e070 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.104+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa994004460 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.104+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99400fba0 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.104+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a00c5450 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.104+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a00c4c30 con 0x7fa988004800 2024-09-13T08:37:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.105+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a00cc6e0 con 0x7fa988004800 2024-09-13T08:37:51.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.106+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fa994021070 con 0x7fa988004800 2024-09-13T08:37:51.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.107+0000 7fa99d600640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 0x7fa98007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:51.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.107+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5824+0+0 (secure 0 0 0) 0x7fa99409b2f0 con 0x7fa988004800 2024-09-13T08:37:51.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.108+0000 7fa9a4a00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 0x7fa98007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:51.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.108+0000 7fa9a4a00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 0x7fa98007ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa98c033000 tx=0x7fa98c001a60 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.111+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa9a00cc6e0 con 0x7fa988004800 2024-09-13T08:37:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:51.308+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa9a006de30 con 0x7fa988004800 2024-09-13T08:37:52.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.166+0000 7fa99d600640 1 -- 172.21.15.31:0/1274719414 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v27) ==== 181+0+0 (secure 0 0 0) 0x7fa994063220 con 0x7fa988004800 2024-09-13T08:37:52.168 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.170+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 msgr2=0x7fa98007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 0x7fa98007ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa98c033000 tx=0x7fa98c001a60 comp rx=0 tx=0).stop 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 msgr2=0x7fa9a01f4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa994011a10 tx=0x7fa994011ed0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 shutdown_connections 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa980078880 0x7fa98007ad40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9a00af120 0x7fa9a0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9a00b1540 0x7fa9a0157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 --2- 172.21.15.31:0/1274719414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa988004800 0x7fa9a01f4320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 >> 172.21.15.31:0/1274719414 conn(0x7fa9a00d0440 msgr2=0x7fa9a015a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:52.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 shutdown_connections 2024-09-13T08:37:52.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.171+0000 7fa9a7e00640 1 -- 172.21.15.31:0/1274719414 wait complete. 2024-09-13T08:37:52.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:37:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 -- 172.21.15.31:0/3502123017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754015f040 msgr2=0x7f75400cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 --2- 172.21.15.31:0/3502123017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754015f040 0x7f75400cb320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f752c005fd0 tx=0x7f752c02a620 comp rx=0 tx=0).stop 2024-09-13T08:37:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 -- 172.21.15.31:0/3502123017 shutdown_connections 2024-09-13T08:37:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 --2- 172.21.15.31:0/3502123017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754015f040 0x7f75400cb320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 --2- 172.21.15.31:0/3502123017 >> v2:172.21.15.45:3300/0 conn(0x7f75400aff80 0x7f75400ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 -- 172.21.15.31:0/3502123017 >> 172.21.15.31:0/3502123017 conn(0x7f75400d0cd0 msgr2=0x7f75400d10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 -- 172.21.15.31:0/3502123017 shutdown_connections 2024-09-13T08:37:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.358+0000 7f7546200640 1 -- 172.21.15.31:0/3502123017 wait complete. 2024-09-13T08:37:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.360+0000 7f7546200640 1 Processor -- start 2024-09-13T08:37:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.360+0000 7f7546200640 1 -- start start 2024-09-13T08:37:52.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75400aff80 0x7f75400b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:52.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f754015f040 0x7f75400c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f754006cc80 con 0x7f75400cb860 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f754006cb00 con 0x7f75400aff80 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f7546200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f754006ce00 con 0x7f754015f040 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57942/0 (socket says 172.21.15.31:57942) 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753ea00640 1 -- 172.21.15.31:0/1153717709 learned_addr learned my addr 172.21.15.31:0/1153717709 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753f400640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75400aff80 0x7f75400b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753fe00640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f754015f040 0x7f75400c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.361+0000 7f753ea00640 1 -- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f754015f040 msgr2=0x7f75400c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f753ea00640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f754015f040 0x7f75400c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f753ea00640 1 -- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75400aff80 msgr2=0x7f75400b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f753ea00640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75400aff80 0x7f75400b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f753ea00640 1 -- 172.21.15.31:0/1153717709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75400b6ae0 con 0x7f75400cb860 2024-09-13T08:37:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f753ea00640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f752c034040 tx=0x7f752c02ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c03e030 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f752c004b20 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c035040 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75401f4dd0 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.362+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75401f5390 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.363+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754006d2b0 con 0x7f75400cb860 2024-09-13T08:37:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.364+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f752c041030 con 0x7f75400cb860 2024-09-13T08:37:52.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.365+0000 7f7537400640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 0x7f752007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:52.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.366+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5386+0+0 (secure 0 0 0) 0x7f752c0babf0 con 0x7f75400cb860 2024-09-13T08:37:52.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.366+0000 7f753f400640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 0x7f752007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.367+0000 7f753f400640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 0x7f752007ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753000bbe0 tx=0x7f753000b390 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:52.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.369+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f752c082cb0 con 0x7f75400cb860 2024-09-13T08:37:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:52.564+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f75400cd4c0 con 0x7f75400cb860 2024-09-13T08:37:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.171+0000 7f7537400640 1 -- 172.21.15.31:0/1153717709 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v28) ==== 169+0+0 (secure 0 0 0) 0x7f752c0876e0 con 0x7f75400cb860 2024-09-13T08:37:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:37:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.175+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 msgr2=0x7f752007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.175+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 0x7f752007ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753000bbe0 tx=0x7f753000b390 comp rx=0 tx=0).stop 2024-09-13T08:37:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.175+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 msgr2=0x7f75400b64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.175+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f752c034040 tx=0x7f752c02ae70 comp rx=0 tx=0).stop 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 shutdown_connections 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7520078880 0x7f752007ad40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f754015f040 0x7f75400c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75400cb860 0x7f75400b64e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 --2- 172.21.15.31:0/1153717709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75400aff80 0x7f75400b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 >> 172.21.15.31:0/1153717709 conn(0x7f75400d0cd0 msgr2=0x7f75400d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 shutdown_connections 2024-09-13T08:37:53.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.176+0000 7f7546200640 1 -- 172.21.15.31:0/1153717709 wait complete. 2024-09-13T08:37:53.199 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:37:53.199 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:37:53.200 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:37:53.200 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:37:53.200 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:37:53.200 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:37:53.200 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:37:53.200 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:37:53.200 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:37:53.201 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:37:53.201 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:37:53.201 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:37:53.201 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:37:53.201 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:37:53.201 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:37:53.201 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:37:53.217 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:37:53.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:37:53.218 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:37:53.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:37:53.219 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:37:53.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:37:53.220 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:37:53.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:37:53.263 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:37:53.263 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:53.264 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:37:53.264 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:53.266 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:37:53.266 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:53.266 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:37:53.267 DEBUG:teuthology.parallel:result is None 2024-09-13T08:37:53.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:37:53.399 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:37:53.399 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:37:53.400 INFO:tasks.ceph.mds.d.smithi045.stderr:2024-09-13T08:37:53.399+0000 7faffe400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-13T08:37:53.407 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:37:53.408 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 -- 172.21.15.31:0/620004897 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f0070820 msgr2=0x7fa0f00b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 --2- 172.21.15.31:0/620004897 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f0070820 0x7fa0f00b3dc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000a940 tx=0x7fa0e002b850 comp rx=0 tx=0).stop 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 -- 172.21.15.31:0/620004897 shutdown_connections 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 --2- 172.21.15.31:0/620004897 >> v2:172.21.15.45:3300/0 conn(0x7fa0f00700d0 0x7fa0f0153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 --2- 172.21.15.31:0/620004897 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f0070820 0x7fa0f00b3dc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 -- 172.21.15.31:0/620004897 >> 172.21.15.31:0/620004897 conn(0x7fa0f00b18f0 msgr2=0x7fa0f00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 -- 172.21.15.31:0/620004897 shutdown_connections 2024-09-13T08:37:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.443+0000 7fa0f5200640 1 -- 172.21.15.31:0/620004897 wait complete. 2024-09-13T08:37:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.445+0000 7fa0f5200640 1 Processor -- start 2024-09-13T08:37:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.445+0000 7fa0f5200640 1 -- start start 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f00700d0 0x7fa0f009d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 0x7fa0f01f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 0x7fa0f0152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f0154840 con 0x7fa0f00ccba0 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f01546c0 con 0x7fa0f0070820 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0f5200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f01549c0 con 0x7fa0f00700d0 2024-09-13T08:37:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 0x7fa0f0152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 0x7fa0f0152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57990/0 (socket says 172.21.15.31:57990) 2024-09-13T08:37:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ed400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 0x7fa0f01f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ee800640 1 -- 172.21.15.31:0/1344547161 learned_addr learned my addr 172.21.15.31:0/1344547161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ede00640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f00700d0 0x7fa0f009d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ed400640 1 -- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f00700d0 msgr2=0x7fa0f009d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.446+0000 7fa0ed400640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f00700d0 0x7fa0f009d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0ed400640 1 -- 172.21.15.31:0/1344547161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 msgr2=0x7fa0f0152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0ed400640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 0x7fa0f0152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0ed400640 1 -- 172.21.15.31:0/1344547161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f009d760 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0ed400640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 0x7fa0f01f9820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c3c0 tx=0x7fa0dc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc011790 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0dc017070 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc011ce0 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01f9ef0 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.447+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f01fa4b0 con 0x7fa0f0070820 2024-09-13T08:37:53.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.448+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f0153ef0 con 0x7fa0f0070820 2024-09-13T08:37:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.450+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fa0dc026430 con 0x7fa0f0070820 2024-09-13T08:37:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.451+0000 7fa0e6000640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 0x7fa0d007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.451+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4943+0+0 (secure 0 0 0) 0x7fa0dc09aca0 con 0x7fa0f0070820 2024-09-13T08:37:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.451+0000 7fa0ede00640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 0x7fa0d007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.452+0000 7fa0ede00640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 0x7fa0d007ad10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0036000 tx=0x7fa0e0004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:53.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.454+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa0dc062fa0 con 0x7fa0f0070820 2024-09-13T08:37:53.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:53.661+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa0f00749f0 con 0x7fa0f0070820 2024-09-13T08:37:54.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.190+0000 7fa0e6000640 1 -- 172.21.15.31:0/1344547161 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) ==== 97+0+0 (secure 0 0 0) 0x7fa0dc0679d0 con 0x7fa0f0070820 2024-09-13T08:37:54.192 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 msgr2=0x7fa0d007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 0x7fa0d007ad10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0036000 tx=0x7fa0e0004730 comp rx=0 tx=0).stop 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 msgr2=0x7fa0f01f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 0x7fa0f01f9820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c3c0 tx=0x7fa0dc00c880 comp rx=0 tx=0).stop 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 shutdown_connections 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0f00ccba0 0x7fa0f0152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f0070820 0x7fa0f01f9820 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa0d0078850 0x7fa0d007ad10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 --2- 172.21.15.31:0/1344547161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0f00700d0 0x7fa0f009d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 >> 172.21.15.31:0/1344547161 conn(0x7fa0f00b18f0 msgr2=0x7fa0f00cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:54.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 shutdown_connections 2024-09-13T08:37:54.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.193+0000 7fa0f5200640 1 -- 172.21.15.31:0/1344547161 wait complete. 2024-09-13T08:37:54.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:37:54.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 -- 172.21.15.31:0/82131692 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0cc720 msgr2=0x7fafcc06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 --2- 172.21.15.31:0/82131692 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0cc720 0x7fafcc06f9b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009830 tx=0x7fafc002a530 comp rx=0 tx=0).stop 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 -- 172.21.15.31:0/82131692 shutdown_connections 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 --2- 172.21.15.31:0/82131692 >> v2:172.21.15.45:3300/0 conn(0x7fafcc0b52d0 0x7fafcc152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 --2- 172.21.15.31:0/82131692 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0cc720 0x7fafcc06f9b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 -- 172.21.15.31:0/82131692 >> 172.21.15.31:0/82131692 conn(0x7fafcc06cb00 msgr2=0x7fafcc06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 -- 172.21.15.31:0/82131692 shutdown_connections 2024-09-13T08:37:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.374+0000 7fafd3000640 1 -- 172.21.15.31:0/82131692 wait complete. 2024-09-13T08:37:54.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.376+0000 7fafd3000640 1 Processor -- start 2024-09-13T08:37:54.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.376+0000 7fafd3000640 1 -- start start 2024-09-13T08:37:54.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0b52d0 0x7fafcc14f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafcc0cc720 0x7fafcc14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc152bc0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc070470 con 0x7fafcc0cc720 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd3000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fafcc152d40 con 0x7fafcc0b52d0 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58000/0 (socket says 172.21.15.31:58000) 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.377+0000 7fafd2600640 1 -- 172.21.15.31:0/2841453271 learned_addr learned my addr 172.21.15.31:0/2841453271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd1c00640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0b52d0 0x7fafcc14f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd1200640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafcc0cc720 0x7fafcc14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 -- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0b52d0 msgr2=0x7fafcc14f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0b52d0 0x7fafcc14f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 -- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafcc0cc720 msgr2=0x7fafcc14cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafcc0cc720 0x7fafcc14cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 -- 172.21.15.31:0/2841453271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc154be0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd2600640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafc400c3c0 tx=0x7fafc400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:54.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc401b070 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc400b630 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400ba60 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc1f6070 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.378+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc1f6630 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.380+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafcc153260 con 0x7fafcc0ccb00 2024-09-13T08:37:54.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.381+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fafc401e070 con 0x7fafcc0ccb00 2024-09-13T08:37:54.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.381+0000 7fafc9600640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 0x7fafac07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.382+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7fafc409acb0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.382+0000 7fafd1c00640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 0x7fafac07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.383+0000 7fafd1c00640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 0x7fafac07ad10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafc0034000 tx=0x7fafc0004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:54.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.386+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fafc40635e0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.649+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fafcc0c98d0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.649+0000 7fafc9600640 1 -- 172.21.15.31:0/2841453271 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) ==== 80+0+6645 (secure 0 0 0) 0x7fafc4062ee0 con 0x7fafcc0ccb00 2024-09-13T08:37:54.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:54.651 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:37:54.652 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.653 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.655 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.658 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:37:54.662 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:37:54.663 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-mds", 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNmqh2FKxAANCTxU1ccvKgIvnEEjBVh+Q==", 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-mds" 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-mgr", 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNm1CiFKxAAezV0STlXxD3wgq+WnxNS7w==", 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-13T08:37:54.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-osd", 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNmuTGFKxAAC4CFFuQmfqBA1GrC2f9Fig==", 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-osd" 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-rbd", 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNmeTqFKxAAHxNOUxDfPjjhmPMmjS+iDw==", 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-13T08:37:54.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNmtUKFKxAAtKBLaZvnw7R9epaq6a1XDA==", 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.bootstrap-rgw", 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCb+eNmJkuFKxAAmvNjfohIBHMOcwbnk6kEeg==", 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-13T08:37:54.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:37:54.668 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 msgr2=0x7fafac07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 0x7fafac07ad10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafc0034000 tx=0x7fafc0004730 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 msgr2=0x7fafcc156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafc400c3c0 tx=0x7fafc400c880 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 shutdown_connections 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafcc0ccb00 0x7fafcc156070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafcc0cc720 0x7fafcc14cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafac078850 0x7fafac07ad10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 --2- 172.21.15.31:0/2841453271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafcc0b52d0 0x7fafcc14f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 >> 172.21.15.31:0/2841453271 conn(0x7fafcc06cb00 msgr2=0x7fafcc06f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 shutdown_connections 2024-09-13T08:37:54.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.652+0000 7fafd3000640 1 -- 172.21.15.31:0/2841453271 wait complete. 2024-09-13T08:37:54.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-13T08:37:54.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 -- 172.21.15.31:0/3915475399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c4073ee0 msgr2=0x7fe3c4074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 --2- 172.21.15.31:0/3915475399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c4073ee0 0x7fe3c4074680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4073760 tx=0x7fe3b002a530 comp rx=0 tx=0).stop 2024-09-13T08:37:54.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 -- 172.21.15.31:0/3915475399 shutdown_connections 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 --2- 172.21.15.31:0/3915475399 >> v2:172.21.15.45:3300/0 conn(0x7fe3c40cea00 0x7fe3c4070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 --2- 172.21.15.31:0/3915475399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c4073ee0 0x7fe3c4074680 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 -- 172.21.15.31:0/3915475399 >> 172.21.15.31:0/3915475399 conn(0x7fe3c40d0e30 msgr2=0x7fe3c40d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 -- 172.21.15.31:0/3915475399 shutdown_connections 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.834+0000 7fe3ca800640 1 -- 172.21.15.31:0/3915475399 wait complete. 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.836+0000 7fe3ca800640 1 Processor -- start 2024-09-13T08:37:54.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.836+0000 7fe3ca800640 1 -- start start 2024-09-13T08:37:54.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 0x7fe3c41f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe3c40cea00 0x7fe3c40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4071030 con 0x7fe3c415e0c0 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c415ea70 con 0x7fe3c4073ee0 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3ca800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c40711b0 con 0x7fe3c40cea00 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 0x7fe3c41f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47620/0 (socket says 172.21.15.31:47620) 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c9e00640 1 -- 172.21.15.31:0/303450853 learned_addr learned my addr 172.21.15.31:0/303450853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 0x7fe3c41f5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:39736/0 (socket says 172.21.15.31:39736) 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.837+0000 7fe3c8a00640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe3c40cea00 0x7fe3c40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 -- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe3c40cea00 msgr2=0x7fe3c40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe3c40cea00 0x7fe3c40b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 -- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 msgr2=0x7fe3c41f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 0x7fe3c41f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:54.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 -- 172.21.15.31:0/303450853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c41f6a00 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c9e00640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00eaa0 tx=0x7fe3bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00cdb0 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3bc004c10 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c41f61c0 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc005020 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.838+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c41f2630 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.840+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c40716a0 con 0x7fe3c415e0c0 2024-09-13T08:37:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.841+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fe3bc0051c0 con 0x7fe3c415e0c0 2024-09-13T08:37:54.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.842+0000 7fe3c1600640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 0x7fe3a407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:54.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.842+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7fe3bc09ae50 con 0x7fe3c415e0c0 2024-09-13T08:37:54.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.842+0000 7fe3c9400640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 0x7fe3a407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:54.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.843+0000 7fe3c9400640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 0x7fe3a407ad10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8006fd0 tx=0x7fe3b8009060 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:54.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:54.846+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe3c41f61c0 con 0x7fe3c415e0c0 2024-09-13T08:37:55.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.114+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fe3c40b6db0 con 0x7fe3c415e0c0 2024-09-13T08:37:55.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.117+0000 7fe3c1600640 1 -- 172.21.15.31:0/303450853 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) ==== 90+0+0 (secure 0 0 0) 0x7fe3bc016030 con 0x7fe3c415e0c0 2024-09-13T08:37:55.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.119+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 msgr2=0x7fe3a407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.119+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 0x7fe3a407ad10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8006fd0 tx=0x7fe3b8009060 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.119+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 msgr2=0x7fe3c40cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.119+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00eaa0 tx=0x7fe3bc00ef60 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.119+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 shutdown_connections 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3c415e0c0 0x7fe3c40cf070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe3a4078850 0x7fe3a407ad10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe3c40cea00 0x7fe3c40b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 --2- 172.21.15.31:0/303450853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3c4073ee0 0x7fe3c41f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 >> 172.21.15.31:0/303450853 conn(0x7fe3c40d0e30 msgr2=0x7fe3c415a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 shutdown_connections 2024-09-13T08:37:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.120+0000 7fe3ca800640 1 -- 172.21.15.31:0/303450853 wait complete. 2024-09-13T08:37:55.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 -- 172.21.15.31:0/2230578458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f380051c0 msgr2=0x7f8f3800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 --2- 172.21.15.31:0/2230578458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f380051c0 0x7f8f3800b9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8f48001e40 tx=0x7f8f38005830 comp rx=0 tx=0).stop 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 -- 172.21.15.31:0/2230578458 shutdown_connections 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 --2- 172.21.15.31:0/2230578458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f380051c0 0x7f8f3800b9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 -- 172.21.15.31:0/2230578458 >> 172.21.15.31:0/2230578458 conn(0x7f8f500d3f50 msgr2=0x7f8f500d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 -- 172.21.15.31:0/2230578458 shutdown_connections 2024-09-13T08:37:55.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.301+0000 7f8f56600640 1 -- 172.21.15.31:0/2230578458 wait complete. 2024-09-13T08:37:55.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.302+0000 7f8f56600640 1 Processor -- start 2024-09-13T08:37:55.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 -- start start 2024-09-13T08:37:55.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f380051c0 0x7f8f5015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f501543d0 0x7f8f501f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500d0b20 con 0x7f8f501f7ff0 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500d09a0 con 0x7f8f501543d0 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.303+0000 7f8f56600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8f500d1110 con 0x7f8f380051c0 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47644/0 (socket says 172.21.15.31:47644) 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 -- 172.21.15.31:0/1991103359 learned_addr learned my addr 172.21.15.31:0/1991103359 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55200640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f380051c0 0x7f8f5015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f4fe00640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f501543d0 0x7f8f501f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 -- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f380051c0 msgr2=0x7f8f5015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f380051c0 0x7f8f5015b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 -- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f501543d0 msgr2=0x7f8f501f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f501543d0 0x7f8f501f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.304+0000 7f8f55c00640 1 -- 172.21.15.31:0/1991103359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5015a180 con 0x7f8f501f7ff0 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f55c00640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8f4802ad70 tx=0x7f8f48006280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48006c40 con 0x7f8f501f7ff0 2024-09-13T08:37:55.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f48006600 con 0x7f8f501f7ff0 2024-09-13T08:37:55.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5015a460 con 0x7f8f501f7ff0 2024-09-13T08:37:55.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48001a60 con 0x7f8f501f7ff0 2024-09-13T08:37:55.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.305+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5015e9c0 con 0x7f8f501f7ff0 2024-09-13T08:37:55.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.306+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5015e7e0 con 0x7f8f501f7ff0 2024-09-13T08:37:55.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.307+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f8f48033460 con 0x7f8f501f7ff0 2024-09-13T08:37:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.309+0000 7f8f4cc00640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 0x7f8f3007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.309+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7f8f480ba420 con 0x7f8f501f7ff0 2024-09-13T08:37:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.309+0000 7f8f55200640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 0x7f8f3007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.310+0000 7f8f55200640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 0x7f8f3007ad40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8f501592e0 tx=0x7f8f44005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:55.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.313+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8f5015e7e0 con 0x7f8f501f7ff0 2024-09-13T08:37:55.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.585+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f8f500cea60 con 0x7f8f501f7ff0 2024-09-13T08:37:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.588+0000 7f8f4cc00640 1 -- 172.21.15.31:0/1991103359 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7f8f48044060 con 0x7f8f501f7ff0 2024-09-13T08:37:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 msgr2=0x7f8f3007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 0x7f8f3007ad40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8f501592e0 tx=0x7f8f44005f90 comp rx=0 tx=0).stop 2024-09-13T08:37:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 msgr2=0x7f8f501f8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8f4802ad70 tx=0x7f8f48006280 comp rx=0 tx=0).stop 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 shutdown_connections 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501f7ff0 0x7f8f501f8470 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f501543d0 0x7f8f501f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8f30078880 0x7f8f3007ad40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 --2- 172.21.15.31:0/1991103359 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f380051c0 0x7f8f5015b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 >> 172.21.15.31:0/1991103359 conn(0x7f8f500d3f50 msgr2=0x7f8f500c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 shutdown_connections 2024-09-13T08:37:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.590+0000 7f8f56600640 1 -- 172.21.15.31:0/1991103359 wait complete. 2024-09-13T08:37:55.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.767+0000 7f821b200640 1 -- 172.21.15.31:0/1921507031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f821415be30 msgr2=0x7f82140cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.767+0000 7f821b200640 1 --2- 172.21.15.31:0/1921507031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f821415be30 0x7f82140cdb80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8204005fd0 tx=0x7f820402aaa0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 -- 172.21.15.31:0/1921507031 shutdown_connections 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 --2- 172.21.15.31:0/1921507031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 0x7f82140ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 --2- 172.21.15.31:0/1921507031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f821415be30 0x7f82140cdb80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 -- 172.21.15.31:0/1921507031 >> 172.21.15.31:0/1921507031 conn(0x7f82140b18f0 msgr2=0x7f82140b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 -- 172.21.15.31:0/1921507031 shutdown_connections 2024-09-13T08:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.768+0000 7f821b200640 1 -- 172.21.15.31:0/1921507031 wait complete. 2024-09-13T08:37:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.769+0000 7f821b200640 1 Processor -- start 2024-09-13T08:37:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.769+0000 7f821b200640 1 -- start start 2024-09-13T08:37:55.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 0x7f821409d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f821415be30 0x7f821409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82140733e0 con 0x7f82140ce600 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8214160160 con 0x7f821415be30 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f821b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f82141602e0 con 0x7f82140ced20 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.770+0000 7f8213e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47662/0 (socket says 172.21.15.31:47662) 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 -- 172.21.15.31:0/3179886608 learned_addr learned my addr 172.21.15.31:0/3179886608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f820b400640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 0x7f821409d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8219200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f821415be30 0x7f821409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 -- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 msgr2=0x7f821409d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 0x7f821409d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 -- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f821415be30 msgr2=0x7f821409d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f821415be30 0x7f821409d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 -- 172.21.15.31:0/3179886608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821409da90 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.771+0000 7f8213e00640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8204006010 tx=0x7f8204003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.772+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204044070 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.772+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8204004250 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.772+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204004660 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.772+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141492c0 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.772+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8214146390 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.773+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8214160790 con 0x7f82140ce600 2024-09-13T08:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.774+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f8204047070 con 0x7f82140ce600 2024-09-13T08:37:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.775+0000 7f8210c00640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 0x7f81f007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.776+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7f82040bb440 con 0x7f82140ce600 2024-09-13T08:37:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.776+0000 7f820b400640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 0x7f81f007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:55.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.777+0000 7f820b400640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 0x7f81f007ad10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81fc005fd0 tx=0x7f81fc005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:55.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:55.780+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8204032030 con 0x7f82140ce600 2024-09-13T08:37:56.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.048+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f82140b25f0 con 0x7f82140ce600 2024-09-13T08:37:56.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.051+0000 7f8210c00640 1 -- 172.21.15.31:0/3179886608 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f8204083df0 con 0x7f82140ce600 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.053+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 msgr2=0x7f81f007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.053+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 0x7f81f007ad10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81fc005fd0 tx=0x7f81fc005850 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 msgr2=0x7f8214148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8204006010 tx=0x7f8204003ce0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 shutdown_connections 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f821415be30 0x7f821409d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f81f0078850 0x7f81f007ad10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f82140ced20 0x7f821409d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 --2- 172.21.15.31:0/3179886608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82140ce600 0x7f8214148d80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 >> 172.21.15.31:0/3179886608 conn(0x7f82140b18f0 msgr2=0x7f82140af210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 shutdown_connections 2024-09-13T08:37:56.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.054+0000 7f821b200640 1 -- 172.21.15.31:0/3179886608 wait complete. 2024-09-13T08:37:56.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-13T08:37:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 -- 172.21.15.31:0/1634829792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 msgr2=0x7ff7ac0cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/1634829792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac0cc0c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0098e0 tx=0x7ff79c02a670 comp rx=0 tx=0).stop 2024-09-13T08:37:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 -- 172.21.15.31:0/1634829792 shutdown_connections 2024-09-13T08:37:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/1634829792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac0cc0c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 -- 172.21.15.31:0/1634829792 >> 172.21.15.31:0/1634829792 conn(0x7ff7ac0cc7a0 msgr2=0x7ff7ac0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 -- 172.21.15.31:0/1634829792 shutdown_connections 2024-09-13T08:37:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.232+0000 7ff7b0a00640 1 -- 172.21.15.31:0/1634829792 wait complete. 2024-09-13T08:37:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.234+0000 7ff7b0a00640 1 Processor -- start 2024-09-13T08:37:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.234+0000 7ff7b0a00640 1 -- start start 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 0x7ff7ac1f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 0x7ff7ac0c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac0cdca0 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac0cdb20 con 0x7ff7ac0b1df0 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7b0a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac0cde20 con 0x7ff7ac1618f0 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47676/0 (socket says 172.21.15.31:47676) 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 -- 172.21.15.31:0/3707303681 learned_addr learned my addr 172.21.15.31:0/3707303681 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 -- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 msgr2=0x7ff7ac0c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7ab200640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 0x7ff7ac0c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7a3e00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 0x7ff7ac1f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 0x7ff7ac0c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 -- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 msgr2=0x7ff7ac1f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 0x7ff7ac1f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7aa800640 1 -- 172.21.15.31:0/3707303681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac0b3030 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7ab200640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 0x7ff7ac0c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.235+0000 7ff7a3e00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 0x7ff7ac1f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7aa800640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff79c02ac60 tx=0x7ff79c0052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c040070 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff79c004a90 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c045460 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac0c9d20 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.236+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac0b3590 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.237+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ac0ce340 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.238+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7ff79c039040 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.239+0000 7ff7a2a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 0x7ff78807ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.239+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7ff79c0bbaf0 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.240+0000 7ff7a3e00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 0x7ff78807ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.240+0000 7ff7a3e00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 0x7ff78807ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0b4d20 tx=0x7ff794009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.243+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff7ac0b3030 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.515+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7ff7ac161260 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.518+0000 7ff7a2a00640 1 -- 172.21.15.31:0/3707303681 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7ff79c0844a0 con 0x7ff7ac0b1a10 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 msgr2=0x7ff78807ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 0x7ff78807ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0b4d20 tx=0x7ff794009040 comp rx=0 tx=0).stop 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 msgr2=0x7ff7ac150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff79c02ac60 tx=0x7ff79c0052f0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 shutdown_connections 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff788078880 0x7ff78807ad40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff7ac1618f0 0x7ff7ac0c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7ac0b1df0 0x7ff7ac1f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 --2- 172.21.15.31:0/3707303681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac0b1a10 0x7ff7ac150860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 >> 172.21.15.31:0/3707303681 conn(0x7ff7ac0cc7a0 msgr2=0x7ff7ac0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 shutdown_connections 2024-09-13T08:37:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.521+0000 7ff7b0a00640 1 -- 172.21.15.31:0/3707303681 wait complete. 2024-09-13T08:37:56.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-13T08:37:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 -- 172.21.15.31:0/1050792434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80af970 msgr2=0x7f32d80688b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 --2- 172.21.15.31:0/1050792434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80af970 0x7f32d80688b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f32d80d16a0 tx=0x7f32c802a520 comp rx=0 tx=0).stop 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 -- 172.21.15.31:0/1050792434 shutdown_connections 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 --2- 172.21.15.31:0/1050792434 >> v2:172.21.15.45:3300/0 conn(0x7f32d806d210 0x7f32d815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 --2- 172.21.15.31:0/1050792434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80af970 0x7f32d80688b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 -- 172.21.15.31:0/1050792434 >> 172.21.15.31:0/1050792434 conn(0x7f32d80b22b0 msgr2=0x7f32d80b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.701+0000 7f32df200640 1 -- 172.21.15.31:0/1050792434 shutdown_connections 2024-09-13T08:37:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.702+0000 7f32df200640 1 -- 172.21.15.31:0/1050792434 wait complete. 2024-09-13T08:37:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.703+0000 7f32df200640 1 Processor -- start 2024-09-13T08:37:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.703+0000 7f32df200640 1 -- start start 2024-09-13T08:37:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f32d806d210 0x7f32d813db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d80af970 0x7f32d80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f32d80cf970 con 0x7f32d80afd50 2024-09-13T08:37:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f32d80cf7f0 con 0x7f32d80af970 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32df200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f32d80cfaf0 con 0x7f32d806d210 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32dd200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.704+0000 7f32dd200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47694/0 (socket says 172.21.15.31:47694) 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 -- 172.21.15.31:0/1926438581 learned_addr learned my addr 172.21.15.31:0/1926438581 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32d7e00640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f32d806d210 0x7f32d813db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32d7400640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d80af970 0x7f32d80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 -- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f32d806d210 msgr2=0x7f32d813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f32d806d210 0x7f32d813db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 -- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d80af970 msgr2=0x7f32d80b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d80af970 0x7f32d80b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 -- 172.21.15.31:0/1926438581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d80ada40 con 0x7f32d80afd50 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32dd200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32d000eaa0 tx=0x7f32d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.705+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d000cdb0 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.706+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32d0004c10 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.706+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0005020 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.706+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d80b48f0 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.706+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d813e070 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.707+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d815fba0 con 0x7f32d80afd50 2024-09-13T08:37:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.708+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f32d00051c0 con 0x7f32d80afd50 2024-09-13T08:37:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.709+0000 7f32cfe00640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 0x7f32b807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.709+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7f32d009af30 con 0x7f32d80afd50 2024-09-13T08:37:56.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.709+0000 7f32d7e00640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 0x7f32b807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.710+0000 7f32d7e00640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 0x7f32b807ad10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f32c8033000 tx=0x7f32c8004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:56.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.713+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f32d0016030 con 0x7f32d80afd50 2024-09-13T08:37:56.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.982+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f32d8144a70 con 0x7f32d80afd50 2024-09-13T08:37:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.985+0000 7f32cfe00640 1 -- 172.21.15.31:0/1926438581 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) ==== 97+0+0 (secure 0 0 0) 0x7f32d0063200 con 0x7f32d80afd50 2024-09-13T08:37:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 msgr2=0x7f32b807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 0x7f32b807ad10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f32c8033000 tx=0x7f32c8004730 comp rx=0 tx=0).stop 2024-09-13T08:37:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 msgr2=0x7f32d815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32d000eaa0 tx=0x7f32d000ef60 comp rx=0 tx=0).stop 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 shutdown_connections 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32d80afd50 0x7f32d815a7e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d80af970 0x7f32d80b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f32b8078850 0x7f32b807ad10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 --2- 172.21.15.31:0/1926438581 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f32d806d210 0x7f32d813db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 >> 172.21.15.31:0/1926438581 conn(0x7f32d80b22b0 msgr2=0x7f32d80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 shutdown_connections 2024-09-13T08:37:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:56.987+0000 7f32df200640 1 -- 172.21.15.31:0/1926438581 wait complete. 2024-09-13T08:37:57.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 -- 172.21.15.31:0/2064737716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f302415e160 msgr2=0x7f30240d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 --2- 172.21.15.31:0/2064737716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f302415e160 0x7f30240d0790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f30240c79d0 tx=0x7f301402a540 comp rx=0 tx=0).stop 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 -- 172.21.15.31:0/2064737716 shutdown_connections 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 --2- 172.21.15.31:0/2064737716 >> v2:172.21.15.45:3300/0 conn(0x7f30240cfa70 0x7f30240d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 --2- 172.21.15.31:0/2064737716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f302415e160 0x7f30240d0790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 -- 172.21.15.31:0/2064737716 >> 172.21.15.31:0/2064737716 conn(0x7f30240d3f50 msgr2=0x7f30240d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:57.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 -- 172.21.15.31:0/2064737716 shutdown_connections 2024-09-13T08:37:57.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.164+0000 7f3028a00640 1 -- 172.21.15.31:0/2064737716 wait complete. 2024-09-13T08:37:57.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.165+0000 7f3028a00640 1 Processor -- start 2024-09-13T08:37:57.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 -- start start 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30240cfa70 0x7f3024160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f302415e160 0x7f30241f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f30240c8bd0 con 0x7f30240d0cd0 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f30240c8a50 con 0x7f302415e160 2024-09-13T08:37:57.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.166+0000 7f3028a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f30240c8d50 con 0x7f30240cfa70 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47708/0 (socket says 172.21.15.31:47708) 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 -- 172.21.15.31:0/1267796596 learned_addr learned my addr 172.21.15.31:0/1267796596 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3023400640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30240cfa70 0x7f3024160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3023e00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f302415e160 0x7f30241f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 -- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30240cfa70 msgr2=0x7f3024160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30240cfa70 0x7f3024160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 -- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f302415e160 msgr2=0x7f30241f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f302415e160 0x7f30241f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.167+0000 7f3022a00640 1 -- 172.21.15.31:0/1267796596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3024160940 con 0x7f30240d0cd0 2024-09-13T08:37:57.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f3022a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f301000c450 tx=0x7f301000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:57.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010016020 con 0x7f30240d0cd0 2024-09-13T08:37:57.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3010012920 con 0x7f30240d0cd0 2024-09-13T08:37:57.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010012d70 con 0x7f30240d0cd0 2024-09-13T08:37:57.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30240cb090 con 0x7f30240d0cd0 2024-09-13T08:37:57.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.168+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30240cb330 con 0x7f30240d0cd0 2024-09-13T08:37:57.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.169+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30240d33c0 con 0x7f30240d0cd0 2024-09-13T08:37:57.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.170+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f3010019020 con 0x7f30240d0cd0 2024-09-13T08:37:57.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.172+0000 7f301b400640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 0x7f300407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.172+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7f301009a2a0 con 0x7f30240d0cd0 2024-09-13T08:37:57.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.172+0000 7f3023400640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 0x7f300407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.173+0000 7f3023400640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 0x7f300407ad10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3014034000 tx=0x7f3014004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:57.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.176+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f30240d33c0 con 0x7f30240d0cd0 2024-09-13T08:37:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.442+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f30240cf080 con 0x7f30240d0cd0 2024-09-13T08:37:57.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.444+0000 7f301b400640 1 -- 172.21.15.31:0/1267796596 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f3010020030 con 0x7f30240d0cd0 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 msgr2=0x7f300407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 0x7f300407ad10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3014034000 tx=0x7f3014004730 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 msgr2=0x7f302414c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f301000c450 tx=0x7f301000c910 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 shutdown_connections 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f302415e160 0x7f30241f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30240d0cd0 0x7f302414c240 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3004078850 0x7f300407ad10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 --2- 172.21.15.31:0/1267796596 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30240cfa70 0x7f3024160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 >> 172.21.15.31:0/1267796596 conn(0x7f30240d3f50 msgr2=0x7f302406e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 shutdown_connections 2024-09-13T08:37:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.447+0000 7f3028a00640 1 -- 172.21.15.31:0/1267796596 wait complete. 2024-09-13T08:37:57.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:37:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 -- 172.21.15.31:0/994894388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3354004050 msgr2=0x7f3354004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 --2- 172.21.15.31:0/994894388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3354004050 0x7f3354004800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f335c009b20 tx=0x7f3354005830 comp rx=0 tx=0).stop 2024-09-13T08:37:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f336e800640 1 -- 172.21.15.31:0/994894388 reap_dead start 2024-09-13T08:37:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 -- 172.21.15.31:0/994894388 shutdown_connections 2024-09-13T08:37:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 -- 172.21.15.31:0/994894388 >> 172.21.15.31:0/994894388 conn(0x7f33700cc910 msgr2=0x7f33700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 -- 172.21.15.31:0/994894388 shutdown_connections 2024-09-13T08:37:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.623+0000 7f3375200640 1 -- 172.21.15.31:0/994894388 wait complete. 2024-09-13T08:37:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.625+0000 7f3375200640 1 Processor -- start 2024-09-13T08:37:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.625+0000 7f3375200640 1 -- start start 2024-09-13T08:37:57.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.625+0000 7f3375200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 0x7f33700b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.625+0000 7f3375200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 0x7f3370146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f3375200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f3375200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33701600f0 con 0x7f33700b4850 2024-09-13T08:37:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f3375200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33700cb850 con 0x7f3354004050 2024-09-13T08:37:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f3375200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3370147870 con 0x7f337009d5e0 2024-09-13T08:37:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f336e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f336e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47728/0 (socket says 172.21.15.31:47728) 2024-09-13T08:37:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f336e800640 1 -- 172.21.15.31:0/1705794585 learned_addr learned my addr 172.21.15.31:0/1705794585 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f336d400640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 0x7f3370146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.626+0000 7f336de00640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 0x7f33700b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 -- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 msgr2=0x7f3370146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 0x7f3370146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 -- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 msgr2=0x7f33700b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 0x7f33700b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 -- 172.21.15.31:0/1705794585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33700b32a0 con 0x7f33700b4850 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336de00640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 0x7f33700b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336d400640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 0x7f3370146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f336e800640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f33700cf330 tx=0x7f3368006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368011900 con 0x7f33700b4850 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3368018070 con 0x7f33700b4850 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368011d40 con 0x7f33700b4850 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33700b5520 con 0x7f33700b4850 2024-09-13T08:37:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.627+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33700b5ae0 con 0x7f33700b4850 2024-09-13T08:37:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.628+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33700cc440 con 0x7f33700b4850 2024-09-13T08:37:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.629+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f336801d460 con 0x7f33700b4850 2024-09-13T08:37:57.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.630+0000 7f3366000640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 0x7f335007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:57.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.630+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7f336809af60 con 0x7f33700b4850 2024-09-13T08:37:57.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.631+0000 7f336de00640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 0x7f335007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:57.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.631+0000 7f336de00640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 0x7f335007ad10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3360006b30 tx=0x7f3360006720 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:57.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.634+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3368063880 con 0x7f33700b4850 2024-09-13T08:37:57.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.832+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f33700ccfd0 con 0x7f33700b4850 2024-09-13T08:37:57.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.832+0000 7f3366000640 1 -- 172.21.15.31:0/1705794585 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) ==== 84+0+65 (secure 0 0 0) 0x7f3368063180 con 0x7f33700b4850 2024-09-13T08:37:57.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:37:57.835 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:37:57.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 msgr2=0x7f335007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 0x7f335007ad10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3360006b30 tx=0x7f3360006720 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 msgr2=0x7f33700b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f33700cf330 tx=0x7f3368006430 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 shutdown_connections 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33700b4850 0x7f33700b4cd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3350078850 0x7f335007ad10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f337009d5e0 0x7f3370146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 --2- 172.21.15.31:0/1705794585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3354004050 0x7f33700b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 >> 172.21.15.31:0/1705794585 conn(0x7f33700cc910 msgr2=0x7f33700acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 shutdown_connections 2024-09-13T08:37:57.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:57.835+0000 7f3375200640 1 -- 172.21.15.31:0/1705794585 wait complete. 2024-09-13T08:37:57.853 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:37:57.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 -- 172.21.15.31:0/267023682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec0741b0 msgr2=0x7fb1ec0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 --2- 172.21.15.31:0/267023682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec0741b0 0x7fb1ec0cd780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0d0cb0 tx=0x7fb1d402a560 comp rx=0 tx=0).stop 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 -- 172.21.15.31:0/267023682 shutdown_connections 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 --2- 172.21.15.31:0/267023682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec0741b0 0x7fb1ec0cd780 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 --2- 172.21.15.31:0/267023682 >> v2:172.21.15.45:3300/0 conn(0x7fb1ec15acb0 0x7fb1ec0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 -- 172.21.15.31:0/267023682 >> 172.21.15.31:0/267023682 conn(0x7fb1ec0cfbb0 msgr2=0x7fb1ec0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 -- 172.21.15.31:0/267023682 shutdown_connections 2024-09-13T08:37:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.016+0000 7fb1f3000640 1 -- 172.21.15.31:0/267023682 wait complete. 2024-09-13T08:37:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.017+0000 7fb1f3000640 1 Processor -- start 2024-09-13T08:37:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.018+0000 7fb1f3000640 1 -- start start 2024-09-13T08:37:58.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.018+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1ec0741b0 0x7fb1ec0af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.018+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 0x7fb1ec14d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f3000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0ce160 con 0x7fb1ec15acb0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0cdfe0 con 0x7fb1ec0745f0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f3000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec0cb0f0 con 0x7fb1ec0741b0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 0x7fb1ec14d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1ec0741b0 0x7fb1ec0af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 0x7fb1ec14d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:39854/0 (socket says 172.21.15.31:39854) 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1e3400640 1 -- 172.21.15.31:0/3158661115 learned_addr learned my addr 172.21.15.31:0/3158661115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47740/0 (socket says 172.21.15.31:47740) 2024-09-13T08:37:58.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 -- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1ec0741b0 msgr2=0x7fb1ec0af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:58.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1ec0741b0 0x7fb1ec0af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 -- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 msgr2=0x7fb1ec14d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 0x7fb1ec14d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.019+0000 7fb1f1000640 1 -- 172.21.15.31:0/3158661115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec1e4680 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1f1000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c780 tx=0x7fb1e400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4016020 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1e400ab40 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec1e48f0 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e400ce30 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.020+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec14d060 con 0x7fb1ec15acb0 2024-09-13T08:37:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.021+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec0cb5c0 con 0x7fb1ec15acb0 2024-09-13T08:37:58.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.022+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fb1e4019020 con 0x7fb1ec15acb0 2024-09-13T08:37:58.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.023+0000 7fb1e8c00640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 0x7fb1c807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:58.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.024+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4857+0+0 (secure 0 0 0) 0x7fb1e409a380 con 0x7fb1ec15acb0 2024-09-13T08:37:58.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.024+0000 7fb1ebe00640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 0x7fb1c807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:58.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.025+0000 7fb1ebe00640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 0x7fb1c807ad10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc004630 tx=0x7fb1dc00b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.028+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb1e4020030 con 0x7fb1ec15acb0 2024-09-13T08:37:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:58.225+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb1ec14d230 con 0x7fb1ec15acb0 2024-09-13T08:37:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.199+0000 7fb1e8c00640 1 -- 172.21.15.31:0/3158661115 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) ==== 138+0+0 (secure 0 0 0) 0x7fb1e40626f0 con 0x7fb1ec15acb0 2024-09-13T08:37:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 msgr2=0x7fb1c807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 0x7fb1c807ad10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc004630 tx=0x7fb1dc00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 msgr2=0x7fb1ec15d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c780 tx=0x7fb1e400cc40 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 shutdown_connections 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ec15acb0 0x7fb1ec15d5b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1ec0745f0 0x7fb1ec14d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb1c8078850 0x7fb1c807ad10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 --2- 172.21.15.31:0/3158661115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1ec0741b0 0x7fb1ec0af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 >> 172.21.15.31:0/3158661115 conn(0x7fb1ec0cfbb0 msgr2=0x7fb1ec0c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.203+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 shutdown_connections 2024-09-13T08:37:59.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.204+0000 7fb1f3000640 1 -- 172.21.15.31:0/3158661115 wait complete. 2024-09-13T08:37:59.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:37:59.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 -- 172.21.15.31:0/287691743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03006fa60 msgr2=0x7fd0300cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:59.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 --2- 172.21.15.31:0/287691743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03006fa60 0x7fd0300cb740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd03006e410 tx=0x7fd01c02a560 comp rx=0 tx=0).stop 2024-09-13T08:37:59.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 -- 172.21.15.31:0/287691743 shutdown_connections 2024-09-13T08:37:59.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 --2- 172.21.15.31:0/287691743 >> v2:172.21.15.45:3300/0 conn(0x7fd03006fec0 0x7fd0300cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 --2- 172.21.15.31:0/287691743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03006fa60 0x7fd0300cb740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 --2- 172.21.15.31:0/287691743 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03015bfc0 0x7fd030072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 -- 172.21.15.31:0/287691743 >> 172.21.15.31:0/287691743 conn(0x7fd0300cfbb0 msgr2=0x7fd0300cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 -- 172.21.15.31:0/287691743 shutdown_connections 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.396+0000 7fd035800640 1 -- 172.21.15.31:0/287691743 wait complete. 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.398+0000 7fd035800640 1 Processor -- start 2024-09-13T08:37:59.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.398+0000 7fd035800640 1 -- start start 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03006fa60 0x7fd0301e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd03015bfc0 0x7fd0301f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0300b2b60 con 0x7fd03013fbe0 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0300b29e0 con 0x7fd03015bfc0 2024-09-13T08:37:59.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd035800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd0300b2ce0 con 0x7fd03006fa60 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd02f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd02f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47756/0 (socket says 172.21.15.31:47756) 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd02f400640 1 -- 172.21.15.31:0/3401380431 learned_addr learned my addr 172.21.15.31:0/3401380431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd02ea00640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03006fa60 0x7fd0301e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.399+0000 7fd02e000640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd03015bfc0 0x7fd0301f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 -- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03006fa60 msgr2=0x7fd0301e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03006fa60 0x7fd0301e4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 -- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd03015bfc0 msgr2=0x7fd0301f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd03015bfc0 0x7fd0301f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 -- 172.21.15.31:0/3401380431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0300c8420 con 0x7fd03013fbe0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd02f400640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd02800eaa0 tx=0x7fd02800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028004b20 con 0x7fd03013fbe0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd028004db0 con 0x7fd03013fbe0 2024-09-13T08:37:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0300c9450 con 0x7fd03013fbe0 2024-09-13T08:37:59.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028005280 con 0x7fd03013fbe0 2024-09-13T08:37:59.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.400+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0300c8970 con 0x7fd03013fbe0 2024-09-13T08:37:59.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.401+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0300b31c0 con 0x7fd03013fbe0 2024-09-13T08:37:59.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.403+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd028005420 con 0x7fd03013fbe0 2024-09-13T08:37:59.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.404+0000 7fd026a00640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 0x7fd01007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:37:59.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.404+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5227+0+0 (secure 0 0 0) 0x7fd02809b700 con 0x7fd03013fbe0 2024-09-13T08:37:59.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.404+0000 7fd02ea00640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 0x7fd01007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:37:59.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.405+0000 7fd02ea00640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 0x7fd01007ad10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd020009ac0 tx=0x7fd020005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:37:59.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.409+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd028063eb0 con 0x7fd03013fbe0 2024-09-13T08:37:59.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:37:59.607+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd0300cbad0 con 0x7fd03013fbe0 2024-09-13T08:38:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.203+0000 7fd026a00640 1 -- 172.21.15.31:0/3401380431 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) ==== 170+0+0 (secure 0 0 0) 0x7fd0280637b0 con 0x7fd03013fbe0 2024-09-13T08:38:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:38:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 msgr2=0x7fd01007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 0x7fd01007ad10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd020009ac0 tx=0x7fd020005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 msgr2=0x7fd0301f2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd02800eaa0 tx=0x7fd02800ef60 comp rx=0 tx=0).stop 2024-09-13T08:38:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 shutdown_connections 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd03013fbe0 0x7fd0301f2700 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd03015bfc0 0x7fd0301f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd010078850 0x7fd01007ad10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 --2- 172.21.15.31:0/3401380431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd03006fa60 0x7fd0301e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 >> 172.21.15.31:0/3401380431 conn(0x7fd0300cfbb0 msgr2=0x7fd030073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 shutdown_connections 2024-09-13T08:38:00.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.207+0000 7fd035800640 1 -- 172.21.15.31:0/3401380431 wait complete. 2024-09-13T08:38:00.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:38:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 -- 172.21.15.31:0/2856069924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd22806fa60 msgr2=0x7fd2280cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 --2- 172.21.15.31:0/2856069924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd22806fa60 0x7fd2280cb740 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd210005fd0 tx=0x7fd21002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 -- 172.21.15.31:0/2856069924 shutdown_connections 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 --2- 172.21.15.31:0/2856069924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd22806fa60 0x7fd2280cb740 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 --2- 172.21.15.31:0/2856069924 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 0x7fd228072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.409+0000 7fd22ce00640 1 -- 172.21.15.31:0/2856069924 >> 172.21.15.31:0/2856069924 conn(0x7fd2280cfbb0 msgr2=0x7fd2280cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.410+0000 7fd22ce00640 1 -- 172.21.15.31:0/2856069924 shutdown_connections 2024-09-13T08:38:00.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.410+0000 7fd22ce00640 1 -- 172.21.15.31:0/2856069924 wait complete. 2024-09-13T08:38:00.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.411+0000 7fd22ce00640 1 Processor -- start 2024-09-13T08:38:00.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.411+0000 7fd22ce00640 1 -- start start 2024-09-13T08:38:00.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd22806fa60 0x7fd2281f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:00.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 0x7fd2281f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:00.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:00.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2280b3270 con 0x7fd2281f8460 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2280b30f0 con 0x7fd22806fa60 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd22ce00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd2280b33f0 con 0x7fd22815bfc0 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.412+0000 7fd226800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47778/0 (socket says 172.21.15.31:47778) 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 -- 172.21.15.31:0/1865622336 learned_addr learned my addr 172.21.15.31:0/1865622336 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd225e00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd22806fa60 0x7fd2281f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd21d400640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 0x7fd2281f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 -- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 msgr2=0x7fd2281f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 0x7fd2281f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 -- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd22806fa60 msgr2=0x7fd2281f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd22806fa60 0x7fd2281f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 -- 172.21.15.31:0/1865622336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2280c8de0 con 0x7fd2281f8460 2024-09-13T08:38:00.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.413+0000 7fd226800640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd22000c7a0 tx=0x7fd22000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.414+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd220003b70 con 0x7fd2281f8460 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.414+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd220003d10 con 0x7fd2281f8460 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.414+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22000aa50 con 0x7fd2281f8460 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.414+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2280c90c0 con 0x7fd2281f8460 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.414+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2280c9680 con 0x7fd2281f8460 2024-09-13T08:38:00.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.415+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2280cbe40 con 0x7fd2281f8460 2024-09-13T08:38:00.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.416+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fd22000abf0 con 0x7fd2281f8460 2024-09-13T08:38:00.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.418+0000 7fd21ea00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 0x7fd20407ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:00.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.418+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5609+0+0 (secure 0 0 0) 0x7fd22009b080 con 0x7fd2281f8460 2024-09-13T08:38:00.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.418+0000 7fd225e00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 0x7fd20407ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:00.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.419+0000 7fd225e00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 0x7fd20407ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd21800bb50 tx=0x7fd21800b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:00.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.421+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd2200630e0 con 0x7fd2281f8460 2024-09-13T08:38:00.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:00.654+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd2280cb310 con 0x7fd2281f8460 2024-09-13T08:38:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.215+0000 7fd21ea00640 1 -- 172.21.15.31:0/1865622336 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7fd220067b10 con 0x7fd2281f8460 2024-09-13T08:38:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 4 and data pool 5 2024-09-13T08:38:01.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.218+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 msgr2=0x7fd20407ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.218+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 0x7fd20407ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd21800bb50 tx=0x7fd21800b2e0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.218+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 msgr2=0x7fd2281f8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.218+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd22000c7a0 tx=0x7fd22000cc60 comp rx=0 tx=0).stop 2024-09-13T08:38:01.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 shutdown_connections 2024-09-13T08:38:01.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2281f8460 0x7fd2281f8910 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd204078880 0x7fd20407ad40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd22815bfc0 0x7fd2281f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 --2- 172.21.15.31:0/1865622336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd22806fa60 0x7fd2281f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 >> 172.21.15.31:0/1865622336 conn(0x7fd2280cfbb0 msgr2=0x7fd2280cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 shutdown_connections 2024-09-13T08:38:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.223+0000 7fd22ce00640 1 -- 172.21.15.31:0/1865622336 wait complete. 2024-09-13T08:38:01.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.423+0000 7f65b1800640 1 -- 172.21.15.31:0/2707744331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15f530 msgr2=0x7f65ac15fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.423+0000 7f65b1800640 1 --2- 172.21.15.31:0/2707744331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15f530 0x7f65ac15fa70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6598009830 tx=0x7f659802a3f0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 -- 172.21.15.31:0/2707744331 shutdown_connections 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 --2- 172.21.15.31:0/2707744331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15f530 0x7f65ac15fa70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 --2- 172.21.15.31:0/2707744331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 0x7f65ac0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 -- 172.21.15.31:0/2707744331 >> 172.21.15.31:0/2707744331 conn(0x7f65ac0d1c70 msgr2=0x7f65ac0d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 -- 172.21.15.31:0/2707744331 shutdown_connections 2024-09-13T08:38:01.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.424+0000 7f65b1800640 1 -- 172.21.15.31:0/2707744331 wait complete. 2024-09-13T08:38:01.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.425+0000 7f65b1800640 1 Processor -- start 2024-09-13T08:38:01.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.425+0000 7f65b1800640 1 -- start start 2024-09-13T08:38:01.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 0x7f65ac1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15ffb0 0x7f65ac0af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1607f0 con 0x7f65ac15f530 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac160670 con 0x7f65ac15ffb0 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.426+0000 7f65b1800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65ac0d0280 con 0x7f65ac15f150 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47780/0 (socket says 172.21.15.31:47780) 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 -- 172.21.15.31:0/2391003502 learned_addr learned my addr 172.21.15.31:0/2391003502 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65abe00640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 0x7f65ac1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65b0e00640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15ffb0 0x7f65ac0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 -- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 msgr2=0x7f65ac1551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 0x7f65ac1551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 -- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15ffb0 msgr2=0x7f65ac0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15ffb0 0x7f65ac0af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 -- 172.21.15.31:0/2391003502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac155710 con 0x7f65ac15f530 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.427+0000 7f65ab400640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6598000c00 tx=0x7f6598032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.428+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598040070 con 0x7f65ac15f530 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.428+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65980038e0 con 0x7f65ac15f530 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.428+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598005c90 con 0x7f65ac15f530 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.428+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac06dfd0 con 0x7f65ac15f530 2024-09-13T08:38:01.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.428+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac06e590 con 0x7f65ac15f530 2024-09-13T08:38:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.429+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac074940 con 0x7f65ac15f530 2024-09-13T08:38:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.430+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f6598039040 con 0x7f65ac15f530 2024-09-13T08:38:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.431+0000 7f65a3e00640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 0x7f658c07ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.431+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f65980bbd20 con 0x7f65ac15f530 2024-09-13T08:38:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.431+0000 7f65abe00640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 0x7f658c07ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.432+0000 7f65abe00640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 0x7f658c07ad70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f659c009bd0 tx=0x7f659c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.436+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6598083dd0 con 0x7f65ac15f530 2024-09-13T08:38:01.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.630+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65ac06ac30 con 0x7f65ac15f530 2024-09-13T08:38:01.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.631+0000 7f65a3e00640 1 -- 172.21.15.31:0/2391003502 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+16634 (secure 0 0 0) 0x7f6598088800 con 0x7f65ac15f530 2024-09-13T08:38:01.633 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:01.634 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:00.656393+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:58.235357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:59.617519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:01.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.633+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 msgr2=0x7f658c07ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.633+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 0x7f658c07ad70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f659c009bd0 tx=0x7f659c005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.633+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 msgr2=0x7f65ac1e7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.633+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6598000c00 tx=0x7f6598032c10 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 shutdown_connections 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ac15ffb0 0x7f65ac0af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac15f530 0x7f65ac1e7620 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f658c0788b0 0x7f658c07ad70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 --2- 172.21.15.31:0/2391003502 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ac15f150 0x7f65ac1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 >> 172.21.15.31:0/2391003502 conn(0x7f65ac0d1c70 msgr2=0x7f65ac071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 shutdown_connections 2024-09-13T08:38:01.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.634+0000 7f65b1800640 1 -- 172.21.15.31:0/2391003502 wait complete. 2024-09-13T08:38:01.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 -- 172.21.15.31:0/3976187261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 msgr2=0x7fc03c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 --2- 172.21.15.31:0/3976187261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c0cc300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009830 tx=0x7fc02c02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 -- 172.21.15.31:0/3976187261 shutdown_connections 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 --2- 172.21.15.31:0/3976187261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c0cc300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 -- 172.21.15.31:0/3976187261 >> 172.21.15.31:0/3976187261 conn(0x7fc03c0cc910 msgr2=0x7fc03c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:01.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 -- 172.21.15.31:0/3976187261 shutdown_connections 2024-09-13T08:38:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.812+0000 7fc042a00640 1 -- 172.21.15.31:0/3976187261 wait complete. 2024-09-13T08:38:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.814+0000 7fc042a00640 1 Processor -- start 2024-09-13T08:38:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.814+0000 7fc042a00640 1 -- start start 2024-09-13T08:38:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 0x7fc03c0adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c0b1d60 con 0x7fc03c0b1910 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c0740a0 con 0x7fc03c15dfc0 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc042a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc03c0b1ee0 con 0x7fc03c0ca5c0 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc03be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc03be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47792/0 (socket says 172.21.15.31:47792) 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc03be00640 1 -- 172.21.15.31:0/785993032 learned_addr learned my addr 172.21.15.31:0/785993032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc033400640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 0x7fc03c0adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc040a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc03be00640 1 -- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 msgr2=0x7fc03c0adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.815+0000 7fc03be00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 0x7fc03c0adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc03be00640 1 -- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 msgr2=0x7fc03c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc03be00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc03be00640 1 -- 172.21.15.31:0/785993032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03c073840 con 0x7fc03c0b1910 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc033400640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 0x7fc03c0adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc040a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc03be00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009950 tx=0x7fc02c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c031d60 con 0x7fc03c0b1910 2024-09-13T08:38:01.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc02c033e10 con 0x7fc03c0b1910 2024-09-13T08:38:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c033310 con 0x7fc03c0b1910 2024-09-13T08:38:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03c0727a0 con 0x7fc03c0b1910 2024-09-13T08:38:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.816+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03c073040 con 0x7fc03c0b1910 2024-09-13T08:38:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.817+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03c072b90 con 0x7fc03c0b1910 2024-09-13T08:38:01.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.818+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fc02c0334b0 con 0x7fc03c0b1910 2024-09-13T08:38:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.820+0000 7fc038c00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 0x7fc01807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.820+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fc02c0bbcb0 con 0x7fc03c0b1910 2024-09-13T08:38:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.820+0000 7fc033400640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 0x7fc01807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.821+0000 7fc033400640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 0x7fc01807ad10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0ca3f0 tx=0x7fc024009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:01.824+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc02c03f030 con 0x7fc03c0b1910 2024-09-13T08:38:02.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.024+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc03c0af5d0 con 0x7fc03c0b1910 2024-09-13T08:38:02.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.025+0000 7fc038c00640 1 -- 172.21.15.31:0/785993032 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+16634 (secure 0 0 0) 0x7fc02c084290 con 0x7fc03c0b1910 2024-09-13T08:38:02.027 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:02.027 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:00.656393+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:58.235357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:59.617519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.027+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 msgr2=0x7fc01807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.027+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 0x7fc01807ad10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0ca3f0 tx=0x7fc024009040 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.027+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 msgr2=0x7fc03c0ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.027+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009950 tx=0x7fc02c033c20 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 shutdown_connections 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03c15dfc0 0x7fc03c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc018078850 0x7fc01807ad10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc03c0ca5c0 0x7fc03c0adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 --2- 172.21.15.31:0/785993032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc03c0b1910 0x7fc03c0ad5c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 >> 172.21.15.31:0/785993032 conn(0x7fc03c0cc910 msgr2=0x7fc03c0af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 shutdown_connections 2024-09-13T08:38:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.028+0000 7fc042a00640 1 -- 172.21.15.31:0/785993032 wait complete. 2024-09-13T08:38:02.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 -- 172.21.15.31:0/3154230770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6518072560 msgr2=0x7f65180cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 --2- 172.21.15.31:0/3154230770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6518072560 0x7f65180cb740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6500009830 tx=0x7f650002a5c0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 -- 172.21.15.31:0/3154230770 shutdown_connections 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 --2- 172.21.15.31:0/3154230770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6518072560 0x7f65180cb740 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 -- 172.21.15.31:0/3154230770 >> 172.21.15.31:0/3154230770 conn(0x7f65180cfbb0 msgr2=0x7f65180cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 -- 172.21.15.31:0/3154230770 shutdown_connections 2024-09-13T08:38:02.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.208+0000 7f651ce00640 1 -- 172.21.15.31:0/3154230770 wait complete. 2024-09-13T08:38:02.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.209+0000 7f651ce00640 1 Processor -- start 2024-09-13T08:38:02.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.210+0000 7f651ce00640 1 -- start start 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.210+0000 7f651ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.210+0000 7f651ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65180cbc80 0x7f6518144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.210+0000 7f651ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 0x7f65181f8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.210+0000 7f651ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f651806e740 con 0x7f6518072560 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f651ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f651806e5c0 con 0x7f65180cbc80 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f651ce00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65180b30f0 con 0x7f651815bfc0 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47806/0 (socket says 172.21.15.31:47806) 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 -- 172.21.15.31:0/2735591804 learned_addr learned my addr 172.21.15.31:0/2735591804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f650d400640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65180cbc80 0x7f6518144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6516800640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 0x7f65181f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 -- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 msgr2=0x7f65181f8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 0x7f65181f8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 -- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65180cbc80 msgr2=0x7f6518144990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65180cbc80 0x7f6518144990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6515e00640 1 -- 172.21.15.31:0/2735591804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65181f8ee0 con 0x7f6518072560 2024-09-13T08:38:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.211+0000 7f6516800640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 0x7f65181f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f6515e00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f650800c430 tx=0x7f650800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508017070 con 0x7f6518072560 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65080053e0 con 0x7f6518072560 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508003850 con 0x7f6518072560 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65181f91c0 con 0x7f6518072560 2024-09-13T08:38:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.212+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65181531b0 con 0x7f6518072560 2024-09-13T08:38:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.213+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65180cb740 con 0x7f6518072560 2024-09-13T08:38:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.214+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f650801a070 con 0x7f6518072560 2024-09-13T08:38:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.215+0000 7f650ea00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 0x7f64f407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.215+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f6508099cf0 con 0x7f6518072560 2024-09-13T08:38:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.216+0000 7f650d400640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 0x7f64f407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.217+0000 7f650d400640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 0x7f64f407ad10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6500034000 tx=0x7f6500004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.219+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6508062350 con 0x7f6518072560 2024-09-13T08:38:02.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:02.459+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f65181f9390 con 0x7f6518072560 2024-09-13T08:38:03.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.247+0000 7f650ea00640 1 -- 172.21.15.31:0/2735591804 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f6508061c50 con 0x7f6518072560 2024-09-13T08:38:03.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 msgr2=0x7f64f407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:03.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 0x7f64f407ad10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6500034000 tx=0x7f6500004730 comp rx=0 tx=0).stop 2024-09-13T08:38:03.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 msgr2=0x7f651815ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f650800c430 tx=0x7f650800c8f0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 shutdown_connections 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f64f4078850 0x7f64f407ad10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f651815bfc0 0x7f65181f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65180cbc80 0x7f6518144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 --2- 172.21.15.31:0/2735591804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6518072560 0x7f651815ca00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 >> 172.21.15.31:0/2735591804 conn(0x7f65180cfbb0 msgr2=0x7f65180c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 shutdown_connections 2024-09-13T08:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.251+0000 7f651ce00640 1 -- 172.21.15.31:0/2735591804 wait complete. 2024-09-13T08:38:03.271 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:38:03.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.438+0000 7f9eb0e00640 1 -- 172.21.15.31:0/1532152765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cc190 msgr2=0x7f9eac0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.438+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/1532152765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cc190 0x7f9eac0caa80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4005fd0 tx=0x7f9ea402a820 comp rx=0 tx=0).stop 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.438+0000 7f9eb0e00640 1 -- 172.21.15.31:0/1532152765 shutdown_connections 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.438+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/1532152765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cc190 0x7f9eac0caa80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.439+0000 7f9eb0e00640 1 -- 172.21.15.31:0/1532152765 >> 172.21.15.31:0/1532152765 conn(0x7f9eac0cfbb0 msgr2=0x7f9eac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.439+0000 7f9eb0e00640 1 -- 172.21.15.31:0/1532152765 shutdown_connections 2024-09-13T08:38:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.439+0000 7f9eb0e00640 1 -- 172.21.15.31:0/1532152765 wait complete. 2024-09-13T08:38:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.440+0000 7f9eb0e00640 1 Processor -- start 2024-09-13T08:38:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.440+0000 7f9eb0e00640 1 -- start start 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac0cc190 0x7f9eac0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 0x7f9eac0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac0cbfd0 con 0x7f9eac0cde90 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac0cbe50 con 0x7f9eac0cc190 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9eb0e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac0cafc0 con 0x7f9eac161480 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9ea1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9ea1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47830/0 (socket says 172.21.15.31:47830) 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.441+0000 7f9ea1400640 1 -- 172.21.15.31:0/2921863622 learned_addr learned my addr 172.21.15.31:0/2921863622 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea9e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac0cc190 0x7f9eac0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9eaa800640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 0x7f9eac0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 -- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 msgr2=0x7f9eac0b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 0x7f9eac0b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 -- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac0cc190 msgr2=0x7f9eac0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac0cc190 0x7f9eac0b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 -- 172.21.15.31:0/2921863622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0b4c50 con 0x7f9eac0cde90 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9eaa800640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 0x7f9eac0b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea1400640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c430 tx=0x7f9e9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94011790 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e94017070 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94011cc0 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.442+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac0c3310 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.443+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac0b2130 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.444+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac153910 con 0x7f9eac0cde90 2024-09-13T08:38:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.445+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f9e9401c460 con 0x7f9eac0cde90 2024-09-13T08:38:03.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.446+0000 7f9ea2a00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 0x7f9e8807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:03.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.446+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f9e9409a6f0 con 0x7f9eac0cde90 2024-09-13T08:38:03.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.446+0000 7f9ea9e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 0x7f9e8807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:03.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.447+0000 7f9ea9e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 0x7f9e8807ad10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0c5960 tx=0x7f9e9c007e90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:03.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.450+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9eac153910 con 0x7f9eac0cde90 2024-09-13T08:38:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:03.684+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9eac0cb400 con 0x7f9eac0cde90 2024-09-13T08:38:04.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.249+0000 7f9ea2a00640 1 -- 172.21.15.31:0/2921863622 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7f9e94062df0 con 0x7f9eac0cde90 2024-09-13T08:38:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 msgr2=0x7f9e8807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 0x7f9e8807ad10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0c5960 tx=0x7f9e9c007e90 comp rx=0 tx=0).stop 2024-09-13T08:38:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 msgr2=0x7f9eac0c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c430 tx=0x7f9e9400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 shutdown_connections 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9e88078850 0x7f9e8807ad10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9eac161480 0x7f9eac0b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eac0cde90 0x7f9eac0c5c00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 --2- 172.21.15.31:0/2921863622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9eac0cc190 0x7f9eac0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 >> 172.21.15.31:0/2921863622 conn(0x7f9eac0cfbb0 msgr2=0x7f9eac0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 shutdown_connections 2024-09-13T08:38:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.252+0000 7f9eb0e00640 1 -- 172.21.15.31:0/2921863622 wait complete. 2024-09-13T08:38:04.271 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:38:04.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 -- 172.21.15.31:0/855984125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0701a0 msgr2=0x7f646c0b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 --2- 172.21.15.31:0/855984125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0701a0 0x7f646c0b5100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f646c0738f0 tx=0x7f646402aa10 comp rx=0 tx=0).stop 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 -- 172.21.15.31:0/855984125 shutdown_connections 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 --2- 172.21.15.31:0/855984125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0701a0 0x7f646c0b5100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 --2- 172.21.15.31:0/855984125 >> v2:172.21.15.45:3300/0 conn(0x7f646c0ccb00 0x7f646c0b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 --2- 172.21.15.31:0/855984125 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0cc720 0x7f646c06f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 -- 172.21.15.31:0/855984125 >> 172.21.15.31:0/855984125 conn(0x7f646c06cb00 msgr2=0x7f646c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 -- 172.21.15.31:0/855984125 shutdown_connections 2024-09-13T08:38:04.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.430+0000 7f6472000640 1 -- 172.21.15.31:0/855984125 wait complete. 2024-09-13T08:38:04.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.432+0000 7f6472000640 1 Processor -- start 2024-09-13T08:38:04.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.432+0000 7f6472000640 1 -- start start 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0701a0 0x7f646c1f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 0x7f646c0b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f646c152a10 con 0x7f646c0cc720 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f646c152890 con 0x7f646c0ccb00 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6472000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f646c152b90 con 0x7f646c0701a0 2024-09-13T08:38:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6463e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6463e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47860/0 (socket says 172.21.15.31:47860) 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6463e00640 1 -- 172.21.15.31:0/2786666329 learned_addr learned my addr 172.21.15.31:0/2786666329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.433+0000 7f6470c00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0701a0 0x7f646c1f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6471600640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 0x7f646c0b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 -- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0701a0 msgr2=0x7f646c1f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0701a0 0x7f646c1f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 -- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 msgr2=0x7f646c0b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 0x7f646c0b0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 -- 172.21.15.31:0/2786666329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c14dc00 con 0x7f646c0cc720 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6471600640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 0x7f646c0b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6463e00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f645400c9a0 tx=0x7f645400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454007bb0 con 0x7f646c0cc720 2024-09-13T08:38:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6454007e40 con 0x7f646c0cc720 2024-09-13T08:38:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646c14deb0 con 0x7f646c0cc720 2024-09-13T08:38:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.434+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645400f6b0 con 0x7f646c0cc720 2024-09-13T08:38:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.435+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646c14e470 con 0x7f646c0cc720 2024-09-13T08:38:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.436+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f646c153040 con 0x7f646c0cc720 2024-09-13T08:38:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.437+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f645400f850 con 0x7f646c0cc720 2024-09-13T08:38:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.438+0000 7f6469600640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 0x7f644807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.438+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f645409b070 con 0x7f646c0cc720 2024-09-13T08:38:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.438+0000 7f6470c00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 0x7f644807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.439+0000 7f6470c00640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 0x7f644807ad10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f645c005940 tx=0x7f645c00a2d0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:04.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.442+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6454014030 con 0x7f646c0cc720 2024-09-13T08:38:04.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.674+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f646c069770 con 0x7f646c0cc720 2024-09-13T08:38:04.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.675+0000 7f6469600640 1 -- 172.21.15.31:0/2786666329 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5214 (secure 0 0 0) 0x7f6454063640 con 0x7f646c0cc720 2024-09-13T08:38:04.677 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:04.677 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"btime":"2024-09-13T08:38:04:249411+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:04.677 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2024-09-13T08:38:04.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 msgr2=0x7f644807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 0x7f644807ad10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f645c005940 tx=0x7f645c00a2d0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 msgr2=0x7f646c14d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f645400c9a0 tx=0x7f645400ce60 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 shutdown_connections 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ccb00 0x7f646c0b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646c0cc720 0x7f646c14d6c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6448078850 0x7f644807ad10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 --2- 172.21.15.31:0/2786666329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f646c0701a0 0x7f646c1f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 >> 172.21.15.31:0/2786666329 conn(0x7f646c06cb00 msgr2=0x7f646c1533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 shutdown_connections 2024-09-13T08:38:04.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.678+0000 7f6472000640 1 -- 172.21.15.31:0/2786666329 wait complete. 2024-09-13T08:38:04.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 -- 172.21.15.31:0/718664800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015bd70 msgr2=0x7fcb700cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 --2- 172.21.15.31:0/718664800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015bd70 0x7fcb700cf850 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcb700c6f10 tx=0x7fcb6002a560 comp rx=0 tx=0).stop 2024-09-13T08:38:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 -- 172.21.15.31:0/718664800 shutdown_connections 2024-09-13T08:38:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 --2- 172.21.15.31:0/718664800 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb700ce3c0 0x7fcb700ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 --2- 172.21.15.31:0/718664800 >> v2:172.21.15.45:3300/0 conn(0x7fcb7015f9e0 0x7fcb7015fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 --2- 172.21.15.31:0/718664800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015bd70 0x7fcb700cf850 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 -- 172.21.15.31:0/718664800 >> 172.21.15.31:0/718664800 conn(0x7fcb700cc910 msgr2=0x7fcb700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 -- 172.21.15.31:0/718664800 shutdown_connections 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.854+0000 7fcb77c00640 1 -- 172.21.15.31:0/718664800 wait complete. 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.856+0000 7fcb77c00640 1 Processor -- start 2024-09-13T08:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.856+0000 7fcb77c00640 1 -- start start 2024-09-13T08:38:04.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb700ce3c0 0x7fcb701535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb7015bd70 0x7fcb700c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb700cb9d0 con 0x7fcb7015f9e0 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb700cb850 con 0x7fcb700ce3c0 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb77c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcb700cbc50 con 0x7fcb7015bd70 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb75c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb75c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46982/0 (socket says 172.21.15.31:46982) 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb75c00640 1 -- 172.21.15.31:0/3851934416 learned_addr learned my addr 172.21.15.31:0/3851934416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb67e00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb7015bd70 0x7fcb700c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.857+0000 7fcb75200640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb700ce3c0 0x7fcb701535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 -- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb7015bd70 msgr2=0x7fcb700c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb7015bd70 0x7fcb700c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 -- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb700ce3c0 msgr2=0x7fcb701535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb700ce3c0 0x7fcb701535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 -- 172.21.15.31:0/3851934416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb700c8940 con 0x7fcb7015f9e0 2024-09-13T08:38:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb75c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800ead0 tx=0x7fcb6800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6800cde0 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb68004c10 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68005020 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb700c8c20 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.858+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb700c91e0 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.859+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb700cf850 con 0x7fcb7015f9e0 2024-09-13T08:38:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.861+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fcb680051c0 con 0x7fcb7015f9e0 2024-09-13T08:38:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.861+0000 7fcb6d600640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 0x7fcb4c07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.862+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fcb6809b280 con 0x7fcb7015f9e0 2024-09-13T08:38:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.862+0000 7fcb75200640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 0x7fcb4c07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.863+0000 7fcb75200640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 0x7fcb4c07ad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb60033000 tx=0x7fcb60004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:04.866+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcb68016030 con 0x7fcb7015f9e0 2024-09-13T08:38:05.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.062+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb700ca560 con 0x7fcb7015f9e0 2024-09-13T08:38:05.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.063+0000 7fcb6d600640 1 -- 172.21.15.31:0/3851934416 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+16621 (secure 0 0 0) 0x7fcb68063290 con 0x7fcb7015f9e0 2024-09-13T08:38:05.065 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:05.065 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:02.230970+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:58.235357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:59.617519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:05.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 msgr2=0x7fcb4c07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 0x7fcb4c07ad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb60033000 tx=0x7fcb60004730 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 msgr2=0x7fcb700c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800ead0 tx=0x7fcb6800ef90 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 shutdown_connections 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7015f9e0 0x7fcb700c5ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcb4c078850 0x7fcb4c07ad10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb7015bd70 0x7fcb700c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 --2- 172.21.15.31:0/3851934416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb700ce3c0 0x7fcb701535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 >> 172.21.15.31:0/3851934416 conn(0x7fcb700cc910 msgr2=0x7fcb700b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 shutdown_connections 2024-09-13T08:38:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.065+0000 7fcb77c00640 1 -- 172.21.15.31:0/3851934416 wait complete. 2024-09-13T08:38:05.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:38:05.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:38:05.192 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-13T08:38:05.190+0000 7fbce7e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.242+0000 7f06cb200640 1 -- 172.21.15.31:0/1952116019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc15dfa0 msgr2=0x7f06cc0d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.242+0000 7f06cb200640 1 --2- 172.21.15.31:0/1952116019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc15dfa0 0x7f06cc0d2500 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0d18f0 tx=0x7f06bc02a560 comp rx=0 tx=0).stop 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 -- 172.21.15.31:0/1952116019 shutdown_connections 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 --2- 172.21.15.31:0/1952116019 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc0b47f0 0x7f06cc0d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 --2- 172.21.15.31:0/1952116019 >> v2:172.21.15.45:3300/0 conn(0x7f06cc0d4110 0x7f06cc1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 --2- 172.21.15.31:0/1952116019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc15dfa0 0x7f06cc0d2500 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 -- 172.21.15.31:0/1952116019 >> 172.21.15.31:0/1952116019 conn(0x7f06cc08d8b0 msgr2=0x7f06cc09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 -- 172.21.15.31:0/1952116019 shutdown_connections 2024-09-13T08:38:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.243+0000 7f06cb200640 1 -- 172.21.15.31:0/1952116019 wait complete. 2024-09-13T08:38:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.244+0000 7f06cb200640 1 Processor -- start 2024-09-13T08:38:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.244+0000 7f06cb200640 1 -- start start 2024-09-13T08:38:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 0x7f06cc0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc15dfa0 0x7f06cc15c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc162270 con 0x7f06cc0b47f0 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc1620f0 con 0x7f06cc0d4110 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06cb200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f06cc1623f0 con 0x7f06cc15dfa0 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.245+0000 7f06c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47002/0 (socket says 172.21.15.31:47002) 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 -- 172.21.15.31:0/2500746381 learned_addr learned my addr 172.21.15.31:0/2500746381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06ca800640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc15dfa0 0x7f06cc15c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9400640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 0x7f06cc0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 -- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc15dfa0 msgr2=0x7f06cc15c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc15dfa0 0x7f06cc15c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 -- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 msgr2=0x7f06cc0d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 0x7f06cc0d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 -- 172.21.15.31:0/2500746381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc0cb020 con 0x7f06cc0b47f0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9400640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 0x7f06cc0d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c9e00640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc03e030 con 0x7f06cc0b47f0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06bc003e00 con 0x7f06cc0b47f0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.246+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc0cb300 con 0x7f06cc0b47f0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.247+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc035040 con 0x7f06cc0b47f0 2024-09-13T08:38:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.247+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc0cb8c0 con 0x7f06cc0b47f0 2024-09-13T08:38:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.248+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06cc0d4cf0 con 0x7f06cc0b47f0 2024-09-13T08:38:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.249+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f06bc041030 con 0x7f06cc0b47f0 2024-09-13T08:38:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.250+0000 7f06c2000640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 0x7f06ac07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.250+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f06bc0bafe0 con 0x7f06cc0b47f0 2024-09-13T08:38:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.250+0000 7f06c9400640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 0x7f06ac07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.251+0000 7f06c9400640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 0x7f06ac07ad10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f06b8005fd0 tx=0x7f06b80094d0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.255+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f06bc083640 con 0x7f06cc0b47f0 2024-09-13T08:38:05.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.449+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06cc0b7eb0 con 0x7f06ac078850 2024-09-13T08:38:05.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.454+0000 7f06c2000640 1 -- 172.21.15.31:0/2500746381 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f06cc0b7eb0 con 0x7f06ac078850 2024-09-13T08:38:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:38:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 msgr2=0x7f06ac07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 0x7f06ac07ad10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f06b8005fd0 tx=0x7f06b80094d0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 msgr2=0x7f06cc156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc003910 comp rx=0 tx=0).stop 2024-09-13T08:38:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 shutdown_connections 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f06ac078850 0x7f06ac07ad10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06cc15dfa0 0x7f06cc15c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06cc0d4110 0x7f06cc0d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 --2- 172.21.15.31:0/2500746381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06cc0b47f0 0x7f06cc156be0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.458+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 >> 172.21.15.31:0/2500746381 conn(0x7f06cc08d8b0 msgr2=0x7f06cc0b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.459+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 shutdown_connections 2024-09-13T08:38:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.459+0000 7f06cb200640 1 -- 172.21.15.31:0/2500746381 wait complete. 2024-09-13T08:38:05.476 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-13T08:38:05.049561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237200,"kb_used_data":4312,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737520,"statfs":{"total":773094113280,"available":772851220480,"internally_reserved":0,"allocated":4415488,"data_stored":2410188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.846823"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544099+0000","last_change":"2024-09-13T08:38:01.265315+0000","last_active":"2024-09-13T08:38:02.544099+0000","last_peered":"2024-09-13T08:38:02.544099+0000","last_clean":"2024-09-13T08:38:02.544099+0000","last_became_active":"2024-09-13T08:38:01.264978+0000","last_became_peered":"2024-09-13T08:38:01.264978+0000","last_unstale":"2024-09-13T08:38:02.544099+0000","last_undegraded":"2024-09-13T08:38:02.544099+0000","last_fullsized":"2024-09-13T08:38:02.544099+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:05.082400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227930+0000","last_change":"2024-09-13T08:38:01.255508+0000","last_active":"2024-09-13T08:38:03.227930+0000","last_peered":"2024-09-13T08:38:03.227930+0000","last_clean":"2024-09-13T08:38:03.227930+0000","last_became_active":"2024-09-13T08:38:01.255095+0000","last_became_peered":"2024-09-13T08:38:01.255095+0000","last_unstale":"2024-09-13T08:38:03.227930+0000","last_undegraded":"2024-09-13T08:38:03.227930+0000","last_fullsized":"2024-09-13T08:38:03.227930+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238865+0000","last_change":"2024-09-13T08:38:01.249031+0000","last_active":"2024-09-13T08:38:02.238865+0000","last_peered":"2024-09-13T08:38:02.238865+0000","last_clean":"2024-09-13T08:38:02.238865+0000","last_became_active":"2024-09-13T08:38:01.248650+0000","last_became_peered":"2024-09-13T08:38:01.248650+0000","last_unstale":"2024-09-13T08:38:02.238865+0000","last_undegraded":"2024-09-13T08:38:02.238865+0000","last_fullsized":"2024-09-13T08:38:02.238865+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:08.135338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238275+0000","last_change":"2024-09-13T08:38:01.259661+0000","last_active":"2024-09-13T08:38:02.238275+0000","last_peered":"2024-09-13T08:38:02.238275+0000","last_clean":"2024-09-13T08:38:02.238275+0000","last_became_active":"2024-09-13T08:38:01.259003+0000","last_became_peered":"2024-09-13T08:38:01.259003+0000","last_unstale":"2024-09-13T08:38:02.238275+0000","last_undegraded":"2024-09-13T08:38:02.238275+0000","last_fullsized":"2024-09-13T08:38:02.238275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:18:31.085119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227780+0000","last_change":"2024-09-13T08:38:01.245776+0000","last_active":"2024-09-13T08:38:03.227780+0000","last_peered":"2024-09-13T08:38:03.227780+0000","last_clean":"2024-09-13T08:38:03.227780+0000","last_became_active":"2024-09-13T08:38:01.245307+0000","last_became_peered":"2024-09-13T08:38:01.245307+0000","last_unstale":"2024-09-13T08:38:03.227780+0000","last_undegraded":"2024-09-13T08:38:03.227780+0000","last_fullsized":"2024-09-13T08:38:03.227780+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:20.208186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237265+0000","last_change":"2024-09-13T08:38:01.250649+0000","last_active":"2024-09-13T08:38:02.237265+0000","last_peered":"2024-09-13T08:38:02.237265+0000","last_clean":"2024-09-13T08:38:02.237265+0000","last_became_active":"2024-09-13T08:38:01.244965+0000","last_became_peered":"2024-09-13T08:38:01.244965+0000","last_unstale":"2024-09-13T08:38:02.237265+0000","last_undegraded":"2024-09-13T08:38:02.237265+0000","last_fullsized":"2024-09-13T08:38:02.237265+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:30.481072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543996+0000","last_change":"2024-09-13T08:38:01.251861+0000","last_active":"2024-09-13T08:38:02.543996+0000","last_peered":"2024-09-13T08:38:02.543996+0000","last_clean":"2024-09-13T08:38:02.543996+0000","last_became_active":"2024-09-13T08:38:01.251572+0000","last_became_peered":"2024-09-13T08:38:01.251572+0000","last_unstale":"2024-09-13T08:38:02.543996+0000","last_undegraded":"2024-09-13T08:38:02.543996+0000","last_fullsized":"2024-09-13T08:38:02.543996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:08.106248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543887+0000","last_change":"2024-09-13T08:38:01.261029+0000","last_active":"2024-09-13T08:38:02.543887+0000","last_peered":"2024-09-13T08:38:02.543887+0000","last_clean":"2024-09-13T08:38:02.543887+0000","last_became_active":"2024-09-13T08:38:01.255507+0000","last_became_peered":"2024-09-13T08:38:01.255507+0000","last_unstale":"2024-09-13T08:38:02.543887+0000","last_undegraded":"2024-09-13T08:38:02.543887+0000","last_fullsized":"2024-09-13T08:38:02.543887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238926+0000","last_change":"2024-09-13T08:38:01.252431+0000","last_active":"2024-09-13T08:38:02.238926+0000","last_peered":"2024-09-13T08:38:02.238926+0000","last_clean":"2024-09-13T08:38:02.238926+0000","last_became_active":"2024-09-13T08:38:01.252109+0000","last_became_peered":"2024-09-13T08:38:01.252109+0000","last_unstale":"2024-09-13T08:38:02.238926+0000","last_undegraded":"2024-09-13T08:38:02.238926+0000","last_fullsized":"2024-09-13T08:38:02.238926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:43.653097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239018+0000","last_change":"2024-09-13T08:38:01.255919+0000","last_active":"2024-09-13T08:38:02.239018+0000","last_peered":"2024-09-13T08:38:02.239018+0000","last_clean":"2024-09-13T08:38:02.239018+0000","last_became_active":"2024-09-13T08:38:01.255635+0000","last_became_peered":"2024-09-13T08:38:01.255635+0000","last_unstale":"2024-09-13T08:38:02.239018+0000","last_undegraded":"2024-09-13T08:38:02.239018+0000","last_fullsized":"2024-09-13T08:38:02.239018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:51.552194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.262826+0000","last_change":"2024-09-13T08:38:01.259150+0000","last_active":"2024-09-13T08:38:01.262826+0000","last_peered":"2024-09-13T08:38:01.262826+0000","last_clean":"2024-09-13T08:38:01.262826+0000","last_became_active":"2024-09-13T08:38:01.258385+0000","last_became_peered":"2024-09-13T08:38:01.258385+0000","last_unstale":"2024-09-13T08:38:01.262826+0000","last_undegraded":"2024-09-13T08:38:01.262826+0000","last_fullsized":"2024-09-13T08:38:01.262826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:27.672690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238640+0000","last_change":"2024-09-13T08:38:01.243100+0000","last_active":"2024-09-13T08:38:02.238640+0000","last_peered":"2024-09-13T08:38:02.238640+0000","last_clean":"2024-09-13T08:38:02.238640+0000","last_became_active":"2024-09-13T08:38:01.242843+0000","last_became_peered":"2024-09-13T08:38:01.242843+0000","last_unstale":"2024-09-13T08:38:02.238640+0000","last_undegraded":"2024-09-13T08:38:02.238640+0000","last_fullsized":"2024-09-13T08:38:02.238640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:26.840165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.260966+0000","last_change":"2024-09-13T08:38:01.256675+0000","last_active":"2024-09-13T08:38:01.260966+0000","last_peered":"2024-09-13T08:38:01.260966+0000","last_clean":"2024-09-13T08:38:01.260966+0000","last_became_active":"2024-09-13T08:38:01.256345+0000","last_became_peered":"2024-09-13T08:38:01.256345+0000","last_unstale":"2024-09-13T08:38:01.260966+0000","last_undegraded":"2024-09-13T08:38:01.260966+0000","last_fullsized":"2024-09-13T08:38:01.260966+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:20.408271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238086+0000","last_change":"2024-09-13T08:38:01.240090+0000","last_active":"2024-09-13T08:38:02.238086+0000","last_peered":"2024-09-13T08:38:02.238086+0000","last_clean":"2024-09-13T08:38:02.238086+0000","last_became_active":"2024-09-13T08:38:01.239708+0000","last_became_peered":"2024-09-13T08:38:01.239708+0000","last_unstale":"2024-09-13T08:38:02.238086+0000","last_undegraded":"2024-09-13T08:38:02.238086+0000","last_fullsized":"2024-09-13T08:38:02.238086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:36.296533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295629+0000","last_change":"2024-09-13T08:38:01.261773+0000","last_active":"2024-09-13T08:38:03.295629+0000","last_peered":"2024-09-13T08:38:03.295629+0000","last_clean":"2024-09-13T08:38:03.295629+0000","last_became_active":"2024-09-13T08:38:01.261429+0000","last_became_peered":"2024-09-13T08:38:01.261429+0000","last_unstale":"2024-09-13T08:38:03.295629+0000","last_undegraded":"2024-09-13T08:38:03.295629+0000","last_fullsized":"2024-09-13T08:38:03.295629+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295546+0000","last_change":"2024-09-13T08:38:01.249657+0000","last_active":"2024-09-13T08:38:03.295546+0000","last_peered":"2024-09-13T08:38:03.295546+0000","last_clean":"2024-09-13T08:38:03.295546+0000","last_became_active":"2024-09-13T08:38:01.249163+0000","last_became_peered":"2024-09-13T08:38:01.249163+0000","last_unstale":"2024-09-13T08:38:03.295546+0000","last_undegraded":"2024-09-13T08:38:03.295546+0000","last_fullsized":"2024-09-13T08:38:03.295546+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:12:17.614756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264896+0000","last_change":"2024-09-13T08:38:01.259313+0000","last_active":"2024-09-13T08:38:01.264896+0000","last_peered":"2024-09-13T08:38:01.264896+0000","last_clean":"2024-09-13T08:38:01.264896+0000","last_became_active":"2024-09-13T08:38:01.259016+0000","last_became_peered":"2024-09-13T08:38:01.259016+0000","last_unstale":"2024-09-13T08:38:01.264896+0000","last_undegraded":"2024-09-13T08:38:01.264896+0000","last_fullsized":"2024-09-13T08:38:01.264896+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:27.180659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238053+0000","last_change":"2024-09-13T08:38:01.243966+0000","last_active":"2024-09-13T08:38:02.238053+0000","last_peered":"2024-09-13T08:38:02.238053+0000","last_clean":"2024-09-13T08:38:02.238053+0000","last_became_active":"2024-09-13T08:38:01.240660+0000","last_became_peered":"2024-09-13T08:38:01.240660+0000","last_unstale":"2024-09-13T08:38:02.238053+0000","last_undegraded":"2024-09-13T08:38:02.238053+0000","last_fullsized":"2024-09-13T08:38:02.238053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:59.979864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237280+0000","last_change":"2024-09-13T08:38:01.245804+0000","last_active":"2024-09-13T08:38:02.237280+0000","last_peered":"2024-09-13T08:38:02.237280+0000","last_clean":"2024-09-13T08:38:02.237280+0000","last_became_active":"2024-09-13T08:38:01.244554+0000","last_became_peered":"2024-09-13T08:38:01.244554+0000","last_unstale":"2024-09-13T08:38:02.237280+0000","last_undegraded":"2024-09-13T08:38:02.237280+0000","last_fullsized":"2024-09-13T08:38:02.237280+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:26.840165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264759+0000","last_change":"2024-09-13T08:38:01.263620+0000","last_active":"2024-09-13T08:38:01.264759+0000","last_peered":"2024-09-13T08:38:01.264759+0000","last_clean":"2024-09-13T08:38:01.264759+0000","last_became_active":"2024-09-13T08:38:01.263332+0000","last_became_peered":"2024-09-13T08:38:01.263332+0000","last_unstale":"2024-09-13T08:38:01.264759+0000","last_undegraded":"2024-09-13T08:38:01.264759+0000","last_fullsized":"2024-09-13T08:38:01.264759+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:06.240183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295714+0000","last_change":"2024-09-13T08:38:01.260291+0000","last_active":"2024-09-13T08:38:03.295714+0000","last_peered":"2024-09-13T08:38:03.295714+0000","last_clean":"2024-09-13T08:38:03.295714+0000","last_became_active":"2024-09-13T08:38:01.259926+0000","last_became_peered":"2024-09-13T08:38:01.259926+0000","last_unstale":"2024-09-13T08:38:03.295714+0000","last_undegraded":"2024-09-13T08:38:03.295714+0000","last_fullsized":"2024-09-13T08:38:03.295714+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:19.986671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544147+0000","last_change":"2024-09-13T08:38:01.259332+0000","last_active":"2024-09-13T08:38:02.544147+0000","last_peered":"2024-09-13T08:38:02.544147+0000","last_clean":"2024-09-13T08:38:02.544147+0000","last_became_active":"2024-09-13T08:38:01.259080+0000","last_became_peered":"2024-09-13T08:38:01.259080+0000","last_unstale":"2024-09-13T08:38:02.544147+0000","last_undegraded":"2024-09-13T08:38:02.544147+0000","last_fullsized":"2024-09-13T08:38:02.544147+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:19.986671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.255885+0000","last_change":"2024-09-13T08:38:01.252742+0000","last_active":"2024-09-13T08:38:01.255885+0000","last_peered":"2024-09-13T08:38:01.255885+0000","last_clean":"2024-09-13T08:38:01.255885+0000","last_became_active":"2024-09-13T08:38:01.251518+0000","last_became_peered":"2024-09-13T08:38:01.251518+0000","last_unstale":"2024-09-13T08:38:01.255885+0000","last_undegraded":"2024-09-13T08:38:01.255885+0000","last_fullsized":"2024-09-13T08:38:01.255885+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:33:18.237176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543809+0000","last_change":"2024-09-13T08:38:01.258035+0000","last_active":"2024-09-13T08:38:02.543809+0000","last_peered":"2024-09-13T08:38:02.543809+0000","last_clean":"2024-09-13T08:38:02.543809+0000","last_became_active":"2024-09-13T08:38:01.257764+0000","last_became_peered":"2024-09-13T08:38:01.257764+0000","last_unstale":"2024-09-13T08:38:02.543809+0000","last_undegraded":"2024-09-13T08:38:02.543809+0000","last_fullsized":"2024-09-13T08:38:02.543809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:08:12.458886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238965+0000","last_change":"2024-09-13T08:38:01.243053+0000","last_active":"2024-09-13T08:38:02.238965+0000","last_peered":"2024-09-13T08:38:02.238965+0000","last_clean":"2024-09-13T08:38:02.238965+0000","last_became_active":"2024-09-13T08:38:01.242519+0000","last_became_peered":"2024-09-13T08:38:01.242519+0000","last_unstale":"2024-09-13T08:38:02.238965+0000","last_undegraded":"2024-09-13T08:38:02.238965+0000","last_fullsized":"2024-09-13T08:38:02.238965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:52:52.670617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239001+0000","last_change":"2024-09-13T08:38:01.243267+0000","last_active":"2024-09-13T08:38:02.239001+0000","last_peered":"2024-09-13T08:38:02.239001+0000","last_clean":"2024-09-13T08:38:02.239001+0000","last_became_active":"2024-09-13T08:38:01.242723+0000","last_became_peered":"2024-09-13T08:38:01.242723+0000","last_unstale":"2024-09-13T08:38:02.239001+0000","last_undegraded":"2024-09-13T08:38:02.239001+0000","last_fullsized":"2024-09-13T08:38:02.239001+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:45:35.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227950+0000","last_change":"2024-09-13T08:38:01.253642+0000","last_active":"2024-09-13T08:38:03.227950+0000","last_peered":"2024-09-13T08:38:03.227950+0000","last_clean":"2024-09-13T08:38:03.227950+0000","last_became_active":"2024-09-13T08:38:01.250652+0000","last_became_peered":"2024-09-13T08:38:01.250652+0000","last_unstale":"2024-09-13T08:38:03.227950+0000","last_undegraded":"2024-09-13T08:38:03.227950+0000","last_fullsized":"2024-09-13T08:38:03.227950+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:39.892681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544145+0000","last_change":"2024-09-13T08:38:01.261029+0000","last_active":"2024-09-13T08:38:02.544145+0000","last_peered":"2024-09-13T08:38:02.544145+0000","last_clean":"2024-09-13T08:38:02.544145+0000","last_became_active":"2024-09-13T08:38:01.260770+0000","last_became_peered":"2024-09-13T08:38:01.260770+0000","last_unstale":"2024-09-13T08:38:02.544145+0000","last_undegraded":"2024-09-13T08:38:02.544145+0000","last_fullsized":"2024-09-13T08:38:02.544145+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:54.694090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295766+0000","last_change":"2024-09-13T08:38:01.253834+0000","last_active":"2024-09-13T08:38:03.295766+0000","last_peered":"2024-09-13T08:38:03.295766+0000","last_clean":"2024-09-13T08:38:03.295766+0000","last_became_active":"2024-09-13T08:38:01.250174+0000","last_became_peered":"2024-09-13T08:38:01.250174+0000","last_unstale":"2024-09-13T08:38:03.295766+0000","last_undegraded":"2024-09-13T08:38:03.295766+0000","last_fullsized":"2024-09-13T08:38:03.295766+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295805+0000","last_change":"2024-09-13T08:38:01.260047+0000","last_active":"2024-09-13T08:38:03.295805+0000","last_peered":"2024-09-13T08:38:03.295805+0000","last_clean":"2024-09-13T08:38:03.295805+0000","last_became_active":"2024-09-13T08:38:01.249658+0000","last_became_peered":"2024-09-13T08:38:01.249658+0000","last_unstale":"2024-09-13T08:38:03.295805+0000","last_undegraded":"2024-09-13T08:38:03.295805+0000","last_fullsized":"2024-09-13T08:38:03.295805+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:56.303136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544295+0000","last_change":"2024-09-13T08:38:01.244840+0000","last_active":"2024-09-13T08:38:02.544295+0000","last_peered":"2024-09-13T08:38:02.544295+0000","last_clean":"2024-09-13T08:38:02.544295+0000","last_became_active":"2024-09-13T08:38:01.244519+0000","last_became_peered":"2024-09-13T08:38:01.244519+0000","last_unstale":"2024-09-13T08:38:02.544295+0000","last_undegraded":"2024-09-13T08:38:02.544295+0000","last_fullsized":"2024-09-13T08:38:02.544295+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:45.642304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.254265+0000","last_change":"2024-09-13T08:38:01.249570+0000","last_active":"2024-09-13T08:38:01.254265+0000","last_peered":"2024-09-13T08:38:01.254265+0000","last_clean":"2024-09-13T08:38:01.254265+0000","last_became_active":"2024-09-13T08:38:01.249177+0000","last_became_peered":"2024-09-13T08:38:01.249177+0000","last_unstale":"2024-09-13T08:38:01.254265+0000","last_undegraded":"2024-09-13T08:38:01.254265+0000","last_fullsized":"2024-09-13T08:38:01.254265+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:35.930050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.250446+0000","last_change":"2024-09-13T08:38:01.246568+0000","last_active":"2024-09-13T08:38:01.250446+0000","last_peered":"2024-09-13T08:38:01.250446+0000","last_clean":"2024-09-13T08:38:01.250446+0000","last_became_active":"2024-09-13T08:38:01.246225+0000","last_became_peered":"2024-09-13T08:38:01.246225+0000","last_unstale":"2024-09-13T08:38:01.250446+0000","last_undegraded":"2024-09-13T08:38:01.250446+0000","last_fullsized":"2024-09-13T08:38:01.250446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:48:04.157436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.246655+0000","last_change":"2024-09-13T08:38:00.230433+0000","last_active":"2024-09-13T08:38:01.246655+0000","last_peered":"2024-09-13T08:38:01.246655+0000","last_clean":"2024-09-13T08:38:01.246655+0000","last_became_active":"2024-09-13T08:38:00.228507+0000","last_became_peered":"2024-09-13T08:38:00.228507+0000","last_unstale":"2024-09-13T08:38:01.246655+0000","last_undegraded":"2024-09-13T08:38:01.246655+0000","last_fullsized":"2024-09-13T08:38:01.246655+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:57.172335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544118+0000","last_change":"2024-09-13T08:38:01.252288+0000","last_active":"2024-09-13T08:38:02.544118+0000","last_peered":"2024-09-13T08:38:02.544118+0000","last_clean":"2024-09-13T08:38:02.544118+0000","last_became_active":"2024-09-13T08:38:01.252013+0000","last_became_peered":"2024-09-13T08:38:01.252013+0000","last_unstale":"2024-09-13T08:38:02.544118+0000","last_undegraded":"2024-09-13T08:38:02.544118+0000","last_fullsized":"2024-09-13T08:38:02.544118+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:25.289292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239146+0000","last_change":"2024-09-13T08:38:00.227234+0000","last_active":"2024-09-13T08:38:02.239146+0000","last_peered":"2024-09-13T08:38:02.239146+0000","last_clean":"2024-09-13T08:38:02.239146+0000","last_became_active":"2024-09-13T08:38:00.226880+0000","last_became_peered":"2024-09-13T08:38:00.226880+0000","last_unstale":"2024-09-13T08:38:02.239146+0000","last_undegraded":"2024-09-13T08:38:02.239146+0000","last_fullsized":"2024-09-13T08:38:02.239146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:21.715485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544266+0000","last_change":"2024-09-13T08:38:01.258862+0000","last_active":"2024-09-13T08:38:02.544266+0000","last_peered":"2024-09-13T08:38:02.544266+0000","last_clean":"2024-09-13T08:38:02.544266+0000","last_became_active":"2024-09-13T08:38:01.258469+0000","last_became_peered":"2024-09-13T08:38:01.258469+0000","last_unstale":"2024-09-13T08:38:02.544266+0000","last_undegraded":"2024-09-13T08:38:02.544266+0000","last_fullsized":"2024-09-13T08:38:02.544266+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:32.007345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237748+0000","last_change":"2024-09-13T08:38:00.224928+0000","last_active":"2024-09-13T08:38:02.237748+0000","last_peered":"2024-09-13T08:38:02.237748+0000","last_clean":"2024-09-13T08:38:02.237748+0000","last_became_active":"2024-09-13T08:38:00.224586+0000","last_became_peered":"2024-09-13T08:38:00.224586+0000","last_unstale":"2024-09-13T08:38:02.237748+0000","last_undegraded":"2024-09-13T08:38:02.237748+0000","last_fullsized":"2024-09-13T08:38:02.237748+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:33.891448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238953+0000","last_change":"2024-09-13T08:38:01.246402+0000","last_active":"2024-09-13T08:38:02.238953+0000","last_peered":"2024-09-13T08:38:02.238953+0000","last_clean":"2024-09-13T08:38:02.238953+0000","last_became_active":"2024-09-13T08:38:01.246074+0000","last_became_peered":"2024-09-13T08:38:01.246074+0000","last_unstale":"2024-09-13T08:38:02.238953+0000","last_undegraded":"2024-09-13T08:38:02.238953+0000","last_fullsized":"2024-09-13T08:38:02.238953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:14.527936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238573+0000","last_change":"2024-09-13T08:38:00.234092+0000","last_active":"2024-09-13T08:38:02.238573+0000","last_peered":"2024-09-13T08:38:02.238573+0000","last_clean":"2024-09-13T08:38:02.238573+0000","last_became_active":"2024-09-13T08:38:00.233741+0000","last_became_peered":"2024-09-13T08:38:00.233741+0000","last_unstale":"2024-09-13T08:38:02.238573+0000","last_undegraded":"2024-09-13T08:38:02.238573+0000","last_fullsized":"2024-09-13T08:38:02.238573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:06.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238493+0000","last_change":"2024-09-13T08:38:01.253342+0000","last_active":"2024-09-13T08:38:02.238493+0000","last_peered":"2024-09-13T08:38:02.238493+0000","last_clean":"2024-09-13T08:38:02.238493+0000","last_became_active":"2024-09-13T08:38:01.252996+0000","last_became_peered":"2024-09-13T08:38:01.252996+0000","last_unstale":"2024-09-13T08:38:02.238493+0000","last_undegraded":"2024-09-13T08:38:02.238493+0000","last_fullsized":"2024-09-13T08:38:02.238493+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:30.481072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543762+0000","last_change":"2024-09-13T08:38:00.234123+0000","last_active":"2024-09-13T08:38:02.543762+0000","last_peered":"2024-09-13T08:38:02.543762+0000","last_clean":"2024-09-13T08:38:02.543762+0000","last_became_active":"2024-09-13T08:38:00.233858+0000","last_became_peered":"2024-09-13T08:38:00.233858+0000","last_unstale":"2024-09-13T08:38:02.543762+0000","last_undegraded":"2024-09-13T08:38:02.543762+0000","last_fullsized":"2024-09-13T08:38:02.543762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:29.474385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543813+0000","last_change":"2024-09-13T08:38:01.260632+0000","last_active":"2024-09-13T08:38:02.543813+0000","last_peered":"2024-09-13T08:38:02.543813+0000","last_clean":"2024-09-13T08:38:02.543813+0000","last_became_active":"2024-09-13T08:38:01.260408+0000","last_became_peered":"2024-09-13T08:38:01.260408+0000","last_unstale":"2024-09-13T08:38:02.543813+0000","last_undegraded":"2024-09-13T08:38:02.543813+0000","last_fullsized":"2024-09-13T08:38:02.543813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:20.698433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543700+0000","last_change":"2024-09-13T08:38:00.231574+0000","last_active":"2024-09-13T08:38:02.543700+0000","last_peered":"2024-09-13T08:38:02.543700+0000","last_clean":"2024-09-13T08:38:02.543700+0000","last_became_active":"2024-09-13T08:38:00.231273+0000","last_became_peered":"2024-09-13T08:38:00.231273+0000","last_unstale":"2024-09-13T08:38:02.543700+0000","last_undegraded":"2024-09-13T08:38:02.543700+0000","last_fullsized":"2024-09-13T08:38:02.543700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:25.833478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238288+0000","last_change":"2024-09-13T08:38:01.246029+0000","last_active":"2024-09-13T08:38:02.238288+0000","last_peered":"2024-09-13T08:38:02.238288+0000","last_clean":"2024-09-13T08:38:02.238288+0000","last_became_active":"2024-09-13T08:38:01.245748+0000","last_became_peered":"2024-09-13T08:38:01.245748+0000","last_unstale":"2024-09-13T08:38:02.238288+0000","last_undegraded":"2024-09-13T08:38:02.238288+0000","last_fullsized":"2024-09-13T08:38:02.238288+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:59.758539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237589+0000","last_change":"2024-09-13T08:38:00.223721+0000","last_active":"2024-09-13T08:38:02.237589+0000","last_peered":"2024-09-13T08:38:02.237589+0000","last_clean":"2024-09-13T08:38:02.237589+0000","last_became_active":"2024-09-13T08:38:00.223183+0000","last_became_peered":"2024-09-13T08:38:00.223183+0000","last_unstale":"2024-09-13T08:38:02.237589+0000","last_undegraded":"2024-09-13T08:38:02.237589+0000","last_fullsized":"2024-09-13T08:38:02.237589+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:31.193415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236776+0000","last_change":"2024-09-13T08:38:01.239689+0000","last_active":"2024-09-13T08:38:02.236776+0000","last_peered":"2024-09-13T08:38:02.236776+0000","last_clean":"2024-09-13T08:38:02.236776+0000","last_became_active":"2024-09-13T08:38:01.239393+0000","last_became_peered":"2024-09-13T08:38:01.239393+0000","last_unstale":"2024-09-13T08:38:02.236776+0000","last_undegraded":"2024-09-13T08:38:02.236776+0000","last_fullsized":"2024-09-13T08:38:02.236776+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:01:35.526012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544006+0000","last_change":"2024-09-13T08:38:00.228561+0000","last_active":"2024-09-13T08:38:02.544006+0000","last_peered":"2024-09-13T08:38:02.544006+0000","last_clean":"2024-09-13T08:38:02.544006+0000","last_became_active":"2024-09-13T08:38:00.225937+0000","last_became_peered":"2024-09-13T08:38:00.225937+0000","last_unstale":"2024-09-13T08:38:02.544006+0000","last_undegraded":"2024-09-13T08:38:02.544006+0000","last_fullsized":"2024-09-13T08:38:02.544006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:58.751851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.247714+0000","last_change":"2024-09-13T08:38:01.239615+0000","last_active":"2024-09-13T08:38:01.247714+0000","last_peered":"2024-09-13T08:38:01.247714+0000","last_clean":"2024-09-13T08:38:01.247714+0000","last_became_active":"2024-09-13T08:38:01.238923+0000","last_became_peered":"2024-09-13T08:38:01.238923+0000","last_unstale":"2024-09-13T08:38:01.247714+0000","last_undegraded":"2024-09-13T08:38:01.247714+0000","last_fullsized":"2024-09-13T08:38:01.247714+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:34.152667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.228008+0000","last_change":"2024-09-13T08:38:00.233030+0000","last_active":"2024-09-13T08:38:03.228008+0000","last_peered":"2024-09-13T08:38:03.228008+0000","last_clean":"2024-09-13T08:38:03.228008+0000","last_became_active":"2024-09-13T08:38:00.232672+0000","last_became_peered":"2024-09-13T08:38:00.232672+0000","last_unstale":"2024-09-13T08:38:03.228008+0000","last_undegraded":"2024-09-13T08:38:03.228008+0000","last_fullsized":"2024-09-13T08:38:03.228008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:05:16.230276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238910+0000","last_change":"2024-09-13T08:38:01.239832+0000","last_active":"2024-09-13T08:38:02.238910+0000","last_peered":"2024-09-13T08:38:02.238910+0000","last_clean":"2024-09-13T08:38:02.238910+0000","last_became_active":"2024-09-13T08:38:01.239494+0000","last_became_peered":"2024-09-13T08:38:01.239494+0000","last_unstale":"2024-09-13T08:38:02.238910+0000","last_undegraded":"2024-09-13T08:38:02.238910+0000","last_fullsized":"2024-09-13T08:38:02.238910+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.065422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227866+0000","last_change":"2024-09-13T08:38:00.228208+0000","last_active":"2024-09-13T08:38:03.227866+0000","last_peered":"2024-09-13T08:38:03.227866+0000","last_clean":"2024-09-13T08:38:03.227866+0000","last_became_active":"2024-09-13T08:38:00.227935+0000","last_became_peered":"2024-09-13T08:38:00.227935+0000","last_unstale":"2024-09-13T08:38:03.227866+0000","last_undegraded":"2024-09-13T08:38:03.227866+0000","last_fullsized":"2024-09-13T08:38:03.227866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:40.619149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238602+0000","last_change":"2024-09-13T08:38:01.259738+0000","last_active":"2024-09-13T08:38:02.238602+0000","last_peered":"2024-09-13T08:38:02.238602+0000","last_clean":"2024-09-13T08:38:02.238602+0000","last_became_active":"2024-09-13T08:38:01.259382+0000","last_became_peered":"2024-09-13T08:38:01.259382+0000","last_unstale":"2024-09-13T08:38:02.238602+0000","last_undegraded":"2024-09-13T08:38:02.238602+0000","last_fullsized":"2024-09-13T08:38:02.238602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:55.850218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264685+0000","last_change":"2024-09-13T08:38:00.233958+0000","last_active":"2024-09-13T08:38:01.264685+0000","last_peered":"2024-09-13T08:38:01.264685+0000","last_clean":"2024-09-13T08:38:01.264685+0000","last_became_active":"2024-09-13T08:38:00.232954+0000","last_became_peered":"2024-09-13T08:38:00.232954+0000","last_unstale":"2024-09-13T08:38:01.264685+0000","last_undegraded":"2024-09-13T08:38:01.264685+0000","last_fullsized":"2024-09-13T08:38:01.264685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:55.296448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.262802+0000","last_change":"2024-09-13T08:38:01.258743+0000","last_active":"2024-09-13T08:38:01.262802+0000","last_peered":"2024-09-13T08:38:01.262802+0000","last_clean":"2024-09-13T08:38:01.262802+0000","last_became_active":"2024-09-13T08:38:01.257969+0000","last_became_peered":"2024-09-13T08:38:01.257969+0000","last_unstale":"2024-09-13T08:38:01.262802+0000","last_undegraded":"2024-09-13T08:38:01.262802+0000","last_fullsized":"2024-09-13T08:38:01.262802+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:53:28.166981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":19,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264574+0000","last_change":"2024-09-13T08:38:00.235585+0000","last_active":"2024-09-13T08:38:01.264574+0000","last_peered":"2024-09-13T08:38:01.264574+0000","last_clean":"2024-09-13T08:38:01.264574+0000","last_became_active":"2024-09-13T08:38:00.235260+0000","last_became_peered":"2024-09-13T08:38:00.235260+0000","last_unstale":"2024-09-13T08:38:01.264574+0000","last_undegraded":"2024-09-13T08:38:01.264574+0000","last_fullsized":"2024-09-13T08:38:01.264574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:39:26.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.251129+0000","last_change":"2024-09-13T08:38:01.245709+0000","last_active":"2024-09-13T08:38:01.251129+0000","last_peered":"2024-09-13T08:38:01.251129+0000","last_clean":"2024-09-13T08:38:01.251129+0000","last_became_active":"2024-09-13T08:38:01.245244+0000","last_became_peered":"2024-09-13T08:38:01.245244+0000","last_unstale":"2024-09-13T08:38:01.251129+0000","last_undegraded":"2024-09-13T08:38:01.251129+0000","last_fullsized":"2024-09-13T08:38:01.251129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:33.546187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543992+0000","last_change":"2024-09-13T08:38:00.226943+0000","last_active":"2024-09-13T08:38:02.543992+0000","last_peered":"2024-09-13T08:38:02.543992+0000","last_clean":"2024-09-13T08:38:02.543992+0000","last_became_active":"2024-09-13T08:38:00.226597+0000","last_became_peered":"2024-09-13T08:38:00.226597+0000","last_unstale":"2024-09-13T08:38:02.543992+0000","last_undegraded":"2024-09-13T08:38:02.543992+0000","last_fullsized":"2024-09-13T08:38:02.543992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:07.128650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238172+0000","last_change":"2024-09-13T08:38:01.243249+0000","last_active":"2024-09-13T08:38:02.238172+0000","last_peered":"2024-09-13T08:38:02.238172+0000","last_clean":"2024-09-13T08:38:02.238172+0000","last_became_active":"2024-09-13T08:38:01.242938+0000","last_became_peered":"2024-09-13T08:38:01.242938+0000","last_unstale":"2024-09-13T08:38:02.238172+0000","last_undegraded":"2024-09-13T08:38:02.238172+0000","last_fullsized":"2024-09-13T08:38:02.238172+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:41:41.441133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237127+0000","last_change":"2024-09-13T08:38:00.230779+0000","last_active":"2024-09-13T08:38:02.237127+0000","last_peered":"2024-09-13T08:38:02.237127+0000","last_clean":"2024-09-13T08:38:02.237127+0000","last_became_active":"2024-09-13T08:38:00.230485+0000","last_became_peered":"2024-09-13T08:38:00.230485+0000","last_unstale":"2024-09-13T08:38:02.237127+0000","last_undegraded":"2024-09-13T08:38:02.237127+0000","last_fullsized":"2024-09-13T08:38:02.237127+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:47.186354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544092+0000","last_change":"2024-09-13T08:38:01.255949+0000","last_active":"2024-09-13T08:38:02.544092+0000","last_peered":"2024-09-13T08:38:02.544092+0000","last_clean":"2024-09-13T08:38:02.544092+0000","last_became_active":"2024-09-13T08:38:01.255155+0000","last_became_peered":"2024-09-13T08:38:01.255155+0000","last_unstale":"2024-09-13T08:38:02.544092+0000","last_undegraded":"2024-09-13T08:38:02.544092+0000","last_fullsized":"2024-09-13T08:38:02.544092+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236977+0000","last_change":"2024-09-13T08:38:00.228323+0000","last_active":"2024-09-13T08:38:02.236977+0000","last_peered":"2024-09-13T08:38:02.236977+0000","last_clean":"2024-09-13T08:38:02.236977+0000","last_became_active":"2024-09-13T08:38:00.223345+0000","last_became_peered":"2024-09-13T08:38:00.223345+0000","last_unstale":"2024-09-13T08:38:02.236977+0000","last_undegraded":"2024-09-13T08:38:02.236977+0000","last_fullsized":"2024-09-13T08:38:02.236977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:28.474359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.245822+0000","last_change":"2024-09-13T08:38:01.242444+0000","last_active":"2024-09-13T08:38:01.245822+0000","last_peered":"2024-09-13T08:38:01.245822+0000","last_clean":"2024-09-13T08:38:01.245822+0000","last_became_active":"2024-09-13T08:38:01.242103+0000","last_became_peered":"2024-09-13T08:38:01.242103+0000","last_unstale":"2024-09-13T08:38:01.245822+0000","last_undegraded":"2024-09-13T08:38:01.245822+0000","last_fullsized":"2024-09-13T08:38:01.245822+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:45.642304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":23,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236859+0000","last_change":"2024-09-13T08:38:00.236587+0000","last_active":"2024-09-13T08:38:02.236859+0000","last_peered":"2024-09-13T08:38:02.236859+0000","last_clean":"2024-09-13T08:38:02.236859+0000","last_became_active":"2024-09-13T08:38:00.236259+0000","last_became_peered":"2024-09-13T08:38:00.236259+0000","last_unstale":"2024-09-13T08:38:02.236859+0000","last_undegraded":"2024-09-13T08:38:02.236859+0000","last_fullsized":"2024-09-13T08:38:02.236859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:52.214727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.259795+0000","last_change":"2024-09-13T08:38:01.255145+0000","last_active":"2024-09-13T08:38:01.259795+0000","last_peered":"2024-09-13T08:38:01.259795+0000","last_clean":"2024-09-13T08:38:01.259795+0000","last_became_active":"2024-09-13T08:38:01.254877+0000","last_became_peered":"2024-09-13T08:38:01.254877+0000","last_unstale":"2024-09-13T08:38:01.259795+0000","last_undegraded":"2024-09-13T08:38:01.259795+0000","last_fullsized":"2024-09-13T08:38:01.259795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:45:09.792867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237746+0000","last_change":"2024-09-13T08:38:00.228903+0000","last_active":"2024-09-13T08:38:02.237746+0000","last_peered":"2024-09-13T08:38:02.237746+0000","last_clean":"2024-09-13T08:38:02.237746+0000","last_became_active":"2024-09-13T08:38:00.227671+0000","last_became_peered":"2024-09-13T08:38:00.227671+0000","last_unstale":"2024-09-13T08:38:02.237746+0000","last_undegraded":"2024-09-13T08:38:02.237746+0000","last_fullsized":"2024-09-13T08:38:02.237746+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:24.940664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239035+0000","last_change":"2024-09-13T08:38:01.245734+0000","last_active":"2024-09-13T08:38:02.239035+0000","last_peered":"2024-09-13T08:38:02.239035+0000","last_clean":"2024-09-13T08:38:02.239035+0000","last_became_active":"2024-09-13T08:38:01.245398+0000","last_became_peered":"2024-09-13T08:38:01.245398+0000","last_unstale":"2024-09-13T08:38:02.239035+0000","last_undegraded":"2024-09-13T08:38:02.239035+0000","last_fullsized":"2024-09-13T08:38:02.239035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:59.979864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543829+0000","last_change":"2024-09-13T08:38:00.233559+0000","last_active":"2024-09-13T08:38:02.543829+0000","last_peered":"2024-09-13T08:38:02.543829+0000","last_clean":"2024-09-13T08:38:02.543829+0000","last_became_active":"2024-09-13T08:38:00.233275+0000","last_became_peered":"2024-09-13T08:38:00.233275+0000","last_unstale":"2024-09-13T08:38:02.543829+0000","last_undegraded":"2024-09-13T08:38:02.543829+0000","last_fullsized":"2024-09-13T08:38:02.543829+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:42.646410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227781+0000","last_change":"2024-09-13T08:38:01.238818+0000","last_active":"2024-09-13T08:38:03.227781+0000","last_peered":"2024-09-13T08:38:03.227781+0000","last_clean":"2024-09-13T08:38:03.227781+0000","last_became_active":"2024-09-13T08:38:01.238499+0000","last_became_peered":"2024-09-13T08:38:01.238499+0000","last_unstale":"2024-09-13T08:38:03.227781+0000","last_undegraded":"2024-09-13T08:38:03.227781+0000","last_fullsized":"2024-09-13T08:38:03.227781+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.246759+0000","last_change":"2024-09-13T08:38:00.234550+0000","last_active":"2024-09-13T08:38:01.246759+0000","last_peered":"2024-09-13T08:38:01.246759+0000","last_clean":"2024-09-13T08:38:01.246759+0000","last_became_active":"2024-09-13T08:38:00.233680+0000","last_became_peered":"2024-09-13T08:38:00.233680+0000","last_unstale":"2024-09-13T08:38:01.246759+0000","last_undegraded":"2024-09-13T08:38:01.246759+0000","last_fullsized":"2024-09-13T08:38:01.246759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:04.075713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237504+0000","last_change":"2024-09-13T08:38:01.247351+0000","last_active":"2024-09-13T08:38:02.237504+0000","last_peered":"2024-09-13T08:38:02.237504+0000","last_clean":"2024-09-13T08:38:02.237504+0000","last_became_active":"2024-09-13T08:38:01.246206+0000","last_became_peered":"2024-09-13T08:38:01.246206+0000","last_unstale":"2024-09-13T08:38:02.237504+0000","last_undegraded":"2024-09-13T08:38:02.237504+0000","last_fullsized":"2024-09-13T08:38:02.237504+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:08.135338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295973+0000","last_change":"2024-09-13T08:38:00.216931+0000","last_active":"2024-09-13T08:38:03.295973+0000","last_peered":"2024-09-13T08:38:03.295973+0000","last_clean":"2024-09-13T08:38:03.295973+0000","last_became_active":"2024-09-13T08:38:00.216603+0000","last_became_peered":"2024-09-13T08:38:00.216603+0000","last_unstale":"2024-09-13T08:38:03.295973+0000","last_undegraded":"2024-09-13T08:38:03.295973+0000","last_fullsized":"2024-09-13T08:38:03.295973+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:36.507927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.263964+0000","last_change":"2024-09-13T08:38:01.262153+0000","last_active":"2024-09-13T08:38:01.263964+0000","last_peered":"2024-09-13T08:38:01.263964+0000","last_clean":"2024-09-13T08:38:01.263964+0000","last_became_active":"2024-09-13T08:38:01.261856+0000","last_became_peered":"2024-09-13T08:38:01.261856+0000","last_unstale":"2024-09-13T08:38:01.263964+0000","last_undegraded":"2024-09-13T08:38:01.263964+0000","last_fullsized":"2024-09-13T08:38:01.263964+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:08:33.091133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227955+0000","last_change":"2024-09-13T08:38:00.232078+0000","last_active":"2024-09-13T08:38:03.227955+0000","last_peered":"2024-09-13T08:38:03.227955+0000","last_clean":"2024-09-13T08:38:03.227955+0000","last_became_active":"2024-09-13T08:38:00.231821+0000","last_became_peered":"2024-09-13T08:38:00.231821+0000","last_unstale":"2024-09-13T08:38:03.227955+0000","last_undegraded":"2024-09-13T08:38:03.227955+0000","last_fullsized":"2024-09-13T08:38:03.227955+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:36.507927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227833+0000","last_change":"2024-09-13T08:38:01.234497+0000","last_active":"2024-09-13T08:38:03.227833+0000","last_peered":"2024-09-13T08:38:03.227833+0000","last_clean":"2024-09-13T08:38:03.227833+0000","last_became_active":"2024-09-13T08:38:01.234202+0000","last_became_peered":"2024-09-13T08:38:01.234202+0000","last_unstale":"2024-09-13T08:38:03.227833+0000","last_undegraded":"2024-09-13T08:38:03.227833+0000","last_fullsized":"2024-09-13T08:38:03.227833+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:08.106248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543897+0000","last_change":"2024-09-13T08:38:00.236286+0000","last_active":"2024-09-13T08:38:02.543897+0000","last_peered":"2024-09-13T08:38:02.543897+0000","last_clean":"2024-09-13T08:38:02.543897+0000","last_became_active":"2024-09-13T08:38:00.236038+0000","last_became_peered":"2024-09-13T08:38:00.236038+0000","last_unstale":"2024-09-13T08:38:02.543897+0000","last_undegraded":"2024-09-13T08:38:02.543897+0000","last_fullsized":"2024-09-13T08:38:02.543897+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:58.973177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.256445+0000","last_change":"2024-09-13T08:38:01.253708+0000","last_active":"2024-09-13T08:38:01.256445+0000","last_peered":"2024-09-13T08:38:01.256445+0000","last_clean":"2024-09-13T08:38:01.256445+0000","last_became_active":"2024-09-13T08:38:01.253387+0000","last_became_peered":"2024-09-13T08:38:01.253387+0000","last_unstale":"2024-09-13T08:38:01.256445+0000","last_undegraded":"2024-09-13T08:38:01.256445+0000","last_fullsized":"2024-09-13T08:38:01.256445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:43.790924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238400+0000","last_change":"2024-09-13T08:38:00.230487+0000","last_active":"2024-09-13T08:38:02.238400+0000","last_peered":"2024-09-13T08:38:02.238400+0000","last_clean":"2024-09-13T08:38:02.238400+0000","last_became_active":"2024-09-13T08:38:00.230118+0000","last_became_peered":"2024-09-13T08:38:00.230118+0000","last_unstale":"2024-09-13T08:38:02.238400+0000","last_undegraded":"2024-09-13T08:38:02.238400+0000","last_fullsized":"2024-09-13T08:38:02.238400+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:31.193415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238374+0000","last_change":"2024-09-13T08:38:01.237796+0000","last_active":"2024-09-13T08:38:02.238374+0000","last_peered":"2024-09-13T08:38:02.238374+0000","last_clean":"2024-09-13T08:38:02.238374+0000","last_became_active":"2024-09-13T08:38:01.237523+0000","last_became_peered":"2024-09-13T08:38:01.237523+0000","last_unstale":"2024-09-13T08:38:02.238374+0000","last_undegraded":"2024-09-13T08:38:02.238374+0000","last_fullsized":"2024-09-13T08:38:02.238374+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:40.687197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.220781+0000","last_change":"2024-09-13T08:38:00.235473+0000","last_active":"2024-09-13T08:38:01.220781+0000","last_peered":"2024-09-13T08:38:01.220781+0000","last_clean":"2024-09-13T08:38:01.220781+0000","last_became_active":"2024-09-13T08:38:00.234870+0000","last_became_peered":"2024-09-13T08:38:00.234870+0000","last_unstale":"2024-09-13T08:38:01.220781+0000","last_undegraded":"2024-09-13T08:38:01.220781+0000","last_fullsized":"2024-09-13T08:38:01.220781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:18.979983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238098+0000","last_change":"2024-09-13T08:38:01.246140+0000","last_active":"2024-09-13T08:38:02.238098+0000","last_peered":"2024-09-13T08:38:02.238098+0000","last_clean":"2024-09-13T08:38:02.238098+0000","last_became_active":"2024-09-13T08:38:01.245801+0000","last_became_peered":"2024-09-13T08:38:01.245801+0000","last_unstale":"2024-09-13T08:38:02.238098+0000","last_undegraded":"2024-09-13T08:38:02.238098+0000","last_fullsized":"2024-09-13T08:38:02.238098+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:05.544931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.263826+0000","last_change":"2024-09-13T08:38:00.230943+0000","last_active":"2024-09-13T08:38:01.263826+0000","last_peered":"2024-09-13T08:38:01.263826+0000","last_clean":"2024-09-13T08:38:01.263826+0000","last_became_active":"2024-09-13T08:38:00.229735+0000","last_became_peered":"2024-09-13T08:38:00.229735+0000","last_unstale":"2024-09-13T08:38:01.263826+0000","last_undegraded":"2024-09-13T08:38:01.263826+0000","last_fullsized":"2024-09-13T08:38:01.263826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:34:28.589744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.244721+0000","last_change":"2024-09-13T08:38:01.241987+0000","last_active":"2024-09-13T08:38:01.244721+0000","last_peered":"2024-09-13T08:38:01.244721+0000","last_clean":"2024-09-13T08:38:01.244721+0000","last_became_active":"2024-09-13T08:38:01.241702+0000","last_became_peered":"2024-09-13T08:38:01.241702+0000","last_unstale":"2024-09-13T08:38:01.244721+0000","last_undegraded":"2024-09-13T08:38:01.244721+0000","last_fullsized":"2024-09-13T08:38:01.244721+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:32.007345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"36'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.229661+0000","last_change":"2024-09-13T08:38:00.225418+0000","last_active":"2024-09-13T08:38:03.229661+0000","last_peered":"2024-09-13T08:38:03.229661+0000","last_clean":"2024-09-13T08:38:03.229661+0000","last_became_active":"2024-09-13T08:38:00.225076+0000","last_became_peered":"2024-09-13T08:38:00.225076+0000","last_unstale":"2024-09-13T08:38:03.229661+0000","last_undegraded":"2024-09-13T08:38:03.229661+0000","last_fullsized":"2024-09-13T08:38:03.229661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:38:15.861161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237790+0000","last_change":"2024-09-13T08:38:01.256868+0000","last_active":"2024-09-13T08:38:02.237790+0000","last_peered":"2024-09-13T08:38:02.237790+0000","last_clean":"2024-09-13T08:38:02.237790+0000","last_became_active":"2024-09-13T08:38:01.253978+0000","last_became_peered":"2024-09-13T08:38:01.253978+0000","last_unstale":"2024-09-13T08:38:02.237790+0000","last_undegraded":"2024-09-13T08:38:02.237790+0000","last_fullsized":"2024-09-13T08:38:02.237790+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:43.653097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.230376+0000","last_change":"2024-09-13T08:38:00.225331+0000","last_active":"2024-09-13T08:38:01.230376+0000","last_peered":"2024-09-13T08:38:01.230376+0000","last_clean":"2024-09-13T08:38:01.230376+0000","last_became_active":"2024-09-13T08:38:00.225053+0000","last_became_peered":"2024-09-13T08:38:00.225053+0000","last_unstale":"2024-09-13T08:38:01.230376+0000","last_undegraded":"2024-09-13T08:38:01.230376+0000","last_fullsized":"2024-09-13T08:38:01.230376+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:17:06.943972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295861+0000","last_change":"2024-09-13T08:38:01.260348+0000","last_active":"2024-09-13T08:38:03.295861+0000","last_peered":"2024-09-13T08:38:03.295861+0000","last_clean":"2024-09-13T08:38:03.295861+0000","last_became_active":"2024-09-13T08:38:01.250067+0000","last_became_peered":"2024-09-13T08:38:01.250067+0000","last_unstale":"2024-09-13T08:38:03.295861+0000","last_undegraded":"2024-09-13T08:38:03.295861+0000","last_fullsized":"2024-09-13T08:38:03.295861+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:21.804316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.230213+0000","last_change":"2024-09-13T08:38:00.227592+0000","last_active":"2024-09-13T08:38:01.230213+0000","last_peered":"2024-09-13T08:38:01.230213+0000","last_clean":"2024-09-13T08:38:01.230213+0000","last_became_active":"2024-09-13T08:38:00.227280+0000","last_became_peered":"2024-09-13T08:38:00.227280+0000","last_unstale":"2024-09-13T08:38:01.230213+0000","last_undegraded":"2024-09-13T08:38:01.230213+0000","last_fullsized":"2024-09-13T08:38:01.230213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:22.801067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237944+0000","last_change":"2024-09-13T08:38:01.243864+0000","last_active":"2024-09-13T08:38:02.237944+0000","last_peered":"2024-09-13T08:38:02.237944+0000","last_clean":"2024-09-13T08:38:02.237944+0000","last_became_active":"2024-09-13T08:38:01.242625+0000","last_became_peered":"2024-09-13T08:38:01.242625+0000","last_unstale":"2024-09-13T08:38:02.237944+0000","last_undegraded":"2024-09-13T08:38:02.237944+0000","last_fullsized":"2024-09-13T08:38:02.237944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:25.627301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.228312+0000","last_change":"2024-09-13T08:38:00.224011+0000","last_active":"2024-09-13T08:38:01.228312+0000","last_peered":"2024-09-13T08:38:01.228312+0000","last_clean":"2024-09-13T08:38:01.228312+0000","last_became_active":"2024-09-13T08:38:00.223453+0000","last_became_peered":"2024-09-13T08:38:00.223453+0000","last_unstale":"2024-09-13T08:38:01.228312+0000","last_undegraded":"2024-09-13T08:38:01.228312+0000","last_fullsized":"2024-09-13T08:38:01.228312+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:28.266039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236924+0000","last_change":"2024-09-13T08:38:01.250323+0000","last_active":"2024-09-13T08:38:02.236924+0000","last_peered":"2024-09-13T08:38:02.236924+0000","last_clean":"2024-09-13T08:38:02.236924+0000","last_became_active":"2024-09-13T08:38:01.249979+0000","last_became_peered":"2024-09-13T08:38:01.249979+0000","last_unstale":"2024-09-13T08:38:02.236924+0000","last_undegraded":"2024-09-13T08:38:02.236924+0000","last_fullsized":"2024-09-13T08:38:02.236924+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:38:46.212355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544106+0000","last_change":"2024-09-13T08:38:00.234498+0000","last_active":"2024-09-13T08:38:02.544106+0000","last_peered":"2024-09-13T08:38:02.544106+0000","last_clean":"2024-09-13T08:38:02.544106+0000","last_became_active":"2024-09-13T08:38:00.233639+0000","last_became_peered":"2024-09-13T08:38:00.233639+0000","last_unstale":"2024-09-13T08:38:02.544106+0000","last_undegraded":"2024-09-13T08:38:02.544106+0000","last_fullsized":"2024-09-13T08:38:02.544106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:39.680510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.239704+0000","last_change":"2024-09-13T08:38:01.232354+0000","last_active":"2024-09-13T08:38:01.239704+0000","last_peered":"2024-09-13T08:38:01.239704+0000","last_clean":"2024-09-13T08:38:01.239704+0000","last_became_active":"2024-09-13T08:38:01.232065+0000","last_became_peered":"2024-09-13T08:38:01.232065+0000","last_unstale":"2024-09-13T08:38:01.239704+0000","last_undegraded":"2024-09-13T08:38:01.239704+0000","last_fullsized":"2024-09-13T08:38:01.239704+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:42.324896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237046+0000","last_change":"2024-09-13T08:38:00.225207+0000","last_active":"2024-09-13T08:38:02.237046+0000","last_peered":"2024-09-13T08:38:02.237046+0000","last_clean":"2024-09-13T08:38:02.237046+0000","last_became_active":"2024-09-13T08:38:00.222287+0000","last_became_peered":"2024-09-13T08:38:00.222287+0000","last_unstale":"2024-09-13T08:38:02.237046+0000","last_undegraded":"2024-09-13T08:38:02.237046+0000","last_fullsized":"2024-09-13T08:38:02.237046+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:02:39.233114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.296024+0000","last_change":"2024-09-13T08:38:01.254391+0000","last_active":"2024-09-13T08:38:03.296024+0000","last_peered":"2024-09-13T08:38:03.296024+0000","last_clean":"2024-09-13T08:38:03.296024+0000","last_became_active":"2024-09-13T08:38:01.249345+0000","last_became_peered":"2024-09-13T08:38:01.249345+0000","last_unstale":"2024-09-13T08:38:03.296024+0000","last_undegraded":"2024-09-13T08:38:03.296024+0000","last_fullsized":"2024-09-13T08:38:03.296024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:58.179023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.221843+0000","last_change":"2024-09-13T08:38:00.234248+0000","last_active":"2024-09-13T08:38:01.221843+0000","last_peered":"2024-09-13T08:38:01.221843+0000","last_clean":"2024-09-13T08:38:01.221843+0000","last_became_active":"2024-09-13T08:38:00.233263+0000","last_became_peered":"2024-09-13T08:38:00.233263+0000","last_unstale":"2024-09-13T08:38:01.221843+0000","last_undegraded":"2024-09-13T08:38:01.221843+0000","last_fullsized":"2024-09-13T08:38:01.221843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:20.797629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":133,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238207+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:38:02.238207+0000","last_peered":"2024-09-13T08:38:02.238207+0000","last_clean":"2024-09-13T08:38:02.238207+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:38:02.238207+0000","last_undegraded":"2024-09-13T08:38:02.238207+0000","last_fullsized":"2024-09-13T08:38:02.238207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":405504,"data_stored":156251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":151541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509456,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":987136,"data_stored":740879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":405504,"data_stored":150705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":405504,"data_stored":156263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":999424,"data_stored":746667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":393216,"data_stored":150019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":417792,"data_stored":157863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:38:05.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:38:05.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 -- 172.21.15.31:0/830443478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1088004800 msgr2=0x7f1088004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 --2- 172.21.15.31:0/830443478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1088004800 0x7f1088004c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f108c009b50 tx=0x7f1088005830 comp rx=0 tx=0).stop 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10a9000640 1 -- 172.21.15.31:0/830443478 reap_dead start 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 -- 172.21.15.31:0/830443478 shutdown_connections 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 -- 172.21.15.31:0/830443478 >> 172.21.15.31:0/830443478 conn(0x7f10a40cc910 msgr2=0x7f10a40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 -- 172.21.15.31:0/830443478 shutdown_connections 2024-09-13T08:38:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.675+0000 7f10ab000640 1 -- 172.21.15.31:0/830443478 wait complete. 2024-09-13T08:38:05.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.677+0000 7f10ab000640 1 Processor -- start 2024-09-13T08:38:05.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.677+0000 7f10ab000640 1 -- start start 2024-09-13T08:38:05.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1088004800 0x7f10a40acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10a409aa10 0x7f10a409ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f10a41600f0 con 0x7f10a409d000 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10a40cb850 con 0x7f10a409aa10 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f10ab000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f10a409d9c0 con 0x7f1088004800 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f109b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f109b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47020/0 (socket says 172.21.15.31:47020) 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.678+0000 7f109b400640 1 -- 172.21.15.31:0/2535060383 learned_addr learned my addr 172.21.15.31:0/2535060383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f10a9000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10a409aa10 0x7f10a409ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 -- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1088004800 msgr2=0x7f10a40acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1088004800 0x7f10a40acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 -- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10a409aa10 msgr2=0x7f10a409ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10a409aa10 0x7f10a409ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 -- 172.21.15.31:0/2535060383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a41f4b60 con 0x7f10a409d000 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f109b400640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f108c036040 tx=0x7f108c033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c038040 con 0x7f10a409d000 2024-09-13T08:38:05.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f108c005020 con 0x7f10a409d000 2024-09-13T08:38:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a41f4e40 con 0x7f10a409d000 2024-09-13T08:38:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.679+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c005470 con 0x7f10a409d000 2024-09-13T08:38:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.680+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a41f4810 con 0x7f10a409d000 2024-09-13T08:38:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.681+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10a40ce4a0 con 0x7f10a409d000 2024-09-13T08:38:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.682+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f108c045020 con 0x7f10a409d000 2024-09-13T08:38:05.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.683+0000 7f10a0c00640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 0x7f108007ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:05.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.683+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f108c0bafb0 con 0x7f10a409d000 2024-09-13T08:38:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.683+0000 7f10a3e00640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 0x7f108007ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.684+0000 7f10a3e00640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 0x7f108007ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f10940047a0 tx=0x7f1094000f70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:05.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.687+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f108c083610 con 0x7f10a409d000 2024-09-13T08:38:05.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:38:05.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:38:05.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.879+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f10a40b0b80 con 0x7f1080078880 2024-09-13T08:38:05.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.884+0000 7f10a0c00640 1 -- 172.21.15.31:0/2535060383 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f10a40b0b80 con 0x7f1080078880 2024-09-13T08:38:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:05.893 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:38:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 msgr2=0x7f108007ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 0x7f108007ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f10940047a0 tx=0x7f1094000f70 comp rx=0 tx=0).stop 2024-09-13T08:38:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 msgr2=0x7f10a409d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f108c036040 tx=0x7f108c033650 comp rx=0 tx=0).stop 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 shutdown_connections 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10a409aa10 0x7f10a409ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10a409d000 0x7f10a409d480 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1080078880 0x7f108007ad40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 --2- 172.21.15.31:0/2535060383 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1088004800 0x7f10a40acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 >> 172.21.15.31:0/2535060383 conn(0x7f10a40cc910 msgr2=0x7f10a40adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 shutdown_connections 2024-09-13T08:38:05.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:05.888+0000 7f10ab000640 1 -- 172.21.15.31:0/2535060383 wait complete. 2024-09-13T08:38:05.905 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-13T08:38:05.049561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237200,"kb_used_data":4312,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737520,"statfs":{"total":773094113280,"available":772851220480,"internally_reserved":0,"allocated":4415488,"data_stored":2410188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.846823"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544099+0000","last_change":"2024-09-13T08:38:01.265315+0000","last_active":"2024-09-13T08:38:02.544099+0000","last_peered":"2024-09-13T08:38:02.544099+0000","last_clean":"2024-09-13T08:38:02.544099+0000","last_became_active":"2024-09-13T08:38:01.264978+0000","last_became_peered":"2024-09-13T08:38:01.264978+0000","last_unstale":"2024-09-13T08:38:02.544099+0000","last_undegraded":"2024-09-13T08:38:02.544099+0000","last_fullsized":"2024-09-13T08:38:02.544099+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:05.082400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227930+0000","last_change":"2024-09-13T08:38:01.255508+0000","last_active":"2024-09-13T08:38:03.227930+0000","last_peered":"2024-09-13T08:38:03.227930+0000","last_clean":"2024-09-13T08:38:03.227930+0000","last_became_active":"2024-09-13T08:38:01.255095+0000","last_became_peered":"2024-09-13T08:38:01.255095+0000","last_unstale":"2024-09-13T08:38:03.227930+0000","last_undegraded":"2024-09-13T08:38:03.227930+0000","last_fullsized":"2024-09-13T08:38:03.227930+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238865+0000","last_change":"2024-09-13T08:38:01.249031+0000","last_active":"2024-09-13T08:38:02.238865+0000","last_peered":"2024-09-13T08:38:02.238865+0000","last_clean":"2024-09-13T08:38:02.238865+0000","last_became_active":"2024-09-13T08:38:01.248650+0000","last_became_peered":"2024-09-13T08:38:01.248650+0000","last_unstale":"2024-09-13T08:38:02.238865+0000","last_undegraded":"2024-09-13T08:38:02.238865+0000","last_fullsized":"2024-09-13T08:38:02.238865+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:08.135338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238275+0000","last_change":"2024-09-13T08:38:01.259661+0000","last_active":"2024-09-13T08:38:02.238275+0000","last_peered":"2024-09-13T08:38:02.238275+0000","last_clean":"2024-09-13T08:38:02.238275+0000","last_became_active":"2024-09-13T08:38:01.259003+0000","last_became_peered":"2024-09-13T08:38:01.259003+0000","last_unstale":"2024-09-13T08:38:02.238275+0000","last_undegraded":"2024-09-13T08:38:02.238275+0000","last_fullsized":"2024-09-13T08:38:02.238275+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:18:31.085119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227780+0000","last_change":"2024-09-13T08:38:01.245776+0000","last_active":"2024-09-13T08:38:03.227780+0000","last_peered":"2024-09-13T08:38:03.227780+0000","last_clean":"2024-09-13T08:38:03.227780+0000","last_became_active":"2024-09-13T08:38:01.245307+0000","last_became_peered":"2024-09-13T08:38:01.245307+0000","last_unstale":"2024-09-13T08:38:03.227780+0000","last_undegraded":"2024-09-13T08:38:03.227780+0000","last_fullsized":"2024-09-13T08:38:03.227780+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:20.208186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237265+0000","last_change":"2024-09-13T08:38:01.250649+0000","last_active":"2024-09-13T08:38:02.237265+0000","last_peered":"2024-09-13T08:38:02.237265+0000","last_clean":"2024-09-13T08:38:02.237265+0000","last_became_active":"2024-09-13T08:38:01.244965+0000","last_became_peered":"2024-09-13T08:38:01.244965+0000","last_unstale":"2024-09-13T08:38:02.237265+0000","last_undegraded":"2024-09-13T08:38:02.237265+0000","last_fullsized":"2024-09-13T08:38:02.237265+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:30.481072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543996+0000","last_change":"2024-09-13T08:38:01.251861+0000","last_active":"2024-09-13T08:38:02.543996+0000","last_peered":"2024-09-13T08:38:02.543996+0000","last_clean":"2024-09-13T08:38:02.543996+0000","last_became_active":"2024-09-13T08:38:01.251572+0000","last_became_peered":"2024-09-13T08:38:01.251572+0000","last_unstale":"2024-09-13T08:38:02.543996+0000","last_undegraded":"2024-09-13T08:38:02.543996+0000","last_fullsized":"2024-09-13T08:38:02.543996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:08.106248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543887+0000","last_change":"2024-09-13T08:38:01.261029+0000","last_active":"2024-09-13T08:38:02.543887+0000","last_peered":"2024-09-13T08:38:02.543887+0000","last_clean":"2024-09-13T08:38:02.543887+0000","last_became_active":"2024-09-13T08:38:01.255507+0000","last_became_peered":"2024-09-13T08:38:01.255507+0000","last_unstale":"2024-09-13T08:38:02.543887+0000","last_undegraded":"2024-09-13T08:38:02.543887+0000","last_fullsized":"2024-09-13T08:38:02.543887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238926+0000","last_change":"2024-09-13T08:38:01.252431+0000","last_active":"2024-09-13T08:38:02.238926+0000","last_peered":"2024-09-13T08:38:02.238926+0000","last_clean":"2024-09-13T08:38:02.238926+0000","last_became_active":"2024-09-13T08:38:01.252109+0000","last_became_peered":"2024-09-13T08:38:01.252109+0000","last_unstale":"2024-09-13T08:38:02.238926+0000","last_undegraded":"2024-09-13T08:38:02.238926+0000","last_fullsized":"2024-09-13T08:38:02.238926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:43.653097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239018+0000","last_change":"2024-09-13T08:38:01.255919+0000","last_active":"2024-09-13T08:38:02.239018+0000","last_peered":"2024-09-13T08:38:02.239018+0000","last_clean":"2024-09-13T08:38:02.239018+0000","last_became_active":"2024-09-13T08:38:01.255635+0000","last_became_peered":"2024-09-13T08:38:01.255635+0000","last_unstale":"2024-09-13T08:38:02.239018+0000","last_undegraded":"2024-09-13T08:38:02.239018+0000","last_fullsized":"2024-09-13T08:38:02.239018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:51.552194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.262826+0000","last_change":"2024-09-13T08:38:01.259150+0000","last_active":"2024-09-13T08:38:01.262826+0000","last_peered":"2024-09-13T08:38:01.262826+0000","last_clean":"2024-09-13T08:38:01.262826+0000","last_became_active":"2024-09-13T08:38:01.258385+0000","last_became_peered":"2024-09-13T08:38:01.258385+0000","last_unstale":"2024-09-13T08:38:01.262826+0000","last_undegraded":"2024-09-13T08:38:01.262826+0000","last_fullsized":"2024-09-13T08:38:01.262826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:27.672690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238640+0000","last_change":"2024-09-13T08:38:01.243100+0000","last_active":"2024-09-13T08:38:02.238640+0000","last_peered":"2024-09-13T08:38:02.238640+0000","last_clean":"2024-09-13T08:38:02.238640+0000","last_became_active":"2024-09-13T08:38:01.242843+0000","last_became_peered":"2024-09-13T08:38:01.242843+0000","last_unstale":"2024-09-13T08:38:02.238640+0000","last_undegraded":"2024-09-13T08:38:02.238640+0000","last_fullsized":"2024-09-13T08:38:02.238640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:26.840165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.260966+0000","last_change":"2024-09-13T08:38:01.256675+0000","last_active":"2024-09-13T08:38:01.260966+0000","last_peered":"2024-09-13T08:38:01.260966+0000","last_clean":"2024-09-13T08:38:01.260966+0000","last_became_active":"2024-09-13T08:38:01.256345+0000","last_became_peered":"2024-09-13T08:38:01.256345+0000","last_unstale":"2024-09-13T08:38:01.260966+0000","last_undegraded":"2024-09-13T08:38:01.260966+0000","last_fullsized":"2024-09-13T08:38:01.260966+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:20.408271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238086+0000","last_change":"2024-09-13T08:38:01.240090+0000","last_active":"2024-09-13T08:38:02.238086+0000","last_peered":"2024-09-13T08:38:02.238086+0000","last_clean":"2024-09-13T08:38:02.238086+0000","last_became_active":"2024-09-13T08:38:01.239708+0000","last_became_peered":"2024-09-13T08:38:01.239708+0000","last_unstale":"2024-09-13T08:38:02.238086+0000","last_undegraded":"2024-09-13T08:38:02.238086+0000","last_fullsized":"2024-09-13T08:38:02.238086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:36.296533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295629+0000","last_change":"2024-09-13T08:38:01.261773+0000","last_active":"2024-09-13T08:38:03.295629+0000","last_peered":"2024-09-13T08:38:03.295629+0000","last_clean":"2024-09-13T08:38:03.295629+0000","last_became_active":"2024-09-13T08:38:01.261429+0000","last_became_peered":"2024-09-13T08:38:01.261429+0000","last_unstale":"2024-09-13T08:38:03.295629+0000","last_undegraded":"2024-09-13T08:38:03.295629+0000","last_fullsized":"2024-09-13T08:38:03.295629+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295546+0000","last_change":"2024-09-13T08:38:01.249657+0000","last_active":"2024-09-13T08:38:03.295546+0000","last_peered":"2024-09-13T08:38:03.295546+0000","last_clean":"2024-09-13T08:38:03.295546+0000","last_became_active":"2024-09-13T08:38:01.249163+0000","last_became_peered":"2024-09-13T08:38:01.249163+0000","last_unstale":"2024-09-13T08:38:03.295546+0000","last_undegraded":"2024-09-13T08:38:03.295546+0000","last_fullsized":"2024-09-13T08:38:03.295546+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:12:17.614756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264896+0000","last_change":"2024-09-13T08:38:01.259313+0000","last_active":"2024-09-13T08:38:01.264896+0000","last_peered":"2024-09-13T08:38:01.264896+0000","last_clean":"2024-09-13T08:38:01.264896+0000","last_became_active":"2024-09-13T08:38:01.259016+0000","last_became_peered":"2024-09-13T08:38:01.259016+0000","last_unstale":"2024-09-13T08:38:01.264896+0000","last_undegraded":"2024-09-13T08:38:01.264896+0000","last_fullsized":"2024-09-13T08:38:01.264896+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:27.180659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238053+0000","last_change":"2024-09-13T08:38:01.243966+0000","last_active":"2024-09-13T08:38:02.238053+0000","last_peered":"2024-09-13T08:38:02.238053+0000","last_clean":"2024-09-13T08:38:02.238053+0000","last_became_active":"2024-09-13T08:38:01.240660+0000","last_became_peered":"2024-09-13T08:38:01.240660+0000","last_unstale":"2024-09-13T08:38:02.238053+0000","last_undegraded":"2024-09-13T08:38:02.238053+0000","last_fullsized":"2024-09-13T08:38:02.238053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:59.979864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237280+0000","last_change":"2024-09-13T08:38:01.245804+0000","last_active":"2024-09-13T08:38:02.237280+0000","last_peered":"2024-09-13T08:38:02.237280+0000","last_clean":"2024-09-13T08:38:02.237280+0000","last_became_active":"2024-09-13T08:38:01.244554+0000","last_became_peered":"2024-09-13T08:38:01.244554+0000","last_unstale":"2024-09-13T08:38:02.237280+0000","last_undegraded":"2024-09-13T08:38:02.237280+0000","last_fullsized":"2024-09-13T08:38:02.237280+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:26.840165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264759+0000","last_change":"2024-09-13T08:38:01.263620+0000","last_active":"2024-09-13T08:38:01.264759+0000","last_peered":"2024-09-13T08:38:01.264759+0000","last_clean":"2024-09-13T08:38:01.264759+0000","last_became_active":"2024-09-13T08:38:01.263332+0000","last_became_peered":"2024-09-13T08:38:01.263332+0000","last_unstale":"2024-09-13T08:38:01.264759+0000","last_undegraded":"2024-09-13T08:38:01.264759+0000","last_fullsized":"2024-09-13T08:38:01.264759+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:06.240183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295714+0000","last_change":"2024-09-13T08:38:01.260291+0000","last_active":"2024-09-13T08:38:03.295714+0000","last_peered":"2024-09-13T08:38:03.295714+0000","last_clean":"2024-09-13T08:38:03.295714+0000","last_became_active":"2024-09-13T08:38:01.259926+0000","last_became_peered":"2024-09-13T08:38:01.259926+0000","last_unstale":"2024-09-13T08:38:03.295714+0000","last_undegraded":"2024-09-13T08:38:03.295714+0000","last_fullsized":"2024-09-13T08:38:03.295714+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:19.986671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544147+0000","last_change":"2024-09-13T08:38:01.259332+0000","last_active":"2024-09-13T08:38:02.544147+0000","last_peered":"2024-09-13T08:38:02.544147+0000","last_clean":"2024-09-13T08:38:02.544147+0000","last_became_active":"2024-09-13T08:38:01.259080+0000","last_became_peered":"2024-09-13T08:38:01.259080+0000","last_unstale":"2024-09-13T08:38:02.544147+0000","last_undegraded":"2024-09-13T08:38:02.544147+0000","last_fullsized":"2024-09-13T08:38:02.544147+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:19.986671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.255885+0000","last_change":"2024-09-13T08:38:01.252742+0000","last_active":"2024-09-13T08:38:01.255885+0000","last_peered":"2024-09-13T08:38:01.255885+0000","last_clean":"2024-09-13T08:38:01.255885+0000","last_became_active":"2024-09-13T08:38:01.251518+0000","last_became_peered":"2024-09-13T08:38:01.251518+0000","last_unstale":"2024-09-13T08:38:01.255885+0000","last_undegraded":"2024-09-13T08:38:01.255885+0000","last_fullsized":"2024-09-13T08:38:01.255885+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:33:18.237176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543809+0000","last_change":"2024-09-13T08:38:01.258035+0000","last_active":"2024-09-13T08:38:02.543809+0000","last_peered":"2024-09-13T08:38:02.543809+0000","last_clean":"2024-09-13T08:38:02.543809+0000","last_became_active":"2024-09-13T08:38:01.257764+0000","last_became_peered":"2024-09-13T08:38:01.257764+0000","last_unstale":"2024-09-13T08:38:02.543809+0000","last_undegraded":"2024-09-13T08:38:02.543809+0000","last_fullsized":"2024-09-13T08:38:02.543809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:08:12.458886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238965+0000","last_change":"2024-09-13T08:38:01.243053+0000","last_active":"2024-09-13T08:38:02.238965+0000","last_peered":"2024-09-13T08:38:02.238965+0000","last_clean":"2024-09-13T08:38:02.238965+0000","last_became_active":"2024-09-13T08:38:01.242519+0000","last_became_peered":"2024-09-13T08:38:01.242519+0000","last_unstale":"2024-09-13T08:38:02.238965+0000","last_undegraded":"2024-09-13T08:38:02.238965+0000","last_fullsized":"2024-09-13T08:38:02.238965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:52:52.670617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239001+0000","last_change":"2024-09-13T08:38:01.243267+0000","last_active":"2024-09-13T08:38:02.239001+0000","last_peered":"2024-09-13T08:38:02.239001+0000","last_clean":"2024-09-13T08:38:02.239001+0000","last_became_active":"2024-09-13T08:38:01.242723+0000","last_became_peered":"2024-09-13T08:38:01.242723+0000","last_unstale":"2024-09-13T08:38:02.239001+0000","last_undegraded":"2024-09-13T08:38:02.239001+0000","last_fullsized":"2024-09-13T08:38:02.239001+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:45:35.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227950+0000","last_change":"2024-09-13T08:38:01.253642+0000","last_active":"2024-09-13T08:38:03.227950+0000","last_peered":"2024-09-13T08:38:03.227950+0000","last_clean":"2024-09-13T08:38:03.227950+0000","last_became_active":"2024-09-13T08:38:01.250652+0000","last_became_peered":"2024-09-13T08:38:01.250652+0000","last_unstale":"2024-09-13T08:38:03.227950+0000","last_undegraded":"2024-09-13T08:38:03.227950+0000","last_fullsized":"2024-09-13T08:38:03.227950+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:39.892681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544145+0000","last_change":"2024-09-13T08:38:01.261029+0000","last_active":"2024-09-13T08:38:02.544145+0000","last_peered":"2024-09-13T08:38:02.544145+0000","last_clean":"2024-09-13T08:38:02.544145+0000","last_became_active":"2024-09-13T08:38:01.260770+0000","last_became_peered":"2024-09-13T08:38:01.260770+0000","last_unstale":"2024-09-13T08:38:02.544145+0000","last_undegraded":"2024-09-13T08:38:02.544145+0000","last_fullsized":"2024-09-13T08:38:02.544145+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:54.694090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295766+0000","last_change":"2024-09-13T08:38:01.253834+0000","last_active":"2024-09-13T08:38:03.295766+0000","last_peered":"2024-09-13T08:38:03.295766+0000","last_clean":"2024-09-13T08:38:03.295766+0000","last_became_active":"2024-09-13T08:38:01.250174+0000","last_became_peered":"2024-09-13T08:38:01.250174+0000","last_unstale":"2024-09-13T08:38:03.295766+0000","last_undegraded":"2024-09-13T08:38:03.295766+0000","last_fullsized":"2024-09-13T08:38:03.295766+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295805+0000","last_change":"2024-09-13T08:38:01.260047+0000","last_active":"2024-09-13T08:38:03.295805+0000","last_peered":"2024-09-13T08:38:03.295805+0000","last_clean":"2024-09-13T08:38:03.295805+0000","last_became_active":"2024-09-13T08:38:01.249658+0000","last_became_peered":"2024-09-13T08:38:01.249658+0000","last_unstale":"2024-09-13T08:38:03.295805+0000","last_undegraded":"2024-09-13T08:38:03.295805+0000","last_fullsized":"2024-09-13T08:38:03.295805+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:56.303136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544295+0000","last_change":"2024-09-13T08:38:01.244840+0000","last_active":"2024-09-13T08:38:02.544295+0000","last_peered":"2024-09-13T08:38:02.544295+0000","last_clean":"2024-09-13T08:38:02.544295+0000","last_became_active":"2024-09-13T08:38:01.244519+0000","last_became_peered":"2024-09-13T08:38:01.244519+0000","last_unstale":"2024-09-13T08:38:02.544295+0000","last_undegraded":"2024-09-13T08:38:02.544295+0000","last_fullsized":"2024-09-13T08:38:02.544295+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:45.642304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.254265+0000","last_change":"2024-09-13T08:38:01.249570+0000","last_active":"2024-09-13T08:38:01.254265+0000","last_peered":"2024-09-13T08:38:01.254265+0000","last_clean":"2024-09-13T08:38:01.254265+0000","last_became_active":"2024-09-13T08:38:01.249177+0000","last_became_peered":"2024-09-13T08:38:01.249177+0000","last_unstale":"2024-09-13T08:38:01.254265+0000","last_undegraded":"2024-09-13T08:38:01.254265+0000","last_fullsized":"2024-09-13T08:38:01.254265+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:35.930050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.250446+0000","last_change":"2024-09-13T08:38:01.246568+0000","last_active":"2024-09-13T08:38:01.250446+0000","last_peered":"2024-09-13T08:38:01.250446+0000","last_clean":"2024-09-13T08:38:01.250446+0000","last_became_active":"2024-09-13T08:38:01.246225+0000","last_became_peered":"2024-09-13T08:38:01.246225+0000","last_unstale":"2024-09-13T08:38:01.250446+0000","last_undegraded":"2024-09-13T08:38:01.250446+0000","last_fullsized":"2024-09-13T08:38:01.250446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:48:04.157436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.246655+0000","last_change":"2024-09-13T08:38:00.230433+0000","last_active":"2024-09-13T08:38:01.246655+0000","last_peered":"2024-09-13T08:38:01.246655+0000","last_clean":"2024-09-13T08:38:01.246655+0000","last_became_active":"2024-09-13T08:38:00.228507+0000","last_became_peered":"2024-09-13T08:38:00.228507+0000","last_unstale":"2024-09-13T08:38:01.246655+0000","last_undegraded":"2024-09-13T08:38:01.246655+0000","last_fullsized":"2024-09-13T08:38:01.246655+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:57.172335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544118+0000","last_change":"2024-09-13T08:38:01.252288+0000","last_active":"2024-09-13T08:38:02.544118+0000","last_peered":"2024-09-13T08:38:02.544118+0000","last_clean":"2024-09-13T08:38:02.544118+0000","last_became_active":"2024-09-13T08:38:01.252013+0000","last_became_peered":"2024-09-13T08:38:01.252013+0000","last_unstale":"2024-09-13T08:38:02.544118+0000","last_undegraded":"2024-09-13T08:38:02.544118+0000","last_fullsized":"2024-09-13T08:38:02.544118+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:22:25.289292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239146+0000","last_change":"2024-09-13T08:38:00.227234+0000","last_active":"2024-09-13T08:38:02.239146+0000","last_peered":"2024-09-13T08:38:02.239146+0000","last_clean":"2024-09-13T08:38:02.239146+0000","last_became_active":"2024-09-13T08:38:00.226880+0000","last_became_peered":"2024-09-13T08:38:00.226880+0000","last_unstale":"2024-09-13T08:38:02.239146+0000","last_undegraded":"2024-09-13T08:38:02.239146+0000","last_fullsized":"2024-09-13T08:38:02.239146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:21.715485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544266+0000","last_change":"2024-09-13T08:38:01.258862+0000","last_active":"2024-09-13T08:38:02.544266+0000","last_peered":"2024-09-13T08:38:02.544266+0000","last_clean":"2024-09-13T08:38:02.544266+0000","last_became_active":"2024-09-13T08:38:01.258469+0000","last_became_peered":"2024-09-13T08:38:01.258469+0000","last_unstale":"2024-09-13T08:38:02.544266+0000","last_undegraded":"2024-09-13T08:38:02.544266+0000","last_fullsized":"2024-09-13T08:38:02.544266+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:32.007345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237748+0000","last_change":"2024-09-13T08:38:00.224928+0000","last_active":"2024-09-13T08:38:02.237748+0000","last_peered":"2024-09-13T08:38:02.237748+0000","last_clean":"2024-09-13T08:38:02.237748+0000","last_became_active":"2024-09-13T08:38:00.224586+0000","last_became_peered":"2024-09-13T08:38:00.224586+0000","last_unstale":"2024-09-13T08:38:02.237748+0000","last_undegraded":"2024-09-13T08:38:02.237748+0000","last_fullsized":"2024-09-13T08:38:02.237748+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:33.891448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238953+0000","last_change":"2024-09-13T08:38:01.246402+0000","last_active":"2024-09-13T08:38:02.238953+0000","last_peered":"2024-09-13T08:38:02.238953+0000","last_clean":"2024-09-13T08:38:02.238953+0000","last_became_active":"2024-09-13T08:38:01.246074+0000","last_became_peered":"2024-09-13T08:38:01.246074+0000","last_unstale":"2024-09-13T08:38:02.238953+0000","last_undegraded":"2024-09-13T08:38:02.238953+0000","last_fullsized":"2024-09-13T08:38:02.238953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:14.527936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238573+0000","last_change":"2024-09-13T08:38:00.234092+0000","last_active":"2024-09-13T08:38:02.238573+0000","last_peered":"2024-09-13T08:38:02.238573+0000","last_clean":"2024-09-13T08:38:02.238573+0000","last_became_active":"2024-09-13T08:38:00.233741+0000","last_became_peered":"2024-09-13T08:38:00.233741+0000","last_unstale":"2024-09-13T08:38:02.238573+0000","last_undegraded":"2024-09-13T08:38:02.238573+0000","last_fullsized":"2024-09-13T08:38:02.238573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:06.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238493+0000","last_change":"2024-09-13T08:38:01.253342+0000","last_active":"2024-09-13T08:38:02.238493+0000","last_peered":"2024-09-13T08:38:02.238493+0000","last_clean":"2024-09-13T08:38:02.238493+0000","last_became_active":"2024-09-13T08:38:01.252996+0000","last_became_peered":"2024-09-13T08:38:01.252996+0000","last_unstale":"2024-09-13T08:38:02.238493+0000","last_undegraded":"2024-09-13T08:38:02.238493+0000","last_fullsized":"2024-09-13T08:38:02.238493+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:30.481072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543762+0000","last_change":"2024-09-13T08:38:00.234123+0000","last_active":"2024-09-13T08:38:02.543762+0000","last_peered":"2024-09-13T08:38:02.543762+0000","last_clean":"2024-09-13T08:38:02.543762+0000","last_became_active":"2024-09-13T08:38:00.233858+0000","last_became_peered":"2024-09-13T08:38:00.233858+0000","last_unstale":"2024-09-13T08:38:02.543762+0000","last_undegraded":"2024-09-13T08:38:02.543762+0000","last_fullsized":"2024-09-13T08:38:02.543762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:29.474385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543813+0000","last_change":"2024-09-13T08:38:01.260632+0000","last_active":"2024-09-13T08:38:02.543813+0000","last_peered":"2024-09-13T08:38:02.543813+0000","last_clean":"2024-09-13T08:38:02.543813+0000","last_became_active":"2024-09-13T08:38:01.260408+0000","last_became_peered":"2024-09-13T08:38:01.260408+0000","last_unstale":"2024-09-13T08:38:02.543813+0000","last_undegraded":"2024-09-13T08:38:02.543813+0000","last_fullsized":"2024-09-13T08:38:02.543813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:06:20.698433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543700+0000","last_change":"2024-09-13T08:38:00.231574+0000","last_active":"2024-09-13T08:38:02.543700+0000","last_peered":"2024-09-13T08:38:02.543700+0000","last_clean":"2024-09-13T08:38:02.543700+0000","last_became_active":"2024-09-13T08:38:00.231273+0000","last_became_peered":"2024-09-13T08:38:00.231273+0000","last_unstale":"2024-09-13T08:38:02.543700+0000","last_undegraded":"2024-09-13T08:38:02.543700+0000","last_fullsized":"2024-09-13T08:38:02.543700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:25.833478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238288+0000","last_change":"2024-09-13T08:38:01.246029+0000","last_active":"2024-09-13T08:38:02.238288+0000","last_peered":"2024-09-13T08:38:02.238288+0000","last_clean":"2024-09-13T08:38:02.238288+0000","last_became_active":"2024-09-13T08:38:01.245748+0000","last_became_peered":"2024-09-13T08:38:01.245748+0000","last_unstale":"2024-09-13T08:38:02.238288+0000","last_undegraded":"2024-09-13T08:38:02.238288+0000","last_fullsized":"2024-09-13T08:38:02.238288+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:59.758539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237589+0000","last_change":"2024-09-13T08:38:00.223721+0000","last_active":"2024-09-13T08:38:02.237589+0000","last_peered":"2024-09-13T08:38:02.237589+0000","last_clean":"2024-09-13T08:38:02.237589+0000","last_became_active":"2024-09-13T08:38:00.223183+0000","last_became_peered":"2024-09-13T08:38:00.223183+0000","last_unstale":"2024-09-13T08:38:02.237589+0000","last_undegraded":"2024-09-13T08:38:02.237589+0000","last_fullsized":"2024-09-13T08:38:02.237589+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:31.193415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236776+0000","last_change":"2024-09-13T08:38:01.239689+0000","last_active":"2024-09-13T08:38:02.236776+0000","last_peered":"2024-09-13T08:38:02.236776+0000","last_clean":"2024-09-13T08:38:02.236776+0000","last_became_active":"2024-09-13T08:38:01.239393+0000","last_became_peered":"2024-09-13T08:38:01.239393+0000","last_unstale":"2024-09-13T08:38:02.236776+0000","last_undegraded":"2024-09-13T08:38:02.236776+0000","last_fullsized":"2024-09-13T08:38:02.236776+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:01:35.526012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544006+0000","last_change":"2024-09-13T08:38:00.228561+0000","last_active":"2024-09-13T08:38:02.544006+0000","last_peered":"2024-09-13T08:38:02.544006+0000","last_clean":"2024-09-13T08:38:02.544006+0000","last_became_active":"2024-09-13T08:38:00.225937+0000","last_became_peered":"2024-09-13T08:38:00.225937+0000","last_unstale":"2024-09-13T08:38:02.544006+0000","last_undegraded":"2024-09-13T08:38:02.544006+0000","last_fullsized":"2024-09-13T08:38:02.544006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:58.751851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.247714+0000","last_change":"2024-09-13T08:38:01.239615+0000","last_active":"2024-09-13T08:38:01.247714+0000","last_peered":"2024-09-13T08:38:01.247714+0000","last_clean":"2024-09-13T08:38:01.247714+0000","last_became_active":"2024-09-13T08:38:01.238923+0000","last_became_peered":"2024-09-13T08:38:01.238923+0000","last_unstale":"2024-09-13T08:38:01.247714+0000","last_undegraded":"2024-09-13T08:38:01.247714+0000","last_fullsized":"2024-09-13T08:38:01.247714+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:34.152667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.228008+0000","last_change":"2024-09-13T08:38:00.233030+0000","last_active":"2024-09-13T08:38:03.228008+0000","last_peered":"2024-09-13T08:38:03.228008+0000","last_clean":"2024-09-13T08:38:03.228008+0000","last_became_active":"2024-09-13T08:38:00.232672+0000","last_became_peered":"2024-09-13T08:38:00.232672+0000","last_unstale":"2024-09-13T08:38:03.228008+0000","last_undegraded":"2024-09-13T08:38:03.228008+0000","last_fullsized":"2024-09-13T08:38:03.228008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:05:16.230276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238910+0000","last_change":"2024-09-13T08:38:01.239832+0000","last_active":"2024-09-13T08:38:02.238910+0000","last_peered":"2024-09-13T08:38:02.238910+0000","last_clean":"2024-09-13T08:38:02.238910+0000","last_became_active":"2024-09-13T08:38:01.239494+0000","last_became_peered":"2024-09-13T08:38:01.239494+0000","last_unstale":"2024-09-13T08:38:02.238910+0000","last_undegraded":"2024-09-13T08:38:02.238910+0000","last_fullsized":"2024-09-13T08:38:02.238910+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:04.065422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227866+0000","last_change":"2024-09-13T08:38:00.228208+0000","last_active":"2024-09-13T08:38:03.227866+0000","last_peered":"2024-09-13T08:38:03.227866+0000","last_clean":"2024-09-13T08:38:03.227866+0000","last_became_active":"2024-09-13T08:38:00.227935+0000","last_became_peered":"2024-09-13T08:38:00.227935+0000","last_unstale":"2024-09-13T08:38:03.227866+0000","last_undegraded":"2024-09-13T08:38:03.227866+0000","last_fullsized":"2024-09-13T08:38:03.227866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:40.619149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238602+0000","last_change":"2024-09-13T08:38:01.259738+0000","last_active":"2024-09-13T08:38:02.238602+0000","last_peered":"2024-09-13T08:38:02.238602+0000","last_clean":"2024-09-13T08:38:02.238602+0000","last_became_active":"2024-09-13T08:38:01.259382+0000","last_became_peered":"2024-09-13T08:38:01.259382+0000","last_unstale":"2024-09-13T08:38:02.238602+0000","last_undegraded":"2024-09-13T08:38:02.238602+0000","last_fullsized":"2024-09-13T08:38:02.238602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:55.850218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264685+0000","last_change":"2024-09-13T08:38:00.233958+0000","last_active":"2024-09-13T08:38:01.264685+0000","last_peered":"2024-09-13T08:38:01.264685+0000","last_clean":"2024-09-13T08:38:01.264685+0000","last_became_active":"2024-09-13T08:38:00.232954+0000","last_became_peered":"2024-09-13T08:38:00.232954+0000","last_unstale":"2024-09-13T08:38:01.264685+0000","last_undegraded":"2024-09-13T08:38:01.264685+0000","last_fullsized":"2024-09-13T08:38:01.264685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:55.296448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.262802+0000","last_change":"2024-09-13T08:38:01.258743+0000","last_active":"2024-09-13T08:38:01.262802+0000","last_peered":"2024-09-13T08:38:01.262802+0000","last_clean":"2024-09-13T08:38:01.262802+0000","last_became_active":"2024-09-13T08:38:01.257969+0000","last_became_peered":"2024-09-13T08:38:01.257969+0000","last_unstale":"2024-09-13T08:38:01.262802+0000","last_undegraded":"2024-09-13T08:38:01.262802+0000","last_fullsized":"2024-09-13T08:38:01.262802+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:53:28.166981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":19,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.264574+0000","last_change":"2024-09-13T08:38:00.235585+0000","last_active":"2024-09-13T08:38:01.264574+0000","last_peered":"2024-09-13T08:38:01.264574+0000","last_clean":"2024-09-13T08:38:01.264574+0000","last_became_active":"2024-09-13T08:38:00.235260+0000","last_became_peered":"2024-09-13T08:38:00.235260+0000","last_unstale":"2024-09-13T08:38:01.264574+0000","last_undegraded":"2024-09-13T08:38:01.264574+0000","last_fullsized":"2024-09-13T08:38:01.264574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:39:26.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.251129+0000","last_change":"2024-09-13T08:38:01.245709+0000","last_active":"2024-09-13T08:38:01.251129+0000","last_peered":"2024-09-13T08:38:01.251129+0000","last_clean":"2024-09-13T08:38:01.251129+0000","last_became_active":"2024-09-13T08:38:01.245244+0000","last_became_peered":"2024-09-13T08:38:01.245244+0000","last_unstale":"2024-09-13T08:38:01.251129+0000","last_undegraded":"2024-09-13T08:38:01.251129+0000","last_fullsized":"2024-09-13T08:38:01.251129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:33.546187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543992+0000","last_change":"2024-09-13T08:38:00.226943+0000","last_active":"2024-09-13T08:38:02.543992+0000","last_peered":"2024-09-13T08:38:02.543992+0000","last_clean":"2024-09-13T08:38:02.543992+0000","last_became_active":"2024-09-13T08:38:00.226597+0000","last_became_peered":"2024-09-13T08:38:00.226597+0000","last_unstale":"2024-09-13T08:38:02.543992+0000","last_undegraded":"2024-09-13T08:38:02.543992+0000","last_fullsized":"2024-09-13T08:38:02.543992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:07.128650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238172+0000","last_change":"2024-09-13T08:38:01.243249+0000","last_active":"2024-09-13T08:38:02.238172+0000","last_peered":"2024-09-13T08:38:02.238172+0000","last_clean":"2024-09-13T08:38:02.238172+0000","last_became_active":"2024-09-13T08:38:01.242938+0000","last_became_peered":"2024-09-13T08:38:01.242938+0000","last_unstale":"2024-09-13T08:38:02.238172+0000","last_undegraded":"2024-09-13T08:38:02.238172+0000","last_fullsized":"2024-09-13T08:38:02.238172+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:41:41.441133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237127+0000","last_change":"2024-09-13T08:38:00.230779+0000","last_active":"2024-09-13T08:38:02.237127+0000","last_peered":"2024-09-13T08:38:02.237127+0000","last_clean":"2024-09-13T08:38:02.237127+0000","last_became_active":"2024-09-13T08:38:00.230485+0000","last_became_peered":"2024-09-13T08:38:00.230485+0000","last_unstale":"2024-09-13T08:38:02.237127+0000","last_undegraded":"2024-09-13T08:38:02.237127+0000","last_fullsized":"2024-09-13T08:38:02.237127+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:47.186354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544092+0000","last_change":"2024-09-13T08:38:01.255949+0000","last_active":"2024-09-13T08:38:02.544092+0000","last_peered":"2024-09-13T08:38:02.544092+0000","last_clean":"2024-09-13T08:38:02.544092+0000","last_became_active":"2024-09-13T08:38:01.255155+0000","last_became_peered":"2024-09-13T08:38:01.255155+0000","last_unstale":"2024-09-13T08:38:02.544092+0000","last_undegraded":"2024-09-13T08:38:02.544092+0000","last_fullsized":"2024-09-13T08:38:02.544092+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.022507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236977+0000","last_change":"2024-09-13T08:38:00.228323+0000","last_active":"2024-09-13T08:38:02.236977+0000","last_peered":"2024-09-13T08:38:02.236977+0000","last_clean":"2024-09-13T08:38:02.236977+0000","last_became_active":"2024-09-13T08:38:00.223345+0000","last_became_peered":"2024-09-13T08:38:00.223345+0000","last_unstale":"2024-09-13T08:38:02.236977+0000","last_undegraded":"2024-09-13T08:38:02.236977+0000","last_fullsized":"2024-09-13T08:38:02.236977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:28.474359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.245822+0000","last_change":"2024-09-13T08:38:01.242444+0000","last_active":"2024-09-13T08:38:01.245822+0000","last_peered":"2024-09-13T08:38:01.245822+0000","last_clean":"2024-09-13T08:38:01.245822+0000","last_became_active":"2024-09-13T08:38:01.242103+0000","last_became_peered":"2024-09-13T08:38:01.242103+0000","last_unstale":"2024-09-13T08:38:01.245822+0000","last_undegraded":"2024-09-13T08:38:01.245822+0000","last_fullsized":"2024-09-13T08:38:01.245822+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:45.642304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":23,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236859+0000","last_change":"2024-09-13T08:38:00.236587+0000","last_active":"2024-09-13T08:38:02.236859+0000","last_peered":"2024-09-13T08:38:02.236859+0000","last_clean":"2024-09-13T08:38:02.236859+0000","last_became_active":"2024-09-13T08:38:00.236259+0000","last_became_peered":"2024-09-13T08:38:00.236259+0000","last_unstale":"2024-09-13T08:38:02.236859+0000","last_undegraded":"2024-09-13T08:38:02.236859+0000","last_fullsized":"2024-09-13T08:38:02.236859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:37:52.214727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.259795+0000","last_change":"2024-09-13T08:38:01.255145+0000","last_active":"2024-09-13T08:38:01.259795+0000","last_peered":"2024-09-13T08:38:01.259795+0000","last_clean":"2024-09-13T08:38:01.259795+0000","last_became_active":"2024-09-13T08:38:01.254877+0000","last_became_peered":"2024-09-13T08:38:01.254877+0000","last_unstale":"2024-09-13T08:38:01.259795+0000","last_undegraded":"2024-09-13T08:38:01.259795+0000","last_fullsized":"2024-09-13T08:38:01.259795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:45:09.792867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237746+0000","last_change":"2024-09-13T08:38:00.228903+0000","last_active":"2024-09-13T08:38:02.237746+0000","last_peered":"2024-09-13T08:38:02.237746+0000","last_clean":"2024-09-13T08:38:02.237746+0000","last_became_active":"2024-09-13T08:38:00.227671+0000","last_became_peered":"2024-09-13T08:38:00.227671+0000","last_unstale":"2024-09-13T08:38:02.237746+0000","last_undegraded":"2024-09-13T08:38:02.237746+0000","last_fullsized":"2024-09-13T08:38:02.237746+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:24.940664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.239035+0000","last_change":"2024-09-13T08:38:01.245734+0000","last_active":"2024-09-13T08:38:02.239035+0000","last_peered":"2024-09-13T08:38:02.239035+0000","last_clean":"2024-09-13T08:38:02.239035+0000","last_became_active":"2024-09-13T08:38:01.245398+0000","last_became_peered":"2024-09-13T08:38:01.245398+0000","last_unstale":"2024-09-13T08:38:02.239035+0000","last_undegraded":"2024-09-13T08:38:02.239035+0000","last_fullsized":"2024-09-13T08:38:02.239035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:59.979864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543829+0000","last_change":"2024-09-13T08:38:00.233559+0000","last_active":"2024-09-13T08:38:02.543829+0000","last_peered":"2024-09-13T08:38:02.543829+0000","last_clean":"2024-09-13T08:38:02.543829+0000","last_became_active":"2024-09-13T08:38:00.233275+0000","last_became_peered":"2024-09-13T08:38:00.233275+0000","last_unstale":"2024-09-13T08:38:02.543829+0000","last_undegraded":"2024-09-13T08:38:02.543829+0000","last_fullsized":"2024-09-13T08:38:02.543829+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:42.646410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227781+0000","last_change":"2024-09-13T08:38:01.238818+0000","last_active":"2024-09-13T08:38:03.227781+0000","last_peered":"2024-09-13T08:38:03.227781+0000","last_clean":"2024-09-13T08:38:03.227781+0000","last_became_active":"2024-09-13T08:38:01.238499+0000","last_became_peered":"2024-09-13T08:38:01.238499+0000","last_unstale":"2024-09-13T08:38:03.227781+0000","last_undegraded":"2024-09-13T08:38:03.227781+0000","last_fullsized":"2024-09-13T08:38:03.227781+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:04.150171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.246759+0000","last_change":"2024-09-13T08:38:00.234550+0000","last_active":"2024-09-13T08:38:01.246759+0000","last_peered":"2024-09-13T08:38:01.246759+0000","last_clean":"2024-09-13T08:38:01.246759+0000","last_became_active":"2024-09-13T08:38:00.233680+0000","last_became_peered":"2024-09-13T08:38:00.233680+0000","last_unstale":"2024-09-13T08:38:01.246759+0000","last_undegraded":"2024-09-13T08:38:01.246759+0000","last_fullsized":"2024-09-13T08:38:01.246759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:04.075713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237504+0000","last_change":"2024-09-13T08:38:01.247351+0000","last_active":"2024-09-13T08:38:02.237504+0000","last_peered":"2024-09-13T08:38:02.237504+0000","last_clean":"2024-09-13T08:38:02.237504+0000","last_became_active":"2024-09-13T08:38:01.246206+0000","last_became_peered":"2024-09-13T08:38:01.246206+0000","last_unstale":"2024-09-13T08:38:02.237504+0000","last_undegraded":"2024-09-13T08:38:02.237504+0000","last_fullsized":"2024-09-13T08:38:02.237504+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:08.135338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295973+0000","last_change":"2024-09-13T08:38:00.216931+0000","last_active":"2024-09-13T08:38:03.295973+0000","last_peered":"2024-09-13T08:38:03.295973+0000","last_clean":"2024-09-13T08:38:03.295973+0000","last_became_active":"2024-09-13T08:38:00.216603+0000","last_became_peered":"2024-09-13T08:38:00.216603+0000","last_unstale":"2024-09-13T08:38:03.295973+0000","last_undegraded":"2024-09-13T08:38:03.295973+0000","last_fullsized":"2024-09-13T08:38:03.295973+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:36.507927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.263964+0000","last_change":"2024-09-13T08:38:01.262153+0000","last_active":"2024-09-13T08:38:01.263964+0000","last_peered":"2024-09-13T08:38:01.263964+0000","last_clean":"2024-09-13T08:38:01.263964+0000","last_became_active":"2024-09-13T08:38:01.261856+0000","last_became_peered":"2024-09-13T08:38:01.261856+0000","last_unstale":"2024-09-13T08:38:01.263964+0000","last_undegraded":"2024-09-13T08:38:01.263964+0000","last_fullsized":"2024-09-13T08:38:01.263964+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:08:33.091133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227955+0000","last_change":"2024-09-13T08:38:00.232078+0000","last_active":"2024-09-13T08:38:03.227955+0000","last_peered":"2024-09-13T08:38:03.227955+0000","last_clean":"2024-09-13T08:38:03.227955+0000","last_became_active":"2024-09-13T08:38:00.231821+0000","last_became_peered":"2024-09-13T08:38:00.231821+0000","last_unstale":"2024-09-13T08:38:03.227955+0000","last_undegraded":"2024-09-13T08:38:03.227955+0000","last_fullsized":"2024-09-13T08:38:03.227955+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:36.507927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.227833+0000","last_change":"2024-09-13T08:38:01.234497+0000","last_active":"2024-09-13T08:38:03.227833+0000","last_peered":"2024-09-13T08:38:03.227833+0000","last_clean":"2024-09-13T08:38:03.227833+0000","last_became_active":"2024-09-13T08:38:01.234202+0000","last_became_peered":"2024-09-13T08:38:01.234202+0000","last_unstale":"2024-09-13T08:38:03.227833+0000","last_undegraded":"2024-09-13T08:38:03.227833+0000","last_fullsized":"2024-09-13T08:38:03.227833+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:08.106248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.543897+0000","last_change":"2024-09-13T08:38:00.236286+0000","last_active":"2024-09-13T08:38:02.543897+0000","last_peered":"2024-09-13T08:38:02.543897+0000","last_clean":"2024-09-13T08:38:02.543897+0000","last_became_active":"2024-09-13T08:38:00.236038+0000","last_became_peered":"2024-09-13T08:38:00.236038+0000","last_unstale":"2024-09-13T08:38:02.543897+0000","last_undegraded":"2024-09-13T08:38:02.543897+0000","last_fullsized":"2024-09-13T08:38:02.543897+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:58.973177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.256445+0000","last_change":"2024-09-13T08:38:01.253708+0000","last_active":"2024-09-13T08:38:01.256445+0000","last_peered":"2024-09-13T08:38:01.256445+0000","last_clean":"2024-09-13T08:38:01.256445+0000","last_became_active":"2024-09-13T08:38:01.253387+0000","last_became_peered":"2024-09-13T08:38:01.253387+0000","last_unstale":"2024-09-13T08:38:01.256445+0000","last_undegraded":"2024-09-13T08:38:01.256445+0000","last_fullsized":"2024-09-13T08:38:01.256445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:43.790924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238400+0000","last_change":"2024-09-13T08:38:00.230487+0000","last_active":"2024-09-13T08:38:02.238400+0000","last_peered":"2024-09-13T08:38:02.238400+0000","last_clean":"2024-09-13T08:38:02.238400+0000","last_became_active":"2024-09-13T08:38:00.230118+0000","last_became_peered":"2024-09-13T08:38:00.230118+0000","last_unstale":"2024-09-13T08:38:02.238400+0000","last_undegraded":"2024-09-13T08:38:02.238400+0000","last_fullsized":"2024-09-13T08:38:02.238400+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:31.193415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238374+0000","last_change":"2024-09-13T08:38:01.237796+0000","last_active":"2024-09-13T08:38:02.238374+0000","last_peered":"2024-09-13T08:38:02.238374+0000","last_clean":"2024-09-13T08:38:02.238374+0000","last_became_active":"2024-09-13T08:38:01.237523+0000","last_became_peered":"2024-09-13T08:38:01.237523+0000","last_unstale":"2024-09-13T08:38:02.238374+0000","last_undegraded":"2024-09-13T08:38:02.238374+0000","last_fullsized":"2024-09-13T08:38:02.238374+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:40.687197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.220781+0000","last_change":"2024-09-13T08:38:00.235473+0000","last_active":"2024-09-13T08:38:01.220781+0000","last_peered":"2024-09-13T08:38:01.220781+0000","last_clean":"2024-09-13T08:38:01.220781+0000","last_became_active":"2024-09-13T08:38:00.234870+0000","last_became_peered":"2024-09-13T08:38:00.234870+0000","last_unstale":"2024-09-13T08:38:01.220781+0000","last_undegraded":"2024-09-13T08:38:01.220781+0000","last_fullsized":"2024-09-13T08:38:01.220781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:18.979983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238098+0000","last_change":"2024-09-13T08:38:01.246140+0000","last_active":"2024-09-13T08:38:02.238098+0000","last_peered":"2024-09-13T08:38:02.238098+0000","last_clean":"2024-09-13T08:38:02.238098+0000","last_became_active":"2024-09-13T08:38:01.245801+0000","last_became_peered":"2024-09-13T08:38:01.245801+0000","last_unstale":"2024-09-13T08:38:02.238098+0000","last_undegraded":"2024-09-13T08:38:02.238098+0000","last_fullsized":"2024-09-13T08:38:02.238098+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:05.544931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.263826+0000","last_change":"2024-09-13T08:38:00.230943+0000","last_active":"2024-09-13T08:38:01.263826+0000","last_peered":"2024-09-13T08:38:01.263826+0000","last_clean":"2024-09-13T08:38:01.263826+0000","last_became_active":"2024-09-13T08:38:00.229735+0000","last_became_peered":"2024-09-13T08:38:00.229735+0000","last_unstale":"2024-09-13T08:38:01.263826+0000","last_undegraded":"2024-09-13T08:38:01.263826+0000","last_fullsized":"2024-09-13T08:38:01.263826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:34:28.589744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.244721+0000","last_change":"2024-09-13T08:38:01.241987+0000","last_active":"2024-09-13T08:38:01.244721+0000","last_peered":"2024-09-13T08:38:01.244721+0000","last_clean":"2024-09-13T08:38:01.244721+0000","last_became_active":"2024-09-13T08:38:01.241702+0000","last_became_peered":"2024-09-13T08:38:01.241702+0000","last_unstale":"2024-09-13T08:38:01.244721+0000","last_undegraded":"2024-09-13T08:38:01.244721+0000","last_fullsized":"2024-09-13T08:38:01.244721+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:32.007345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"36'1","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.229661+0000","last_change":"2024-09-13T08:38:00.225418+0000","last_active":"2024-09-13T08:38:03.229661+0000","last_peered":"2024-09-13T08:38:03.229661+0000","last_clean":"2024-09-13T08:38:03.229661+0000","last_became_active":"2024-09-13T08:38:00.225076+0000","last_became_peered":"2024-09-13T08:38:00.225076+0000","last_unstale":"2024-09-13T08:38:03.229661+0000","last_undegraded":"2024-09-13T08:38:03.229661+0000","last_fullsized":"2024-09-13T08:38:03.229661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:38:15.861161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237790+0000","last_change":"2024-09-13T08:38:01.256868+0000","last_active":"2024-09-13T08:38:02.237790+0000","last_peered":"2024-09-13T08:38:02.237790+0000","last_clean":"2024-09-13T08:38:02.237790+0000","last_became_active":"2024-09-13T08:38:01.253978+0000","last_became_peered":"2024-09-13T08:38:01.253978+0000","last_unstale":"2024-09-13T08:38:02.237790+0000","last_undegraded":"2024-09-13T08:38:02.237790+0000","last_fullsized":"2024-09-13T08:38:02.237790+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:43.653097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.230376+0000","last_change":"2024-09-13T08:38:00.225331+0000","last_active":"2024-09-13T08:38:01.230376+0000","last_peered":"2024-09-13T08:38:01.230376+0000","last_clean":"2024-09-13T08:38:01.230376+0000","last_became_active":"2024-09-13T08:38:00.225053+0000","last_became_peered":"2024-09-13T08:38:00.225053+0000","last_unstale":"2024-09-13T08:38:01.230376+0000","last_undegraded":"2024-09-13T08:38:01.230376+0000","last_fullsized":"2024-09-13T08:38:01.230376+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:17:06.943972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.295861+0000","last_change":"2024-09-13T08:38:01.260348+0000","last_active":"2024-09-13T08:38:03.295861+0000","last_peered":"2024-09-13T08:38:03.295861+0000","last_clean":"2024-09-13T08:38:03.295861+0000","last_became_active":"2024-09-13T08:38:01.250067+0000","last_became_peered":"2024-09-13T08:38:01.250067+0000","last_unstale":"2024-09-13T08:38:03.295861+0000","last_undegraded":"2024-09-13T08:38:03.295861+0000","last_fullsized":"2024-09-13T08:38:03.295861+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:21.804316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.230213+0000","last_change":"2024-09-13T08:38:00.227592+0000","last_active":"2024-09-13T08:38:01.230213+0000","last_peered":"2024-09-13T08:38:01.230213+0000","last_clean":"2024-09-13T08:38:01.230213+0000","last_became_active":"2024-09-13T08:38:00.227280+0000","last_became_peered":"2024-09-13T08:38:00.227280+0000","last_unstale":"2024-09-13T08:38:01.230213+0000","last_undegraded":"2024-09-13T08:38:01.230213+0000","last_fullsized":"2024-09-13T08:38:01.230213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:22.801067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237944+0000","last_change":"2024-09-13T08:38:01.243864+0000","last_active":"2024-09-13T08:38:02.237944+0000","last_peered":"2024-09-13T08:38:02.237944+0000","last_clean":"2024-09-13T08:38:02.237944+0000","last_became_active":"2024-09-13T08:38:01.242625+0000","last_became_peered":"2024-09-13T08:38:01.242625+0000","last_unstale":"2024-09-13T08:38:02.237944+0000","last_undegraded":"2024-09-13T08:38:02.237944+0000","last_fullsized":"2024-09-13T08:38:02.237944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:25.627301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.228312+0000","last_change":"2024-09-13T08:38:00.224011+0000","last_active":"2024-09-13T08:38:01.228312+0000","last_peered":"2024-09-13T08:38:01.228312+0000","last_clean":"2024-09-13T08:38:01.228312+0000","last_became_active":"2024-09-13T08:38:00.223453+0000","last_became_peered":"2024-09-13T08:38:00.223453+0000","last_unstale":"2024-09-13T08:38:01.228312+0000","last_undegraded":"2024-09-13T08:38:01.228312+0000","last_fullsized":"2024-09-13T08:38:01.228312+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:28.266039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.236924+0000","last_change":"2024-09-13T08:38:01.250323+0000","last_active":"2024-09-13T08:38:02.236924+0000","last_peered":"2024-09-13T08:38:02.236924+0000","last_clean":"2024-09-13T08:38:02.236924+0000","last_became_active":"2024-09-13T08:38:01.249979+0000","last_became_peered":"2024-09-13T08:38:01.249979+0000","last_unstale":"2024-09-13T08:38:02.236924+0000","last_undegraded":"2024-09-13T08:38:02.236924+0000","last_fullsized":"2024-09-13T08:38:02.236924+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:38:46.212355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.544106+0000","last_change":"2024-09-13T08:38:00.234498+0000","last_active":"2024-09-13T08:38:02.544106+0000","last_peered":"2024-09-13T08:38:02.544106+0000","last_clean":"2024-09-13T08:38:02.544106+0000","last_became_active":"2024-09-13T08:38:00.233639+0000","last_became_peered":"2024-09-13T08:38:00.233639+0000","last_unstale":"2024-09-13T08:38:02.544106+0000","last_undegraded":"2024-09-13T08:38:02.544106+0000","last_fullsized":"2024-09-13T08:38:02.544106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:39.680510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.239704+0000","last_change":"2024-09-13T08:38:01.232354+0000","last_active":"2024-09-13T08:38:01.239704+0000","last_peered":"2024-09-13T08:38:01.239704+0000","last_clean":"2024-09-13T08:38:01.239704+0000","last_became_active":"2024-09-13T08:38:01.232065+0000","last_became_peered":"2024-09-13T08:38:01.232065+0000","last_unstale":"2024-09-13T08:38:01.239704+0000","last_undegraded":"2024-09-13T08:38:01.239704+0000","last_fullsized":"2024-09-13T08:38:01.239704+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:42.324896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.237046+0000","last_change":"2024-09-13T08:38:00.225207+0000","last_active":"2024-09-13T08:38:02.237046+0000","last_peered":"2024-09-13T08:38:02.237046+0000","last_clean":"2024-09-13T08:38:02.237046+0000","last_became_active":"2024-09-13T08:38:00.222287+0000","last_became_peered":"2024-09-13T08:38:00.222287+0000","last_unstale":"2024-09-13T08:38:02.237046+0000","last_undegraded":"2024-09-13T08:38:02.237046+0000","last_fullsized":"2024-09-13T08:38:02.237046+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:02:39.233114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:03.296024+0000","last_change":"2024-09-13T08:38:01.254391+0000","last_active":"2024-09-13T08:38:03.296024+0000","last_peered":"2024-09-13T08:38:03.296024+0000","last_clean":"2024-09-13T08:38:03.296024+0000","last_became_active":"2024-09-13T08:38:01.249345+0000","last_became_peered":"2024-09-13T08:38:01.249345+0000","last_unstale":"2024-09-13T08:38:03.296024+0000","last_undegraded":"2024-09-13T08:38:03.296024+0000","last_fullsized":"2024-09-13T08:38:03.296024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:00.201359+0000","last_clean_scrub_stamp":"2024-09-13T08:38:00.201359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:58.179023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T08:38:01.221843+0000","last_change":"2024-09-13T08:38:00.234248+0000","last_active":"2024-09-13T08:38:01.221843+0000","last_peered":"2024-09-13T08:38:01.221843+0000","last_clean":"2024-09-13T08:38:01.221843+0000","last_became_active":"2024-09-13T08:38:00.233263+0000","last_became_peered":"2024-09-13T08:38:00.233263+0000","last_unstale":"2024-09-13T08:38:01.221843+0000","last_undegraded":"2024-09-13T08:38:01.221843+0000","last_fullsized":"2024-09-13T08:38:01.221843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:37:59.194672+0000","last_clean_scrub_stamp":"2024-09-13T08:37:59.194672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:20.797629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":133,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T08:38:02.238207+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:38:02.238207+0000","last_peered":"2024-09-13T08:38:02.238207+0000","last_clean":"2024-09-13T08:38:02.238207+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:38:02.238207+0000","last_undegraded":"2024-09-13T08:38:02.238207+0000","last_fullsized":"2024-09-13T08:38:02.238207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":405504,"data_stored":156251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":151541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509456,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":987136,"data_stored":740879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":405504,"data_stored":150705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":405504,"data_stored":156263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":999424,"data_stored":746667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":393216,"data_stored":150019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":417792,"data_stored":157863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:38:05.909 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:38:05.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.110+0000 7f5f59200640 1 -- 172.21.15.31:0/3819470233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5406fa60 msgr2=0x7f5f540cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.110+0000 7f5f59200640 1 --2- 172.21.15.31:0/3819470233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5406fa60 0x7f5f540cb740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5f54069dd0 tx=0x7f5f3c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 -- 172.21.15.31:0/3819470233 shutdown_connections 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 --2- 172.21.15.31:0/3819470233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5406fa60 0x7f5f540cb740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 --2- 172.21.15.31:0/3819470233 >> v2:172.21.15.45:3300/0 conn(0x7f5f5415bfc0 0x7f5f54072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 -- 172.21.15.31:0/3819470233 >> 172.21.15.31:0/3819470233 conn(0x7f5f540cfbb0 msgr2=0x7f5f540cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 -- 172.21.15.31:0/3819470233 shutdown_connections 2024-09-13T08:38:06.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.111+0000 7f5f59200640 1 -- 172.21.15.31:0/3819470233 wait complete. 2024-09-13T08:38:06.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.112+0000 7f5f59200640 1 Processor -- start 2024-09-13T08:38:06.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.112+0000 7f5f59200640 1 -- start start 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5f5415bfc0 0x7f5f541f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 0x7f5f5415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f540b3270 con 0x7f5f540c5900 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f540b30f0 con 0x7f5f5406fa60 2024-09-13T08:38:06.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.113+0000 7f5f59200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5f540b33f0 con 0x7f5f5415bfc0 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f52800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 0x7f5f5415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:45604/0 (socket says 172.21.15.31:45604) 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 -- 172.21.15.31:0/3765586094 learned_addr learned my addr 172.21.15.31:0/3765586094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f49400640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5f5415bfc0 0x7f5f541f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 -- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5f5415bfc0 msgr2=0x7f5f541f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5f5415bfc0 0x7f5f541f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 -- 172.21.15.31:0/3765586094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 msgr2=0x7f5f5415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 0x7f5f5415c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f51e00640 1 -- 172.21.15.31:0/3765586094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f540c5140 con 0x7f5f5406fa60 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.114+0000 7f5f52800640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 0x7f5f5415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:06.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f51e00640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e790 tx=0x7f5f4400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400eea0 con 0x7f5f5406fa60 2024-09-13T08:38:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f4400cdb0 con 0x7f5f5406fa60 2024-09-13T08:38:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44010690 con 0x7f5f5406fa60 2024-09-13T08:38:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f540c9a20 con 0x7f5f5406fa60 2024-09-13T08:38:06.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.115+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f541f2a60 con 0x7f5f5406fa60 2024-09-13T08:38:06.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.116+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f540cbf20 con 0x7f5f5406fa60 2024-09-13T08:38:06.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.118+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f5f44010830 con 0x7f5f5406fa60 2024-09-13T08:38:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.118+0000 7f5f4aa00640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 0x7f5f3007ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.119+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f5f4409bb30 con 0x7f5f5406fa60 2024-09-13T08:38:06.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.119+0000 7f5f49400640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 0x7f5f3007ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.120+0000 7f5f49400640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 0x7f5f3007ad10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c033000 tx=0x7f5f3c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.127+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5f44063b40 con 0x7f5f5406fa60 2024-09-13T08:38:06.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.397+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5f540cb310 con 0x7f5f5406fa60 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.403+0000 7f5f4aa00640 1 -- 172.21.15.31:0/3765586094 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) ==== 190+0+147 (secure 0 0 0) 0x7f5f44068570 con 0x7f5f5406fa60 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:38:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:38:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 msgr2=0x7f5f3007ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 0x7f5f3007ad10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c033000 tx=0x7f5f3c004730 comp rx=0 tx=0).stop 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 msgr2=0x7f5f540c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e790 tx=0x7f5f4400ec50 comp rx=0 tx=0).stop 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 shutdown_connections 2024-09-13T08:38:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f540c5900 0x7f5f5415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5f30078850 0x7f5f3007ad10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5f5415bfc0 0x7f5f541f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 --2- 172.21.15.31:0/3765586094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f5406fa60 0x7f5f540c6c80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 >> 172.21.15.31:0/3765586094 conn(0x7f5f540cfbb0 msgr2=0x7f5f540ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 shutdown_connections 2024-09-13T08:38:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.406+0000 7f5f59200640 1 -- 172.21.15.31:0/3765586094 wait complete. 2024-09-13T08:38:06.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 -- 172.21.15.31:0/2844694766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0cad40 msgr2=0x7f9b6c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2844694766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0cad40 0x7f9b6c0cb1b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9b64005fd0 tx=0x7f9b6402aaa0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 -- 172.21.15.31:0/2844694766 shutdown_connections 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2844694766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0cad40 0x7f9b6c0cb1b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2844694766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c151a80 0x7f9b6c0cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 -- 172.21.15.31:0/2844694766 >> 172.21.15.31:0/2844694766 conn(0x7f9b6c06cb00 msgr2=0x7f9b6c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 -- 172.21.15.31:0/2844694766 shutdown_connections 2024-09-13T08:38:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.582+0000 7f9b70e00640 1 -- 172.21.15.31:0/2844694766 wait complete. 2024-09-13T08:38:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.584+0000 7f9b70e00640 1 Processor -- start 2024-09-13T08:38:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.584+0000 7f9b70e00640 1 -- start start 2024-09-13T08:38:06.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c0cad40 0x7f9b6c149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c151a80 0x7f9b6c14ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c06e900 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c0cb8e0 con 0x7f9b6c151a80 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b70e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c06ea80 con 0x7f9b6c0cad40 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b62a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b62a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47062/0 (socket says 172.21.15.31:47062) 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b62a00640 1 -- 172.21.15.31:0/2502059143 learned_addr learned my addr 172.21.15.31:0/2502059143 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b6b400640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c0cad40 0x7f9b6c149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.585+0000 7f9b6be00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c151a80 0x7f9b6c14ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 -- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c0cad40 msgr2=0x7f9b6c149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c0cad40 0x7f9b6c149db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 -- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c151a80 msgr2=0x7f9b6c14ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c151a80 0x7f9b6c14ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 -- 172.21.15.31:0/2502059143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c14ef30 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b62a00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c450 tx=0x7f9b5400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b540052d0 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b54005470 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54012870 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c14f210 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.586+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c14f7d0 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.587+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c0cb1b0 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.588+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f9b540035f0 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.589+0000 7f9b63e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 0x7f9b4807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.589+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f9b5409a380 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.590+0000 7f9b6b400640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 0x7f9b4807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:06.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.590+0000 7f9b6b400640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 0x7f9b4807ad10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0c6520 tx=0x7f9b5c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:06.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.593+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9b54062300 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.824+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b6c0ca180 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.825+0000 7f9b63e00640 1 -- 172.21.15.31:0/2502059143 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5214 (secure 0 0 0) 0x7f9b54066d30 con 0x7f9b6c0ceab0 2024-09-13T08:38:06.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:06.827 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"btime":"2024-09-13T08:38:04:249411+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 msgr2=0x7f9b4807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 0x7f9b4807ad10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0c6520 tx=0x7f9b5c005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 msgr2=0x7f9b6c14ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c450 tx=0x7f9b5400c910 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 shutdown_connections 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c151a80 0x7f9b6c14ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b6c0ceab0 0x7f9b6c14ffd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9b48078850 0x7f9b4807ad10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 --2- 172.21.15.31:0/2502059143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b6c0cad40 0x7f9b6c149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 >> 172.21.15.31:0/2502059143 conn(0x7f9b6c06cb00 msgr2=0x7f9b6c073350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 shutdown_connections 2024-09-13T08:38:06.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:06.827+0000 7f9b70e00640 1 -- 172.21.15.31:0/2502059143 wait complete. 2024-09-13T08:38:07.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:38:00.656360+0000', 'modified': '2024-09-13T08:38:03.686476+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4878}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4878': {'gid': 4878, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6834/47525403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 47525403}, {'type': 'v1', 'addr': '172.21.15.31:6834', 'nonce': 47525403}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4878, 'qdb_cluster': [4878]} 2024-09-13T08:38:07.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:38:07.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T08:38:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 -- 172.21.15.31:0/946231907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0b16a0 msgr2=0x7f1bfc0b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 --2- 172.21.15.31:0/946231907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0b16a0 0x7f1bfc0b1ad0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009830 tx=0x7f1bf402a510 comp rx=0 tx=0).stop 2024-09-13T08:38:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 -- 172.21.15.31:0/946231907 shutdown_connections 2024-09-13T08:38:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 --2- 172.21.15.31:0/946231907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0b16a0 0x7f1bfc0b1ad0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 --2- 172.21.15.31:0/946231907 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0ceb40 0x7f1bfc1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 -- 172.21.15.31:0/946231907 >> 172.21.15.31:0/946231907 conn(0x7f1bfc0d1e80 msgr2=0x7f1bfc0d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 -- 172.21.15.31:0/946231907 shutdown_connections 2024-09-13T08:38:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.410+0000 7f1c03e00640 1 -- 172.21.15.31:0/946231907 wait complete. 2024-09-13T08:38:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.412+0000 7f1c03e00640 1 Processor -- start 2024-09-13T08:38:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.412+0000 7f1c03e00640 1 -- start start 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0b16a0 0x7f1bfc154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 0x7f1bfc0b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc0c57b0 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc0c5630 con 0x7f1bfc0ce760 2024-09-13T08:38:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c03e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc0b1270 con 0x7f1bfc0b16a0 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47078/0 (socket says 172.21.15.31:47078) 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 -- 172.21.15.31:0/867374012 learned_addr learned my addr 172.21.15.31:0/867374012 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c00a00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 0x7f1bfc0b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01400640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0b16a0 0x7f1bfc154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 -- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0b16a0 msgr2=0x7f1bfc154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0b16a0 0x7f1bfc154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 -- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 msgr2=0x7f1bfc0b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.413+0000 7f1c01e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 0x7f1bfc0b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1c01e00640 1 -- 172.21.15.31:0/867374012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc0b5b00 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1c00a00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 0x7f1bfc0b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1c01e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1bf402ab00 tx=0x7f1bf4032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1bf9600640 1 -- 172.21.15.31:0/867374012 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4040070 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1bf9600640 1 -- 172.21.15.31:0/867374012 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bf4005310 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1bf9600640 1 -- 172.21.15.31:0/867374012 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40047a0 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc0b72c0 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.414+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc0b7880 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.417+0000 7f1bf9600640 1 -- 172.21.15.31:0/867374012 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f1bf4039040 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.417+0000 7f1bf9600640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 0x7f1bdc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.418+0000 7f1bf9600640 1 -- 172.21.15.31:0/867374012 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f1bf40bb760 con 0x7f1bfc0ceb40 2024-09-13T08:38:07.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.418+0000 7f1c01400640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 0x7f1bdc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.418+0000 7f1c06e84740 1 Processor -- start 2024-09-13T08:38:07.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.418+0000 7f1c06e84740 1 -- start start 2024-09-13T08:38:07.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.418+0000 7f1c01400640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 0x7f1bdc07ad10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001840 tx=0x7f1bf00017d0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563b88912430 con 0x563b8898ffa0 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563b889025d0 con 0x563b889d48b0 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563b888f1a00 con 0x563b8898fa20 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c01400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c00a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c00a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:48818/0 (socket says 172.21.15.31:48818) 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.419+0000 7f1c00a00640 1 -- 172.21.15.31:0/1036124352 learned_addr learned my addr 172.21.15.31:0/1036124352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 -- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 msgr2=0x563b889d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 -- 172.21.15.31:0/1036124352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 msgr2=0x563b889d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 -- 172.21.15.31:0/1036124352 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b888ba520 con 0x563b8898fa20 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009f30 tx=0x7f1be80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80aca26a8905c1d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1bedc00640 1 -- 172.21.15.31:0/1036124352 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8004030 con 0x563b8898fa20 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1bedc00640 1 -- 172.21.15.31:0/1036124352 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1be80042c0 con 0x563b8898fa20 2024-09-13T08:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.420+0000 7f1bedc00640 1 -- 172.21.15.31:0/1036124352 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8005220 con 0x563b8898fa20 2024-09-13T08:38:07.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 -- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 msgr2=0x563b889d3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009f30 tx=0x7f1be80118d0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 -- 172.21.15.31:0/1036124352 shutdown_connections 2024-09-13T08:38:07.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b8898fa20 0x563b889d3390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1036124352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 -- 172.21.15.31:0/1036124352 >> 172.21.15.31:0/1036124352 conn(0x563b8898aff0 msgr2=0x563b8898ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 -- 172.21.15.31:0/1036124352 shutdown_connections 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.421+0000 7f1c06e84740 1 -- 172.21.15.31:0/1036124352 wait complete. 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.422+0000 7f1c06e84740 1 -- start start 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.422+0000 7f1c06e84740 1 Processor -- start 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563b88898330 con 0x563b8898ffa0 2024-09-13T08:38:07.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563b888f5810 con 0x563b889d48b0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563b88662620 con 0x563b889d2180 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c06e84740 10 client.0 _ll_register_callbacks cb 0x563b8889d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47086/0 (socket says 172.21.15.31:47086) 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c01400640 1 -- 172.21.15.31:0/1155008548 learned_addr learned my addr 172.21.15.31:0/1155008548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.423+0000 7f1c01e00640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 -- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 msgr2=0x563b889b7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 -- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 msgr2=0x563b889d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 -- 172.21.15.31:0/1155008548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b88653850 con 0x563b8898ffa0 2024-09-13T08:38:07.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01e00640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c01400640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bf001dec0 tx=0x7f1bf0001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2305f1fa2aea6a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1be2000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf001d0d0 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be80042c0 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.424+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f1bf001d270 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f1bf001b4f0 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 10 client.5043 fetch_fsmap learned FSMap version 17 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x563b886ca6b0 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1be80042c0 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7f1bf0037080 con 0x563b8898ffa0 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 10 client.5043 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 10 client.5043 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T08:38:07.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 20 client.5043 populate_metadata read hostname 'smithi031' 2024-09-13T08:38:07.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b889a3840 0x563b889a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 4 client.5043 mds_command: new command op to 4878 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:38:07.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 --> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b88904520 con 0x563b889a3840 2024-09-13T08:38:07.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.425+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b889a3840 0x563b889a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.426+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b889a3840 0x563b889a3c20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94bf4327dea945af server_cookie=827528afa9b4732 in_seq=0 out_seq=0 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.426+0000 7f1be2000640 10 client.5043 ms_handle_connect on v2:172.21.15.31:6832/47525403 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1be2000640 1 -- 172.21.15.31:0/1155008548 <== mds.0 v2:172.21.15.31:6832/47525403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563b88904520 con 0x563b889a3840 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1be2000640 10 client.5043 handle_command_reply: tid=0 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 client.5043 shutdown 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 msgr2=0x563b889d3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bf001dec0 tx=0x7f1bf0001ad0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 shutdown_connections 2024-09-13T08:38:07.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b889a3840 0x563b889a3c20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b889d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1155008548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889d3be0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1be2000640 0 client.5043 ms_handle_reset on v2:172.21.15.31:6832/47525403 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.427+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 >> 172.21.15.31:0/1155008548 conn(0x563b8898fa20 msgr2=0x563b889d6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.428+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 shutdown_connections 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.428+0000 7f1c06e84740 1 -- 172.21.15.31:0/1155008548 wait complete. 2024-09-13T08:38:07.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.428+0000 7f1c06e84740 20 client.5043 trim_cache size 0 max 16384 2024-09-13T08:38:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.447+0000 7f1c06e84740 1 Processor -- start 2024-09-13T08:38:07.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.447+0000 7f1c06e84740 1 -- start start 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b8898ffa0 0x563b889d6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc06eea0 con 0x563b889d48b0 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc0cadc0 con 0x563b8898ffa0 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc0cb410 con 0x563b889d2180 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47090/0 (socket says 172.21.15.31:47090) 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 -- 172.21.15.31:0/1107941720 learned_addr learned my addr 172.21.15.31:0/1107941720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01400640 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b8898ffa0 0x563b889d6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c00a00640 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 -- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 msgr2=0x563b889b7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 -- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b8898ffa0 msgr2=0x563b889d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b8898ffa0 0x563b889d6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.448+0000 7f1c01e00640 1 -- 172.21.15.31:0/1107941720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b889e68a0 con 0x563b889d48b0 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.449+0000 7f1c01e00640 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40809e0 tx=0x7f1bf402a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d2ab781ca65de9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.449+0000 7f1bed200640 1 -- 172.21.15.31:0/1107941720 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4049040 con 0x563b889d48b0 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.449+0000 7f1bed200640 1 -- 172.21.15.31:0/1107941720 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bf4083bd0 con 0x563b889d48b0 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.449+0000 7f1bed200640 1 -- 172.21.15.31:0/1107941720 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf408ce20 con 0x563b889d48b0 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- 172.21.15.31:0/1107941720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 msgr2=0x563b889b7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40809e0 tx=0x7f1bf402a510 comp rx=0 tx=0).stop 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- 172.21.15.31:0/1107941720 shutdown_connections 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b889d48b0 0x563b889b7650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889b7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 --2- 172.21.15.31:0/1107941720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b8898ffa0 0x563b889d6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- 172.21.15.31:0/1107941720 >> 172.21.15.31:0/1107941720 conn(0x563b8898fa20 msgr2=0x563b88984560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- 172.21.15.31:0/1107941720 shutdown_connections 2024-09-13T08:38:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- 172.21.15.31:0/1107941720 wait complete. 2024-09-13T08:38:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.450+0000 7f1c06e84740 1 -- start start 2024-09-13T08:38:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 Processor -- start 2024-09-13T08:38:07.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b88987e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563b889a6610 con 0x563b8898ffa0 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563b889a6a70 con 0x563b889d48b0 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563b889e71d0 con 0x563b889d2180 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.451+0000 7f1c06e84740 10 client.0 _ll_register_callbacks cb 0x7f1bf001db90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47098/0 (socket says 172.21.15.31:47098) 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 -- 172.21.15.31:0/3346453776 learned_addr learned my addr 172.21.15.31:0/3346453776 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01e00640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b88987e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c00a00640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 -- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 msgr2=0x563b889f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 -- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 msgr2=0x563b88987e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b88987e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 -- 172.21.15.31:0/3346453776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b8898e2a0 con 0x563b8898ffa0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.452+0000 7f1c01400640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1bf001f010 tx=0x7f1bf0015630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3332c418c706320b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0004a40 con 0x563b8898ffa0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f1bf0015a30 con 0x563b8898ffa0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bf4083bd0 con 0x563b8898ffa0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f1bf0045050 con 0x563b8898ffa0 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 10 client.5049 fetch_fsmap learned FSMap version 17 2024-09-13T08:38:07.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x563b8898a1d0 con 0x563b8898ffa0 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bf4083bd0 con 0x563b8898ffa0 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7f1bf0015c70 con 0x563b8898ffa0 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 10 client.5049 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 10 client.5049 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 20 client.5049 populate_metadata read hostname 'smithi031' 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b88a53ee0 0x563b88a57890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 4 client.5049 mds_command: new command op to 4878 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.453+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 --> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] -- command(tid 0: {"prefix": "status"}) -- 0x563b888fffd0 con 0x563b88a53ee0 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1c00a00640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b88a53ee0 0x563b88a57890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1c00a00640 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b88a53ee0 0x563b88a57890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30b9d6ee2eb4a013 server_cookie=6cac635fa0b04fd2 in_seq=0 out_seq=0 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1be2000640 10 client.5049 ms_handle_connect on v2:172.21.15.31:6832/47525403 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1be2000640 1 -- 172.21.15.31:0/3346453776 <== mds.0 v2:172.21.15.31:6832/47525403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x563b888fffd0 con 0x563b88a53ee0 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1be2000640 10 client.5049 handle_command_reply: tid=0 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.454+0000 7f1c06e84740 1 client.5049 shutdown 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 msgr2=0x563b889ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1bf001f010 tx=0x7f1bf0015630 comp rx=0 tx=0).stop 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 shutdown_connections 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:6832/47525403,v1:172.21.15.31:6834/47525403] conn(0x563b88a53ee0 0x563b88a57890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b889d48b0 0x563b88987e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563b889d2180 0x563b889f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 --2- 172.21.15.31:0/3346453776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b8898ffa0 0x563b889ccff0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 >> 172.21.15.31:0/3346453776 conn(0x563b8898fa20 msgr2=0x563b889890f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1be2000640 0 client.5049 ms_handle_reset on v2:172.21.15.31:6832/47525403 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 shutdown_connections 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 1 -- 172.21.15.31:0/3346453776 wait complete. 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.455+0000 7f1c06e84740 20 client.5049 trim_cache size 0 max 16384 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4878, 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:38:07.462 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2382043769999997, 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 17, 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 36, 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 36, 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 14.043955487 2024-09-13T08:38:07.463 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 msgr2=0x7f1bdc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 0x7f1bdc07ad10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001840 tx=0x7f1bf00017d0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 msgr2=0x7f1bfc1f25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1bf402ab00 tx=0x7f1bf4032b90 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 shutdown_connections 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc0ceb40 0x7f1bfc1f25d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bfc0ce760 0x7f1bfc0b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1bdc078850 0x7f1bdc07ad10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 --2- 172.21.15.31:0/867374012 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1bfc0b16a0 0x7f1bfc154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 >> 172.21.15.31:0/867374012 conn(0x7f1bfc0d1e80 msgr2=0x7f1bfc0b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 shutdown_connections 2024-09-13T08:38:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.456+0000 7f1c03e00640 1 -- 172.21.15.31:0/867374012 wait complete. 2024-09-13T08:38:07.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:38:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 -- 172.21.15.31:0/4194375298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 msgr2=0x7f4a280ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/4194375298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a280ce850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4a280cd9e0 tx=0x7f4a2002aa10 comp rx=0 tx=0).stop 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 -- 172.21.15.31:0/4194375298 shutdown_connections 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/4194375298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a280ce850 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/4194375298 >> v2:172.21.15.45:3300/0 conn(0x7f4a2815f9e0 0x7f4a2815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/4194375298 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815bd70 0x7f4a280cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 -- 172.21.15.31:0/4194375298 >> 172.21.15.31:0/4194375298 conn(0x7f4a280cc910 msgr2=0x7f4a280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 -- 172.21.15.31:0/4194375298 shutdown_connections 2024-09-13T08:38:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.631+0000 7f4a2fa00640 1 -- 172.21.15.31:0/4194375298 wait complete. 2024-09-13T08:38:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.633+0000 7f4a2fa00640 1 Processor -- start 2024-09-13T08:38:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.633+0000 7f4a2fa00640 1 -- start start 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 0x7f4a280b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 0x7f4a280b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280cb9d0 con 0x7f4a280ce3c0 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280cb850 con 0x7f4a2815bd70 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2fa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4a280cbc50 con 0x7f4a2815f9e0 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47102/0 (socket says 172.21.15.31:47102) 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2d000640 1 -- 172.21.15.31:0/2401098632 learned_addr learned my addr 172.21.15.31:0/2401098632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a1fe00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 0x7f4a280b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.634+0000 7f4a2da00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 0x7f4a280b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 -- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 msgr2=0x7f4a280b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 0x7f4a280b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 -- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 msgr2=0x7f4a280b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 0x7f4a280b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 -- 172.21.15.31:0/2401098632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2809c330 con 0x7f4a280ce3c0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a1fe00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 0x7f4a280b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2da00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 0x7f4a280b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2d000640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800c920 tx=0x7f4a1800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18007c70 con 0x7f4a280ce3c0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a1800f450 con 0x7f4a280ce3c0 2024-09-13T08:38:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2809c610 con 0x7f4a280ce3c0 2024-09-13T08:38:07.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800f8a0 con 0x7f4a280ce3c0 2024-09-13T08:38:07.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.635+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2809cbd0 con 0x7f4a280ce3c0 2024-09-13T08:38:07.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.636+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a280cf850 con 0x7f4a280ce3c0 2024-09-13T08:38:07.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.637+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4a1800fa40 con 0x7f4a280ce3c0 2024-09-13T08:38:07.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.638+0000 7f4a25600640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 0x7f4a0407ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:07.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.639+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f4a1809b1d0 con 0x7f4a280ce3c0 2024-09-13T08:38:07.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.639+0000 7f4a1fe00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 0x7f4a0407ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:07.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.640+0000 7f4a1fe00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 0x7f4a0407ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4a10005fd0 tx=0x7f4a10005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:07.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.643+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4a18014030 con 0x7f4a280ce3c0 2024-09-13T08:38:07.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.875+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4a280b5e10 con 0x7f4a280ce3c0 2024-09-13T08:38:07.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.876+0000 7f4a25600640 1 -- 172.21.15.31:0/2401098632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f4a18063830 con 0x7f4a280ce3c0 2024-09-13T08:38:07.878 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:38:07.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 msgr2=0x7f4a0407ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 0x7f4a0407ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4a10005fd0 tx=0x7f4a10005850 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 msgr2=0x7f4a28155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800c920 tx=0x7f4a1800cde0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 shutdown_connections 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a04078880 0x7f4a0407ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a2815f9e0 0x7f4a280b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a2815bd70 0x7f4a280b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 --2- 172.21.15.31:0/2401098632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280ce3c0 0x7f4a28155f40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 >> 172.21.15.31:0/2401098632 conn(0x7f4a280cc910 msgr2=0x7f4a280cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 shutdown_connections 2024-09-13T08:38:07.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:07.878+0000 7f4a2fa00640 1 -- 172.21.15.31:0/2401098632 wait complete. 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:38:07.897 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:07.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:07.898 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:38:07.933 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:38:07.933 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:38:07.933 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42644sec preferred_lft 42644sec 2024-09-13T08:38:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:07.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:07.936 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:38:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:38:08.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:08.030 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:38:08.066 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:38:08.067 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:38:08.067 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:38:08.067 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:38:08.067 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:38:08.067 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:38:08.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:38:08.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:08.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:08.266 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:38:08.301 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:38:08.301 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:38:08.302 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:08.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:08.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:38:08.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:08.400 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:08.436 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:08.436 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:08.436 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:08.510 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:38:08.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:08.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:08.640 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:08.640 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:08.667 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:08.667 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:08.667 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:38:08.668 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:38:08.668 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:38:08.669 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:38:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:38:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:38:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:38:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:38:08.673 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:38:08.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:08.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:08.730 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:38:08.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:38:08.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:08.827 INFO:teuthology.orchestra.run.smithi031.stdout:client4799 2024-09-13T08:38:08.827 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:08.827 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:08.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 -- 172.21.15.31:0/1788874184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 msgr2=0x7f42fc0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 --2- 172.21.15.31:0/1788874184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0d3bd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0d19a0 tx=0x7f42e802a560 comp rx=0 tx=0).stop 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 -- 172.21.15.31:0/1788874184 shutdown_connections 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 --2- 172.21.15.31:0/1788874184 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 --2- 172.21.15.31:0/1788874184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0d3bd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 --2- 172.21.15.31:0/1788874184 >> v2:172.21.15.45:3300/0 conn(0x7f42fc15dfa0 0x7f42fc0d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 -- 172.21.15.31:0/1788874184 >> 172.21.15.31:0/1788874184 conn(0x7f42fc08d8b0 msgr2=0x7f42fc09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:09.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 -- 172.21.15.31:0/1788874184 shutdown_connections 2024-09-13T08:38:09.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.062+0000 7f4302200640 1 -- 172.21.15.31:0/1788874184 wait complete. 2024-09-13T08:38:09.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.063+0000 7f4302200640 1 Processor -- start 2024-09-13T08:38:09.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.064+0000 7f4302200640 1 -- start start 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc0b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42fc15dfa0 0x7f42fc0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc162270 con 0x7f42fc0b47f0 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1620f0 con 0x7f42fc15dfa0 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4302200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc1623f0 con 0x7f42fc0d4110 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47116/0 (socket says 172.21.15.31:47116) 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 -- 172.21.15.31:0/3198304899 learned_addr learned my addr 172.21.15.31:0/3198304899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f42fbe00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc0b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4301800640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42fc15dfa0 0x7f42fc0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 -- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 msgr2=0x7f42fc0b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc0b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 -- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42fc15dfa0 msgr2=0x7f42fc0d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42fc15dfa0 0x7f42fc0d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f4300e00640 1 -- 172.21.15.31:0/3198304899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc0b1ef0 con 0x7f42fc0b47f0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.065+0000 7f42fbe00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc0b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f4300e00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f42f000d770 tx=0x7f42f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0017070 con 0x7f42fc0b47f0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42f0009cc0 con 0x7f42fc0b47f0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f00106c0 con 0x7f42fc0b47f0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc1f4cd0 con 0x7f42fc0b47f0 2024-09-13T08:38:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.066+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc0ccc40 con 0x7f42fc0b47f0 2024-09-13T08:38:09.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.067+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc0d4cf0 con 0x7f42fc0b47f0 2024-09-13T08:38:09.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.068+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f42f001a070 con 0x7f42fc0b47f0 2024-09-13T08:38:09.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.069+0000 7f42f8c00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 0x7f42dc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:09.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.069+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f42f009aa20 con 0x7f42fc0b47f0 2024-09-13T08:38:09.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.069+0000 7f42fbe00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 0x7f42dc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:09.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.070+0000 7f42fbe00640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 0x7f42dc07ad10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42e8004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:09.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.073+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f42f0063080 con 0x7f42fc0b47f0 2024-09-13T08:38:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.313+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f42fc161980 con 0x7f42fc0b47f0 2024-09-13T08:38:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.314+0000 7f42f8c00640 1 -- 172.21.15.31:0/3198304899 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f42f0062980 con 0x7f42fc0b47f0 2024-09-13T08:38:09.316 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 msgr2=0x7f42dc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 0x7f42dc07ad10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42e8004730 comp rx=0 tx=0).stop 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 msgr2=0x7f42fc0cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f42f000d770 tx=0x7f42f000dc30 comp rx=0 tx=0).stop 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 shutdown_connections 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42fc15dfa0 0x7f42fc0d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f42dc078850 0x7f42dc07ad10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42fc0d4110 0x7f42fc0b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 --2- 172.21.15.31:0/3198304899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42fc0b47f0 0x7f42fc0cc700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 >> 172.21.15.31:0/3198304899 conn(0x7f42fc08d8b0 msgr2=0x7f42fc0b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 shutdown_connections 2024-09-13T08:38:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:09.316+0000 7f4302200640 1 -- 172.21.15.31:0/3198304899 wait complete. 2024-09-13T08:38:09.334 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:38:09.334 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:38:09.334 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:38:09.334 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:38:09.334 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:09.335 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:38:09.335 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:38:09.335 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:38:09.335 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:09.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.335 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:38:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:38:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42647sec preferred_lft 42647sec 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:38:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:09.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.379 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:38:09.477 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:38:09.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.480 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:38:09.518 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:38:09.518 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:38:09.518 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:38:09.519 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:38:09.519 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:38:09.519 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:38:09.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:38:09.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:09.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.717 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:38:09.752 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:38:09.752 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:38:09.752 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:09.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:38:09.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:09.848 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:09.883 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:09.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:09.883 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:09.958 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:38:09.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:09.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:10.087 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:10.087 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:38:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:38:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:38:10.118 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:38:10.118 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:38:10.118 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:38:10.118 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:38:10.118 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:38:10.121 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:38:10.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:10.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:10.177 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:38:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:38:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:10.268 INFO:teuthology.orchestra.run.smithi045.stdout:client4978 2024-09-13T08:38:10.269 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:10.269 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:10.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 -- 172.21.15.31:0/1051552647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cd8b0 msgr2=0x7f4ecc0cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 --2- 172.21.15.31:0/1051552647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cd8b0 0x7f4ecc0cdd20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4005fd0 tx=0x7f4ec402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 -- 172.21.15.31:0/1051552647 shutdown_connections 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 --2- 172.21.15.31:0/1051552647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cd8b0 0x7f4ecc0cdd20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 -- 172.21.15.31:0/1051552647 >> 172.21.15.31:0/1051552647 conn(0x7f4ecc0cf9b0 msgr2=0x7f4ecc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 -- 172.21.15.31:0/1051552647 shutdown_connections 2024-09-13T08:38:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.481+0000 7f4ed1000640 1 -- 172.21.15.31:0/1051552647 wait complete. 2024-09-13T08:38:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.482+0000 7f4ed1000640 1 Processor -- start 2024-09-13T08:38:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.483+0000 7f4ed1000640 1 -- start start 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cbe30 0x7f4ecc1f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ecc0cd8b0 0x7f4ecc152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc0ca390 con 0x7f4ecc15b890 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc0ca210 con 0x7f4ecc0cbe30 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ed1000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4ecc0ca510 con 0x7f4ecc0cd8b0 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47124/0 (socket says 172.21.15.31:47124) 2024-09-13T08:38:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 -- 172.21.15.31:0/231591323 learned_addr learned my addr 172.21.15.31:0/231591323 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ec2a00640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ecc0cd8b0 0x7f4ecc152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecb400640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cbe30 0x7f4ecc1f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 -- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ecc0cd8b0 msgr2=0x7f4ecc152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ecc0cd8b0 0x7f4ecc152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 -- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cbe30 msgr2=0x7f4ecc1f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cbe30 0x7f4ecc1f7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.484+0000 7f4ecbe00640 1 -- 172.21.15.31:0/231591323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ecc1f86d0 con 0x7f4ecc15b890 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ecbe00640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4006010 tx=0x7f4ec40316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4040070 con 0x7f4ecc15b890 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ec4004040 con 0x7f4ecc15b890 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4004450 con 0x7f4ecc15b890 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ecc1f89b0 con 0x7f4ecc15b890 2024-09-13T08:38:10.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.485+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ecc0c6c00 con 0x7f4ecc15b890 2024-09-13T08:38:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.486+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ecc0ca9f0 con 0x7f4ecc15b890 2024-09-13T08:38:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.487+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f4ec4039040 con 0x7f4ecc15b890 2024-09-13T08:38:10.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.489+0000 7f4ec3e00640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 0x7f4ea807ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:10.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.489+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f4ec40bbe80 con 0x7f4ecc15b890 2024-09-13T08:38:10.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.489+0000 7f4ecb400640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 0x7f4ea807ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:10.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.490+0000 7f4ecb400640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 0x7f4ea807ad10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0046b0 tx=0x7f4ebc007cc0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:10.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.493+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4ec4083e10 con 0x7f4ecc15b890 2024-09-13T08:38:10.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:10.730+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f4ecc0aea10 con 0x7f4ecc15b890 2024-09-13T08:38:11.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.421+0000 7f4ec3e00640 1 -- 172.21.15.31:0/231591323 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) ==== 136+0+0 (secure 0 0 0) 0x7f4ec4088840 con 0x7f4ecc15b890 2024-09-13T08:38:11.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 msgr2=0x7f4ea807ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 0x7f4ea807ad10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0046b0 tx=0x7f4ebc007cc0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 msgr2=0x7f4ecc0c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4006010 tx=0x7f4ec40316e0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 shutdown_connections 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ecc15b890 0x7f4ecc0c8e20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4ea8078850 0x7f4ea807ad10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ecc0cd8b0 0x7f4ecc152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 --2- 172.21.15.31:0/231591323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ecc0cbe30 0x7f4ecc1f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 >> 172.21.15.31:0/231591323 conn(0x7f4ecc0cf9b0 msgr2=0x7f4ecc0af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 shutdown_connections 2024-09-13T08:38:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.424+0000 7f4ed1000640 1 -- 172.21.15.31:0/231591323 wait complete. 2024-09-13T08:38:11.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:38:11.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 -- 172.21.15.31:0/3663178614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd301446e0 msgr2=0x7fbd3015bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 --2- 172.21.15.31:0/3663178614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd301446e0 0x7fbd3015bdc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbd24005fd0 tx=0x7fbd2402a530 comp rx=0 tx=0).stop 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 -- 172.21.15.31:0/3663178614 shutdown_connections 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 --2- 172.21.15.31:0/3663178614 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd3015e930 0x7fbd3015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 --2- 172.21.15.31:0/3663178614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd301446e0 0x7fbd3015bdc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 -- 172.21.15.31:0/3663178614 >> 172.21.15.31:0/3663178614 conn(0x7fbd300b18f0 msgr2=0x7fbd300b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 -- 172.21.15.31:0/3663178614 shutdown_connections 2024-09-13T08:38:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.608+0000 7fbd38800640 1 -- 172.21.15.31:0/3663178614 wait complete. 2024-09-13T08:38:11.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.610+0000 7fbd38800640 1 Processor -- start 2024-09-13T08:38:11.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.610+0000 7fbd38800640 1 -- start start 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd300cec90 0x7fbd3015a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd301446e0 0x7fbd300743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd300cf2c0 con 0x7fbd3015e930 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbd300cf140 con 0x7fbd301446e0 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd38800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbd300cf440 con 0x7fbd300cec90 2024-09-13T08:38:11.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47142/0 (socket says 172.21.15.31:47142) 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd36800640 1 -- 172.21.15.31:0/1765578148 learned_addr learned my addr 172.21.15.31:0/1765578148 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd35400640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd301446e0 0x7fbd300743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.611+0000 7fbd35e00640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd300cec90 0x7fbd3015a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 -- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd300cec90 msgr2=0x7fbd3015a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd300cec90 0x7fbd3015a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 -- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd301446e0 msgr2=0x7fbd300743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd301446e0 0x7fbd300743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 -- 172.21.15.31:0/1765578148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd301f5b20 con 0x7fbd3015e930 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd36800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00c780 tx=0x7fbd2c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c004b60 con 0x7fbd3015e930 2024-09-13T08:38:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd2c004df0 con 0x7fbd3015e930 2024-09-13T08:38:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd30074930 con 0x7fbd3015e930 2024-09-13T08:38:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00aa70 con 0x7fbd3015e930 2024-09-13T08:38:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.612+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd301f5840 con 0x7fbd3015e930 2024-09-13T08:38:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.613+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd3015bdc0 con 0x7fbd3015e930 2024-09-13T08:38:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.615+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fbd2c00ac10 con 0x7fbd3015e930 2024-09-13T08:38:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.615+0000 7fbd2a000640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 0x7fbd1407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.616+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fbd2c09ac30 con 0x7fbd3015e930 2024-09-13T08:38:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.616+0000 7fbd35e00640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 0x7fbd1407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:11.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.617+0000 7fbd35e00640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 0x7fbd1407ad10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd24033000 tx=0x7fbd24004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.620+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbd2c020070 con 0x7fbd3015e930 2024-09-13T08:38:11.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.838+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbd301f55f0 con 0x7fbd14078850 2024-09-13T08:38:11.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.840+0000 7fbd2a000640 1 -- 172.21.15.31:0/1765578148 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fbd301f55f0 con 0x7fbd14078850 2024-09-13T08:38:11.842 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:38:11.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:11.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:38:11.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:11.842 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:38:11.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 msgr2=0x7fbd1407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 0x7fbd1407ad10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd24033000 tx=0x7fbd24004730 comp rx=0 tx=0).stop 2024-09-13T08:38:11.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 msgr2=0x7fbd30158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:11.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00c780 tx=0x7fbd2c00cc40 comp rx=0 tx=0).stop 2024-09-13T08:38:11.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 shutdown_connections 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd3015e930 0x7fbd30158ef0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd301446e0 0x7fbd300743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd14078850 0x7fbd1407ad10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 --2- 172.21.15.31:0/1765578148 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbd300cec90 0x7fbd3015a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 >> 172.21.15.31:0/1765578148 conn(0x7fbd300b18f0 msgr2=0x7fbd300ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 shutdown_connections 2024-09-13T08:38:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:11.842+0000 7fbd38800640 1 -- 172.21.15.31:0/1765578148 wait complete. 2024-09-13T08:38:11.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:38:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 -- 172.21.15.31:0/1059464817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7d0003a30 msgr2=0x7fc7d0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1059464817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7d0003a30 0x7fc7d0003e50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00c310 tx=0x7fc7d0005830 comp rx=0 tx=0).stop 2024-09-13T08:38:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7ea800640 1 -- 172.21.15.31:0/1059464817 reap_dead start 2024-09-13T08:38:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 -- 172.21.15.31:0/1059464817 shutdown_connections 2024-09-13T08:38:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 -- 172.21.15.31:0/1059464817 >> 172.21.15.31:0/1059464817 conn(0x7fc7ec0b1520 msgr2=0x7fc7ec0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 -- 172.21.15.31:0/1059464817 shutdown_connections 2024-09-13T08:38:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.022+0000 7fc7eb200640 1 -- 172.21.15.31:0/1059464817 wait complete. 2024-09-13T08:38:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.023+0000 7fc7eb200640 1 Processor -- start 2024-09-13T08:38:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 -- start start 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7d0003a30 0x7fc7ec0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7ec15dee0 0x7fc7ec15e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec0cc560 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec0cc3e0 con 0x7fc7d0003a30 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.024+0000 7fc7eb200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ec0cc6e0 con 0x7fc7ec15dee0 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47166/0 (socket says 172.21.15.31:47166) 2024-09-13T08:38:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 -- 172.21.15.31:0/1820064764 learned_addr learned my addr 172.21.15.31:0/1820064764 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7ea800640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7ec15dee0 0x7fc7ec15e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9e00640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7d0003a30 0x7fc7ec0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 -- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7ec15dee0 msgr2=0x7fc7ec15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7ec15dee0 0x7fc7ec15e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 -- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7d0003a30 msgr2=0x7fc7ec0b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7d0003a30 0x7fc7ec0b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 -- 172.21.15.31:0/1820064764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec1f57e0 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.025+0000 7fc7e9400640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8011ae0 tx=0x7fc7d8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.026+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d801e070 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.026+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7d800fe70 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.026+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8004290 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.026+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ec1f5ac0 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.026+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ec0b0ad0 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.027+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ec06e660 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.028+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fc7d8021070 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.029+0000 7fc7e2000640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 0x7fc7cc07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.029+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fc7d809b270 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.030+0000 7fc7e9e00640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 0x7fc7cc07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.030+0000 7fc7e9e00640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 0x7fc7cc07ad10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc033000 tx=0x7fc7ec0cbcb0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:12.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.033+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc7d8063340 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.230+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc7ec15a950 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.230+0000 7fc7e2000640 1 -- 172.21.15.31:0/1820064764 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7fc7d8067d70 con 0x7fc7ec1f7b20 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 msgr2=0x7fc7cc07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 0x7fc7cc07ad10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc033000 tx=0x7fc7ec0cbcb0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 msgr2=0x7fc7ec1f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8011ae0 tx=0x7fc7d8009e90 comp rx=0 tx=0).stop 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 shutdown_connections 2024-09-13T08:38:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc7cc078850 0x7fc7cc07ad10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7ec15dee0 0x7fc7ec15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7ec1f7b20 0x7fc7ec1f7fa0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 --2- 172.21.15.31:0/1820064764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7d0003a30 0x7fc7ec0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 >> 172.21.15.31:0/1820064764 conn(0x7fc7ec0b1520 msgr2=0x7fc7ec0b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 shutdown_connections 2024-09-13T08:38:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.233+0000 7fc7eb200640 1 -- 172.21.15.31:0/1820064764 wait complete. 2024-09-13T08:38:12.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:12.251 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp87300/subvol80330) 2024-09-13T08:38:12.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:12.316 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:38:12.353 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:38:12.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:12.353 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp87300/subvol80330) 2024-09-13T08:38:12.461 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:12.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp87300/subvol80330/.meta 2024-09-13T08:38:12.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol80330 subvol_grp87300 2024-09-13T08:38:12.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 -- 172.21.15.31:0/255857649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf4003aa0 msgr2=0x7fbaf4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 --2- 172.21.15.31:0/255857649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf4003aa0 0x7fbaf4003ec0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbb00006f90 tx=0x7fbaf4005830 comp rx=0 tx=0).stop 2024-09-13T08:38:12.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 -- 172.21.15.31:0/255857649 shutdown_connections 2024-09-13T08:38:12.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 --2- 172.21.15.31:0/255857649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf4003aa0 0x7fbaf4003ec0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 -- 172.21.15.31:0/255857649 >> 172.21.15.31:0/255857649 conn(0x7fbb100cfbb0 msgr2=0x7fbb100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 -- 172.21.15.31:0/255857649 shutdown_connections 2024-09-13T08:38:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.731+0000 7fbb15c00640 1 -- 172.21.15.31:0/255857649 wait complete. 2024-09-13T08:38:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.732+0000 7fbb15c00640 1 Processor -- start 2024-09-13T08:38:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 -- start start 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbaf4003aa0 0x7fbb100c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1015a040 0x7fbb1015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb100cb8c0 con 0x7fbb101500c0 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb100cb740 con 0x7fbb1015a040 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.733+0000 7fbb15c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbb100cba40 con 0x7fbaf4003aa0 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47190/0 (socket says 172.21.15.31:47190) 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 -- 172.21.15.31:0/1356992850 learned_addr learned my addr 172.21.15.31:0/1356992850 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb0ea00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbaf4003aa0 0x7fbb100c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb0f400640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1015a040 0x7fbb1015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 -- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbaf4003aa0 msgr2=0x7fbb100c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbaf4003aa0 0x7fbb100c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 -- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1015a040 msgr2=0x7fbb1015a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1015a040 0x7fbb1015a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.734+0000 7fbb06000640 1 -- 172.21.15.31:0/1356992850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb101f9870 con 0x7fbb101500c0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb06000640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8003f50 tx=0x7fbaf80016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8011640 con 0x7fbb101500c0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbaf8016020 con 0x7fbb101500c0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8011a60 con 0x7fbb101500c0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb100c4df0 con 0x7fbb101500c0 2024-09-13T08:38:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.735+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb101f5770 con 0x7fbb101500c0 2024-09-13T08:38:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.736+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1006fa60 con 0x7fbb101500c0 2024-09-13T08:38:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.737+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7fbaf8011c00 con 0x7fbb101500c0 2024-09-13T08:38:12.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.739+0000 7fbb07400640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 0x7fbaec07ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:12.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.739+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fbaf809b560 con 0x7fbb101500c0 2024-09-13T08:38:12.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.739+0000 7fbb0ea00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 0x7fbaec07ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:12.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.740+0000 7fbb0ea00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 0x7fbaec07ad10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbb00033000 tx=0x7fbb1006eb90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:12.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.743+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbaf8020020 con 0x7fbb101500c0 2024-09-13T08:38:12.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:12.971+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol80330", "group_name": "subvol_grp87300", "target": ["mon-mgr", ""]}) -- 0x7fbb100cb310 con 0x7fbaec078850 2024-09-13T08:38:12.980 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:12.979+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:12.980 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:12.979+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:12.980 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:12.979+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:12.981 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:12.979+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:12.981 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:12.979+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.027+0000 7fbb07400640 1 -- 172.21.15.31:0/1356992850 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbb100cb310 con 0x7fbaec078850 2024-09-13T08:38:13.029 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp87300/subvol80330 2024-09-13T08:38:13.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 msgr2=0x7fbaec07ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 0x7fbaec07ad10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbb00033000 tx=0x7fbb1006eb90 comp rx=0 tx=0).stop 2024-09-13T08:38:13.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 msgr2=0x7fbb101f9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8003f50 tx=0x7fbaf80016f0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 shutdown_connections 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1015a040 0x7fbb1015a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb101500c0 0x7fbb101f9330 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbaec078850 0x7fbaec07ad10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 --2- 172.21.15.31:0/1356992850 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbaf4003aa0 0x7fbb100c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 >> 172.21.15.31:0/1356992850 conn(0x7fbb100cfbb0 msgr2=0x7fbb100b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 shutdown_connections 2024-09-13T08:38:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.030+0000 7fbb15c00640 1 -- 172.21.15.31:0/1356992850 wait complete. 2024-09-13T08:38:13.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol80330 subvol_grp87300 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 -- 172.21.15.31:0/2724344890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 msgr2=0x7f71740d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 --2- 172.21.15.31:0/2724344890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f71740d28f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f71640098e0 tx=0x7f716402a530 comp rx=0 tx=0).stop 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 -- 172.21.15.31:0/2724344890 shutdown_connections 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 --2- 172.21.15.31:0/2724344890 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 0x7f71740ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 --2- 172.21.15.31:0/2724344890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f71740d28f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 -- 172.21.15.31:0/2724344890 >> 172.21.15.31:0/2724344890 conn(0x7f71740d1dd0 msgr2=0x7f71740d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:13.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 -- 172.21.15.31:0/2724344890 shutdown_connections 2024-09-13T08:38:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.213+0000 7f7179e00640 1 -- 172.21.15.31:0/2724344890 wait complete. 2024-09-13T08:38:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.215+0000 7f7179e00640 1 Processor -- start 2024-09-13T08:38:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.215+0000 7f7179e00640 1 -- start start 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 0x7f71741f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 0x7f71740caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71740ccfb0 con 0x7f717415bec0 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71740b6770 con 0x7f71740ce980 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7179e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f71740cd130 con 0x7f71740cf050 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7173400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7172a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 0x7f71741f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7172000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 0x7f71740caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7173400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47200/0 (socket says 172.21.15.31:47200) 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7172a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 0x7f71741f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:45772/0 (socket says 172.21.15.31:45772) 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.216+0000 7f7173400640 1 -- 172.21.15.31:0/1766926216 learned_addr learned my addr 172.21.15.31:0/1766926216 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 -- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 msgr2=0x7f71740caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 0x7f71740caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 -- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 msgr2=0x7f71741f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 0x7f71741f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 -- 172.21.15.31:0/1766926216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71740cb3f0 con 0x7f717415bec0 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7173400640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c710 tx=0x7f716c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c016020 con 0x7f717415bec0 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f716c00ce20 con 0x7f717415bec0 2024-09-13T08:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c006e60 con 0x7f717415bec0 2024-09-13T08:38:13.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71740cb690 con 0x7f717415bec0 2024-09-13T08:38:13.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.217+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71740cbc50 con 0x7f717415bec0 2024-09-13T08:38:13.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.219+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71740cd630 con 0x7f717415bec0 2024-09-13T08:38:13.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.220+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 101287+0+0 (secure 0 0 0) 0x7f716c003bd0 con 0x7f717415bec0 2024-09-13T08:38:13.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.221+0000 7f716b400640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 0x7f715407ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:13.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.221+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f716c09a4d0 con 0x7f717415bec0 2024-09-13T08:38:13.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.222+0000 7f7172a00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 0x7f715407ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.222+0000 7f7172a00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 0x7f715407ad10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f7164004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:13.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.225+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f716c020020 con 0x7f717415bec0 2024-09-13T08:38:13.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.433+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 6) ==== 101338+0+0 (secure 0 0 0) 0x7f716c05e750 con 0x7f717415bec0 2024-09-13T08:38:13.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.451+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol80330", "group_name": "subvol_grp87300", "target": ["mon-mgr", ""]}) -- 0x7f71741456a0 con 0x7f7154078850 2024-09-13T08:38:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.456+0000 7f716b400640 1 -- 172.21.15.31:0/1766926216 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f71741456a0 con 0x7f7154078850 2024-09-13T08:38:13.458 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp87300/subvol80330 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.458+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 msgr2=0x7f715407ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.458+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 0x7f715407ad10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f7164004730 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.458+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 msgr2=0x7f717406dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c710 tx=0x7f716c00cbd0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 shutdown_connections 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f717415bec0 0x7f717406dc10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7154078850 0x7f715407ad10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71740cf050 0x7f71740caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 --2- 172.21.15.31:0/1766926216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71740ce980 0x7f71741f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:13.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 >> 172.21.15.31:0/1766926216 conn(0x7f71740d1dd0 msgr2=0x7f71740cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:13.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 shutdown_connections 2024-09-13T08:38:13.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:13.459+0000 7f7179e00640 1 -- 172.21.15.31:0/1766926216 wait complete. 2024-09-13T08:38:13.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:13.477 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/8e7f1f41355a3b8d190e60182c3ada98.meta) 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/subvol_grp87300/subvol80330 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:38:18.425 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:18.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol80330 subvol_grp87300 2024-09-13T08:38:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 -- 172.21.15.31:0/3183140311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 msgr2=0x7fe4a40b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 --2- 172.21.15.31:0/3183140311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a40b2820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe490009830 tx=0x7fe49002a5c0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 -- 172.21.15.31:0/3183140311 shutdown_connections 2024-09-13T08:38:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 --2- 172.21.15.31:0/3183140311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a40b2820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 -- 172.21.15.31:0/3183140311 >> 172.21.15.31:0/3183140311 conn(0x7fe4a40b1520 msgr2=0x7fe4a40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 -- 172.21.15.31:0/3183140311 shutdown_connections 2024-09-13T08:38:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.602+0000 7fe4a9600640 1 -- 172.21.15.31:0/3183140311 wait complete. 2024-09-13T08:38:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.604+0000 7fe4a9600640 1 Processor -- start 2024-09-13T08:38:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.604+0000 7fe4a9600640 1 -- start start 2024-09-13T08:38:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4a40cc390 0x7fe4a41f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40cae00 con 0x7fe4a415bae0 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40cac80 con 0x7fe4a4073ac0 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a9600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a40caf80 con 0x7fe4a40cc390 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34866/0 (socket says 172.21.15.31:34866) 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a8c00640 1 -- 172.21.15.31:0/4286197008 learned_addr learned my addr 172.21.15.31:0/4286197008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a3400640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4a40cc390 0x7fe4a41f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.605+0000 7fe4a3e00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 -- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4a40cc390 msgr2=0x7fe4a41f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4a40cc390 0x7fe4a41f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 -- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 msgr2=0x7fe4a4152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a4152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 -- 172.21.15.31:0/4286197008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a41f54e0 con 0x7fe4a415bae0 2024-09-13T08:38:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a8c00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c430 tx=0x7fe49c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c017070 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe49c0053c0 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c003850 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.606+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a415eaf0 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.607+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a415f060 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.607+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a40cb400 con 0x7fe4a415bae0 2024-09-13T08:38:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.609+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 6) ==== 101338+0+0 (secure 0 0 0) 0x7fe49c01a070 con 0x7fe4a415bae0 2024-09-13T08:38:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.609+0000 7fe49be00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 0x7fe48407ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.610+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fe49c099e60 con 0x7fe4a415bae0 2024-09-13T08:38:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.610+0000 7fe4a3e00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 0x7fe48407ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.611+0000 7fe4a3e00640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 0x7fe48407ada0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe494004a30 tx=0x7fe494001a10 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.614+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe4a41f54e0 con 0x7fe4a415bae0 2024-09-13T08:38:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.836+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol80330", "group_name": "subvol_grp87300", "target": ["mon-mgr", ""]}) -- 0x7fe4a406ff10 con 0x7fe4840788e0 2024-09-13T08:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.867+0000 7fe49be00640 1 -- 172.21.15.31:0/4286197008 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe4a406ff10 con 0x7fe4840788e0 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 msgr2=0x7fe48407ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 0x7fe48407ada0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe494004a30 tx=0x7fe494001a10 comp rx=0 tx=0).stop 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 msgr2=0x7fe4a415c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c430 tx=0x7fe49c00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 shutdown_connections 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4a415bae0 0x7fe4a415c180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe4840788e0 0x7fe48407ada0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4a40cc390 0x7fe4a41f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 --2- 172.21.15.31:0/4286197008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4a4073ac0 0x7fe4a4152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 >> 172.21.15.31:0/4286197008 conn(0x7fe4a40b1520 msgr2=0x7fe4a40af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 shutdown_connections 2024-09-13T08:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18.870+0000 7fe4a9600640 1 -- 172.21.15.31:0/4286197008 wait complete. 2024-09-13T08:38:18.873 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.872+0000 7fe26b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.874 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.872+0000 7fe26b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.874 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.872+0000 7fe26b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.874 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.872+0000 7fe26b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.874 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.872+0000 7fe26b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.887 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:38:18.893 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.892+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.893 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.892+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.893 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.892+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.893 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.892+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.893 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:18.892+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:38:18.974 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:38:18.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:18.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp87300 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.147+0000 7f6d1fe00640 1 -- 172.21.15.31:0/1685452956 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200af450 msgr2=0x7f6d200b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.147+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/1685452956 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200af450 0x7f6d200b4940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6d20068a30 tx=0x7f6d0802a520 comp rx=0 tx=0).stop 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 -- 172.21.15.31:0/1685452956 shutdown_connections 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/1685452956 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200af450 0x7f6d200b4940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/1685452956 >> v2:172.21.15.45:3300/0 conn(0x7f6d200ce180 0x7f6d200cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 -- 172.21.15.31:0/1685452956 >> 172.21.15.31:0/1685452956 conn(0x7f6d200ccd00 msgr2=0x7f6d200cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 -- 172.21.15.31:0/1685452956 shutdown_connections 2024-09-13T08:38:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.148+0000 7f6d1fe00640 1 -- 172.21.15.31:0/1685452956 wait complete. 2024-09-13T08:38:19.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.149+0000 7f6d1fe00640 1 Processor -- start 2024-09-13T08:38:19.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.149+0000 7f6d1fe00640 1 -- start start 2024-09-13T08:38:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200af450 0x7f6d201e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200ce180 0x7f6d20074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d200d0080 con 0x7f6d200b4e80 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d200cff00 con 0x7f6d200af450 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.150+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6d200cec70 con 0x7f6d200ce180 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34870/0 (socket says 172.21.15.31:34870) 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 -- 172.21.15.31:0/659058308 learned_addr learned my addr 172.21.15.31:0/659058308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d1f400640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200ce180 0x7f6d20074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d1ea00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200af450 0x7f6d201e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 -- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200ce180 msgr2=0x7f6d20074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200ce180 0x7f6d20074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 -- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200af450 msgr2=0x7f6d201e8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200af450 0x7f6d201e8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 -- 172.21.15.31:0/659058308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d200c3e50 con 0x7f6d200b4e80 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.151+0000 7f6d16000640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6d08000c00 tx=0x7f6d08004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.152+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0803f070 con 0x7f6d200b4e80 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.152+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d080038e0 con 0x7f6d200b4e80 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.152+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08035030 con 0x7f6d200b4e80 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.152+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d200c4130 con 0x7f6d200b4e80 2024-09-13T08:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.152+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d201f8c00 con 0x7f6d200b4e80 2024-09-13T08:38:19.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.153+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d200b4940 con 0x7f6d200b4e80 2024-09-13T08:38:19.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.154+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 6) ==== 101338+0+0 (secure 0 0 0) 0x7f6d08042070 con 0x7f6d200b4e80 2024-09-13T08:38:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.155+0000 7f6d17400640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 0x7f6cfc07ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.155+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f6d080bb890 con 0x7f6d200b4e80 2024-09-13T08:38:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.155+0000 7f6d1ea00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 0x7f6cfc07ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.156+0000 7f6d1ea00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 0x7f6cfc07ada0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009ac0 tx=0x7f6d10005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:19.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.159+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6d08083d90 con 0x7f6d200b4e80 2024-09-13T08:38:19.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.379+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp87300", "target": ["mon-mgr", ""]}) -- 0x7f6d200c4650 con 0x7f6cfc0788e0 2024-09-13T08:38:19.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.384+0000 7f6d17400640 1 -- 172.21.15.31:0/659058308 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6d10009710 con 0x7f6cfc0788e0 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 msgr2=0x7f6cfc07ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 0x7f6cfc07ada0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009ac0 tx=0x7f6d10005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 msgr2=0x7f6d20072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6d08000c00 tx=0x7f6d08004c40 comp rx=0 tx=0).stop 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 shutdown_connections 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6cfc0788e0 0x7f6cfc07ada0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d200ce180 0x7f6d20074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d200b4e80 0x7f6d20072ba0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 --2- 172.21.15.31:0/659058308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200af450 0x7f6d201e8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 >> 172.21.15.31:0/659058308 conn(0x7f6d200ccd00 msgr2=0x7f6d200cb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 shutdown_connections 2024-09-13T08:38:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.386+0000 7f6d1fe00640 1 -- 172.21.15.31:0/659058308 wait complete. 2024-09-13T08:38:19.404 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:38:19.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:38:19.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:38:19.590 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:38:19.591 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:38:19.591 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:19.591 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:19.626 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:19.626 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:19.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:38:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 -- 172.21.15.31:0/3345483913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad415be30 msgr2=0x7fcad40cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 --2- 172.21.15.31:0/3345483913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad415be30 0x7fcad40cdb80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009830 tx=0x7fcac402a510 comp rx=0 tx=0).stop 2024-09-13T08:38:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 -- 172.21.15.31:0/3345483913 shutdown_connections 2024-09-13T08:38:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 --2- 172.21.15.31:0/3345483913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcad4073860 0x7fcad4073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 --2- 172.21.15.31:0/3345483913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad415be30 0x7fcad40cdb80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 -- 172.21.15.31:0/3345483913 >> 172.21.15.31:0/3345483913 conn(0x7fcad40b18f0 msgr2=0x7fcad40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 -- 172.21.15.31:0/3345483913 shutdown_connections 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.903+0000 7fcad9400640 1 -- 172.21.15.31:0/3345483913 wait complete. 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.905+0000 7fcad9400640 1 Processor -- start 2024-09-13T08:38:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.905+0000 7fcad9400640 1 -- start start 2024-09-13T08:38:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad40ced20 0x7fcad4156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcad415be30 0x7fcad40b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcad40733e0 con 0x7fcad4073860 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcad40ce9e0 con 0x7fcad40ced20 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad9400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcad4074230 con 0x7fcad415be30 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34890/0 (socket says 172.21.15.31:34890) 2024-09-13T08:38:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 -- 172.21.15.31:0/1913365749 learned_addr learned my addr 172.21.15.31:0/1913365749 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 -- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcad415be30 msgr2=0x7fcad40b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcad415be30 0x7fcad40b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 -- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad40ced20 msgr2=0x7fcad4156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad40ced20 0x7fcad4156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.906+0000 7fcad2800640 1 -- 172.21.15.31:0/1913365749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad41f8e00 con 0x7fcad4073860 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcad2800640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcac4000c00 tx=0x7fcac4033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4031d60 con 0x7fcad4073860 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcac4033070 con 0x7fcad4073860 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac40334c0 con 0x7fcad4073860 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad4154e00 con 0x7fcad4073860 2024-09-13T08:38:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.907+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad4155c70 con 0x7fcad4073860 2024-09-13T08:38:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.908+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad4074720 con 0x7fcad4073860 2024-09-13T08:38:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.909+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fcac4033660 con 0x7fcad4073860 2024-09-13T08:38:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.910+0000 7fcacb400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 0x7fcab007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.910+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fcac40bc1d0 con 0x7fcad4073860 2024-09-13T08:38:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.911+0000 7fcac9e00640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 0x7fcab007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.912+0000 7fcac9e00640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 0x7fcab007aec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcad40cbd50 tx=0x7fcabc0078d0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:19.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:19.915+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcac403f030 con 0x7fcad4073860 2024-09-13T08:38:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.120+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcad40b25f0 con 0x7fcad4073860 2024-09-13T08:38:20.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.121+0000 7fcacb400640 1 -- 172.21.15.31:0/1913365749 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7fcac40844a0 con 0x7fcad4073860 2024-09-13T08:38:20.122 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:38:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 msgr2=0x7fcab007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 0x7fcab007aec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcad40cbd50 tx=0x7fcabc0078d0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 msgr2=0x7fcad41548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcac4000c00 tx=0x7fcac4033d60 comp rx=0 tx=0).stop 2024-09-13T08:38:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 shutdown_connections 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcab0078a00 0x7fcab007aec0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcad415be30 0x7fcad40b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad40ced20 0x7fcad4156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 --2- 172.21.15.31:0/1913365749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad4073860 0x7fcad41548c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 >> 172.21.15.31:0/1913365749 conn(0x7fcad40b18f0 msgr2=0x7fcad40af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 shutdown_connections 2024-09-13T08:38:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.123+0000 7fcad9400640 1 -- 172.21.15.31:0/1913365749 wait complete. 2024-09-13T08:38:20.140 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:38:20.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:20.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:20.237 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:38:20.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:20.237 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:38:20.287 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:38:20.287 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:20.287 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:20.325 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:20.325 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:20.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 -- 172.21.15.31:0/1432325488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 msgr2=0x7efc440d0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 --2- 172.21.15.31:0/1432325488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc440d0ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efc440c79d0 tx=0x7efc2c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 -- 172.21.15.31:0/1432325488 shutdown_connections 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 --2- 172.21.15.31:0/1432325488 >> v2:172.21.15.45:3300/0 conn(0x7efc440cea60 0x7efc440ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 --2- 172.21.15.31:0/1432325488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc440d0ea0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 -- 172.21.15.31:0/1432325488 >> 172.21.15.31:0/1432325488 conn(0x7efc440d3f50 msgr2=0x7efc440d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 -- 172.21.15.31:0/1432325488 shutdown_connections 2024-09-13T08:38:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.563+0000 7efc43e00640 1 -- 172.21.15.31:0/1432325488 wait complete. 2024-09-13T08:38:20.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.564+0000 7efc43e00640 1 Processor -- start 2024-09-13T08:38:20.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.564+0000 7efc43e00640 1 -- start start 2024-09-13T08:38:20.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc440cea60 0x7efc440b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4415e160 0x7efc440b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc440cf430 con 0x7efc440c9350 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efc440d15f0 con 0x7efc4415e160 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.565+0000 7efc43e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efc440cf5b0 con 0x7efc440cea60 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34900/0 (socket says 172.21.15.31:34900) 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 -- 172.21.15.31:0/724104760 learned_addr learned my addr 172.21.15.31:0/724104760 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc43400640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4415e160 0x7efc440b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc3a000640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc440cea60 0x7efc440b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 -- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc440cea60 msgr2=0x7efc440b8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc440cea60 0x7efc440b8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 -- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4415e160 msgr2=0x7efc440b92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4415e160 0x7efc440b92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 -- 172.21.15.31:0/724104760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441f8b90 con 0x7efc440c9350 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.566+0000 7efc42a00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c3c0 tx=0x7efc3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.567+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34017070 con 0x7efc440c9350 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.567+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc340035f0 con 0x7efc440c9350 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.567+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34003a80 con 0x7efc440c9350 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.567+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4415f9b0 con 0x7efc440c9350 2024-09-13T08:38:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.567+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4415ff70 con 0x7efc440c9350 2024-09-13T08:38:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.568+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc440ceef0 con 0x7efc440c9350 2024-09-13T08:38:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.569+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7efc3401a070 con 0x7efc440c9350 2024-09-13T08:38:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.570+0000 7efc3b400640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 0x7efc2007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.570+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7efc34099e70 con 0x7efc440c9350 2024-09-13T08:38:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.570+0000 7efc3a000640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 0x7efc2007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.571+0000 7efc3a000640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 0x7efc2007aec0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc2c033000 tx=0x7efc2c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:20.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.574+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efc4415e540 con 0x7efc440c9350 2024-09-13T08:38:20.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.780+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efc4406d6e0 con 0x7efc440c9350 2024-09-13T08:38:20.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.780+0000 7efc3b400640 1 -- 172.21.15.31:0/724104760 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7efc34061ba0 con 0x7efc440c9350 2024-09-13T08:38:20.781 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:38:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 msgr2=0x7efc2007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 0x7efc2007aec0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc2c033000 tx=0x7efc2c004730 comp rx=0 tx=0).stop 2024-09-13T08:38:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 msgr2=0x7efc44152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c3c0 tx=0x7efc3400c880 comp rx=0 tx=0).stop 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 shutdown_connections 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4415e160 0x7efc440b92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc20078a00 0x7efc2007aec0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc440cea60 0x7efc440b8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 --2- 172.21.15.31:0/724104760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc440c9350 0x7efc44152ab0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 >> 172.21.15.31:0/724104760 conn(0x7efc440d3f50 msgr2=0x7efc440d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 shutdown_connections 2024-09-13T08:38:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:20.783+0000 7efc43e00640 1 -- 172.21.15.31:0/724104760 wait complete. 2024-09-13T08:38:20.799 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:38:20.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:20.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:20.871 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:38:20.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:20.871 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:38:20.908 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:38:20.909 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:20.909 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:20.944 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:38:20.944 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:38:20.944 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:20.944 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:20.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:38:20.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 -- 172.21.15.31:0/978819423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800cc720 msgr2=0x7fa48006f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 --2- 172.21.15.31:0/978819423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800cc720 0x7fa48006f9b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa470005fd0 tx=0x7fa47002aa90 comp rx=0 tx=0).stop 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 -- 172.21.15.31:0/978819423 shutdown_connections 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 --2- 172.21.15.31:0/978819423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ccb00 0x7fa4800b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 --2- 172.21.15.31:0/978819423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800cc720 0x7fa48006f9b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 -- 172.21.15.31:0/978819423 >> 172.21.15.31:0/978819423 conn(0x7fa48006cb00 msgr2=0x7fa48006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 -- 172.21.15.31:0/978819423 shutdown_connections 2024-09-13T08:38:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.147+0000 7fa485c00640 1 -- 172.21.15.31:0/978819423 wait complete. 2024-09-13T08:38:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.148+0000 7fa485c00640 1 Processor -- start 2024-09-13T08:38:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.148+0000 7fa485c00640 1 -- start start 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800701a0 0x7fa48014d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800cc720 0x7fa48014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4800b5490 con 0x7fa4800ccb00 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4800b5310 con 0x7fa4800701a0 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.149+0000 7fa485c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa480152d90 con 0x7fa4800cc720 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34912/0 (socket says 172.21.15.31:34912) 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 -- 172.21.15.31:0/4194120583 learned_addr learned my addr 172.21.15.31:0/4194120583 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa477400640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800cc720 0x7fa48014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa47fe00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800701a0 0x7fa48014d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 -- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800cc720 msgr2=0x7fa48014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800cc720 0x7fa48014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 -- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800701a0 msgr2=0x7fa48014d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800701a0 0x7fa48014d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 -- 172.21.15.31:0/4194120583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48014d6e0 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.150+0000 7fa485200640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa47800c3c0 tx=0x7fa47800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.151+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478017070 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.151+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4780053c0 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.151+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478003850 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.151+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4801f2470 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.151+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4801f2a30 con 0x7fa4800ccb00 2024-09-13T08:38:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.152+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa48006d130 con 0x7fa4800ccb00 2024-09-13T08:38:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.153+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fa47801a070 con 0x7fa4800ccb00 2024-09-13T08:38:21.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.154+0000 7fa47cc00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 0x7fa45c07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.154+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fa478099ff0 con 0x7fa4800ccb00 2024-09-13T08:38:21.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.154+0000 7fa47fe00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 0x7fa45c07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.155+0000 7fa47fe00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 0x7fa45c07aef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa470035000 tx=0x7fa470039040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.158+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa48014d6e0 con 0x7fa4800ccb00 2024-09-13T08:38:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.398+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa480069770 con 0x7fa4800ccb00 2024-09-13T08:38:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.399+0000 7fa47cc00640 1 -- 172.21.15.31:0/4194120583 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5214 (secure 0 0 0) 0x7fa478061cb0 con 0x7fa4800ccb00 2024-09-13T08:38:21.400 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:21.400 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"btime":"2024-09-13T08:38:11:416801+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2024-09-13T08:38:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 msgr2=0x7fa45c07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 0x7fa45c07aef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa470035000 tx=0x7fa470039040 comp rx=0 tx=0).stop 2024-09-13T08:38:21.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 msgr2=0x7fa48014f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa47800c3c0 tx=0x7fa47800c880 comp rx=0 tx=0).stop 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 shutdown_connections 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800ccb00 0x7fa48014f950 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa45c078a30 0x7fa45c07aef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800cc720 0x7fa48014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 --2- 172.21.15.31:0/4194120583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800701a0 0x7fa48014d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 >> 172.21.15.31:0/4194120583 conn(0x7fa48006cb00 msgr2=0x7fa4800b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 shutdown_connections 2024-09-13T08:38:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.401+0000 7fa485c00640 1 -- 172.21.15.31:0/4194120583 wait complete. 2024-09-13T08:38:21.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:21.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.578+0000 7f3e52600640 1 -- 172.21.15.31:0/2106247675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 msgr2=0x7f3e4c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.578+0000 7f3e52600640 1 --2- 172.21.15.31:0/2106247675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c0cf850 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c005fd0 tx=0x7f3e3c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.579+0000 7f3e52600640 1 -- 172.21.15.31:0/2106247675 shutdown_connections 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.579+0000 7f3e52600640 1 --2- 172.21.15.31:0/2106247675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c0cf850 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.579+0000 7f3e52600640 1 -- 172.21.15.31:0/2106247675 >> 172.21.15.31:0/2106247675 conn(0x7f3e4c0cc910 msgr2=0x7f3e4c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.579+0000 7f3e52600640 1 -- 172.21.15.31:0/2106247675 shutdown_connections 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.579+0000 7f3e52600640 1 -- 172.21.15.31:0/2106247675 wait complete. 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.580+0000 7f3e52600640 1 Processor -- start 2024-09-13T08:38:21.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.580+0000 7f3e52600640 1 -- start start 2024-09-13T08:38:21.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 0x7f3e4c14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e4c15fe20 0x7f3e4c1f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c0cec50 con 0x7f3e4c15bd70 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c0cead0 con 0x7f3e4c15f9e0 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e52600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4c0cb850 con 0x7f3e4c15fe20 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.581+0000 7f3e4b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34936/0 (socket says 172.21.15.31:34936) 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 -- 172.21.15.31:0/3557979071 learned_addr learned my addr 172.21.15.31:0/3557979071 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 -- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e4c15fe20 msgr2=0x7f3e4c1f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e4c15fe20 0x7f3e4c1f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 -- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 msgr2=0x7f3e4c14e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4aa00640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 0x7f3e4c14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 0x7f3e4c14e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 -- 172.21.15.31:0/3557979071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c0b54e0 con 0x7f3e4c15bd70 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4aa00640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 0x7f3e4c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e4b400640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009850 tx=0x7f3e3c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.582+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c03f070 con 0x7f3e4c15bd70 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.583+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e3c0047a0 con 0x7f3e4c15bd70 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.583+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c035030 con 0x7f3e4c15bd70 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.583+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c14ea60 con 0x7f3e4c15bd70 2024-09-13T08:38:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.583+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c0b57a0 con 0x7f3e4c15bd70 2024-09-13T08:38:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.584+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4c0cf850 con 0x7f3e4c15bd70 2024-09-13T08:38:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.585+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f3e3c042070 con 0x7f3e4c15bd70 2024-09-13T08:38:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.586+0000 7f3e43400640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 0x7f3e2c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.586+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f3e3c0bba40 con 0x7f3e4c15bd70 2024-09-13T08:38:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.586+0000 7f3e4aa00640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 0x7f3e2c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.587+0000 7f3e4aa00640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 0x7f3e2c07aec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0c85a0 tx=0x7f3e38009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.590+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3e3c083d10 con 0x7f3e4c15bd70 2024-09-13T08:38:21.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.822+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e4c0b5df0 con 0x7f3e4c15bd70 2024-09-13T08:38:21.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.823+0000 7f3e43400640 1 -- 172.21.15.31:0/3557979071 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5214 (secure 0 0 0) 0x7f3e3c083610 con 0x7f3e4c15bd70 2024-09-13T08:38:21.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:21.824 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"btime":"2024-09-13T08:38:11:416801+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:21.824 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 msgr2=0x7f3e2c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 0x7f3e2c07aec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0c85a0 tx=0x7f3e38009040 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 msgr2=0x7f3e4c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009850 tx=0x7f3e3c0038a0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 shutdown_connections 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3e2c078a00 0x7f3e2c07aec0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e4c15fe20 0x7f3e4c1f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e4c15f9e0 0x7f3e4c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 --2- 172.21.15.31:0/3557979071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e4c15bd70 0x7f3e4c14dfe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 >> 172.21.15.31:0/3557979071 conn(0x7f3e4c0cc910 msgr2=0x7f3e4c15abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 shutdown_connections 2024-09-13T08:38:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:21.825+0000 7f3e52600640 1 -- 172.21.15.31:0/3557979071 wait complete. 2024-09-13T08:38:21.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 -- 172.21.15.31:0/350606450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0cf850 msgr2=0x7f8fec0ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 --2- 172.21.15.31:0/350606450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0cf850 0x7f8fec0ce3c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0ca470 tx=0x7f8fd402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 -- 172.21.15.31:0/350606450 shutdown_connections 2024-09-13T08:38:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 --2- 172.21.15.31:0/350606450 >> v2:172.21.15.45:3300/0 conn(0x7f8fec0ce900 0x7f8fec0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 --2- 172.21.15.31:0/350606450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0cf850 0x7f8fec0ce3c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 -- 172.21.15.31:0/350606450 >> 172.21.15.31:0/350606450 conn(0x7f8fec0cc910 msgr2=0x7f8fec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 -- 172.21.15.31:0/350606450 shutdown_connections 2024-09-13T08:38:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.003+0000 7f8ff3600640 1 -- 172.21.15.31:0/350606450 wait complete. 2024-09-13T08:38:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.005+0000 7f8ff3600640 1 Processor -- start 2024-09-13T08:38:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.005+0000 7f8ff3600640 1 -- start start 2024-09-13T08:38:22.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 0x7f8fec06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 0x7f8fec06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec1600f0 con 0x7f8fec0ce900 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec0cb850 con 0x7f8fec15bd70 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff3600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec06f440 con 0x7f8fec0cf850 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34948/0 (socket says 172.21.15.31:34948) 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff0c00640 1 -- 172.21.15.31:0/986064103 learned_addr learned my addr 172.21.15.31:0/986064103 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8ff1600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 0x7f8fec06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.006+0000 7f8fe3e00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 0x7f8fec06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 -- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 msgr2=0x7f8fec06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 0x7f8fec06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 -- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 msgr2=0x7f8fec06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 0x7f8fec06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 -- 172.21.15.31:0/986064103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fec0c5b20 con 0x7f8fec0ce900 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff1600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 0x7f8fec06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8fe3e00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 0x7f8fec06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff0c00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c3f0 tx=0x7f8fdc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc017070 con 0x7f8fec0ce900 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fdc0035f0 con 0x7f8fec0ce900 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fec0c68a0 con 0x7f8fec0ce900 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc003a40 con 0x7f8fec0ce900 2024-09-13T08:38:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.007+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fec06f6a0 con 0x7f8fec0ce900 2024-09-13T08:38:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.008+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fec0acef0 con 0x7f8fec0ce900 2024-09-13T08:38:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.010+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f8fdc01a070 con 0x7f8fec0ce900 2024-09-13T08:38:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.011+0000 7f8fe9600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 0x7f8fc807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.011+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f8fdc099fd0 con 0x7f8fec0ce900 2024-09-13T08:38:22.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.011+0000 7f8fe3e00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 0x7f8fc807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.012+0000 7f8fe3e00640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 0x7f8fc807aec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4033000 tx=0x7f8fd4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.015+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8fec0acef0 con 0x7f8fec0ce900 2024-09-13T08:38:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.210+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8fec0ce3c0 con 0x7f8fec0ce900 2024-09-13T08:38:22.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.211+0000 7f8fe9600640 1 -- 172.21.15.31:0/986064103 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+16621 (secure 0 0 0) 0x7f8fdc062340 con 0x7f8fec0ce900 2024-09-13T08:38:22.212 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:22.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:02.230970+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:58.235357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:59.617519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.213+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 msgr2=0x7f8fc807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.213+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 0x7f8fc807aec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4033000 tx=0x7f8fd4004730 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 msgr2=0x7f8fec14f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c3f0 tx=0x7f8fdc00c8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 shutdown_connections 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fec15bd70 0x7f8fec06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8fc8078a00 0x7f8fc807aec0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8fec0cf850 0x7f8fec06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 --2- 172.21.15.31:0/986064103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fec0ce900 0x7f8fec14f0f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 >> 172.21.15.31:0/986064103 conn(0x7f8fec0cc910 msgr2=0x7f8fec0cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:22.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 shutdown_connections 2024-09-13T08:38:22.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.214+0000 7f8ff3600640 1 -- 172.21.15.31:0/986064103 wait complete. 2024-09-13T08:38:22.230 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:38:22.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.390+0000 7fb763200640 1 -- 172.21.15.31:0/1821552777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 msgr2=0x7fb76406f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.390+0000 7fb763200640 1 --2- 172.21.15.31:0/1821552777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb76406f9b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb754005fd0 tx=0x7fb75402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 -- 172.21.15.31:0/1821552777 shutdown_connections 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 --2- 172.21.15.31:0/1821552777 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640ccb00 0x7fb7640b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 --2- 172.21.15.31:0/1821552777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb76406f9b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 -- 172.21.15.31:0/1821552777 >> 172.21.15.31:0/1821552777 conn(0x7fb76406cb00 msgr2=0x7fb76406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 -- 172.21.15.31:0/1821552777 shutdown_connections 2024-09-13T08:38:22.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.391+0000 7fb763200640 1 -- 172.21.15.31:0/1821552777 wait complete. 2024-09-13T08:38:22.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.392+0000 7fb763200640 1 Processor -- start 2024-09-13T08:38:22.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.392+0000 7fb763200640 1 -- start start 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 0x7fb764150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7640ccb00 0x7fb76415dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7640b5490 con 0x7fb7640cc720 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7640b5310 con 0x7fb7640ccb00 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.393+0000 7fb763200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb764152d90 con 0x7fb7640701a0 2024-09-13T08:38:22.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34972/0 (socket says 172.21.15.31:34972) 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 -- 172.21.15.31:0/868204650 learned_addr learned my addr 172.21.15.31:0/868204650 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb762800640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7640ccb00 0x7fb76415dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb761e00640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 0x7fb764150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 -- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 msgr2=0x7fb764150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 0x7fb764150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 -- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7640ccb00 msgr2=0x7fb76415dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7640ccb00 0x7fb76415dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 -- 172.21.15.31:0/868204650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7641501f0 con 0x7fb7640cc720 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb761e00640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 0x7fb764150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.394+0000 7fb759400640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009fd0 tx=0x7fb74c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.395+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00ccc0 con 0x7fb7640cc720 2024-09-13T08:38:22.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.395+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb74c00ee70 con 0x7fb7640cc720 2024-09-13T08:38:22.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.395+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7641504d0 con 0x7fb7640cc720 2024-09-13T08:38:22.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.395+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00ce60 con 0x7fb7640cc720 2024-09-13T08:38:22.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.395+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7641f6c80 con 0x7fb7640cc720 2024-09-13T08:38:22.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.396+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb764153260 con 0x7fb7640cc720 2024-09-13T08:38:22.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.397+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fb74c004280 con 0x7fb7640cc720 2024-09-13T08:38:22.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.399+0000 7fb75aa00640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 0x7fb74007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.399+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fb74c09b710 con 0x7fb7640cc720 2024-09-13T08:38:22.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.399+0000 7fb761e00640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 0x7fb74007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.400+0000 7fb761e00640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 0x7fb74007aec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb754033000 tx=0x7fb754004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.403+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb74c063950 con 0x7fb7640cc720 2024-09-13T08:38:22.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.635+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7640cd210 con 0x7fb7640cc720 2024-09-13T08:38:22.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.635+0000 7fb75aa00640 1 -- 172.21.15.31:0/868204650 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5214 (secure 0 0 0) 0x7fb74c063250 con 0x7fb7640cc720 2024-09-13T08:38:22.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:22.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"btime":"2024-09-13T08:38:11:416801+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:22.637 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.637+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 msgr2=0x7fb74007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.637+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 0x7fb74007aec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb754033000 tx=0x7fb754004730 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.637+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 msgr2=0x7fb764158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.637+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009fd0 tx=0x7fb74c00ec20 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 shutdown_connections 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7640ccb00 0x7fb76415dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7640cc720 0x7fb764158200 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb740078a00 0x7fb74007aec0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 --2- 172.21.15.31:0/868204650 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7640701a0 0x7fb764150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 >> 172.21.15.31:0/868204650 conn(0x7fb76406cb00 msgr2=0x7fb76406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:22.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 shutdown_connections 2024-09-13T08:38:22.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.638+0000 7fb763200640 1 -- 172.21.15.31:0/868204650 wait complete. 2024-09-13T08:38:22.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 -- 172.21.15.31:0/263768948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7100b47f0 msgr2=0x7fc7100d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 --2- 172.21.15.31:0/263768948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7100b47f0 0x7fc7100d3bd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc7100d4c90 tx=0x7fc6fc02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 -- 172.21.15.31:0/263768948 shutdown_connections 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 --2- 172.21.15.31:0/263768948 >> v2:172.21.15.45:3300/0 conn(0x7fc7100d4110 0x7fc7100b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 --2- 172.21.15.31:0/263768948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7100b47f0 0x7fc7100d3bd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 -- 172.21.15.31:0/263768948 >> 172.21.15.31:0/263768948 conn(0x7fc71008d8b0 msgr2=0x7fc71009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 -- 172.21.15.31:0/263768948 shutdown_connections 2024-09-13T08:38:22.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.814+0000 7fc715400640 1 -- 172.21.15.31:0/263768948 wait complete. 2024-09-13T08:38:22.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.815+0000 7fc715400640 1 Processor -- start 2024-09-13T08:38:22.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.816+0000 7fc715400640 1 -- start start 2024-09-13T08:38:22.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 0x7fc7101e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 0x7fc7100d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100b1850 con 0x7fc71015dfa0 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100b16d0 con 0x7fc7100b47f0 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc715400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc7100b19d0 con 0x7fc7100d4110 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34986/0 (socket says 172.21.15.31:34986) 2024-09-13T08:38:22.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 -- 172.21.15.31:0/2492280102 learned_addr learned my addr 172.21.15.31:0/2492280102 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc70f400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 0x7fc7100d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 -- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 msgr2=0x7fc7100d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc70fe00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 0x7fc7101e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 0x7fc7100d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 -- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 msgr2=0x7fc7101e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 0x7fc7101e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc714a00640 1 -- 172.21.15.31:0/2492280102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7101f8ff0 con 0x7fc71015dfa0 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc70f400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 0x7fc7100d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.817+0000 7fc70fe00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 0x7fc7101e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc714a00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc70800eb10 tx=0x7fc70800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708004b20 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc708004ce0 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708005280 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7101f4df0 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.818+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7101f5690 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.819+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7100b1f20 con 0x7fc71015dfa0 2024-09-13T08:38:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.820+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fc708005420 con 0x7fc71015dfa0 2024-09-13T08:38:22.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.822+0000 7fc707e00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 0x7fc6f007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:22.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.822+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7fc70809ba20 con 0x7fc71015dfa0 2024-09-13T08:38:22.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.822+0000 7fc70fe00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 0x7fc6f007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:22.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.823+0000 7fc70fe00640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 0x7fc6f007aec0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc7000016a0 tx=0x7fc700009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:22.826+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc708063cf0 con 0x7fc71015dfa0 2024-09-13T08:38:23.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.057+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7101f4fc0 con 0x7fc71015dfa0 2024-09-13T08:38:23.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.057+0000 7fc707e00640 1 -- 172.21.15.31:0/2492280102 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5214 (secure 0 0 0) 0x7fc7080635f0 con 0x7fc71015dfa0 2024-09-13T08:38:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:23.059 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"btime":"2024-09-13T08:38:11:416801+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:00.656360+0000","modified":"2024-09-13T08:38:03.686476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4878":{"gid":4878,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/47525403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":47525403},{"type":"v1","addr":"172.21.15.31:6834","nonce":47525403}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4878,"qdb_cluster":[4878]},"id":2}]} 2024-09-13T08:38:23.059 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 msgr2=0x7fc6f007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 0x7fc6f007aec0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc7000016a0 tx=0x7fc700009040 comp rx=0 tx=0).stop 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 msgr2=0x7fc7100d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc70800eb10 tx=0x7fc70800c2d0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 shutdown_connections 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc71015dfa0 0x7fc7100d07c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc6f0078a00 0x7fc6f007aec0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7100d4110 0x7fc7100d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 --2- 172.21.15.31:0/2492280102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7100b47f0 0x7fc7101e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 >> 172.21.15.31:0/2492280102 conn(0x7fc71008d8b0 msgr2=0x7fc7100b6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:23.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 shutdown_connections 2024-09-13T08:38:23.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.060+0000 7fc715400640 1 -- 172.21.15.31:0/2492280102 wait complete. 2024-09-13T08:38:23.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 -- 172.21.15.31:0/1799985575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180ca5c0 msgr2=0x7f67180cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 --2- 172.21.15.31:0/1799985575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180ca5c0 0x7f67180cc300 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6700009830 tx=0x7f670002a510 comp rx=0 tx=0).stop 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 -- 172.21.15.31:0/1799985575 shutdown_connections 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 --2- 172.21.15.31:0/1799985575 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67180b1910 0x7f67180b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 --2- 172.21.15.31:0/1799985575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180ca5c0 0x7f67180cc300 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.237+0000 7f671e800640 1 -- 172.21.15.31:0/1799985575 >> 172.21.15.31:0/1799985575 conn(0x7f67180cc910 msgr2=0x7f67180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.238+0000 7f671e800640 1 -- 172.21.15.31:0/1799985575 shutdown_connections 2024-09-13T08:38:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.238+0000 7f671e800640 1 -- 172.21.15.31:0/1799985575 wait complete. 2024-09-13T08:38:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.239+0000 7f671e800640 1 Processor -- start 2024-09-13T08:38:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.239+0000 7f671e800640 1 -- start start 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67180ca5c0 0x7f67181f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f671815dfc0 0x7f671815d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6718073b10 con 0x7f67180b1910 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6718073990 con 0x7f671815dfc0 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.240+0000 7f671e800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6718073c90 con 0x7f67180ca5c0 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35002/0 (socket says 172.21.15.31:35002) 2024-09-13T08:38:23.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 -- 172.21.15.31:0/3792053768 learned_addr learned my addr 172.21.15.31:0/3792053768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 -- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67180ca5c0 msgr2=0x7f67181f5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67180ca5c0 0x7f67181f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 -- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f671815dfc0 msgr2=0x7f671815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f671815dfc0 0x7f671815d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 -- 172.21.15.31:0/3792053768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67181f5930 con 0x7f67180b1910 2024-09-13T08:38:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.241+0000 7f6717400640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f670800c3f0 tx=0x7f670800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.242+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708017070 con 0x7f67180b1910 2024-09-13T08:38:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.242+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67080035f0 con 0x7f67180b1910 2024-09-13T08:38:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.242+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67181f5c70 con 0x7f67180b1910 2024-09-13T08:38:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.242+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708003a40 con 0x7f67180b1910 2024-09-13T08:38:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.242+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67181f6520 con 0x7f67180b1910 2024-09-13T08:38:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.243+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67180ceef0 con 0x7f67180b1910 2024-09-13T08:38:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.244+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f670801a070 con 0x7f67180b1910 2024-09-13T08:38:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.245+0000 7f670fe00640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 0x7f66f407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.245+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f670809a0c0 con 0x7f67180b1910 2024-09-13T08:38:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.245+0000 7f670ea00640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 0x7f66f407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.246+0000 7f670ea00640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 0x7f66f407aec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6700034000 tx=0x7f67000046c0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.249+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f67180ceef0 con 0x7f67180b1910 2024-09-13T08:38:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.448+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67181f66f0 con 0x7f67180b1910 2024-09-13T08:38:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.449+0000 7f670fe00640 1 -- 172.21.15.31:0/3792053768 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+16621 (secure 0 0 0) 0x7f67181f5c70 con 0x7f67180b1910 2024-09-13T08:38:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:23.450 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:02.230970+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:37:58.235357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T08:37:59.617519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.451+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 msgr2=0x7f66f407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.451+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 0x7f66f407aec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6700034000 tx=0x7f67000046c0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.451+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 msgr2=0x7f67181f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.451+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f670800c3f0 tx=0x7f670800c8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 shutdown_connections 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f671815dfc0 0x7f671815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f66f4078a00 0x7f66f407aec0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67180ca5c0 0x7f67181f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 --2- 172.21.15.31:0/3792053768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67180b1910 0x7f67181f9170 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 >> 172.21.15.31:0/3792053768 conn(0x7f67180cc910 msgr2=0x7f67180b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 shutdown_connections 2024-09-13T08:38:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.452+0000 7f671e800640 1 -- 172.21.15.31:0/3792053768 wait complete. 2024-09-13T08:38:23.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:38:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 -- 172.21.15.31:0/770200896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 msgr2=0x7f77940cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 --2- 172.21.15.31:0/770200896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cdb80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f77940b5020 tx=0x7f778402a600 comp rx=0 tx=0).stop 2024-09-13T08:38:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 -- 172.21.15.31:0/770200896 shutdown_connections 2024-09-13T08:38:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 --2- 172.21.15.31:0/770200896 >> v2:172.21.15.45:3300/0 conn(0x7f7794073860 0x7f7794073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 --2- 172.21.15.31:0/770200896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cdb80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 -- 172.21.15.31:0/770200896 >> 172.21.15.31:0/770200896 conn(0x7f77940b18f0 msgr2=0x7f77940b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 -- 172.21.15.31:0/770200896 shutdown_connections 2024-09-13T08:38:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.632+0000 7f7799800640 1 -- 172.21.15.31:0/770200896 wait complete. 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.634+0000 7f7799800640 1 Processor -- start 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.634+0000 7f7799800640 1 -- start start 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77940ced20 0x7f77940cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77940733e0 con 0x7f779415be30 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f77940ce9e0 con 0x7f77940ced20 2024-09-13T08:38:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7799800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f77940741f0 con 0x7f7794073860 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7792a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7793400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7792a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:44926/0 (socket says 172.21.15.31:44926) 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.635+0000 7f7792a00640 1 -- 172.21.15.31:0/760454227 learned_addr learned my addr 172.21.15.31:0/760454227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f778a000640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77940ced20 0x7f77940cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 -- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77940ced20 msgr2=0x7f77940cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77940ced20 0x7f77940cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 -- 172.21.15.31:0/760454227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 msgr2=0x7f77940cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 -- 172.21.15.31:0/760454227 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779409c4a0 con 0x7f7794073860 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7793400640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.636+0000 7f7792a00640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7784034040 tx=0x7f77840039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.637+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778403e030 con 0x7f7794073860 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.637+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77840047a0 con 0x7f7794073860 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.637+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7794159210 con 0x7f7794073860 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.637+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784035040 con 0x7f7794073860 2024-09-13T08:38:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.637+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77941597d0 con 0x7f7794073860 2024-09-13T08:38:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.638+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7794073cb0 con 0x7f7794073860 2024-09-13T08:38:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.639+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f7784041030 con 0x7f7794073860 2024-09-13T08:38:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.640+0000 7f778b400640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 0x7f777007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.640+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 5709+0+0 (secure 0 0 0) 0x7f77840bb3b0 con 0x7f7794073860 2024-09-13T08:38:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.640+0000 7f778a000640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 0x7f777007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:23.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.641+0000 7f778a000640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 0x7f777007aec0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7794158f90 tx=0x7f777c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.646+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f778402ae20 con 0x7f7794073860 2024-09-13T08:38:23.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.880+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f77940b25f0 con 0x7f7794073860 2024-09-13T08:38:23.882 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:38:23.881+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:38:23.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.888+0000 7f778b400640 1 -- 172.21.15.31:0/760454227 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7f77840830e0 con 0x7f7794073860 2024-09-13T08:38:23.889 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 msgr2=0x7f777007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 0x7f777007aec0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7794158f90 tx=0x7f777c009040 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 msgr2=0x7f7794156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7784034040 tx=0x7f77840039e0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 shutdown_connections 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f779415be30 0x7f77940cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77940ced20 0x7f77940cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7770078a00 0x7f777007aec0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 --2- 172.21.15.31:0/760454227 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7794073860 0x7f7794156850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 >> 172.21.15.31:0/760454227 conn(0x7f77940b18f0 msgr2=0x7f77940aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:23.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 shutdown_connections 2024-09-13T08:38:23.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:23.892+0000 7f7799800640 1 -- 172.21.15.31:0/760454227 wait complete. 2024-09-13T08:38:23.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:38:23.960 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:38:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 -- 172.21.15.31:0/3164203366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c815df00 msgr2=0x7fd4c80b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3164203366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c815df00 0x7fd4c80b0970 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009830 tx=0x7fd4bc02a530 comp rx=0 tx=0).stop 2024-09-13T08:38:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 -- 172.21.15.31:0/3164203366 shutdown_connections 2024-09-13T08:38:24.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3164203366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c815df00 0x7fd4c80b0970 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 -- 172.21.15.31:0/3164203366 >> 172.21.15.31:0/3164203366 conn(0x7fd4c80d3e10 msgr2=0x7fd4c80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:24.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 -- 172.21.15.31:0/3164203366 shutdown_connections 2024-09-13T08:38:24.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.078+0000 7fd4cec00640 1 -- 172.21.15.31:0/3164203366 wait complete. 2024-09-13T08:38:24.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.080+0000 7fd4cec00640 1 Processor -- start 2024-09-13T08:38:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.080+0000 7fd4cec00640 1 -- start start 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 0x7fd4c80ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 0x7fd4c80cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c80d4cc0 con 0x7fd4c815df00 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c80d4b40 con 0x7fd4c80d0d20 2024-09-13T08:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cec00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c80d2740 con 0x7fd4c80b2a20 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35036/0 (socket says 172.21.15.31:35036) 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 -- 172.21.15.31:0/3414977206 learned_addr learned my addr 172.21.15.31:0/3414977206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cd800640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 0x7fd4c80ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cce00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 0x7fd4c80cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 -- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 msgr2=0x7fd4c80ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 0x7fd4c80ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 -- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 msgr2=0x7fd4c80cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 0x7fd4c80cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4ce200640 1 -- 172.21.15.31:0/3414977206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c809dd40 con 0x7fd4c815df00 2024-09-13T08:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cd800640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 0x7fd4c80ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.081+0000 7fd4cce00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 0x7fd4c80cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4ce200640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c7a0 tx=0x7fd4c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0003b70 con 0x7fd4c815df00 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4c0003d10 con 0x7fd4c815df00 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c000aa50 con 0x7fd4c815df00 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c809e020 con 0x7fd4c815df00 2024-09-13T08:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.082+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c809e8c0 con 0x7fd4c815df00 2024-09-13T08:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.083+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c80d2c60 con 0x7fd4c815df00 2024-09-13T08:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.084+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fd4c000abf0 con 0x7fd4c815df00 2024-09-13T08:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.085+0000 7fd4c5600640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 0x7fd4a807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.085+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5795+0+0 (secure 0 0 0) 0x7fd4c009aba0 con 0x7fd4c815df00 2024-09-13T08:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.086+0000 7fd4cd800640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 0x7fd4a807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.086+0000 7fd4cd800640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 0x7fd4a807aec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc034000 tx=0x7fd4bc004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:24.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.090+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd4c0062e20 con 0x7fd4c815df00 2024-09-13T08:38:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.328+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd4c809e1f0 con 0x7fd4c815df00 2024-09-13T08:38:24.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.885+0000 7fd4c5600640 1 -- 172.21.15.31:0/3414977206 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) ==== 104+0+0 (secure 0 0 0) 0x7fd4c0062720 con 0x7fd4c815df00 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 msgr2=0x7fd4a807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 0x7fd4a807aec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc034000 tx=0x7fd4bc004730 comp rx=0 tx=0).stop 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 msgr2=0x7fd4c80cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c7a0 tx=0x7fd4c000cc60 comp rx=0 tx=0).stop 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 shutdown_connections 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c815df00 0x7fd4c80cd6d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c80d0d20 0x7fd4c80cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd4a8078a00 0x7fd4a807aec0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 --2- 172.21.15.31:0/3414977206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd4c80b2a20 0x7fd4c80ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:24.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 >> 172.21.15.31:0/3414977206 conn(0x7fd4c80d3e10 msgr2=0x7fd4c80d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:24.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 shutdown_connections 2024-09-13T08:38:24.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:24.888+0000 7fd4cec00640 1 -- 172.21.15.31:0/3414977206 wait complete. 2024-09-13T08:38:24.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:38:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 -- 172.21.15.31:0/771507439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 msgr2=0x7fa3580cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 --2- 172.21.15.31:0/771507439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580cd780 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa348005fd0 tx=0x7fa34802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 -- 172.21.15.31:0/771507439 shutdown_connections 2024-09-13T08:38:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 --2- 172.21.15.31:0/771507439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580cd780 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 -- 172.21.15.31:0/771507439 >> 172.21.15.31:0/771507439 conn(0x7fa3580cfbb0 msgr2=0x7fa3580cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 -- 172.21.15.31:0/771507439 shutdown_connections 2024-09-13T08:38:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.072+0000 7fa35d000640 1 -- 172.21.15.31:0/771507439 wait complete. 2024-09-13T08:38:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.074+0000 7fa35d000640 1 Processor -- start 2024-09-13T08:38:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.074+0000 7fa35d000640 1 -- start start 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3580cb380 0x7fa3581543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 0x7fa3580c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580cbdc0 con 0x7fa358161480 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580cbc40 con 0x7fa3580cb380 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa35d000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa3580cbf40 con 0x7fa3580cc190 2024-09-13T08:38:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa356800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa356800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37456/0 (socket says 172.21.15.31:37456) 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa356800640 1 -- 172.21.15.31:0/999453337 learned_addr learned my addr 172.21.15.31:0/999453337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa34d400640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 0x7fa3580c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.075+0000 7fa355e00640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3580cb380 0x7fa3581543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 -- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 msgr2=0x7fa3580c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 0x7fa3580c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 -- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3580cb380 msgr2=0x7fa3581543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3580cb380 0x7fa3581543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 -- 172.21.15.31:0/999453337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35815f020 con 0x7fa358161480 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa34d400640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 0x7fa3580c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa356800640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c7d0 tx=0x7fa35000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350003b70 con 0x7fa358161480 2024-09-13T08:38:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa350003d10 con 0x7fa358161480 2024-09-13T08:38:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35000aab0 con 0x7fa358161480 2024-09-13T08:38:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3580b4300 con 0x7fa358161480 2024-09-13T08:38:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.076+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3580b48c0 con 0x7fa358161480 2024-09-13T08:38:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.077+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3580b4130 con 0x7fa358161480 2024-09-13T08:38:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.079+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fa35000ac50 con 0x7fa358161480 2024-09-13T08:38:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.079+0000 7fa34ea00640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 0x7fa33407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.080+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5795+0+0 (secure 0 0 0) 0x7fa35009ac60 con 0x7fa358161480 2024-09-13T08:38:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.080+0000 7fa355e00640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 0x7fa33407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.081+0000 7fa355e00640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 0x7fa33407aec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa348033000 tx=0x7fa348004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.083+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa350062e40 con 0x7fa358161480 2024-09-13T08:38:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.284+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa3580cde50 con 0x7fa358161480 2024-09-13T08:38:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.891+0000 7fa34ea00640 1 -- 172.21.15.31:0/999453337 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) ==== 181+0+0 (secure 0 0 0) 0x7fa350062740 con 0x7fa358161480 2024-09-13T08:38:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:38:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 msgr2=0x7fa33407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 0x7fa33407aec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa348033000 tx=0x7fa348004730 comp rx=0 tx=0).stop 2024-09-13T08:38:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 msgr2=0x7fa3580b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c7d0 tx=0x7fa35000cc90 comp rx=0 tx=0).stop 2024-09-13T08:38:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 shutdown_connections 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa358161480 0x7fa3580b1230 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa334078a00 0x7fa33407aec0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3580cc190 0x7fa3580c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 --2- 172.21.15.31:0/999453337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3580cb380 0x7fa3581543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.895+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 >> 172.21.15.31:0/999453337 conn(0x7fa3580cfbb0 msgr2=0x7fa358072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.896+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 shutdown_connections 2024-09-13T08:38:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:25.896+0000 7fa35d000640 1 -- 172.21.15.31:0/999453337 wait complete. 2024-09-13T08:38:25.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:38:26.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 -- 172.21.15.31:0/2226377612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f4004d40 msgr2=0x7f16f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:26.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 --2- 172.21.15.31:0/2226377612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f4004d40 0x7f16f40051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1708005fd0 tx=0x7f16f4005830 comp rx=0 tx=0).stop 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f170e800640 1 -- 172.21.15.31:0/2226377612 reap_dead start 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 -- 172.21.15.31:0/2226377612 shutdown_connections 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 -- 172.21.15.31:0/2226377612 >> 172.21.15.31:0/2226377612 conn(0x7f17100cfbb0 msgr2=0x7f17100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 -- 172.21.15.31:0/2226377612 shutdown_connections 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.084+0000 7f1714c00640 1 -- 172.21.15.31:0/2226377612 wait complete. 2024-09-13T08:38:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.086+0000 7f1714c00640 1 Processor -- start 2024-09-13T08:38:26.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.086+0000 7f1714c00640 1 -- start start 2024-09-13T08:38:26.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f4004d40 0x7f17100cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:26.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:26.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17101e8ac0 0x7f171014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:26.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f171006e740 con 0x7f17100c7ea0 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f171006e5c0 con 0x7f16f4004d40 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f1714c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f17100b29e0 con 0x7f17101e8ac0 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f170d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f170d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37462/0 (socket says 172.21.15.31:37462) 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f170d400640 1 -- 172.21.15.31:0/3240528867 learned_addr learned my addr 172.21.15.31:0/3240528867 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f170de00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f4004d40 0x7f17100cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.087+0000 7f170e800640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17101e8ac0 0x7f171014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 -- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17101e8ac0 msgr2=0x7f171014dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17101e8ac0 0x7f171014dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 -- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f4004d40 msgr2=0x7f17100cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f4004d40 0x7f17100cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 -- 172.21.15.31:0/3240528867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17101f8670 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f170d400640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00f9d0 tx=0x7f16fc00fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00bcb0 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16fc011440 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17101f8950 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc011870 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.088+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17101f8f10 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.089+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17100cb740 con 0x7f17100c7ea0 2024-09-13T08:38:26.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.091+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f16fc019020 con 0x7f17100c7ea0 2024-09-13T08:38:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.092+0000 7f1706000640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 0x7f16f007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:26.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.092+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5357+0+0 (secure 0 0 0) 0x7f16fc09b5a0 con 0x7f17100c7ea0 2024-09-13T08:38:26.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.092+0000 7f170de00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 0x7f16f007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:26.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.093+0000 7f170de00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 0x7f16f007aec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f17101e9410 tx=0x7f1700001b90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:26.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.096+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f16fc0632f0 con 0x7f17100c7ea0 2024-09-13T08:38:26.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.289+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f17100cb310 con 0x7f17100c7ea0 2024-09-13T08:38:26.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.892+0000 7f1706000640 1 -- 172.21.15.31:0/3240528867 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) ==== 169+0+0 (secure 0 0 0) 0x7f16fc067d20 con 0x7f17100c7ea0 2024-09-13T08:38:26.894 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 msgr2=0x7f16f007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 0x7f16f007aec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f17101e9410 tx=0x7f1700001b90 comp rx=0 tx=0).stop 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 msgr2=0x7f17101f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00f9d0 tx=0x7f16fc00fdd0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 shutdown_connections 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f16f0078a00 0x7f16f007aec0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17101e8ac0 0x7f171014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17100c7ea0 0x7f17101f5980 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 --2- 172.21.15.31:0/3240528867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f4004d40 0x7f17100cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 >> 172.21.15.31:0/3240528867 conn(0x7f17100cfbb0 msgr2=0x7f171015aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 shutdown_connections 2024-09-13T08:38:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:26.897+0000 7f1714c00640 1 -- 172.21.15.31:0/3240528867 wait complete. 2024-09-13T08:38:26.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.089+0000 7fb6ec400640 1 -- 172.21.15.31:0/3573366960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4158700 msgr2=0x7fb6e406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.089+0000 7fb6ec400640 1 --2- 172.21.15.31:0/3573366960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4158700 0x7fb6e406e660 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8005fd0 tx=0x7fb6d802aaa0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 -- 172.21.15.31:0/3573366960 shutdown_connections 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 --2- 172.21.15.31:0/3573366960 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406eba0 0x7fb6e406f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 --2- 172.21.15.31:0/3573366960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4158700 0x7fb6e406e660 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 -- 172.21.15.31:0/3573366960 >> 172.21.15.31:0/3573366960 conn(0x7fb6e40b18f0 msgr2=0x7fb6e40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 -- 172.21.15.31:0/3573366960 shutdown_connections 2024-09-13T08:38:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.090+0000 7fb6ec400640 1 -- 172.21.15.31:0/3573366960 wait complete. 2024-09-13T08:38:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.091+0000 7fb6ec400640 1 Processor -- start 2024-09-13T08:38:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.091+0000 7fb6ec400640 1 -- start start 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 0x7fb6e40cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 0x7fb6e40cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40b4b40 con 0x7fb6e406eba0 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40b49c0 con 0x7fb6e4158700 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.092+0000 7fb6ec400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e40ccd70 con 0x7fb6e406f8c0 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37490/0 (socket says 172.21.15.31:37490) 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 -- 172.21.15.31:0/1857724642 learned_addr learned my addr 172.21.15.31:0/1857724642 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9000640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 0x7fb6e40cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 -- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 msgr2=0x7fb6e40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 0x7fb6e40cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 -- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 msgr2=0x7fb6e40cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6ea400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 0x7fb6e40cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 0x7fb6e40cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 -- 172.21.15.31:0/1857724642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e41f8a40 con 0x7fb6e406eba0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9000640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 0x7fb6e40cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6ea400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 0x7fb6e40cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.093+0000 7fb6e9a00640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8006010 tx=0x7fb6d8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.094+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8044070 con 0x7fb6e406eba0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.094+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6d8003e00 con 0x7fb6e406eba0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.094+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8004490 con 0x7fb6e406eba0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.094+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e4072bc0 con 0x7fb6e406eba0 2024-09-13T08:38:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.094+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e41f8fc0 con 0x7fb6e406eba0 2024-09-13T08:38:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.095+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e40cd330 con 0x7fb6e406eba0 2024-09-13T08:38:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.096+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fb6d802c040 con 0x7fb6e406eba0 2024-09-13T08:38:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.097+0000 7fb6e1600640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 0x7fb6c407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.097+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7fb6d80bb620 con 0x7fb6e406eba0 2024-09-13T08:38:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.097+0000 7fb6e9000640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 0x7fb6c407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.098+0000 7fb6e9000640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 0x7fb6c407aec0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb6e406e870 tx=0x7fb6d0005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.101+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb6d8032030 con 0x7fb6e406eba0 2024-09-13T08:38:27.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.299+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fb6e415fce0 con 0x7fb6e406eba0 2024-09-13T08:38:27.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.303+0000 7fb6e1600640 1 -- 172.21.15.31:0/1857724642 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v29) ==== 71+0+0 (secure 0 0 0) 0x7fb6d8083b60 con 0x7fb6e406eba0 2024-09-13T08:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.305+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 msgr2=0x7fb6c407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.305+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 0x7fb6c407aec0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb6e406e870 tx=0x7fb6d0005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.305+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 msgr2=0x7fb6e41f8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.305+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8006010 tx=0x7fb6d8003c10 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 shutdown_connections 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e4158700 0x7fb6e40cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb6c4078a00 0x7fb6c407aec0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e406f8c0 0x7fb6e40cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 --2- 172.21.15.31:0/1857724642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e406eba0 0x7fb6e41f8500 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 >> 172.21.15.31:0/1857724642 conn(0x7fb6e40b18f0 msgr2=0x7fb6e41560b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 shutdown_connections 2024-09-13T08:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.306+0000 7fb6ec400640 1 -- 172.21.15.31:0/1857724642 wait complete. 2024-09-13T08:38:27.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:38:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 -- 172.21.15.31:0/1237267560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5fc004800 msgr2=0x7fb5fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 --2- 172.21.15.31:0/1237267560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5fc004800 0x7fb5fc004c80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb600009b20 tx=0x7fb5fc005830 comp rx=0 tx=0).stop 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb617e00640 1 -- 172.21.15.31:0/1237267560 reap_dead start 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 -- 172.21.15.31:0/1237267560 shutdown_connections 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 -- 172.21.15.31:0/1237267560 >> 172.21.15.31:0/1237267560 conn(0x7fb6180cc910 msgr2=0x7fb6180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 -- 172.21.15.31:0/1237267560 shutdown_connections 2024-09-13T08:38:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.483+0000 7fb61e600640 1 -- 172.21.15.31:0/1237267560 wait complete. 2024-09-13T08:38:27.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.485+0000 7fb61e600640 1 Processor -- start 2024-09-13T08:38:27.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.485+0000 7fb61e600640 1 -- start start 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5fc004800 0x7fb6180b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6181539a0 0x7fb6180b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6181600f0 con 0x7fb61809d6c0 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6180cb850 con 0x7fb6181539a0 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb61e600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb61809d230 con 0x7fb5fc004800 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37512/0 (socket says 172.21.15.31:37512) 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 -- 172.21.15.31:0/2155223312 learned_addr learned my addr 172.21.15.31:0/2155223312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617400640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5fc004800 0x7fb6180b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb60ea00640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6181539a0 0x7fb6180b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 -- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5fc004800 msgr2=0x7fb6180b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5fc004800 0x7fb6180b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 -- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6181539a0 msgr2=0x7fb6180b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6181539a0 0x7fb6180b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.486+0000 7fb617e00640 1 -- 172.21.15.31:0/2155223312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6180b51e0 con 0x7fb61809d6c0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb617e00640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb6180cf330 tx=0x7fb6100019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610011870 con 0x7fb61809d6c0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb610017070 con 0x7fb61809d6c0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6180b5770 con 0x7fb61809d6c0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610011dc0 con 0x7fb61809d6c0 2024-09-13T08:38:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.487+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61809cec0 con 0x7fb61809d6c0 2024-09-13T08:38:27.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.488+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6180ce700 con 0x7fb61809d6c0 2024-09-13T08:38:27.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.489+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fb61001c460 con 0x7fb61809d6c0 2024-09-13T08:38:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.491+0000 7fb60fe00640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 0x7fb5f407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.491+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7fb61009b280 con 0x7fb61809d6c0 2024-09-13T08:38:27.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.491+0000 7fb617400640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 0x7fb5f407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:27.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.492+0000 7fb617400640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 0x7fb5f407aec0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6180748f0 tx=0x7fb608001660 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:27.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.495+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb610063850 con 0x7fb61809d6c0 2024-09-13T08:38:27.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.766+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fb6180cc490 con 0x7fb61809d6c0 2024-09-13T08:38:27.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.906+0000 7fb60fe00640 1 -- 172.21.15.31:0/2155223312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v98) ==== 158+0+0 (secure 0 0 0) 0x7fb610063150 con 0x7fb61809d6c0 2024-09-13T08:38:27.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 msgr2=0x7fb5f407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 0x7fb5f407aec0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6180748f0 tx=0x7fb608001660 comp rx=0 tx=0).stop 2024-09-13T08:38:27.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 msgr2=0x7fb6180b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb6180cf330 tx=0x7fb6100019b0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 shutdown_connections 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb61809d6c0 0x7fb6180b08c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6181539a0 0x7fb6180b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb5f4078a00 0x7fb5f407aec0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 --2- 172.21.15.31:0/2155223312 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5fc004800 0x7fb6180b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 >> 172.21.15.31:0/2155223312 conn(0x7fb6180cc910 msgr2=0x7fb6180ad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 shutdown_connections 2024-09-13T08:38:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:27.908+0000 7fb61e600640 1 -- 172.21.15.31:0/2155223312 wait complete. 2024-09-13T08:38:27.926 INFO:tasks.cephfs_test_runner:Validate binary .meta file on legacy subvol root doesn't break the system ... ok 2024-09-13T08:38:27.926 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:38:27.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 -- 172.21.15.31:0/614936707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 msgr2=0x7f71f40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 --2- 172.21.15.31:0/614936707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f40cf850 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f71f40c6f10 tx=0x7f71e402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 -- 172.21.15.31:0/614936707 shutdown_connections 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 --2- 172.21.15.31:0/614936707 >> v2:172.21.15.45:3300/0 conn(0x7f71f415f9e0 0x7f71f415fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 --2- 172.21.15.31:0/614936707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f40cf850 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 -- 172.21.15.31:0/614936707 >> 172.21.15.31:0/614936707 conn(0x7f71f40cc910 msgr2=0x7f71f40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:28.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 -- 172.21.15.31:0/614936707 shutdown_connections 2024-09-13T08:38:28.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.087+0000 7f71f9a00640 1 -- 172.21.15.31:0/614936707 wait complete. 2024-09-13T08:38:28.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.088+0000 7f71f9a00640 1 Processor -- start 2024-09-13T08:38:28.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 -- start start 2024-09-13T08:38:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40ce3c0 0x7f71f4159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71f415f9e0 0x7f71f40c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71f40cb9d0 con 0x7f71f415bd70 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71f40cb850 con 0x7f71f40ce3c0 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.089+0000 7f71f9a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f71f40cc360 con 0x7f71f415f9e0 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37542/0 (socket says 172.21.15.31:37542) 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 -- 172.21.15.31:0/2337870563 learned_addr learned my addr 172.21.15.31:0/2337870563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40ce3c0 0x7f71f4159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f3400640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71f415f9e0 0x7f71f40c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 -- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71f415f9e0 msgr2=0x7f71f40c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71f415f9e0 0x7f71f40c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 -- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40ce3c0 msgr2=0x7f71f4159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40ce3c0 0x7f71f4159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.090+0000 7f71f2000640 1 -- 172.21.15.31:0/2337870563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f41f4b20 con 0x7f71f415bd70 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71f2000640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f71e000d700 tx=0x7f71e000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0017070 con 0x7f71f415bd70 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71e0005310 con 0x7f71f415bd70 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e00097c0 con 0x7f71f415bd70 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f41f4e00 con 0x7f71f415bd70 2024-09-13T08:38:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.091+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f41f53c0 con 0x7f71f415bd70 2024-09-13T08:38:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.092+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f40cf850 con 0x7f71f415bd70 2024-09-13T08:38:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.093+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f71e0009960 con 0x7f71f415bd70 2024-09-13T08:38:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.094+0000 7f71eaa00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 0x7f71d407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.094+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7f71e009aa10 con 0x7f71f415bd70 2024-09-13T08:38:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.095+0000 7f71f2a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 0x7f71d407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.095+0000 7f71f2a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 0x7f71d407aec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71e4033000 tx=0x7f71e4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.098+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f71e00629b0 con 0x7f71f415bd70 2024-09-13T08:38:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.295+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71f40cce90 con 0x7f71f415bd70 2024-09-13T08:38:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.296+0000 7f71eaa00640 1 -- 172.21.15.31:0/2337870563 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+12980 (secure 0 0 0) 0x7f71e00673e0 con 0x7f71f415bd70 2024-09-13T08:38:28.297 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:28.297 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":39,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:26.891640+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6834/47525403":"2024-09-14T08:38:23.881873+0000","172.21.15.31:6832/47525403":"2024-09-14T08:38:23.881873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 msgr2=0x7f71d407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 0x7f71d407aec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71e4033000 tx=0x7f71e4004730 comp rx=0 tx=0).stop 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 msgr2=0x7f71f415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f71e000d700 tx=0x7f71e000dbc0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 shutdown_connections 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f71d4078a00 0x7f71d407aec0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71f415f9e0 0x7f71f40c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f415bd70 0x7f71f415c990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 --2- 172.21.15.31:0/2337870563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40ce3c0 0x7f71f4159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 >> 172.21.15.31:0/2337870563 conn(0x7f71f40cc910 msgr2=0x7f71f415b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 shutdown_connections 2024-09-13T08:38:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.299+0000 7f71f9a00640 1 -- 172.21.15.31:0/2337870563 wait complete. 2024-09-13T08:38:28.316 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:38:28.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 -- 172.21.15.31:0/3309011762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 msgr2=0x7f56fc0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 --2- 172.21.15.31:0/3309011762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc0b4840 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f56f4005fd0 tx=0x7f56f402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 -- 172.21.15.31:0/3309011762 shutdown_connections 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 --2- 172.21.15.31:0/3309011762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc0b4840 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 --2- 172.21.15.31:0/3309011762 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc0aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 -- 172.21.15.31:0/3309011762 >> 172.21.15.31:0/3309011762 conn(0x7f56fc0cf9b0 msgr2=0x7f56fc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:28.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 -- 172.21.15.31:0/3309011762 shutdown_connections 2024-09-13T08:38:28.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.481+0000 7f5701000640 1 -- 172.21.15.31:0/3309011762 wait complete. 2024-09-13T08:38:28.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.483+0000 7f5701000640 1 Processor -- start 2024-09-13T08:38:28.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.483+0000 7f5701000640 1 -- start start 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 0x7f56fc0c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc0b5150 con 0x7f56fc0742b0 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc0b4fd0 con 0x7f56fc072450 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f5701000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f56fc0b52d0 con 0x7f56fc152a30 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37556/0 (socket says 172.21.15.31:37556) 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 -- 172.21.15.31:0/2660914299 learned_addr learned my addr 172.21.15.31:0/2660914299 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56fbe00640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56fb400640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 0x7f56fc0c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 -- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 msgr2=0x7f56fc148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 -- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 msgr2=0x7f56fc0c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 0x7f56fc0c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56faa00640 1 -- 172.21.15.31:0/2660914299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1f8730 con 0x7f56fc0742b0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56fbe00640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.484+0000 7f56fb400640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 0x7f56fc0c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f56faa00640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c400 tx=0x7f56e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011790 con 0x7f56fc0742b0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56e8017070 con 0x7f56fc0742b0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc147ed0 con 0x7f56fc0742b0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011cc0 con 0x7f56fc0742b0 2024-09-13T08:38:28.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.485+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc148490 con 0x7f56fc0742b0 2024-09-13T08:38:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.486+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc0b4840 con 0x7f56fc0742b0 2024-09-13T08:38:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.487+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f56e801c460 con 0x7f56fc0742b0 2024-09-13T08:38:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.488+0000 7f56f3400640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 0x7f56dc07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.488+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7f56e809a7c0 con 0x7f56fc0742b0 2024-09-13T08:38:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.489+0000 7f56fb400640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 0x7f56dc07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.489+0000 7f56fb400640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 0x7f56dc07aec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0aea40 tx=0x7f56ec005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.492+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f56fc1f8730 con 0x7f56fc0742b0 2024-09-13T08:38:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.685+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f56fc0008d0 con 0x7f56fc0742b0 2024-09-13T08:38:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.685+0000 7f56f3400640 1 -- 172.21.15.31:0/2660914299 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f56e8062e50 con 0x7f56fc0742b0 2024-09-13T08:38:28.687 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:38:28.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 msgr2=0x7f56dc07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 0x7f56dc07aec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0aea40 tx=0x7f56ec005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:28.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 msgr2=0x7f56fc1f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c400 tx=0x7f56e800c8c0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 shutdown_connections 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f56dc078a00 0x7f56dc07aec0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56fc152a30 0x7f56fc148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc0742b0 0x7f56fc1f81f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 --2- 172.21.15.31:0/2660914299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56fc072450 0x7f56fc0c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 >> 172.21.15.31:0/2660914299 conn(0x7f56fc0cf9b0 msgr2=0x7f56fc0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 shutdown_connections 2024-09-13T08:38:28.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.688+0000 7f5701000640 1 -- 172.21.15.31:0/2660914299 wait complete. 2024-09-13T08:38:28.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-13T08:38:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 -- 172.21.15.31:0/1661701311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a815f9e0 msgr2=0x7f09a815fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 --2- 172.21.15.31:0/1661701311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a815f9e0 0x7f09a815fe20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f09a809e090 tx=0x7f099402a560 comp rx=0 tx=0).stop 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 -- 172.21.15.31:0/1661701311 shutdown_connections 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 --2- 172.21.15.31:0/1661701311 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a80ce3c0 0x7f09a80ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 --2- 172.21.15.31:0/1661701311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a815f9e0 0x7f09a815fe20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 --2- 172.21.15.31:0/1661701311 >> v2:172.21.15.45:3300/0 conn(0x7f09a815bd70 0x7f09a80cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 -- 172.21.15.31:0/1661701311 >> 172.21.15.31:0/1661701311 conn(0x7f09a80cc910 msgr2=0x7f09a80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 -- 172.21.15.31:0/1661701311 shutdown_connections 2024-09-13T08:38:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.867+0000 7f09aee00640 1 -- 172.21.15.31:0/1661701311 wait complete. 2024-09-13T08:38:28.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.869+0000 7f09aee00640 1 Processor -- start 2024-09-13T08:38:28.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.869+0000 7f09aee00640 1 -- start start 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a815bd70 0x7f09a80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a815f9e0 0x7f09a806f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80cb9d0 con 0x7f09a80ce3c0 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80cb850 con 0x7f09a815f9e0 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09aee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f09a80cbc50 con 0x7f09a815bd70 2024-09-13T08:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37560/0 (socket says 172.21.15.31:37560) 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.870+0000 7f09a7e00640 1 -- 172.21.15.31:0/4024687755 learned_addr learned my addr 172.21.15.31:0/4024687755 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7400640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a815bd70 0x7f09a80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09ace00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a815f9e0 0x7f09a806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 -- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a815bd70 msgr2=0x7f09a80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a815bd70 0x7f09a80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 -- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a815f9e0 msgr2=0x7f09a806f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a815f9e0 0x7f09a806f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 -- 172.21.15.31:0/4024687755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a81f4b70 con 0x7f09a80ce3c0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09a7e00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f099800ea20 tx=0x7f099800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099800cd80 con 0x7f09a80ce3c0 2024-09-13T08:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0998010430 con 0x7f09a80ce3c0 2024-09-13T08:38:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998010840 con 0x7f09a80ce3c0 2024-09-13T08:38:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81f4e50 con 0x7f09a80ce3c0 2024-09-13T08:38:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.871+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81f5410 con 0x7f09a80ce3c0 2024-09-13T08:38:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.872+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a80b4c00 con 0x7f09a80ce3c0 2024-09-13T08:38:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.873+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f09980109e0 con 0x7f09a80ce3c0 2024-09-13T08:38:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.874+0000 7f099fe00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 0x7f098807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.875+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7f099809b5e0 con 0x7f09a80ce3c0 2024-09-13T08:38:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.875+0000 7f09a7400640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 0x7f098807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:28.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.876+0000 7f09a7400640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 0x7f098807aec0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0994033000 tx=0x7f0994004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:28.879+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0998063b20 con 0x7f09a80ce3c0 2024-09-13T08:38:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.146+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7f09a80cf850 con 0x7f09a80ce3c0 2024-09-13T08:38:29.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.913+0000 7f099fe00640 1 -- 172.21.15.31:0/4024687755 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v100) ==== 143+0+0 (secure 0 0 0) 0x7f0998063420 con 0x7f09a80ce3c0 2024-09-13T08:38:29.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 msgr2=0x7f098807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:29.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 0x7f098807aec0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0994033000 tx=0x7f0994004730 comp rx=0 tx=0).stop 2024-09-13T08:38:29.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 msgr2=0x7f09a80b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f099800ea20 tx=0x7f099800eee0 comp rx=0 tx=0).stop 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 shutdown_connections 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a815f9e0 0x7f09a806f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0988078a00 0x7f098807aec0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09a815bd70 0x7f09a80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 --2- 172.21.15.31:0/4024687755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a80ce3c0 0x7f09a80b0b30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 >> 172.21.15.31:0/4024687755 conn(0x7f09a80cc910 msgr2=0x7f09a815e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 shutdown_connections 2024-09-13T08:38:29.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:29.916+0000 7f09aee00640 1 -- 172.21.15.31:0/4024687755 wait complete. 2024-09-13T08:38:29.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:38:30.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 -- 172.21.15.31:0/2330944469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 msgr2=0x7f87280cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 --2- 172.21.15.31:0/2330944469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f87280cc300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8718005fd0 tx=0x7f871802a530 comp rx=0 tx=0).stop 2024-09-13T08:38:30.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 -- 172.21.15.31:0/2330944469 shutdown_connections 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 --2- 172.21.15.31:0/2330944469 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87280ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 --2- 172.21.15.31:0/2330944469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f87280cc300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 -- 172.21.15.31:0/2330944469 >> 172.21.15.31:0/2330944469 conn(0x7f87280cc910 msgr2=0x7f87280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 -- 172.21.15.31:0/2330944469 shutdown_connections 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.100+0000 7f872e000640 1 -- 172.21.15.31:0/2330944469 wait complete. 2024-09-13T08:38:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.102+0000 7f872e000640 1 Processor -- start 2024-09-13T08:38:30.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.102+0000 7f872e000640 1 -- start start 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87280b1910 0x7f8728147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87281f9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8728074010 con 0x7f872815dfc0 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8728073e90 con 0x7f87280b1910 2024-09-13T08:38:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f872e000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f872815b7b0 con 0x7f87280ca5c0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f8727e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f8727e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37578/0 (socket says 172.21.15.31:37578) 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f8727e00640 1 -- 172.21.15.31:0/393839510 learned_addr learned my addr 172.21.15.31:0/393839510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f8727400640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87280b1910 0x7f8728147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.103+0000 7f871ea00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87281f9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 -- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 msgr2=0x7f87281f9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87281f9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 -- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87280b1910 msgr2=0x7f8728147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87280b1910 0x7f8728147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 -- 172.21.15.31:0/393839510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281f97a0 con 0x7f872815dfc0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f871ea00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87281f9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f8727e00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f872000c7b0 tx=0x7f872000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720017070 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8720005070 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87280c8b80 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87200054c0 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.104+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87280c9140 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.105+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87280b2150 con 0x7f872815dfc0 2024-09-13T08:38:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.106+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f872001a070 con 0x7f872815dfc0 2024-09-13T08:38:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.108+0000 7f871fe00640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 0x7f870407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.108+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7f8720099fe0 con 0x7f872815dfc0 2024-09-13T08:38:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.108+0000 7f8727400640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 0x7f870407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.109+0000 7f8727400640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 0x7f870407aec0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8718033000 tx=0x7f8718004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.112+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f87281f97a0 con 0x7f872815dfc0 2024-09-13T08:38:30.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.308+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f87280747e0 con 0x7f872815dfc0 2024-09-13T08:38:30.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.308+0000 7f871fe00640 1 -- 172.21.15.31:0/393839510 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v29) ==== 59+0+1998 (secure 0 0 0) 0x7f8720061fa0 con 0x7f872815dfc0 2024-09-13T08:38:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:38:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:38:30.309 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:38:30.310 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:38:30.311 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:38:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 msgr2=0x7f870407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 0x7f870407aec0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8718033000 tx=0x7f8718004730 comp rx=0 tx=0).stop 2024-09-13T08:38:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 msgr2=0x7f8728148360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f872000c7b0 tx=0x7f872000cc70 comp rx=0 tx=0).stop 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 shutdown_connections 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872815dfc0 0x7f8728148360 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8704078a00 0x7f870407aec0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87280ca5c0 0x7f87281f9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 --2- 172.21.15.31:0/393839510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87280b1910 0x7f8728147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 >> 172.21.15.31:0/393839510 conn(0x7f87280cc910 msgr2=0x7f87280b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 shutdown_connections 2024-09-13T08:38:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.311+0000 7f872e000640 1 -- 172.21.15.31:0/393839510 wait complete. 2024-09-13T08:38:30.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 -- 172.21.15.31:0/258814974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61406fec0 msgr2=0x7fc6140b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 --2- 172.21.15.31:0/258814974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61406fec0 0x7fc6140b29e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc60c005fd0 tx=0x7fc60c02aaa0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 -- 172.21.15.31:0/258814974 shutdown_connections 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 --2- 172.21.15.31:0/258814974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61406fec0 0x7fc6140b29e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 --2- 172.21.15.31:0/258814974 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61415bfc0 0x7fc614072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 -- 172.21.15.31:0/258814974 >> 172.21.15.31:0/258814974 conn(0x7fc6140cfbb0 msgr2=0x7fc6140cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 -- 172.21.15.31:0/258814974 shutdown_connections 2024-09-13T08:38:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.490+0000 7fc61c600640 1 -- 172.21.15.31:0/258814974 wait complete. 2024-09-13T08:38:30.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.491+0000 7fc61c600640 1 Processor -- start 2024-09-13T08:38:30.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.492+0000 7fc61c600640 1 -- start start 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61406fa60 0x7fc61414ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61414f410 0x7fc6140c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc6140cbfd0 con 0x7fc61415bfc0 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6140cbe50 con 0x7fc61414f410 2024-09-13T08:38:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61c600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc6140b2f20 con 0x7fc61406fa60 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37590/0 (socket says 172.21.15.31:37590) 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 -- 172.21.15.31:0/2662223970 learned_addr learned my addr 172.21.15.31:0/2662223970 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619c00640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61406fa60 0x7fc61414ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc61a600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61414f410 0x7fc6140c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 -- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61406fa60 msgr2=0x7fc61414ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61406fa60 0x7fc61414ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 -- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61414f410 msgr2=0x7fc6140c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61414f410 0x7fc6140c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.493+0000 7fc619200640 1 -- 172.21.15.31:0/2662223970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6140c8180 con 0x7fc61415bfc0 2024-09-13T08:38:30.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc619200640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc60000c420 tx=0x7fc60000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600017070 con 0x7fc61415bfc0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6140c89e0 con 0x7fc61415bfc0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6000053c0 con 0x7fc61415bfc0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600003850 con 0x7fc61415bfc0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.494+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6140c8fa0 con 0x7fc61415bfc0 2024-09-13T08:38:30.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.495+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6140c9490 con 0x7fc61415bfc0 2024-09-13T08:38:30.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.496+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fc60001a070 con 0x7fc61415bfc0 2024-09-13T08:38:30.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.497+0000 7fc611600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 0x7fc5f407aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:30.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.497+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7fc600099e80 con 0x7fc61415bfc0 2024-09-13T08:38:30.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.498+0000 7fc619c00640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 0x7fc5f407aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:30.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.499+0000 7fc619c00640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 0x7fc5f407aef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc608009ac0 tx=0x7fc608005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:30.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.501+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc600062340 con 0x7fc61415bfc0 2024-09-13T08:38:30.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.696+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc6140b33d0 con 0x7fc61415bfc0 2024-09-13T08:38:30.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.700+0000 7fc611600640 1 -- 172.21.15.31:0/2662223970 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7fc600061c40 con 0x7fc61415bfc0 2024-09-13T08:38:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 msgr2=0x7fc5f407aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 0x7fc5f407aef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc608009ac0 tx=0x7fc608005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 msgr2=0x7fc6140c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc60000c420 tx=0x7fc60000c8e0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 shutdown_connections 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc61414f410 0x7fc6140c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc61415bfc0 0x7fc6140c78a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc5f4078a30 0x7fc5f407aef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 --2- 172.21.15.31:0/2662223970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc61406fa60 0x7fc61414ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.702+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 >> 172.21.15.31:0/2662223970 conn(0x7fc6140cfbb0 msgr2=0x7fc614157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.703+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 shutdown_connections 2024-09-13T08:38:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:30.703+0000 7fc61c600640 1 -- 172.21.15.31:0/2662223970 wait complete. 2024-09-13T08:38:30.719 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:38:30.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:38:30.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:38:30.917 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:38:30.917 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:30.917 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:30.954 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:38:30.954 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:30.954 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:30.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:38:30.993 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:30.993 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:31.030 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:38:31.030 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:31.030 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:31.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:38:31.071 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:38:31.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:31.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 -- 172.21.15.31:0/2986353783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6688004d90 msgr2=0x7f6688005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2986353783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6688004d90 0x7f6688005210 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f669c001f30 tx=0x7f6688005830 comp rx=0 tx=0).stop 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a2800640 1 -- 172.21.15.31:0/2986353783 reap_dead start 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 -- 172.21.15.31:0/2986353783 shutdown_connections 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 -- 172.21.15.31:0/2986353783 >> 172.21.15.31:0/2986353783 conn(0x7f66a40cc910 msgr2=0x7f66a40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 -- 172.21.15.31:0/2986353783 shutdown_connections 2024-09-13T08:38:31.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.231+0000 7f66a8c00640 1 -- 172.21.15.31:0/2986353783 wait complete. 2024-09-13T08:38:31.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.233+0000 7f66a8c00640 1 Processor -- start 2024-09-13T08:38:31.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.233+0000 7f66a8c00640 1 -- start start 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6688004d90 0x7f66a41f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66a409aa10 0x7f66a415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4153220 con 0x7f66a4070000 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66a41530a0 con 0x7f66a409aa10 2024-09-13T08:38:31.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a8c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f66a40b1910 con 0x7f6688004d90 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f6699400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f6699400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37616/0 (socket says 172.21.15.31:37616) 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f6699400640 1 -- 172.21.15.31:0/2320168821 learned_addr learned my addr 172.21.15.31:0/2320168821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a1e00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6688004d90 0x7f66a41f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.234+0000 7f66a2800640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66a409aa10 0x7f66a415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 -- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6688004d90 msgr2=0x7f66a41f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6688004d90 0x7f66a41f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 -- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66a409aa10 msgr2=0x7f66a415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66a409aa10 0x7f66a415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 -- 172.21.15.31:0/2320168821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a41f4e90 con 0x7f66a4070000 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f6699400640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f668c0066e0 tx=0x7f668c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:31.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c015070 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f668c011840 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a41f5170 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c011c90 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.235+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a4070460 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.236+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a40743f0 con 0x7f66a4070000 2024-09-13T08:38:31.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.238+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f668c018070 con 0x7f66a4070000 2024-09-13T08:38:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.239+0000 7f669aa00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 0x7f668007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.239+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7f668c09b5f0 con 0x7f66a4070000 2024-09-13T08:38:31.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.239+0000 7f66a1e00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 0x7f668007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.240+0000 7f66a1e00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 0x7f668007aec0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6694009c50 tx=0x7f6694005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:31.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.243+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f668c0634f0 con 0x7f66a4070000 2024-09-13T08:38:31.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.473+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66a406f7b0 con 0x7f66a4070000 2024-09-13T08:38:31.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.474+0000 7f669aa00640 1 -- 172.21.15.31:0/2320168821 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+3908 (secure 0 0 0) 0x7f668c067f20 con 0x7f66a4070000 2024-09-13T08:38:31.475 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:31.475 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-13T08:38:24:883995+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/485787773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":485787773},{"type":"v1","addr":"172.21.15.45:6836","nonce":485787773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4164780796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":4164780796},{"type":"v1","addr":"172.21.15.45:6837","nonce":4164780796}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3860592811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3860592811},{"type":"v1","addr":"172.21.15.31:6835","nonce":3860592811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/4167010747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":4167010747},{"type":"v1","addr":"172.21.15.31:6834","nonce":4167010747}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-09-13T08:38:31.475 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-13T08:38:31.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 msgr2=0x7f668007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 0x7f668007aec0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6694009c50 tx=0x7f6694005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 msgr2=0x7f66a40b5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f668c0066e0 tx=0x7f668c0039c0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 shutdown_connections 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66a409aa10 0x7f66a415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a4070000 0x7f66a40b5920 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6680078a00 0x7f668007aec0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 --2- 172.21.15.31:0/2320168821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6688004d90 0x7f66a41f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 >> 172.21.15.31:0/2320168821 conn(0x7f66a40cc910 msgr2=0x7f66a40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 shutdown_connections 2024-09-13T08:38:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.477+0000 7f66a8c00640 1 -- 172.21.15.31:0/2320168821 wait complete. 2024-09-13T08:38:31.493 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:38:31.493 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:38:31.493 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:38:31.493 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:38:31.494 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:38:31.494 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:38:31.494 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:38:31.494 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:38:31.494 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:38:31.494 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:38:31.494 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:38:31.495 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:38:31.495 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:38:31.495 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:38:31.495 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:38:31.495 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:38:31.508 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:38:31.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:38:31.509 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:38:31.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:38:31.510 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:38:31.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:38:31.511 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:38:31.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:38:31.552 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:38:31.553 DEBUG:teuthology.parallel:result is None 2024-09-13T08:38:31.554 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:38:31.554 DEBUG:teuthology.parallel:result is None 2024-09-13T08:38:31.555 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:38:31.555 DEBUG:teuthology.parallel:result is None 2024-09-13T08:38:31.557 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:38:31.557 DEBUG:teuthology.parallel:result is None 2024-09-13T08:38:31.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:38:31.685 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:38:31.686 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:38:31.687 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:38:31.687 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:38:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.725+0000 7fb9e9a00640 1 -- 172.21.15.31:0/2384469638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40b3dc0 msgr2=0x7fb9e406f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.725+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/2384469638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e406f9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009830 tx=0x7fb9cc02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.726+0000 7fb9e9a00640 1 -- 172.21.15.31:0/2384469638 shutdown_connections 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.726+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/2384469638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e406f9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.726+0000 7fb9e9a00640 1 -- 172.21.15.31:0/2384469638 >> 172.21.15.31:0/2384469638 conn(0x7fb9e40b18f0 msgr2=0x7fb9e40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.726+0000 7fb9e9a00640 1 -- 172.21.15.31:0/2384469638 shutdown_connections 2024-09-13T08:38:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.726+0000 7fb9e9a00640 1 -- 172.21.15.31:0/2384469638 wait complete. 2024-09-13T08:38:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.727+0000 7fb9e9a00640 1 Processor -- start 2024-09-13T08:38:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.727+0000 7fb9e9a00640 1 -- start start 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e406ff00 0x7fb9e414b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb9e4070820 0x7fb9e414c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e4154990 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e4154810 con 0x7fb9e406ff00 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.728+0000 7fb9e9a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e4154b10 con 0x7fb9e4070820 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9da000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb9e4070820 0x7fb9e414c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37648/0 (socket says 172.21.15.31:37648) 2024-09-13T08:38:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 -- 172.21.15.31:0/1576224884 learned_addr learned my addr 172.21.15.31:0/1576224884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e2a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e406ff00 0x7fb9e414b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 -- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb9e4070820 msgr2=0x7fb9e414c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb9e4070820 0x7fb9e414c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 -- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e406ff00 msgr2=0x7fb9e414b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e406ff00 0x7fb9e414b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 -- 172.21.15.31:0/1576224884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e414a1f0 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.729+0000 7fb9e3400640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00cb40 tx=0x7fb9dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.730+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc016020 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.730+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9dc00adb0 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.730+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e4148620 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.730+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc012680 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.730+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e41553e0 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.731+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e406f9c0 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.732+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fb9dc019020 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.733+0000 7fb9db400640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 0x7fb9c007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.733+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7fb9dc09afb0 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.734+0000 7fb9e2a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 0x7fb9c007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.734+0000 7fb9e2a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 0x7fb9c007aec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4000c00 tx=0x7fb9d4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.737+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb9dc020020 con 0x7fb9e40b3dc0 2024-09-13T08:38:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:31.948+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb9e40749f0 con 0x7fb9e40b3dc0 2024-09-13T08:38:32.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.938+0000 7fb9db400640 1 -- 172.21.15.31:0/1576224884 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) ==== 97+0+0 (secure 0 0 0) 0x7fb9dc062ea0 con 0x7fb9e40b3dc0 2024-09-13T08:38:32.939 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 msgr2=0x7fb9c007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 0x7fb9c007aec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4000c00 tx=0x7fb9d4009040 comp rx=0 tx=0).stop 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 msgr2=0x7fb9e409ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00cb40 tx=0x7fb9dc00a060 comp rx=0 tx=0).stop 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 shutdown_connections 2024-09-13T08:38:32.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e40b3dc0 0x7fb9e409ce50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb9c0078a00 0x7fb9c007aec0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb9e4070820 0x7fb9e414c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 --2- 172.21.15.31:0/1576224884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e406ff00 0x7fb9e414b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 >> 172.21.15.31:0/1576224884 conn(0x7fb9e40b18f0 msgr2=0x7fb9e40ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 shutdown_connections 2024-09-13T08:38:32.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:32.942+0000 7fb9e9a00640 1 -- 172.21.15.31:0/1576224884 wait complete. 2024-09-13T08:38:32.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:38:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 -- 172.21.15.31:0/3839097856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd038004800 msgr2=0x7fd038004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 --2- 172.21.15.31:0/3839097856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd038004800 0x7fd038004c80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c2e0 tx=0x7fd038005830 comp rx=0 tx=0).stop 2024-09-13T08:38:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 -- 172.21.15.31:0/3839097856 shutdown_connections 2024-09-13T08:38:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 --2- 172.21.15.31:0/3839097856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd038004800 0x7fd038004c80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 -- 172.21.15.31:0/3839097856 >> 172.21.15.31:0/3839097856 conn(0x7fd0500d3e70 msgr2=0x7fd0500d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 -- 172.21.15.31:0/3839097856 shutdown_connections 2024-09-13T08:38:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.128+0000 7fd056600640 1 -- 172.21.15.31:0/3839097856 wait complete. 2024-09-13T08:38:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.130+0000 7fd056600640 1 Processor -- start 2024-09-13T08:38:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.130+0000 7fd056600640 1 -- start start 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd038004800 0x7fd0501f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 0x7fd0500cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500d0670 con 0x7fd0500b95d0 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500d2e60 con 0x7fd038004800 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd056600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd0500d07f0 con 0x7fd0500cd0d0 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37666/0 (socket says 172.21.15.31:37666) 2024-09-13T08:38:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 -- 172.21.15.31:0/856244659 learned_addr learned my addr 172.21.15.31:0/856244659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd055200640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd038004800 0x7fd0501f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd055c00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 0x7fd0500cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 -- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 msgr2=0x7fd0500cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 0x7fd0500cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 -- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd038004800 msgr2=0x7fd0501f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd038004800 0x7fd0501f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd04fe00640 1 -- 172.21.15.31:0/856244659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0501f5c70 con 0x7fd0500b95d0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.131+0000 7fd055c00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 0x7fd0500cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd04fe00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c400 tx=0x7fd03c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c031040 con 0x7fd0500b95d0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd03c0065e0 con 0x7fd0500b95d0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003a60 con 0x7fd0500b95d0 2024-09-13T08:38:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0500b7e20 con 0x7fd0500b95d0 2024-09-13T08:38:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.132+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0500b83e0 con 0x7fd0500b95d0 2024-09-13T08:38:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.133+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0500d0d00 con 0x7fd0500b95d0 2024-09-13T08:38:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.134+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fd03c03e030 con 0x7fd0500b95d0 2024-09-13T08:38:33.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.135+0000 7fd04cc00640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 0x7fd03007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.135+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7fd03c0ba3e0 con 0x7fd0500b95d0 2024-09-13T08:38:33.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.136+0000 7fd055200640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 0x7fd03007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.136+0000 7fd055200640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 0x7fd03007aec0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd044006fd0 tx=0x7fd044006d10 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.139+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd0501f5c70 con 0x7fd0500b95d0 2024-09-13T08:38:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.404+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fd0500d1340 con 0x7fd0500b95d0 2024-09-13T08:38:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.404+0000 7fd04cc00640 1 -- 172.21.15.31:0/856244659 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) ==== 80+0+5304 (secure 0 0 0) 0x7fd03c0823b0 con 0x7fd0500b95d0 2024-09-13T08:38:33.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:38:33.410 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:38:33.412 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:38:33.414 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:38:33.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.416 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:38:33.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:38:33.419 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:38:33.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 msgr2=0x7fd03007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 0x7fd03007aec0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd044006fd0 tx=0x7fd044006d10 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 msgr2=0x7fd050154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c400 tx=0x7fd03c001d80 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 shutdown_connections 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd030078a00 0x7fd03007aec0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0500cd0d0 0x7fd0500cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0500b95d0 0x7fd050154070 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 --2- 172.21.15.31:0/856244659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd038004800 0x7fd0501f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 >> 172.21.15.31:0/856244659 conn(0x7fd0500d3e70 msgr2=0x7fd05015fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 shutdown_connections 2024-09-13T08:38:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.407+0000 7fd056600640 1 -- 172.21.15.31:0/856244659 wait complete. 2024-09-13T08:38:33.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 -- 172.21.15.31:0/226219376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 msgr2=0x7f36600cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 --2- 172.21.15.31:0/226219376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600cf850 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f36600ca470 tx=0x7f365002a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 -- 172.21.15.31:0/226219376 shutdown_connections 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 --2- 172.21.15.31:0/226219376 >> v2:172.21.15.45:3300/0 conn(0x7f366015fe20 0x7f36600cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 --2- 172.21.15.31:0/226219376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600cf850 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.584+0000 7f3665200640 1 -- 172.21.15.31:0/226219376 >> 172.21.15.31:0/226219376 conn(0x7f36600cc910 msgr2=0x7f36600cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.585+0000 7f3665200640 1 -- 172.21.15.31:0/226219376 shutdown_connections 2024-09-13T08:38:33.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.585+0000 7f3665200640 1 -- 172.21.15.31:0/226219376 wait complete. 2024-09-13T08:38:33.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.586+0000 7f3665200640 1 Processor -- start 2024-09-13T08:38:33.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.586+0000 7f3665200640 1 -- start start 2024-09-13T08:38:33.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 0x7f36600b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 0x7f36600c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36600cec50 con 0x7f366015bd70 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f36600cead0 con 0x7f366015f9e0 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f3665200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f36600cb850 con 0x7f366015fe20 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.587+0000 7f365de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37682/0 (socket says 172.21.15.31:37682) 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 -- 172.21.15.31:0/1575467124 learned_addr learned my addr 172.21.15.31:0/1575467124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3657e00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 0x7f36600b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365e800640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 0x7f36600c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 -- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 msgr2=0x7f36600c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 0x7f36600c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 -- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 msgr2=0x7f36600b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 0x7f36600b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 -- 172.21.15.31:0/1575467124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36600c5420 con 0x7f366015bd70 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365e800640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 0x7f36600c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3657e00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 0x7f36600b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f365de00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3650009850 tx=0x7f3650003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365003f070 con 0x7f366015bd70 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36500047a0 con 0x7f366015bd70 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650035030 con 0x7f366015bd70 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.588+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36600c5700 con 0x7f366015bd70 2024-09-13T08:38:33.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.589+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36600c5060 con 0x7f366015bd70 2024-09-13T08:38:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.590+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36600cf850 con 0x7f366015bd70 2024-09-13T08:38:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.591+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f3650042070 con 0x7f366015bd70 2024-09-13T08:38:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.592+0000 7f3656000640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 0x7f363c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.592+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f36500bb870 con 0x7f366015bd70 2024-09-13T08:38:33.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.592+0000 7f3657e00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 0x7f363c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.593+0000 7f3657e00640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 0x7f363c07aec0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f36600c6e10 tx=0x7f3648009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.596+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3650083e00 con 0x7f366015bd70 2024-09-13T08:38:33.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.796+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f36600ccd90 con 0x7f366015bd70 2024-09-13T08:38:33.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.796+0000 7f3656000640 1 -- 172.21.15.31:0/1575467124 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) ==== 84+0+65 (secure 0 0 0) 0x7f3650083700 con 0x7f366015bd70 2024-09-13T08:38:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:33.798 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:38:33.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 msgr2=0x7f363c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 0x7f363c07aec0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f36600c6e10 tx=0x7f3648009040 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 msgr2=0x7f36600b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3650009850 tx=0x7f3650003820 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 shutdown_connections 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f363c078a00 0x7f363c07aec0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f366015fe20 0x7f36600c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366015f9e0 0x7f36600b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 --2- 172.21.15.31:0/1575467124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f366015bd70 0x7f36600b0840 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 >> 172.21.15.31:0/1575467124 conn(0x7f36600cc910 msgr2=0x7f36600ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 shutdown_connections 2024-09-13T08:38:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.799+0000 7f3665200640 1 -- 172.21.15.31:0/1575467124 wait complete. 2024-09-13T08:38:33.815 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:38:33.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:38:33.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 -- 172.21.15.31:0/2841911593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b00ce8c0 msgr2=0x7f49b00cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 --2- 172.21.15.31:0/2841911593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cecc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f49b00b69e0 tx=0x7f499c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 -- 172.21.15.31:0/2841911593 shutdown_connections 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 --2- 172.21.15.31:0/2841911593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cecc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 --2- 172.21.15.31:0/2841911593 >> v2:172.21.15.45:3300/0 conn(0x7f49b01613f0 0x7f49b00d49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 -- 172.21.15.31:0/2841911593 >> 172.21.15.31:0/2841911593 conn(0x7f49b00d3df0 msgr2=0x7f49b00d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 -- 172.21.15.31:0/2841911593 shutdown_connections 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.983+0000 7f49b6800640 1 -- 172.21.15.31:0/2841911593 wait complete. 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.985+0000 7f49b6800640 1 Processor -- start 2024-09-13T08:38:33.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.985+0000 7f49b6800640 1 -- start start 2024-09-13T08:38:33.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 0x7f49b00ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49b00d0c70 con 0x7f49b01613f0 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49b00b8420 con 0x7f49b00ce8c0 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b6800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f49b00d0df0 con 0x7f49b00cf200 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37696/0 (socket says 172.21.15.31:37696) 2024-09-13T08:38:33.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 -- 172.21.15.31:0/1790405570 learned_addr learned my addr 172.21.15.31:0/1790405570 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5400640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b4a00640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 0x7f49b00ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 -- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 msgr2=0x7f49b00ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 0x7f49b00ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 -- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 msgr2=0x7f49b00cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5e00640 1 -- 172.21.15.31:0/1790405570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b00c7760 con 0x7f49b01613f0 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b5400640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.986+0000 7f49b4a00640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 0x7f49b00ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49b5e00640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c770 tx=0x7f49a800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8017070 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49a8005070 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a80054c0 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b00b47d0 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.987+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b00c8180 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.988+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b00d1310 con 0x7f49b01613f0 2024-09-13T08:38:33.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.989+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f49a801a070 con 0x7f49b01613f0 2024-09-13T08:38:33.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.990+0000 7f49ad600640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 0x7f499007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:33.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.990+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f49a809a060 con 0x7f49b01613f0 2024-09-13T08:38:33.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.990+0000 7f49b5400640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 0x7f499007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:33.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.991+0000 7f49b5400640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 0x7f499007aec0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009ab0 tx=0x7f49a4005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:33.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:33.995+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f49a8062680 con 0x7f49b01613f0 2024-09-13T08:38:34.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.189+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f49b00b5e40 con 0x7f49b01613f0 2024-09-13T08:38:34.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.949+0000 7f49ad600640 1 -- 172.21.15.31:0/1790405570 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) ==== 138+0+0 (secure 0 0 0) 0x7f49a8061f80 con 0x7f49b01613f0 2024-09-13T08:38:34.950 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:38:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 msgr2=0x7f499007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 0x7f499007aec0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009ab0 tx=0x7f49a4005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 msgr2=0x7f49b00ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c770 tx=0x7f49a800cc30 comp rx=0 tx=0).stop 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 shutdown_connections 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49b01613f0 0x7f49b00ca5b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4990078a00 0x7f499007aec0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49b00cf200 0x7f49b00ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 --2- 172.21.15.31:0/1790405570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49b00ce8c0 0x7f49b00cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 >> 172.21.15.31:0/1790405570 conn(0x7f49b00d3df0 msgr2=0x7f49b006d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 shutdown_connections 2024-09-13T08:38:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:34.952+0000 7f49b6800640 1 -- 172.21.15.31:0/1790405570 wait complete. 2024-09-13T08:38:34.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:38:35.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 -- 172.21.15.31:0/1652753991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe690003a50 msgr2=0x7fe690003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 --2- 172.21.15.31:0/1652753991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe690003a50 0x7fe690003e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0016c0 tx=0x7fe690005830 comp rx=0 tx=0).stop 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 -- 172.21.15.31:0/1652753991 shutdown_connections 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 --2- 172.21.15.31:0/1652753991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe690003a50 0x7fe690003e70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 -- 172.21.15.31:0/1652753991 >> 172.21.15.31:0/1652753991 conn(0x7fe6ac0cfbb0 msgr2=0x7fe6ac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 -- 172.21.15.31:0/1652753991 shutdown_connections 2024-09-13T08:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.151+0000 7fe6b2200640 1 -- 172.21.15.31:0/1652753991 wait complete. 2024-09-13T08:38:35.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.153+0000 7fe6b2200640 1 Processor -- start 2024-09-13T08:38:35.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.153+0000 7fe6b2200640 1 -- start start 2024-09-13T08:38:35.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe690003a50 0x7fe6ac14fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:35.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 0x7fe6ac0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac0cb8c0 con 0x7fe6ac156fb0 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac0cb740 con 0x7fe6ac1f6df0 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6b2200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ac0cba40 con 0x7fe690003a50 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6a2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6a2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44602/0 (socket says 172.21.15.31:44602) 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6a2a00640 1 -- 172.21.15.31:0/2888473134 learned_addr learned my addr 172.21.15.31:0/2888473134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6ab400640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe690003a50 0x7fe6ac14fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.154+0000 7fe6abe00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 0x7fe6ac0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 -- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe690003a50 msgr2=0x7fe6ac14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe690003a50 0x7fe6ac14fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 -- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 msgr2=0x7fe6ac0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:35.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 0x7fe6ac0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 -- 172.21.15.31:0/2888473134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac1503d0 con 0x7fe6ac156fb0 2024-09-13T08:38:35.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6abe00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 0x7fe6ac0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a2a00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe694000fb0 tx=0x7fe694001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694015070 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe694010980 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694010dd0 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.155+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac153940 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.156+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac153f00 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.157+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ac1f6180 con 0x7fe6ac156fb0 2024-09-13T08:38:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.158+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fe694018070 con 0x7fe6ac156fb0 2024-09-13T08:38:35.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.159+0000 7fe6a3e00640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 0x7fe68807aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:35.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.159+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5227+0+0 (secure 0 0 0) 0x7fe69409ae30 con 0x7fe6ac156fb0 2024-09-13T08:38:35.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.159+0000 7fe6ab400640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 0x7fe68807aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:35.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.160+0000 7fe6ab400640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 0x7fe68807aef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe69c032000 tx=0x7fe69c02c040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.164+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe694062c00 con 0x7fe6ac156fb0 2024-09-13T08:38:35.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.365+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe6ac0c9fc0 con 0x7fe6ac156fb0 2024-09-13T08:38:35.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:38:35.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.954+0000 7fe6a3e00640 1 -- 172.21.15.31:0/2888473134 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) ==== 170+0+0 (secure 0 0 0) 0x7fe694067630 con 0x7fe6ac156fb0 2024-09-13T08:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:38:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 msgr2=0x7fe68807aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 0x7fe68807aef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe69c032000 tx=0x7fe69c02c040 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 msgr2=0x7fe6ac1f68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe694000fb0 tx=0x7fe694001820 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 shutdown_connections 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6ac1f6df0 0x7fe6ac0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6ac156fb0 0x7fe6ac1f68b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe688078a30 0x7fe68807aef0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 --2- 172.21.15.31:0/2888473134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe690003a50 0x7fe6ac14fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 >> 172.21.15.31:0/2888473134 conn(0x7fe6ac0cfbb0 msgr2=0x7fe6ac15e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:35.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.958+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 shutdown_connections 2024-09-13T08:38:35.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:35.959+0000 7fe6b2200640 1 -- 172.21.15.31:0/2888473134 wait complete. 2024-09-13T08:38:35.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 -- 172.21.15.31:0/1076280676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200741b0 msgr2=0x7fb2200cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 --2- 172.21.15.31:0/1076280676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200741b0 0x7fb2200cd780 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb2200d0cb0 tx=0x7fb21002a560 comp rx=0 tx=0).stop 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 -- 172.21.15.31:0/1076280676 shutdown_connections 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 --2- 172.21.15.31:0/1076280676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200741b0 0x7fb2200cd780 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 --2- 172.21.15.31:0/1076280676 >> v2:172.21.15.45:3300/0 conn(0x7fb22015acb0 0x7fb2200cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.193+0000 7fb228800640 1 -- 172.21.15.31:0/1076280676 >> 172.21.15.31:0/1076280676 conn(0x7fb2200cfbb0 msgr2=0x7fb2200cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.194+0000 7fb228800640 1 -- 172.21.15.31:0/1076280676 shutdown_connections 2024-09-13T08:38:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.194+0000 7fb228800640 1 -- 172.21.15.31:0/1076280676 wait complete. 2024-09-13T08:38:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.195+0000 7fb228800640 1 Processor -- start 2024-09-13T08:38:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.195+0000 7fb228800640 1 -- start start 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2200741b0 0x7fb22014d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb22015acb0 0x7fb2201f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200ce100 con 0x7fb2200745f0 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200cdf80 con 0x7fb22015acb0 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb228800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb2200cb0f0 con 0x7fb2200741b0 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb225400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb225400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44622/0 (socket says 172.21.15.31:44622) 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb225400640 1 -- 172.21.15.31:0/2540991901 learned_addr learned my addr 172.21.15.31:0/2540991901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.196+0000 7fb225e00640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2200741b0 0x7fb22014d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb226800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb22015acb0 0x7fb2201f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 -- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2200741b0 msgr2=0x7fb22014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2200741b0 0x7fb22014d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 -- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb22015acb0 msgr2=0x7fb2201f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb22015acb0 0x7fb2201f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 -- 172.21.15.31:0/2540991901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22014ccc0 con 0x7fb2200745f0 2024-09-13T08:38:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb225400640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb210009850 tx=0x7fb210003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.197+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21003f070 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.198+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2100047a0 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.198+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210035030 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.198+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2201f8470 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.198+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22014cf90 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.199+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2200cb5a0 con 0x7fb2200745f0 2024-09-13T08:38:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.200+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fb210042070 con 0x7fb2200745f0 2024-09-13T08:38:36.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.201+0000 7fb21a000640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 0x7fb20407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:36.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.201+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5609+0+0 (secure 0 0 0) 0x7fb2100bba10 con 0x7fb2200745f0 2024-09-13T08:38:36.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.201+0000 7fb225e00640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 0x7fb20407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:36.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.202+0000 7fb225e00640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 0x7fb20407aec0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb21400bb50 tx=0x7fb21400b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.206+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb2100837a0 con 0x7fb2200745f0 2024-09-13T08:38:36.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.442+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb220150ab0 con 0x7fb2200745f0 2024-09-13T08:38:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.957+0000 7fb21a000640 1 -- 172.21.15.31:0/2540991901 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v23) ==== 172+0+0 (secure 0 0 0) 0x7fb2100881d0 con 0x7fb2200745f0 2024-09-13T08:38:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 6 and data pool 7 2024-09-13T08:38:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.963+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 msgr2=0x7fb20407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.963+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 0x7fb20407aec0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb21400bb50 tx=0x7fb21400b2e0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.963+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 msgr2=0x7fb2200c63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.963+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb210009850 tx=0x7fb210003910 comp rx=0 tx=0).stop 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 shutdown_connections 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb22015acb0 0x7fb2201f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2200745f0 0x7fb2200c63d0 secure :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb210009850 tx=0x7fb210003910 comp rx=0 tx=0).stop 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb204078a00 0x7fb20407aec0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 --2- 172.21.15.31:0/2540991901 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2200741b0 0x7fb22014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 >> 172.21.15.31:0/2540991901 conn(0x7fb2200cfbb0 msgr2=0x7fb2200cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 shutdown_connections 2024-09-13T08:38:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:36.964+0000 7fb228800640 1 -- 172.21.15.31:0/2540991901 wait complete. 2024-09-13T08:38:37.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:37.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 -- 172.21.15.31:0/3457772680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700aef70 msgr2=0x7f8a700cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 --2- 172.21.15.31:0/3457772680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700aef70 0x7f8a700cc270 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a58005fd0 tx=0x7f8a5802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 -- 172.21.15.31:0/3457772680 shutdown_connections 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 --2- 172.21.15.31:0/3457772680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700aef70 0x7f8a700cc270 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 --2- 172.21.15.31:0/3457772680 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a7015a010 0x7f8a700ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 -- 172.21.15.31:0/3457772680 >> 172.21.15.31:0/3457772680 conn(0x7f8a700cc910 msgr2=0x7f8a700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 -- 172.21.15.31:0/3457772680 shutdown_connections 2024-09-13T08:38:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.168+0000 7f8a75c00640 1 -- 172.21.15.31:0/3457772680 wait complete. 2024-09-13T08:38:37.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.170+0000 7f8a75c00640 1 Processor -- start 2024-09-13T08:38:37.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.170+0000 7f8a75c00640 1 -- start start 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a700aef70 0x7f8a70073700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a7015a010 0x7f8a7015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7006e4f0 con 0x7f8a700b1910 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70070ab0 con 0x7f8a7015a010 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a75c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7006e670 con 0x7f8a700aef70 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a66000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a66000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44646/0 (socket says 172.21.15.31:44646) 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a66000640 1 -- 172.21.15.31:0/4040103208 learned_addr learned my addr 172.21.15.31:0/4040103208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a6ea00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a700aef70 0x7f8a70073700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.171+0000 7f8a6f400640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a7015a010 0x7f8a7015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 -- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a700aef70 msgr2=0x7f8a70073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a700aef70 0x7f8a70073700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 -- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a7015a010 msgr2=0x7f8a7015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a7015a010 0x7f8a7015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 -- 172.21.15.31:0/4040103208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a70158ba0 con 0x7f8a700b1910 2024-09-13T08:38:37.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a66000640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a58005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58040020 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a58004320 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58035070 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.172+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7015be00 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.173+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7015c6a0 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.173+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a700cc270 con 0x7f8a700b1910 2024-09-13T08:38:37.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.175+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f8a58043020 con 0x7f8a700b1910 2024-09-13T08:38:37.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.176+0000 7f8a67400640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 0x7f8a4c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.176+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7f8a580bb980 con 0x7f8a700b1910 2024-09-13T08:38:37.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.176+0000 7f8a6ea00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 0x7f8a4c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.177+0000 7f8a6ea00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 0x7f8a4c07aec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000bc50 tx=0x7f8a6000b3e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.180+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8a5804a030 con 0x7f8a700b1910 2024-09-13T08:38:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.375+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a700b5e10 con 0x7f8a700b1910 2024-09-13T08:38:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.376+0000 7f8a67400640 1 -- 172.21.15.31:0/4040103208 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16634 (secure 0 0 0) 0x7f8a58083570 con 0x7f8a700b1910 2024-09-13T08:38:37.378 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:36.443080+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:38:34.198287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T08:38:35.374739+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 msgr2=0x7f8a4c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 0x7f8a4c07aec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000bc50 tx=0x7f8a6000b3e0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 msgr2=0x7f8a70158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a58005d50 comp rx=0 tx=0).stop 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 shutdown_connections 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a7015a010 0x7f8a7015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a700b1910 0x7f8a70158660 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8a4c078a00 0x7f8a4c07aec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 --2- 172.21.15.31:0/4040103208 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a700aef70 0x7f8a70073700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 >> 172.21.15.31:0/4040103208 conn(0x7f8a700cc910 msgr2=0x7f8a701e7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 shutdown_connections 2024-09-13T08:38:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.379+0000 7f8a75c00640 1 -- 172.21.15.31:0/4040103208 wait complete. 2024-09-13T08:38:37.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 -- 172.21.15.31:0/3335827276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400ce900 msgr2=0x7f04400cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 --2- 172.21.15.31:0/3335827276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400ce900 0x7f04400cbc50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0438005fd0 tx=0x7f043802a8a0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 -- 172.21.15.31:0/3335827276 shutdown_connections 2024-09-13T08:38:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 --2- 172.21.15.31:0/3335827276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400ce900 0x7f04400cbc50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 --2- 172.21.15.31:0/3335827276 >> v2:172.21.15.45:3300/0 conn(0x7f04400cf850 0x7f04400ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 -- 172.21.15.31:0/3335827276 >> 172.21.15.31:0/3335827276 conn(0x7f04400cc910 msgr2=0x7f04400cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.558+0000 7f0445800640 1 -- 172.21.15.31:0/3335827276 shutdown_connections 2024-09-13T08:38:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.559+0000 7f0445800640 1 -- 172.21.15.31:0/3335827276 wait complete. 2024-09-13T08:38:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.560+0000 7f0445800640 1 Processor -- start 2024-09-13T08:38:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.560+0000 7f0445800640 1 -- start start 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 0x7f04400c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04401600f0 con 0x7f044015bd70 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04400cb850 con 0x7f04400ce900 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f0445800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f044015f240 con 0x7f04400cf850 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f043f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f043ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 0x7f04400c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:55890/0 (socket says 172.21.15.31:55890) 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 -- 172.21.15.31:0/2866633404 learned_addr learned my addr 172.21.15.31:0/2866633404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.561+0000 7f043f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44652/0 (socket says 172.21.15.31:44652) 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 -- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 msgr2=0x7f04400c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 0x7f04400c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 -- 172.21.15.31:0/2866633404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 msgr2=0x7f044009bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 -- 172.21.15.31:0/2866633404 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04401f5630 con 0x7f04400cf850 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f043ea00640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 0x7f04400c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f043f400640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.562+0000 7f0436000640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f042800e9f0 tx=0x7f042800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.563+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800cd80 con 0x7f04400cf850 2024-09-13T08:38:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.563+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0428010430 con 0x7f04400cf850 2024-09-13T08:38:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.563+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428010840 con 0x7f04400cf850 2024-09-13T08:38:37.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.563+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04401f4870 con 0x7f04400cf850 2024-09-13T08:38:37.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.563+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04401f4e30 con 0x7f04400cf850 2024-09-13T08:38:37.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.564+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04400ce3c0 con 0x7f04400cf850 2024-09-13T08:38:37.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.565+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f04280109e0 con 0x7f04400cf850 2024-09-13T08:38:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.566+0000 7f0437400640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 0x7f041c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.566+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7f042809baa0 con 0x7f04400cf850 2024-09-13T08:38:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.566+0000 7f043ea00640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 0x7f041c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.567+0000 7f043ea00640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 0x7f041c07aec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0430007f20 tx=0x7f0430007eb0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.574+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0428063910 con 0x7f04400cf850 2024-09-13T08:38:37.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.771+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04400acd40 con 0x7f04400cf850 2024-09-13T08:38:37.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.773+0000 7f0437400640 1 -- 172.21.15.31:0/2866633404 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16634 (secure 0 0 0) 0x7f0428068340 con 0x7f04400cf850 2024-09-13T08:38:37.775 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:37.775 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:36.443080+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:38:34.198287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T08:38:35.374739+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 msgr2=0x7f041c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 0x7f041c07aec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0430007f20 tx=0x7f0430007eb0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 msgr2=0x7f044009b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f042800e9f0 tx=0x7f042800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 shutdown_connections 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f044015bd70 0x7f044009bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f041c078a00 0x7f041c07aec0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04400cf850 0x7f044009b5d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 --2- 172.21.15.31:0/2866633404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04400ce900 0x7f04400c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.776+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 >> 172.21.15.31:0/2866633404 conn(0x7f04400cc910 msgr2=0x7f04400cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.777+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 shutdown_connections 2024-09-13T08:38:37.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.777+0000 7f0445800640 1 -- 172.21.15.31:0/2866633404 wait complete. 2024-09-13T08:38:37.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 -- 172.21.15.31:0/4145541407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13380b1f70 msgr2=0x7f13380cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 --2- 172.21.15.31:0/4145541407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13380b1f70 0x7f13380cea50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1328009830 tx=0x7f132802a530 comp rx=0 tx=0).stop 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 -- 172.21.15.31:0/4145541407 shutdown_connections 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 --2- 172.21.15.31:0/4145541407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13380b1f70 0x7f13380cea50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 -- 172.21.15.31:0/4145541407 >> 172.21.15.31:0/4145541407 conn(0x7f13380ccd00 msgr2=0x7f13380cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 -- 172.21.15.31:0/4145541407 shutdown_connections 2024-09-13T08:38:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.957+0000 7f133d600640 1 -- 172.21.15.31:0/4145541407 wait complete. 2024-09-13T08:38:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.959+0000 7f133d600640 1 Processor -- start 2024-09-13T08:38:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.959+0000 7f133d600640 1 -- start start 2024-09-13T08:38:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13380b1f70 0x7f13380c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13380b40c0 0x7f133809c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13380b24d0 con 0x7f1338070140 2024-09-13T08:38:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13380b2350 con 0x7f13380b1f70 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133d600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f13380b2650 con 0x7f13380b40c0 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f1337e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f1337e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44666/0 (socket says 172.21.15.31:44666) 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f1337e00640 1 -- 172.21.15.31:0/2667755390 learned_addr learned my addr 172.21.15.31:0/2667755390 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f132fe00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13380b1f70 0x7f13380c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.960+0000 7f133cc00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13380b40c0 0x7f133809c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 -- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13380b40c0 msgr2=0x7f133809c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13380b40c0 0x7f133809c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 -- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13380b1f70 msgr2=0x7f13380c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13380b1f70 0x7f13380c2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 -- 172.21.15.31:0/2667755390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13380cc220 con 0x7f1338070140 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1337e00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f132802ab20 tx=0x7f1328032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328040070 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1328005330 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328003800 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.961+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f133809cf50 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.962+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13380cbf40 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.963+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13380d13d0 con 0x7f1338070140 2024-09-13T08:38:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.964+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f1328039040 con 0x7f1338070140 2024-09-13T08:38:37.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.965+0000 7f1334c00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 0x7f131407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:37.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.965+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7f13280bbb90 con 0x7f1338070140 2024-09-13T08:38:37.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.965+0000 7f132fe00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 0x7f131407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:37.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.966+0000 7f132fe00640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 0x7f131407aec0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f13380c2db0 tx=0x7f1320009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:37.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:37.969+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f13380d13d0 con 0x7f1338070140 2024-09-13T08:38:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.208+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f133809d120 con 0x7f1338070140 2024-09-13T08:38:38.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.982+0000 7f1334c00640 1 -- 172.21.15.31:0/2667755390 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f1328083810 con 0x7f1338070140 2024-09-13T08:38:38.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 msgr2=0x7f131407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:38.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 0x7f131407aec0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f13380c2db0 tx=0x7f1320009040 comp rx=0 tx=0).stop 2024-09-13T08:38:38.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 msgr2=0x7f133809d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f132802ab20 tx=0x7f1328032d50 comp rx=0 tx=0).stop 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 shutdown_connections 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1314078a00 0x7f131407aec0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13380b40c0 0x7f133809c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13380b1f70 0x7f13380c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 --2- 172.21.15.31:0/2667755390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1338070140 0x7f133809d430 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 >> 172.21.15.31:0/2667755390 conn(0x7f13380ccd00 msgr2=0x7f133806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 shutdown_connections 2024-09-13T08:38:38.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:38.985+0000 7f133d600640 1 -- 172.21.15.31:0/2667755390 wait complete. 2024-09-13T08:38:39.006 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:38:39.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 -- 172.21.15.31:0/1441248393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b40cc190 msgr2=0x7f42b40cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 --2- 172.21.15.31:0/1441248393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b40cc190 0x7f42b40cdcc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005fd0 tx=0x7f42a002a4a0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 -- 172.21.15.31:0/1441248393 shutdown_connections 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 --2- 172.21.15.31:0/1441248393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b40cc190 0x7f42b40cdcc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 --2- 172.21.15.31:0/1441248393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b4161480 0x7f42b40cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 -- 172.21.15.31:0/1441248393 >> 172.21.15.31:0/1441248393 conn(0x7f42b40cfbb0 msgr2=0x7f42b40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 -- 172.21.15.31:0/1441248393 shutdown_connections 2024-09-13T08:38:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.175+0000 7f42ba600640 1 -- 172.21.15.31:0/1441248393 wait complete. 2024-09-13T08:38:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.177+0000 7f42ba600640 1 Processor -- start 2024-09-13T08:38:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.177+0000 7f42ba600640 1 -- start start 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42b40cbc40 0x7f42b40c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b40cc190 0x7f42b40c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42b40cb500 con 0x7f42b4161480 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42b40cb380 con 0x7f42b40cbc40 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42ba600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f42b40cb680 con 0x7f42b40cc190 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44672/0 (socket says 172.21.15.31:44672) 2024-09-13T08:38:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 -- 172.21.15.31:0/2290509542 learned_addr learned my addr 172.21.15.31:0/2290509542 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3400640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42b40cbc40 0x7f42b40c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b2a00640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b40cc190 0x7f42b40c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 -- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b40cc190 msgr2=0x7f42b40c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b40cc190 0x7f42b40c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 -- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42b40cbc40 msgr2=0x7f42b40c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42b40cbc40 0x7f42b40c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.178+0000 7f42b3e00640 1 -- 172.21.15.31:0/2290509542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b40c95a0 con 0x7f42b4161480 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42b3e00640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c780 tx=0x7f42ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac004b60 con 0x7f42b4161480 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42ac004df0 con 0x7f42b4161480 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac00aa90 con 0x7f42b4161480 2024-09-13T08:38:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b40c9880 con 0x7f42b4161480 2024-09-13T08:38:39.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.179+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b40c9e40 con 0x7f42b4161480 2024-09-13T08:38:39.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.180+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b41625f0 con 0x7f42b4161480 2024-09-13T08:38:39.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.181+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f42ac00ac30 con 0x7f42b4161480 2024-09-13T08:38:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.182+0000 7f42ab400640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 0x7f429407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.182+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f42ac09ae70 con 0x7f42b4161480 2024-09-13T08:38:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.183+0000 7f42b3400640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 0x7f429407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.183+0000 7f42b3400640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 0x7f429407aec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42b415a8e0 tx=0x7f42a4009060 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.187+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f42ac020070 con 0x7f42b4161480 2024-09-13T08:38:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.418+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f42b415eef0 con 0x7f42b4161480 2024-09-13T08:38:39.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.987+0000 7f42ab400640 1 -- 172.21.15.31:0/2290509542 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) ==== 122+0+0 (secure 0 0 0) 0x7f42ac062af0 con 0x7f42b4161480 2024-09-13T08:38:39.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 msgr2=0x7f429407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:39.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 0x7f429407aec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42b415a8e0 tx=0x7f42a4009060 comp rx=0 tx=0).stop 2024-09-13T08:38:39.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 msgr2=0x7f42b41f8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:39.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c780 tx=0x7f42ac00cc40 comp rx=0 tx=0).stop 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 shutdown_connections 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42b4161480 0x7f42b41f8080 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4294078a00 0x7f429407aec0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42b40cc190 0x7f42b40c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 --2- 172.21.15.31:0/2290509542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42b40cbc40 0x7f42b40c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 >> 172.21.15.31:0/2290509542 conn(0x7f42b40cfbb0 msgr2=0x7f42b415d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 shutdown_connections 2024-09-13T08:38:39.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:39.990+0000 7f42ba600640 1 -- 172.21.15.31:0/2290509542 wait complete. 2024-09-13T08:38:40.009 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:38:40.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 -- 172.21.15.31:0/3001754895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10d0003c60 msgr2=0x7f10d0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 --2- 172.21.15.31:0/3001754895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10d0003c60 0x7f10d0004060 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00c3c0 tx=0x7f10d0005830 comp rx=0 tx=0).stop 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10ede00640 1 -- 172.21.15.31:0/3001754895 reap_dead start 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 -- 172.21.15.31:0/3001754895 shutdown_connections 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 -- 172.21.15.31:0/3001754895 >> 172.21.15.31:0/3001754895 conn(0x7f10e80cc910 msgr2=0x7f10e80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 -- 172.21.15.31:0/3001754895 shutdown_connections 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.171+0000 7f10efe00640 1 -- 172.21.15.31:0/3001754895 wait complete. 2024-09-13T08:38:40.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.173+0000 7f10efe00640 1 Processor -- start 2024-09-13T08:38:40.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.173+0000 7f10efe00640 1 -- start start 2024-09-13T08:38:40.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f10d0003c60 0x7f10e80b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10e80c8790 0x7f10e80c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80cc6c0 con 0x7f10e8154740 2024-09-13T08:38:40.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80cc540 con 0x7f10e80c8790 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10efe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f10e8073780 con 0x7f10d0003c60 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10eca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10eca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44700/0 (socket says 172.21.15.31:44700) 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10eca00640 1 -- 172.21.15.31:0/3311743105 learned_addr learned my addr 172.21.15.31:0/3311743105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10ed400640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f10d0003c60 0x7f10e80b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.174+0000 7f10ede00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10e80c8790 0x7f10e80c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 -- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f10d0003c60 msgr2=0x7f10e80b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f10d0003c60 0x7f10e80b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 -- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10e80c8790 msgr2=0x7f10e80c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10e80c8790 0x7f10e80c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 -- 172.21.15.31:0/3311743105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e80b5b10 con 0x7f10e8154740 2024-09-13T08:38:40.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10eca00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f10d4004cc0 tx=0x7f10d4001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4015020 con 0x7f10e8154740 2024-09-13T08:38:40.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10d4011960 con 0x7f10e8154740 2024-09-13T08:38:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4011d70 con 0x7f10e8154740 2024-09-13T08:38:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e80c9460 con 0x7f10e8154740 2024-09-13T08:38:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.175+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e80c9a20 con 0x7f10e8154740 2024-09-13T08:38:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.177+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e8073ce0 con 0x7f10e8154740 2024-09-13T08:38:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.178+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f10d4018020 con 0x7f10e8154740 2024-09-13T08:38:40.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.178+0000 7f10e5600640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 0x7f10c807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.179+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f10d409b720 con 0x7f10e8154740 2024-09-13T08:38:40.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.179+0000 7f10ed400640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 0x7f10c807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.180+0000 7f10ed400640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 0x7f10c807aec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f10dc033000 tx=0x7f10dc02d070 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.183+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f10d401f020 con 0x7f10e8154740 2024-09-13T08:38:40.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.413+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10e80af5d0 con 0x7f10e8154740 2024-09-13T08:38:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.414+0000 7f10e5600640 1 -- 172.21.15.31:0/3311743105 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5223 (secure 0 0 0) 0x7f10d4063960 con 0x7f10e8154740 2024-09-13T08:38:40.415 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:40.415 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"btime":"2024-09-13T08:38:39:983238+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-09-13T08:38:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 msgr2=0x7f10c807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 0x7f10c807aec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f10dc033000 tx=0x7f10dc02d070 comp rx=0 tx=0).stop 2024-09-13T08:38:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 msgr2=0x7f10e81597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f10d4004cc0 tx=0x7f10d4001800 comp rx=0 tx=0).stop 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 shutdown_connections 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10e80c8790 0x7f10e80c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10e8154740 0x7f10e81597e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f10c8078a00 0x7f10c807aec0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 --2- 172.21.15.31:0/3311743105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f10d0003c60 0x7f10e80b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.416+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 >> 172.21.15.31:0/3311743105 conn(0x7f10e80cc910 msgr2=0x7f10e806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.417+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 shutdown_connections 2024-09-13T08:38:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.417+0000 7f10efe00640 1 -- 172.21.15.31:0/3311743105 wait complete. 2024-09-13T08:38:40.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.595+0000 7f1f3ca00640 1 -- 172.21.15.31:0/3987694116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f1c004d40 msgr2=0x7f1f1c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.595+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/3987694116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f1c004d40 0x7f1f1c0051c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009ed0 tx=0x7f1f1c005830 comp rx=0 tx=0).stop 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.595+0000 7f1f36800640 1 -- 172.21.15.31:0/3987694116 reap_dead start 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.595+0000 7f1f3ca00640 1 -- 172.21.15.31:0/3987694116 shutdown_connections 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.595+0000 7f1f3ca00640 1 -- 172.21.15.31:0/3987694116 >> 172.21.15.31:0/3987694116 conn(0x7f1f38074260 msgr2=0x7f1f38074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.596+0000 7f1f3ca00640 1 -- 172.21.15.31:0/3987694116 shutdown_connections 2024-09-13T08:38:40.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.596+0000 7f1f3ca00640 1 -- 172.21.15.31:0/3987694116 wait complete. 2024-09-13T08:38:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.597+0000 7f1f3ca00640 1 Processor -- start 2024-09-13T08:38:40.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.597+0000 7f1f3ca00640 1 -- start start 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f1c004d40 0x7f1f3806fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f380b58b0 0x7f1f380cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f380b1880 con 0x7f1f380cfda0 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f380b1700 con 0x7f1f380b58b0 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.598+0000 7f1f3ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1f380b1a00 con 0x7f1f1c004d40 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44724/0 (socket says 172.21.15.31:44724) 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 -- 172.21.15.31:0/2723340711 learned_addr learned my addr 172.21.15.31:0/2723340711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f35e00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f1c004d40 0x7f1f3806fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f2d400640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f380b58b0 0x7f1f380cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 -- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f1c004d40 msgr2=0x7f1f3806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f1c004d40 0x7f1f3806fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 -- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f380b58b0 msgr2=0x7f1f380cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f380b58b0 0x7f1f380cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 -- 172.21.15.31:0/2723340711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f380b5fd0 con 0x7f1f380cfda0 2024-09-13T08:38:40.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.599+0000 7f1f36800640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009ed0 tx=0x7f1f30033bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.600+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30043070 con 0x7f1f380cfda0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.600+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f30033ce0 con 0x7f1f380cfda0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.600+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30038040 con 0x7f1f380cfda0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.600+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f380b5d10 con 0x7f1f380cfda0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.600+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f380b51c0 con 0x7f1f380cfda0 2024-09-13T08:38:40.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.601+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f380008d0 con 0x7f1f380cfda0 2024-09-13T08:38:40.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.602+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f1f30046070 con 0x7f1f380cfda0 2024-09-13T08:38:40.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.603+0000 7f1f2ea00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 0x7f1f1407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.603+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f1f300ba7b0 con 0x7f1f380cfda0 2024-09-13T08:38:40.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.603+0000 7f1f35e00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 0x7f1f1407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.604+0000 7f1f35e00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 0x7f1f1407aec0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f28000fb0 tx=0x7f1f28006720 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.607+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1f30082450 con 0x7f1f380cfda0 2024-09-13T08:38:40.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.802+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f380afe00 con 0x7f1f380cfda0 2024-09-13T08:38:40.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.803+0000 7f1f2ea00640 1 -- 172.21.15.31:0/2723340711 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16621 (secure 0 0 0) 0x7f1f30086e80 con 0x7f1f380cfda0 2024-09-13T08:38:40.805 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:40.805 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:37.970201+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:38:34.198287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T08:38:35.374739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 msgr2=0x7f1f1407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 0x7f1f1407aec0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f28000fb0 tx=0x7f1f28006720 comp rx=0 tx=0).stop 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 msgr2=0x7f1f380b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009ed0 tx=0x7f1f30033bf0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 shutdown_connections 2024-09-13T08:38:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f380cfda0 0x7f1f380b20a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f380b58b0 0x7f1f380cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1f14078a00 0x7f1f1407aec0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 --2- 172.21.15.31:0/2723340711 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f1c004d40 0x7f1f3806fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 >> 172.21.15.31:0/2723340711 conn(0x7f1f38074260 msgr2=0x7f1f3815b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 shutdown_connections 2024-09-13T08:38:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.806+0000 7f1f3ca00640 1 -- 172.21.15.31:0/2723340711 wait complete. 2024-09-13T08:38:40.823 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:38:40.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:38:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 -- 172.21.15.31:0/1717627516 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 msgr2=0x7f42780cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 --2- 172.21.15.31:0/1717627516 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f42780cf850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4268009830 tx=0x7f426802a510 comp rx=0 tx=0).stop 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 -- 172.21.15.31:0/1717627516 shutdown_connections 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 --2- 172.21.15.31:0/1717627516 >> v2:172.21.15.45:3300/0 conn(0x7f427815f9e0 0x7f427815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 --2- 172.21.15.31:0/1717627516 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f42780cf850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 -- 172.21.15.31:0/1717627516 >> 172.21.15.31:0/1717627516 conn(0x7f42780cc910 msgr2=0x7f42780cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.985+0000 7f427ea00640 1 -- 172.21.15.31:0/1717627516 shutdown_connections 2024-09-13T08:38:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.986+0000 7f427ea00640 1 -- 172.21.15.31:0/1717627516 wait complete. 2024-09-13T08:38:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.987+0000 7f427ea00640 1 Processor -- start 2024-09-13T08:38:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.987+0000 7f427ea00640 1 -- start start 2024-09-13T08:38:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f427815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f427815f9e0 0x7f427809ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42780cb9d0 con 0x7f42780ce3c0 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42780cb850 con 0x7f427815f9e0 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f427ea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f42780cc360 con 0x7f427815bd70 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f4277e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f4277e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44734/0 (socket says 172.21.15.31:44734) 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.988+0000 7f4277e00640 1 -- 172.21.15.31:0/749466070 learned_addr learned my addr 172.21.15.31:0/749466070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f427ca00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f427815f9e0 0x7f427809ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f426f400640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f427815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 -- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 msgr2=0x7f427815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f427815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 -- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f427815f9e0 msgr2=0x7f427809ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f427815f9e0 0x7f427809ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 -- 172.21.15.31:0/749466070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42780c3b40 con 0x7f42780ce3c0 2024-09-13T08:38:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.989+0000 7f4277e00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4268009950 tx=0x7f4268032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.991+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268040070 con 0x7f42780ce3c0 2024-09-13T08:38:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.991+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4268005310 con 0x7f42780ce3c0 2024-09-13T08:38:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.991+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268005c90 con 0x7f42780ce3c0 2024-09-13T08:38:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.991+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427809d930 con 0x7f42780ce3c0 2024-09-13T08:38:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.991+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427809c8b0 con 0x7f42780ce3c0 2024-09-13T08:38:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.993+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42780cf850 con 0x7f42780ce3c0 2024-09-13T08:38:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.994+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f4268039040 con 0x7f42780ce3c0 2024-09-13T08:38:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.994+0000 7f4274c00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 0x7f425407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.995+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f42680bbb10 con 0x7f42780ce3c0 2024-09-13T08:38:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.995+0000 7f426f400640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 0x7f425407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.996+0000 7f426f400640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 0x7f425407aec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f42600095f0 tx=0x7f4260005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:40.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:40.999+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4268083830 con 0x7f42780ce3c0 2024-09-13T08:38:41.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.190+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f427806ac50 con 0x7f4254078a00 2024-09-13T08:38:41.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.196+0000 7f4274c00640 1 -- 172.21.15.31:0/749466070 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f427806ac50 con 0x7f4254078a00 2024-09-13T08:38:41.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:38:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 msgr2=0x7f425407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 0x7f425407aec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f42600095f0 tx=0x7f4260005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 msgr2=0x7f42780c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4268009950 tx=0x7f4268032c10 comp rx=0 tx=0).stop 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 shutdown_connections 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f427815f9e0 0x7f427809ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4254078a00 0x7f425407aec0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f427815bd70 0x7f427815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 --2- 172.21.15.31:0/749466070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42780ce3c0 0x7f42780c51f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 >> 172.21.15.31:0/749466070 conn(0x7f42780cc910 msgr2=0x7f42780b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 shutdown_connections 2024-09-13T08:38:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.200+0000 7f427ea00640 1 -- 172.21.15.31:0/749466070 wait complete. 2024-09-13T08:38:41.216 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-13T08:38:41.057239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238336,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736384,"statfs":{"total":773094113280,"available":772850057216,"internally_reserved":0,"allocated":5578752,"data_stored":2924629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001459"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310452+0000","last_change":"2024-09-13T08:38:37.001717+0000","last_active":"2024-09-13T08:38:38.310452+0000","last_peered":"2024-09-13T08:38:38.310452+0000","last_clean":"2024-09-13T08:38:38.310452+0000","last_became_active":"2024-09-13T08:38:37.001417+0000","last_became_peered":"2024-09-13T08:38:37.001417+0000","last_unstale":"2024-09-13T08:38:38.310452+0000","last_undegraded":"2024-09-13T08:38:38.310452+0000","last_fullsized":"2024-09-13T08:38:38.310452+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:10.615827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310140+0000","last_change":"2024-09-13T08:38:36.982983+0000","last_active":"2024-09-13T08:38:38.310140+0000","last_peered":"2024-09-13T08:38:38.310140+0000","last_clean":"2024-09-13T08:38:38.310140+0000","last_became_active":"2024-09-13T08:38:36.982706+0000","last_became_peered":"2024-09-13T08:38:36.982706+0000","last_unstale":"2024-09-13T08:38:38.310140+0000","last_undegraded":"2024-09-13T08:38:38.310140+0000","last_fullsized":"2024-09-13T08:38:38.310140+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309206+0000","last_change":"2024-09-13T08:38:36.980093+0000","last_active":"2024-09-13T08:38:38.309206+0000","last_peered":"2024-09-13T08:38:38.309206+0000","last_clean":"2024-09-13T08:38:38.309206+0000","last_became_active":"2024-09-13T08:38:36.976661+0000","last_became_peered":"2024-09-13T08:38:36.976661+0000","last_unstale":"2024-09-13T08:38:38.309206+0000","last_undegraded":"2024-09-13T08:38:38.309206+0000","last_fullsized":"2024-09-13T08:38:38.309206+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:30.442843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309369+0000","last_change":"2024-09-13T08:38:36.991627+0000","last_active":"2024-09-13T08:38:38.309369+0000","last_peered":"2024-09-13T08:38:38.309369+0000","last_clean":"2024-09-13T08:38:38.309369+0000","last_became_active":"2024-09-13T08:38:36.991331+0000","last_became_peered":"2024-09-13T08:38:36.991331+0000","last_unstale":"2024-09-13T08:38:38.309369+0000","last_undegraded":"2024-09-13T08:38:38.309369+0000","last_fullsized":"2024-09-13T08:38:38.309369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:29.953483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309637+0000","last_change":"2024-09-13T08:38:36.994148+0000","last_active":"2024-09-13T08:38:38.309637+0000","last_peered":"2024-09-13T08:38:38.309637+0000","last_clean":"2024-09-13T08:38:38.309637+0000","last_became_active":"2024-09-13T08:38:36.993838+0000","last_became_peered":"2024-09-13T08:38:36.993838+0000","last_unstale":"2024-09-13T08:38:38.309637+0000","last_undegraded":"2024-09-13T08:38:38.309637+0000","last_fullsized":"2024-09-13T08:38:38.309637+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:16.831423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277595+0000","last_change":"2024-09-13T08:38:36.994944+0000","last_active":"2024-09-13T08:38:38.277595+0000","last_peered":"2024-09-13T08:38:38.277595+0000","last_clean":"2024-09-13T08:38:38.277595+0000","last_became_active":"2024-09-13T08:38:36.994637+0000","last_became_peered":"2024-09-13T08:38:36.994637+0000","last_unstale":"2024-09-13T08:38:38.277595+0000","last_undegraded":"2024-09-13T08:38:38.277595+0000","last_fullsized":"2024-09-13T08:38:38.277595+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:31.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977522+0000","last_change":"2024-09-13T08:38:36.992617+0000","last_active":"2024-09-13T08:38:37.977522+0000","last_peered":"2024-09-13T08:38:37.977522+0000","last_clean":"2024-09-13T08:38:37.977522+0000","last_became_active":"2024-09-13T08:38:36.991986+0000","last_became_peered":"2024-09-13T08:38:36.991986+0000","last_unstale":"2024-09-13T08:38:37.977522+0000","last_undegraded":"2024-09-13T08:38:37.977522+0000","last_fullsized":"2024-09-13T08:38:37.977522+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:59:54.439228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308970+0000","last_change":"2024-09-13T08:38:36.987688+0000","last_active":"2024-09-13T08:38:38.308970+0000","last_peered":"2024-09-13T08:38:38.308970+0000","last_clean":"2024-09-13T08:38:38.308970+0000","last_became_active":"2024-09-13T08:38:36.986338+0000","last_became_peered":"2024-09-13T08:38:36.986338+0000","last_unstale":"2024-09-13T08:38:38.308970+0000","last_undegraded":"2024-09-13T08:38:38.308970+0000","last_fullsized":"2024-09-13T08:38:38.308970+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:33.875455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277819+0000","last_change":"2024-09-13T08:38:36.984422+0000","last_active":"2024-09-13T08:38:38.277819+0000","last_peered":"2024-09-13T08:38:38.277819+0000","last_clean":"2024-09-13T08:38:38.277819+0000","last_became_active":"2024-09-13T08:38:36.984123+0000","last_became_peered":"2024-09-13T08:38:36.984123+0000","last_unstale":"2024-09-13T08:38:38.277819+0000","last_undegraded":"2024-09-13T08:38:38.277819+0000","last_fullsized":"2024-09-13T08:38:38.277819+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:27.393269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.982556+0000","last_change":"2024-09-13T08:38:36.980412+0000","last_active":"2024-09-13T08:38:36.982556+0000","last_peered":"2024-09-13T08:38:36.982556+0000","last_clean":"2024-09-13T08:38:36.982556+0000","last_became_active":"2024-09-13T08:38:36.980088+0000","last_became_peered":"2024-09-13T08:38:36.980088+0000","last_unstale":"2024-09-13T08:38:36.982556+0000","last_undegraded":"2024-09-13T08:38:36.982556+0000","last_fullsized":"2024-09-13T08:38:36.982556+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:02.764241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309445+0000","last_change":"2024-09-13T08:38:36.991525+0000","last_active":"2024-09-13T08:38:38.309445+0000","last_peered":"2024-09-13T08:38:38.309445+0000","last_clean":"2024-09-13T08:38:38.309445+0000","last_became_active":"2024-09-13T08:38:36.990939+0000","last_became_peered":"2024-09-13T08:38:36.990939+0000","last_unstale":"2024-09-13T08:38:38.309445+0000","last_undegraded":"2024-09-13T08:38:38.309445+0000","last_fullsized":"2024-09-13T08:38:38.309445+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:54.057235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309706+0000","last_change":"2024-09-13T08:38:36.979115+0000","last_active":"2024-09-13T08:38:38.309706+0000","last_peered":"2024-09-13T08:38:38.309706+0000","last_clean":"2024-09-13T08:38:38.309706+0000","last_became_active":"2024-09-13T08:38:36.978828+0000","last_became_peered":"2024-09-13T08:38:36.978828+0000","last_unstale":"2024-09-13T08:38:38.309706+0000","last_undegraded":"2024-09-13T08:38:38.309706+0000","last_fullsized":"2024-09-13T08:38:38.309706+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:16.831423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277485+0000","last_change":"2024-09-13T08:38:36.989066+0000","last_active":"2024-09-13T08:38:38.277485+0000","last_peered":"2024-09-13T08:38:38.277485+0000","last_clean":"2024-09-13T08:38:38.277485+0000","last_became_active":"2024-09-13T08:38:36.988720+0000","last_became_peered":"2024-09-13T08:38:36.988720+0000","last_unstale":"2024-09-13T08:38:38.277485+0000","last_undegraded":"2024-09-13T08:38:38.277485+0000","last_fullsized":"2024-09-13T08:38:38.277485+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:34.821199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309077+0000","last_change":"2024-09-13T08:38:36.986743+0000","last_active":"2024-09-13T08:38:38.309077+0000","last_peered":"2024-09-13T08:38:38.309077+0000","last_clean":"2024-09-13T08:38:38.309077+0000","last_became_active":"2024-09-13T08:38:36.986307+0000","last_became_peered":"2024-09-13T08:38:36.986307+0000","last_unstale":"2024-09-13T08:38:38.309077+0000","last_undegraded":"2024-09-13T08:38:38.309077+0000","last_fullsized":"2024-09-13T08:38:38.309077+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309002+0000","last_change":"2024-09-13T08:38:36.993262+0000","last_active":"2024-09-13T08:38:38.309002+0000","last_peered":"2024-09-13T08:38:38.309002+0000","last_clean":"2024-09-13T08:38:38.309002+0000","last_became_active":"2024-09-13T08:38:36.992919+0000","last_became_peered":"2024-09-13T08:38:36.992919+0000","last_unstale":"2024-09-13T08:38:38.309002+0000","last_undegraded":"2024-09-13T08:38:38.309002+0000","last_fullsized":"2024-09-13T08:38:38.309002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:52.991190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309351+0000","last_change":"2024-09-13T08:38:36.995098+0000","last_active":"2024-09-13T08:38:38.309351+0000","last_peered":"2024-09-13T08:38:38.309351+0000","last_clean":"2024-09-13T08:38:38.309351+0000","last_became_active":"2024-09-13T08:38:36.992649+0000","last_became_peered":"2024-09-13T08:38:36.992649+0000","last_unstale":"2024-09-13T08:38:38.309351+0000","last_undegraded":"2024-09-13T08:38:38.309351+0000","last_fullsized":"2024-09-13T08:38:38.309351+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:12.768061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310264+0000","last_change":"2024-09-13T08:38:36.991658+0000","last_active":"2024-09-13T08:38:38.310264+0000","last_peered":"2024-09-13T08:38:38.310264+0000","last_clean":"2024-09-13T08:38:38.310264+0000","last_became_active":"2024-09-13T08:38:36.991322+0000","last_became_peered":"2024-09-13T08:38:36.991322+0000","last_unstale":"2024-09-13T08:38:38.310264+0000","last_undegraded":"2024-09-13T08:38:38.310264+0000","last_fullsized":"2024-09-13T08:38:38.310264+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:10.615827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309262+0000","last_change":"2024-09-13T08:38:36.987380+0000","last_active":"2024-09-13T08:38:38.309262+0000","last_peered":"2024-09-13T08:38:38.309262+0000","last_clean":"2024-09-13T08:38:38.309262+0000","last_became_active":"2024-09-13T08:38:36.987139+0000","last_became_peered":"2024-09-13T08:38:36.987139+0000","last_unstale":"2024-09-13T08:38:38.309262+0000","last_undegraded":"2024-09-13T08:38:38.309262+0000","last_fullsized":"2024-09-13T08:38:38.309262+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:45.406855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310298+0000","last_change":"2024-09-13T08:38:36.992766+0000","last_active":"2024-09-13T08:38:38.310298+0000","last_peered":"2024-09-13T08:38:38.310298+0000","last_clean":"2024-09-13T08:38:38.310298+0000","last_became_active":"2024-09-13T08:38:36.989989+0000","last_became_peered":"2024-09-13T08:38:36.989989+0000","last_unstale":"2024-09-13T08:38:38.310298+0000","last_undegraded":"2024-09-13T08:38:38.310298+0000","last_fullsized":"2024-09-13T08:38:38.310298+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:53.221145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.981634+0000","last_change":"2024-09-13T08:38:36.972726+0000","last_active":"2024-09-13T08:38:36.981634+0000","last_peered":"2024-09-13T08:38:36.981634+0000","last_clean":"2024-09-13T08:38:36.981634+0000","last_became_active":"2024-09-13T08:38:36.972471+0000","last_became_peered":"2024-09-13T08:38:36.972471+0000","last_unstale":"2024-09-13T08:38:36.981634+0000","last_undegraded":"2024-09-13T08:38:36.981634+0000","last_fullsized":"2024-09-13T08:38:36.981634+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:14.854884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309683+0000","last_change":"2024-09-13T08:38:36.983169+0000","last_active":"2024-09-13T08:38:38.309683+0000","last_peered":"2024-09-13T08:38:38.309683+0000","last_clean":"2024-09-13T08:38:38.309683+0000","last_became_active":"2024-09-13T08:38:36.982575+0000","last_became_peered":"2024-09-13T08:38:36.982575+0000","last_unstale":"2024-09-13T08:38:38.309683+0000","last_undegraded":"2024-09-13T08:38:38.309683+0000","last_fullsized":"2024-09-13T08:38:38.309683+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:45.289837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978446+0000","last_change":"2024-09-13T08:38:36.969942+0000","last_active":"2024-09-13T08:38:37.978446+0000","last_peered":"2024-09-13T08:38:37.978446+0000","last_clean":"2024-09-13T08:38:37.978446+0000","last_became_active":"2024-09-13T08:38:36.969649+0000","last_became_peered":"2024-09-13T08:38:36.969649+0000","last_unstale":"2024-09-13T08:38:37.978446+0000","last_undegraded":"2024-09-13T08:38:37.978446+0000","last_fullsized":"2024-09-13T08:38:37.978446+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:01.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977554+0000","last_change":"2024-09-13T08:38:36.992980+0000","last_active":"2024-09-13T08:38:37.977554+0000","last_peered":"2024-09-13T08:38:37.977554+0000","last_clean":"2024-09-13T08:38:37.977554+0000","last_became_active":"2024-09-13T08:38:36.992339+0000","last_became_peered":"2024-09-13T08:38:36.992339+0000","last_unstale":"2024-09-13T08:38:37.977554+0000","last_undegraded":"2024-09-13T08:38:37.977554+0000","last_fullsized":"2024-09-13T08:38:37.977554+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:56.392500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309013+0000","last_change":"2024-09-13T08:38:36.991547+0000","last_active":"2024-09-13T08:38:38.309013+0000","last_peered":"2024-09-13T08:38:38.309013+0000","last_clean":"2024-09-13T08:38:38.309013+0000","last_became_active":"2024-09-13T08:38:36.991253+0000","last_became_peered":"2024-09-13T08:38:36.991253+0000","last_unstale":"2024-09-13T08:38:38.309013+0000","last_undegraded":"2024-09-13T08:38:38.309013+0000","last_fullsized":"2024-09-13T08:38:38.309013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:17.801741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310427+0000","last_change":"2024-09-13T08:38:37.001718+0000","last_active":"2024-09-13T08:38:38.310427+0000","last_peered":"2024-09-13T08:38:38.310427+0000","last_clean":"2024-09-13T08:38:38.310427+0000","last_became_active":"2024-09-13T08:38:37.001378+0000","last_became_peered":"2024-09-13T08:38:37.001378+0000","last_unstale":"2024-09-13T08:38:38.310427+0000","last_undegraded":"2024-09-13T08:38:38.310427+0000","last_fullsized":"2024-09-13T08:38:38.310427+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309350+0000","last_change":"2024-09-13T08:38:36.990170+0000","last_active":"2024-09-13T08:38:38.309350+0000","last_peered":"2024-09-13T08:38:38.309350+0000","last_clean":"2024-09-13T08:38:38.309350+0000","last_became_active":"2024-09-13T08:38:36.989751+0000","last_became_peered":"2024-09-13T08:38:36.989751+0000","last_unstale":"2024-09-13T08:38:38.309350+0000","last_undegraded":"2024-09-13T08:38:38.309350+0000","last_fullsized":"2024-09-13T08:38:38.309350+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:54.329849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978009+0000","last_change":"2024-09-13T08:38:37.002661+0000","last_active":"2024-09-13T08:38:37.978009+0000","last_peered":"2024-09-13T08:38:37.978009+0000","last_clean":"2024-09-13T08:38:37.978009+0000","last_became_active":"2024-09-13T08:38:37.002380+0000","last_became_peered":"2024-09-13T08:38:37.002380+0000","last_unstale":"2024-09-13T08:38:37.978009+0000","last_undegraded":"2024-09-13T08:38:37.978009+0000","last_fullsized":"2024-09-13T08:38:37.978009+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:09.877487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977949+0000","last_change":"2024-09-13T08:38:36.987371+0000","last_active":"2024-09-13T08:38:37.977949+0000","last_peered":"2024-09-13T08:38:37.977949+0000","last_clean":"2024-09-13T08:38:37.977949+0000","last_became_active":"2024-09-13T08:38:36.987046+0000","last_became_peered":"2024-09-13T08:38:36.987046+0000","last_unstale":"2024-09-13T08:38:37.977949+0000","last_undegraded":"2024-09-13T08:38:37.977949+0000","last_fullsized":"2024-09-13T08:38:37.977949+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:04.546377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309829+0000","last_change":"2024-09-13T08:38:36.996341+0000","last_active":"2024-09-13T08:38:38.309829+0000","last_peered":"2024-09-13T08:38:38.309829+0000","last_clean":"2024-09-13T08:38:38.309829+0000","last_became_active":"2024-09-13T08:38:36.990607+0000","last_became_peered":"2024-09-13T08:38:36.990607+0000","last_unstale":"2024-09-13T08:38:38.309829+0000","last_undegraded":"2024-09-13T08:38:38.309829+0000","last_fullsized":"2024-09-13T08:38:38.309829+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309239+0000","last_change":"2024-09-13T08:38:36.989533+0000","last_active":"2024-09-13T08:38:38.309239+0000","last_peered":"2024-09-13T08:38:38.309239+0000","last_clean":"2024-09-13T08:38:38.309239+0000","last_became_active":"2024-09-13T08:38:36.989291+0000","last_became_peered":"2024-09-13T08:38:36.989291+0000","last_unstale":"2024-09-13T08:38:38.309239+0000","last_undegraded":"2024-09-13T08:38:38.309239+0000","last_fullsized":"2024-09-13T08:38:38.309239+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:12.768061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277465+0000","last_change":"2024-09-13T08:38:36.991591+0000","last_active":"2024-09-13T08:38:38.277465+0000","last_peered":"2024-09-13T08:38:38.277465+0000","last_clean":"2024-09-13T08:38:38.277465+0000","last_became_active":"2024-09-13T08:38:36.991318+0000","last_became_peered":"2024-09-13T08:38:36.991318+0000","last_unstale":"2024-09-13T08:38:38.277465+0000","last_undegraded":"2024-09-13T08:38:38.277465+0000","last_fullsized":"2024-09-13T08:38:38.277465+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:21.688201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309010+0000","last_change":"2024-09-13T08:38:36.986408+0000","last_active":"2024-09-13T08:38:38.309010+0000","last_peered":"2024-09-13T08:38:38.309010+0000","last_clean":"2024-09-13T08:38:38.309010+0000","last_became_active":"2024-09-13T08:38:36.983578+0000","last_became_peered":"2024-09-13T08:38:36.983578+0000","last_unstale":"2024-09-13T08:38:38.309010+0000","last_undegraded":"2024-09-13T08:38:38.309010+0000","last_fullsized":"2024-09-13T08:38:38.309010+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979280+0000","last_change":"2024-09-13T08:38:36.972913+0000","last_active":"2024-09-13T08:38:37.979280+0000","last_peered":"2024-09-13T08:38:37.979280+0000","last_clean":"2024-09-13T08:38:37.979280+0000","last_became_active":"2024-09-13T08:38:36.972673+0000","last_became_peered":"2024-09-13T08:38:36.972673+0000","last_unstale":"2024-09-13T08:38:37.979280+0000","last_undegraded":"2024-09-13T08:38:37.979280+0000","last_fullsized":"2024-09-13T08:38:37.979280+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:39.966579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277800+0000","last_change":"2024-09-13T08:38:35.976185+0000","last_active":"2024-09-13T08:38:38.277800+0000","last_peered":"2024-09-13T08:38:38.277800+0000","last_clean":"2024-09-13T08:38:38.277800+0000","last_became_active":"2024-09-13T08:38:35.975896+0000","last_became_peered":"2024-09-13T08:38:35.975896+0000","last_unstale":"2024-09-13T08:38:38.277800+0000","last_undegraded":"2024-09-13T08:38:38.277800+0000","last_fullsized":"2024-09-13T08:38:38.277800+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:35.170683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310531+0000","last_change":"2024-09-13T08:38:36.990308+0000","last_active":"2024-09-13T08:38:38.310531+0000","last_peered":"2024-09-13T08:38:38.310531+0000","last_clean":"2024-09-13T08:38:38.310531+0000","last_became_active":"2024-09-13T08:38:36.989885+0000","last_became_peered":"2024-09-13T08:38:36.989885+0000","last_unstale":"2024-09-13T08:38:38.310531+0000","last_undegraded":"2024-09-13T08:38:38.310531+0000","last_fullsized":"2024-09-13T08:38:38.310531+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:32:11.072827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977587+0000","last_change":"2024-09-13T08:38:35.982559+0000","last_active":"2024-09-13T08:38:37.977587+0000","last_peered":"2024-09-13T08:38:37.977587+0000","last_clean":"2024-09-13T08:38:37.977587+0000","last_became_active":"2024-09-13T08:38:35.980833+0000","last_became_peered":"2024-09-13T08:38:35.980833+0000","last_unstale":"2024-09-13T08:38:37.977587+0000","last_undegraded":"2024-09-13T08:38:37.977587+0000","last_fullsized":"2024-09-13T08:38:37.977587+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:41.245329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309951+0000","last_change":"2024-09-13T08:38:36.983729+0000","last_active":"2024-09-13T08:38:38.309951+0000","last_peered":"2024-09-13T08:38:38.309951+0000","last_clean":"2024-09-13T08:38:38.309951+0000","last_became_active":"2024-09-13T08:38:36.983394+0000","last_became_peered":"2024-09-13T08:38:36.983394+0000","last_unstale":"2024-09-13T08:38:38.309951+0000","last_undegraded":"2024-09-13T08:38:38.309951+0000","last_fullsized":"2024-09-13T08:38:38.309951+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277577+0000","last_change":"2024-09-13T08:38:35.972025+0000","last_active":"2024-09-13T08:38:38.277577+0000","last_peered":"2024-09-13T08:38:38.277577+0000","last_clean":"2024-09-13T08:38:38.277577+0000","last_became_active":"2024-09-13T08:38:35.971648+0000","last_became_peered":"2024-09-13T08:38:35.971648+0000","last_unstale":"2024-09-13T08:38:38.277577+0000","last_undegraded":"2024-09-13T08:38:38.277577+0000","last_fullsized":"2024-09-13T08:38:38.277577+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:10.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309663+0000","last_change":"2024-09-13T08:38:36.991221+0000","last_active":"2024-09-13T08:38:38.309663+0000","last_peered":"2024-09-13T08:38:38.309663+0000","last_clean":"2024-09-13T08:38:38.309663+0000","last_became_active":"2024-09-13T08:38:36.990544+0000","last_became_peered":"2024-09-13T08:38:36.990544+0000","last_unstale":"2024-09-13T08:38:38.309663+0000","last_undegraded":"2024-09-13T08:38:38.309663+0000","last_fullsized":"2024-09-13T08:38:38.309663+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:20.944856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978065+0000","last_change":"2024-09-13T08:38:35.980735+0000","last_active":"2024-09-13T08:38:37.978065+0000","last_peered":"2024-09-13T08:38:37.978065+0000","last_clean":"2024-09-13T08:38:37.978065+0000","last_became_active":"2024-09-13T08:38:35.980417+0000","last_became_peered":"2024-09-13T08:38:35.980417+0000","last_unstale":"2024-09-13T08:38:37.978065+0000","last_undegraded":"2024-09-13T08:38:37.978065+0000","last_fullsized":"2024-09-13T08:38:37.978065+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:19.935473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977824+0000","last_change":"2024-09-13T08:38:36.998184+0000","last_active":"2024-09-13T08:38:37.977824+0000","last_peered":"2024-09-13T08:38:37.977824+0000","last_clean":"2024-09-13T08:38:37.977824+0000","last_became_active":"2024-09-13T08:38:36.997768+0000","last_became_peered":"2024-09-13T08:38:36.997768+0000","last_unstale":"2024-09-13T08:38:37.977824+0000","last_undegraded":"2024-09-13T08:38:37.977824+0000","last_fullsized":"2024-09-13T08:38:37.977824+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:31.035669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277680+0000","last_change":"2024-09-13T08:38:35.971969+0000","last_active":"2024-09-13T08:38:38.277680+0000","last_peered":"2024-09-13T08:38:38.277680+0000","last_clean":"2024-09-13T08:38:38.277680+0000","last_became_active":"2024-09-13T08:38:35.971551+0000","last_became_peered":"2024-09-13T08:38:35.971551+0000","last_unstale":"2024-09-13T08:38:38.277680+0000","last_undegraded":"2024-09-13T08:38:38.277680+0000","last_fullsized":"2024-09-13T08:38:38.277680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:01.920028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.992719+0000","last_change":"2024-09-13T08:38:36.985088+0000","last_active":"2024-09-13T08:38:36.992719+0000","last_peered":"2024-09-13T08:38:36.992719+0000","last_clean":"2024-09-13T08:38:36.992719+0000","last_became_active":"2024-09-13T08:38:36.984815+0000","last_became_peered":"2024-09-13T08:38:36.984815+0000","last_unstale":"2024-09-13T08:38:36.992719+0000","last_undegraded":"2024-09-13T08:38:36.992719+0000","last_fullsized":"2024-09-13T08:38:36.992719+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:39.810743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977550+0000","last_change":"2024-09-13T08:38:35.982396+0000","last_active":"2024-09-13T08:38:37.977550+0000","last_peered":"2024-09-13T08:38:37.977550+0000","last_clean":"2024-09-13T08:38:37.977550+0000","last_became_active":"2024-09-13T08:38:35.981259+0000","last_became_peered":"2024-09-13T08:38:35.981259+0000","last_unstale":"2024-09-13T08:38:37.977550+0000","last_undegraded":"2024-09-13T08:38:37.977550+0000","last_fullsized":"2024-09-13T08:38:37.977550+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:53.047851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.995997+0000","last_change":"2024-09-13T08:38:36.987818+0000","last_active":"2024-09-13T08:38:36.995997+0000","last_peered":"2024-09-13T08:38:36.995997+0000","last_clean":"2024-09-13T08:38:36.995997+0000","last_became_active":"2024-09-13T08:38:36.987556+0000","last_became_peered":"2024-09-13T08:38:36.987556+0000","last_unstale":"2024-09-13T08:38:36.995997+0000","last_undegraded":"2024-09-13T08:38:36.995997+0000","last_fullsized":"2024-09-13T08:38:36.995997+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:12:22.223773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309220+0000","last_change":"2024-09-13T08:38:35.980406+0000","last_active":"2024-09-13T08:38:38.309220+0000","last_peered":"2024-09-13T08:38:38.309220+0000","last_clean":"2024-09-13T08:38:38.309220+0000","last_became_active":"2024-09-13T08:38:35.973870+0000","last_became_peered":"2024-09-13T08:38:35.973870+0000","last_unstale":"2024-09-13T08:38:38.309220+0000","last_undegraded":"2024-09-13T08:38:38.309220+0000","last_fullsized":"2024-09-13T08:38:38.309220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977761+0000","last_change":"2024-09-13T08:38:36.989200+0000","last_active":"2024-09-13T08:38:37.977761+0000","last_peered":"2024-09-13T08:38:37.977761+0000","last_clean":"2024-09-13T08:38:37.977761+0000","last_became_active":"2024-09-13T08:38:36.987601+0000","last_became_peered":"2024-09-13T08:38:36.987601+0000","last_unstale":"2024-09-13T08:38:37.977761+0000","last_undegraded":"2024-09-13T08:38:37.977761+0000","last_fullsized":"2024-09-13T08:38:37.977761+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:25.936306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.965600+0000","last_change":"2024-09-13T08:38:35.982758+0000","last_active":"2024-09-13T08:38:36.965600+0000","last_peered":"2024-09-13T08:38:36.965600+0000","last_clean":"2024-09-13T08:38:36.965600+0000","last_became_active":"2024-09-13T08:38:35.982164+0000","last_became_peered":"2024-09-13T08:38:35.982164+0000","last_unstale":"2024-09-13T08:38:36.965600+0000","last_undegraded":"2024-09-13T08:38:36.965600+0000","last_fullsized":"2024-09-13T08:38:36.965600+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:57:04.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309335+0000","last_change":"2024-09-13T08:38:36.969592+0000","last_active":"2024-09-13T08:38:38.309335+0000","last_peered":"2024-09-13T08:38:38.309335+0000","last_clean":"2024-09-13T08:38:38.309335+0000","last_became_active":"2024-09-13T08:38:36.969303+0000","last_became_peered":"2024-09-13T08:38:36.969303+0000","last_unstale":"2024-09-13T08:38:38.309335+0000","last_undegraded":"2024-09-13T08:38:38.309335+0000","last_fullsized":"2024-09-13T08:38:38.309335+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:18.930654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309142+0000","last_change":"2024-09-13T08:38:35.972396+0000","last_active":"2024-09-13T08:38:38.309142+0000","last_peered":"2024-09-13T08:38:38.309142+0000","last_clean":"2024-09-13T08:38:38.309142+0000","last_became_active":"2024-09-13T08:38:35.972103+0000","last_became_peered":"2024-09-13T08:38:35.972103+0000","last_unstale":"2024-09-13T08:38:38.309142+0000","last_undegraded":"2024-09-13T08:38:38.309142+0000","last_fullsized":"2024-09-13T08:38:38.309142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:17.203949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309450+0000","last_change":"2024-09-13T08:38:36.992997+0000","last_active":"2024-09-13T08:38:38.309450+0000","last_peered":"2024-09-13T08:38:38.309450+0000","last_clean":"2024-09-13T08:38:38.309450+0000","last_became_active":"2024-09-13T08:38:36.992729+0000","last_became_peered":"2024-09-13T08:38:36.992729+0000","last_unstale":"2024-09-13T08:38:38.309450+0000","last_undegraded":"2024-09-13T08:38:38.309450+0000","last_fullsized":"2024-09-13T08:38:38.309450+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:18.297013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.965473+0000","last_change":"2024-09-13T08:38:35.982428+0000","last_active":"2024-09-13T08:38:36.965473+0000","last_peered":"2024-09-13T08:38:36.965473+0000","last_clean":"2024-09-13T08:38:36.965473+0000","last_became_active":"2024-09-13T08:38:35.981773+0000","last_became_peered":"2024-09-13T08:38:35.981773+0000","last_unstale":"2024-09-13T08:38:36.965473+0000","last_undegraded":"2024-09-13T08:38:36.965473+0000","last_fullsized":"2024-09-13T08:38:36.965473+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:12.594169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309354+0000","last_change":"2024-09-13T08:38:36.989498+0000","last_active":"2024-09-13T08:38:38.309354+0000","last_peered":"2024-09-13T08:38:38.309354+0000","last_clean":"2024-09-13T08:38:38.309354+0000","last_became_active":"2024-09-13T08:38:36.985877+0000","last_became_peered":"2024-09-13T08:38:36.985877+0000","last_unstale":"2024-09-13T08:38:38.309354+0000","last_undegraded":"2024-09-13T08:38:38.309354+0000","last_fullsized":"2024-09-13T08:38:38.309354+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:20.539807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":18,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309873+0000","last_change":"2024-09-13T08:38:35.971378+0000","last_active":"2024-09-13T08:38:38.309873+0000","last_peered":"2024-09-13T08:38:38.309873+0000","last_clean":"2024-09-13T08:38:38.309873+0000","last_became_active":"2024-09-13T08:38:35.971078+0000","last_became_peered":"2024-09-13T08:38:35.971078+0000","last_unstale":"2024-09-13T08:38:38.309873+0000","last_undegraded":"2024-09-13T08:38:38.309873+0000","last_fullsized":"2024-09-13T08:38:38.309873+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:19.825456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977964+0000","last_change":"2024-09-13T08:38:36.995852+0000","last_active":"2024-09-13T08:38:37.977964+0000","last_peered":"2024-09-13T08:38:37.977964+0000","last_clean":"2024-09-13T08:38:37.977964+0000","last_became_active":"2024-09-13T08:38:36.995581+0000","last_became_peered":"2024-09-13T08:38:36.995581+0000","last_unstale":"2024-09-13T08:38:37.977964+0000","last_undegraded":"2024-09-13T08:38:37.977964+0000","last_fullsized":"2024-09-13T08:38:37.977964+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:34:20.397099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.006895+0000","last_change":"2024-09-13T08:38:35.986508+0000","last_active":"2024-09-13T08:38:37.006895+0000","last_peered":"2024-09-13T08:38:37.006895+0000","last_clean":"2024-09-13T08:38:37.006895+0000","last_became_active":"2024-09-13T08:38:35.986183+0000","last_became_peered":"2024-09-13T08:38:35.986183+0000","last_unstale":"2024-09-13T08:38:37.006895+0000","last_undegraded":"2024-09-13T08:38:37.006895+0000","last_fullsized":"2024-09-13T08:38:37.006895+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:47:18.762193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309265+0000","last_change":"2024-09-13T08:38:36.986089+0000","last_active":"2024-09-13T08:38:38.309265+0000","last_peered":"2024-09-13T08:38:38.309265+0000","last_clean":"2024-09-13T08:38:38.309265+0000","last_became_active":"2024-09-13T08:38:36.985807+0000","last_became_peered":"2024-09-13T08:38:36.985807+0000","last_unstale":"2024-09-13T08:38:38.309265+0000","last_undegraded":"2024-09-13T08:38:38.309265+0000","last_fullsized":"2024-09-13T08:38:38.309265+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:03:21.391056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978783+0000","last_change":"2024-09-13T08:38:35.976190+0000","last_active":"2024-09-13T08:38:37.978783+0000","last_peered":"2024-09-13T08:38:37.978783+0000","last_clean":"2024-09-13T08:38:37.978783+0000","last_became_active":"2024-09-13T08:38:35.975860+0000","last_became_peered":"2024-09-13T08:38:35.975860+0000","last_unstale":"2024-09-13T08:38:37.978783+0000","last_undegraded":"2024-09-13T08:38:37.978783+0000","last_fullsized":"2024-09-13T08:38:37.978783+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:01.551693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978678+0000","last_change":"2024-09-13T08:38:36.977472+0000","last_active":"2024-09-13T08:38:37.978678+0000","last_peered":"2024-09-13T08:38:37.978678+0000","last_clean":"2024-09-13T08:38:37.978678+0000","last_became_active":"2024-09-13T08:38:36.977126+0000","last_became_peered":"2024-09-13T08:38:36.977126+0000","last_unstale":"2024-09-13T08:38:37.978678+0000","last_undegraded":"2024-09-13T08:38:37.978678+0000","last_fullsized":"2024-09-13T08:38:37.978678+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:30.442843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310193+0000","last_change":"2024-09-13T08:38:35.970380+0000","last_active":"2024-09-13T08:38:38.310193+0000","last_peered":"2024-09-13T08:38:38.310193+0000","last_clean":"2024-09-13T08:38:38.310193+0000","last_became_active":"2024-09-13T08:38:35.969233+0000","last_became_peered":"2024-09-13T08:38:35.969233+0000","last_unstale":"2024-09-13T08:38:38.310193+0000","last_undegraded":"2024-09-13T08:38:38.310193+0000","last_fullsized":"2024-09-13T08:38:38.310193+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:46.376760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978369+0000","last_change":"2024-09-13T08:38:36.972042+0000","last_active":"2024-09-13T08:38:37.978369+0000","last_peered":"2024-09-13T08:38:37.978369+0000","last_clean":"2024-09-13T08:38:37.978369+0000","last_became_active":"2024-09-13T08:38:36.971784+0000","last_became_peered":"2024-09-13T08:38:36.971784+0000","last_unstale":"2024-09-13T08:38:37.978369+0000","last_undegraded":"2024-09-13T08:38:37.978369+0000","last_fullsized":"2024-09-13T08:38:37.978369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:06:15.641433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277323+0000","last_change":"2024-09-13T08:38:35.976982+0000","last_active":"2024-09-13T08:38:38.277323+0000","last_peered":"2024-09-13T08:38:38.277323+0000","last_clean":"2024-09-13T08:38:38.277323+0000","last_became_active":"2024-09-13T08:38:35.976727+0000","last_became_peered":"2024-09-13T08:38:35.976727+0000","last_unstale":"2024-09-13T08:38:38.277323+0000","last_undegraded":"2024-09-13T08:38:38.277323+0000","last_fullsized":"2024-09-13T08:38:38.277323+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:45:44.532236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309010+0000","last_change":"2024-09-13T08:38:36.997329+0000","last_active":"2024-09-13T08:38:38.309010+0000","last_peered":"2024-09-13T08:38:38.309010+0000","last_clean":"2024-09-13T08:38:38.309010+0000","last_became_active":"2024-09-13T08:38:36.997059+0000","last_became_peered":"2024-09-13T08:38:36.997059+0000","last_unstale":"2024-09-13T08:38:38.309010+0000","last_undegraded":"2024-09-13T08:38:38.309010+0000","last_fullsized":"2024-09-13T08:38:38.309010+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.455558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":21,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.000752+0000","last_change":"2024-09-13T08:38:35.974864+0000","last_active":"2024-09-13T08:38:37.000752+0000","last_peered":"2024-09-13T08:38:37.000752+0000","last_clean":"2024-09-13T08:38:37.000752+0000","last_became_active":"2024-09-13T08:38:35.974501+0000","last_became_peered":"2024-09-13T08:38:35.974501+0000","last_unstale":"2024-09-13T08:38:37.000752+0000","last_undegraded":"2024-09-13T08:38:37.000752+0000","last_fullsized":"2024-09-13T08:38:37.000752+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:46.238519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979319+0000","last_change":"2024-09-13T08:38:36.970506+0000","last_active":"2024-09-13T08:38:37.979319+0000","last_peered":"2024-09-13T08:38:37.979319+0000","last_clean":"2024-09-13T08:38:37.979319+0000","last_became_active":"2024-09-13T08:38:36.970223+0000","last_became_peered":"2024-09-13T08:38:36.970223+0000","last_unstale":"2024-09-13T08:38:37.979319+0000","last_undegraded":"2024-09-13T08:38:37.979319+0000","last_fullsized":"2024-09-13T08:38:37.979319+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:45.406855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309747+0000","last_change":"2024-09-13T08:38:35.981247+0000","last_active":"2024-09-13T08:38:38.309747+0000","last_peered":"2024-09-13T08:38:38.309747+0000","last_clean":"2024-09-13T08:38:38.309747+0000","last_became_active":"2024-09-13T08:38:35.980838+0000","last_became_peered":"2024-09-13T08:38:35.980838+0000","last_unstale":"2024-09-13T08:38:38.309747+0000","last_undegraded":"2024-09-13T08:38:38.309747+0000","last_fullsized":"2024-09-13T08:38:38.309747+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:02.412059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979678+0000","last_change":"2024-09-13T08:38:36.969498+0000","last_active":"2024-09-13T08:38:37.979678+0000","last_peered":"2024-09-13T08:38:37.979678+0000","last_clean":"2024-09-13T08:38:37.979678+0000","last_became_active":"2024-09-13T08:38:36.969200+0000","last_became_peered":"2024-09-13T08:38:36.969200+0000","last_unstale":"2024-09-13T08:38:37.979678+0000","last_undegraded":"2024-09-13T08:38:37.979678+0000","last_fullsized":"2024-09-13T08:38:37.979678+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:30:09.901419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309374+0000","last_change":"2024-09-13T08:38:35.972332+0000","last_active":"2024-09-13T08:38:38.309374+0000","last_peered":"2024-09-13T08:38:38.309374+0000","last_clean":"2024-09-13T08:38:38.309374+0000","last_became_active":"2024-09-13T08:38:35.971977+0000","last_became_peered":"2024-09-13T08:38:35.971977+0000","last_unstale":"2024-09-13T08:38:38.309374+0000","last_undegraded":"2024-09-13T08:38:38.309374+0000","last_fullsized":"2024-09-13T08:38:38.309374+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:51.164377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309964+0000","last_change":"2024-09-13T08:38:36.994845+0000","last_active":"2024-09-13T08:38:38.309964+0000","last_peered":"2024-09-13T08:38:38.309964+0000","last_clean":"2024-09-13T08:38:38.309964+0000","last_became_active":"2024-09-13T08:38:36.994549+0000","last_became_peered":"2024-09-13T08:38:36.994549+0000","last_unstale":"2024-09-13T08:38:38.309964+0000","last_undegraded":"2024-09-13T08:38:38.309964+0000","last_fullsized":"2024-09-13T08:38:38.309964+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:53.221145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308964+0000","last_change":"2024-09-13T08:38:35.983889+0000","last_active":"2024-09-13T08:38:38.308964+0000","last_peered":"2024-09-13T08:38:38.308964+0000","last_clean":"2024-09-13T08:38:38.308964+0000","last_became_active":"2024-09-13T08:38:35.982289+0000","last_became_peered":"2024-09-13T08:38:35.982289+0000","last_unstale":"2024-09-13T08:38:38.308964+0000","last_undegraded":"2024-09-13T08:38:38.308964+0000","last_fullsized":"2024-09-13T08:38:38.308964+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:32.480346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308892+0000","last_change":"2024-09-13T08:38:36.977348+0000","last_active":"2024-09-13T08:38:38.308892+0000","last_peered":"2024-09-13T08:38:38.308892+0000","last_clean":"2024-09-13T08:38:38.308892+0000","last_became_active":"2024-09-13T08:38:36.977084+0000","last_became_peered":"2024-09-13T08:38:36.977084+0000","last_unstale":"2024-09-13T08:38:38.308892+0000","last_undegraded":"2024-09-13T08:38:38.308892+0000","last_fullsized":"2024-09-13T08:38:38.308892+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:39.966579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277502+0000","last_change":"2024-09-13T08:38:35.978119+0000","last_active":"2024-09-13T08:38:38.277502+0000","last_peered":"2024-09-13T08:38:38.277502+0000","last_clean":"2024-09-13T08:38:38.277502+0000","last_became_active":"2024-09-13T08:38:35.976443+0000","last_became_peered":"2024-09-13T08:38:35.976443+0000","last_unstale":"2024-09-13T08:38:38.277502+0000","last_undegraded":"2024-09-13T08:38:38.277502+0000","last_fullsized":"2024-09-13T08:38:38.277502+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:54:02.906350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.983829+0000","last_change":"2024-09-13T08:38:36.981947+0000","last_active":"2024-09-13T08:38:36.983829+0000","last_peered":"2024-09-13T08:38:36.983829+0000","last_clean":"2024-09-13T08:38:36.983829+0000","last_became_active":"2024-09-13T08:38:36.980929+0000","last_became_peered":"2024-09-13T08:38:36.980929+0000","last_unstale":"2024-09-13T08:38:36.983829+0000","last_undegraded":"2024-09-13T08:38:36.983829+0000","last_fullsized":"2024-09-13T08:38:36.983829+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:45.228779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.002522+0000","last_change":"2024-09-13T08:38:35.982084+0000","last_active":"2024-09-13T08:38:37.002522+0000","last_peered":"2024-09-13T08:38:37.002522+0000","last_clean":"2024-09-13T08:38:37.002522+0000","last_became_active":"2024-09-13T08:38:35.981666+0000","last_became_peered":"2024-09-13T08:38:35.981666+0000","last_unstale":"2024-09-13T08:38:37.002522+0000","last_undegraded":"2024-09-13T08:38:37.002522+0000","last_fullsized":"2024-09-13T08:38:37.002522+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:50.240478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277502+0000","last_change":"2024-09-13T08:38:36.978758+0000","last_active":"2024-09-13T08:38:38.277502+0000","last_peered":"2024-09-13T08:38:38.277502+0000","last_clean":"2024-09-13T08:38:38.277502+0000","last_became_active":"2024-09-13T08:38:36.978436+0000","last_became_peered":"2024-09-13T08:38:36.978436+0000","last_unstale":"2024-09-13T08:38:38.277502+0000","last_undegraded":"2024-09-13T08:38:38.277502+0000","last_fullsized":"2024-09-13T08:38:38.277502+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.455558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310099+0000","last_change":"2024-09-13T08:38:35.979632+0000","last_active":"2024-09-13T08:38:38.310099+0000","last_peered":"2024-09-13T08:38:38.310099+0000","last_clean":"2024-09-13T08:38:38.310099+0000","last_became_active":"2024-09-13T08:38:35.979339+0000","last_became_peered":"2024-09-13T08:38:35.979339+0000","last_unstale":"2024-09-13T08:38:38.310099+0000","last_undegraded":"2024-09-13T08:38:38.310099+0000","last_fullsized":"2024-09-13T08:38:38.310099+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:35.170683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309220+0000","last_change":"2024-09-13T08:38:36.980687+0000","last_active":"2024-09-13T08:38:38.309220+0000","last_peered":"2024-09-13T08:38:38.309220+0000","last_clean":"2024-09-13T08:38:38.309220+0000","last_became_active":"2024-09-13T08:38:36.980408+0000","last_became_peered":"2024-09-13T08:38:36.980408+0000","last_unstale":"2024-09-13T08:38:38.309220+0000","last_undegraded":"2024-09-13T08:38:38.309220+0000","last_fullsized":"2024-09-13T08:38:38.309220+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:35:09.103775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.963487+0000","last_change":"2024-09-13T08:38:35.976033+0000","last_active":"2024-09-13T08:38:36.963487+0000","last_peered":"2024-09-13T08:38:36.963487+0000","last_clean":"2024-09-13T08:38:36.963487+0000","last_became_active":"2024-09-13T08:38:35.975464+0000","last_became_peered":"2024-09-13T08:38:35.975464+0000","last_unstale":"2024-09-13T08:38:36.963487+0000","last_undegraded":"2024-09-13T08:38:36.963487+0000","last_fullsized":"2024-09-13T08:38:36.963487+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:01.658298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309071+0000","last_change":"2024-09-13T08:38:36.983639+0000","last_active":"2024-09-13T08:38:38.309071+0000","last_peered":"2024-09-13T08:38:38.309071+0000","last_clean":"2024-09-13T08:38:38.309071+0000","last_became_active":"2024-09-13T08:38:36.983327+0000","last_became_peered":"2024-09-13T08:38:36.983327+0000","last_unstale":"2024-09-13T08:38:38.309071+0000","last_undegraded":"2024-09-13T08:38:38.309071+0000","last_fullsized":"2024-09-13T08:38:38.309071+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:31.782847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977738+0000","last_change":"2024-09-13T08:38:35.977525+0000","last_active":"2024-09-13T08:38:37.977738+0000","last_peered":"2024-09-13T08:38:37.977738+0000","last_clean":"2024-09-13T08:38:37.977738+0000","last_became_active":"2024-09-13T08:38:35.977346+0000","last_became_peered":"2024-09-13T08:38:35.977346+0000","last_unstale":"2024-09-13T08:38:37.977738+0000","last_undegraded":"2024-09-13T08:38:37.977738+0000","last_fullsized":"2024-09-13T08:38:37.977738+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:28.944099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978039+0000","last_change":"2024-09-13T08:38:36.988070+0000","last_active":"2024-09-13T08:38:37.978039+0000","last_peered":"2024-09-13T08:38:37.978039+0000","last_clean":"2024-09-13T08:38:37.978039+0000","last_became_active":"2024-09-13T08:38:36.987808+0000","last_became_peered":"2024-09-13T08:38:36.987808+0000","last_unstale":"2024-09-13T08:38:37.978039+0000","last_undegraded":"2024-09-13T08:38:37.978039+0000","last_fullsized":"2024-09-13T08:38:37.978039+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:36.635788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309479+0000","last_change":"2024-09-13T08:38:35.986462+0000","last_active":"2024-09-13T08:38:38.309479+0000","last_peered":"2024-09-13T08:38:38.309479+0000","last_clean":"2024-09-13T08:38:38.309479+0000","last_became_active":"2024-09-13T08:38:35.986216+0000","last_became_peered":"2024-09-13T08:38:35.986216+0000","last_unstale":"2024-09-13T08:38:38.309479+0000","last_undegraded":"2024-09-13T08:38:38.309479+0000","last_fullsized":"2024-09-13T08:38:38.309479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:54:02.906350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.997393+0000","last_change":"2024-09-13T08:38:36.994000+0000","last_active":"2024-09-13T08:38:36.997393+0000","last_peered":"2024-09-13T08:38:36.997393+0000","last_clean":"2024-09-13T08:38:36.997393+0000","last_became_active":"2024-09-13T08:38:36.992137+0000","last_became_peered":"2024-09-13T08:38:36.992137+0000","last_unstale":"2024-09-13T08:38:36.997393+0000","last_undegraded":"2024-09-13T08:38:36.997393+0000","last_fullsized":"2024-09-13T08:38:36.997393+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:38.542302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310261+0000","last_change":"2024-09-13T08:38:35.976885+0000","last_active":"2024-09-13T08:38:38.310261+0000","last_peered":"2024-09-13T08:38:38.310261+0000","last_clean":"2024-09-13T08:38:38.310261+0000","last_became_active":"2024-09-13T08:38:35.976270+0000","last_became_peered":"2024-09-13T08:38:35.976270+0000","last_unstale":"2024-09-13T08:38:38.310261+0000","last_undegraded":"2024-09-13T08:38:38.310261+0000","last_fullsized":"2024-09-13T08:38:38.310261+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:13.773728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978146+0000","last_change":"2024-09-13T08:38:36.995799+0000","last_active":"2024-09-13T08:38:37.978146+0000","last_peered":"2024-09-13T08:38:37.978146+0000","last_clean":"2024-09-13T08:38:37.978146+0000","last_became_active":"2024-09-13T08:38:36.995276+0000","last_became_peered":"2024-09-13T08:38:36.995276+0000","last_unstale":"2024-09-13T08:38:37.978146+0000","last_undegraded":"2024-09-13T08:38:37.978146+0000","last_fullsized":"2024-09-13T08:38:37.978146+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:37.521180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978991+0000","last_change":"2024-09-13T08:38:35.972242+0000","last_active":"2024-09-13T08:38:37.978991+0000","last_peered":"2024-09-13T08:38:37.978991+0000","last_clean":"2024-09-13T08:38:37.978991+0000","last_became_active":"2024-09-13T08:38:35.971769+0000","last_became_peered":"2024-09-13T08:38:35.971769+0000","last_unstale":"2024-09-13T08:38:37.978991+0000","last_undegraded":"2024-09-13T08:38:37.978991+0000","last_fullsized":"2024-09-13T08:38:37.978991+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:52:36.718111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.983456+0000","last_change":"2024-09-13T08:38:36.981500+0000","last_active":"2024-09-13T08:38:36.983456+0000","last_peered":"2024-09-13T08:38:36.983456+0000","last_clean":"2024-09-13T08:38:36.983456+0000","last_became_active":"2024-09-13T08:38:36.980588+0000","last_became_peered":"2024-09-13T08:38:36.980588+0000","last_unstale":"2024-09-13T08:38:36.983456+0000","last_undegraded":"2024-09-13T08:38:36.983456+0000","last_fullsized":"2024-09-13T08:38:36.983456+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:21.041457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309712+0000","last_change":"2024-09-13T08:38:35.973799+0000","last_active":"2024-09-13T08:38:38.309712+0000","last_peered":"2024-09-13T08:38:38.309712+0000","last_clean":"2024-09-13T08:38:38.309712+0000","last_became_active":"2024-09-13T08:38:35.973523+0000","last_became_peered":"2024-09-13T08:38:35.973523+0000","last_unstale":"2024-09-13T08:38:38.309712+0000","last_undegraded":"2024-09-13T08:38:38.309712+0000","last_fullsized":"2024-09-13T08:38:38.309712+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:10.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309594+0000","last_change":"2024-09-13T08:38:36.982987+0000","last_active":"2024-09-13T08:38:38.309594+0000","last_peered":"2024-09-13T08:38:38.309594+0000","last_clean":"2024-09-13T08:38:38.309594+0000","last_became_active":"2024-09-13T08:38:36.982676+0000","last_became_peered":"2024-09-13T08:38:36.982676+0000","last_unstale":"2024-09-13T08:38:38.309594+0000","last_undegraded":"2024-09-13T08:38:38.309594+0000","last_fullsized":"2024-09-13T08:38:38.309594+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:42.254713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977667+0000","last_change":"2024-09-13T08:38:35.986366+0000","last_active":"2024-09-13T08:38:37.977667+0000","last_peered":"2024-09-13T08:38:37.977667+0000","last_clean":"2024-09-13T08:38:37.977667+0000","last_became_active":"2024-09-13T08:38:35.986083+0000","last_became_peered":"2024-09-13T08:38:35.986083+0000","last_unstale":"2024-09-13T08:38:37.977667+0000","last_undegraded":"2024-09-13T08:38:37.977667+0000","last_fullsized":"2024-09-13T08:38:37.977667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:29.172988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308695+0000","last_change":"2024-09-13T08:38:36.986721+0000","last_active":"2024-09-13T08:38:38.308695+0000","last_peered":"2024-09-13T08:38:38.308695+0000","last_clean":"2024-09-13T08:38:38.308695+0000","last_became_active":"2024-09-13T08:38:36.986450+0000","last_became_peered":"2024-09-13T08:38:36.986450+0000","last_unstale":"2024-09-13T08:38:38.308695+0000","last_undegraded":"2024-09-13T08:38:38.308695+0000","last_fullsized":"2024-09-13T08:38:38.308695+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:01.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277422+0000","last_change":"2024-09-13T08:38:35.982648+0000","last_active":"2024-09-13T08:38:38.277422+0000","last_peered":"2024-09-13T08:38:38.277422+0000","last_clean":"2024-09-13T08:38:38.277422+0000","last_became_active":"2024-09-13T08:38:35.982370+0000","last_became_peered":"2024-09-13T08:38:35.982370+0000","last_unstale":"2024-09-13T08:38:38.277422+0000","last_undegraded":"2024-09-13T08:38:38.277422+0000","last_fullsized":"2024-09-13T08:38:38.277422+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:02.412059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309013+0000","last_change":"2024-09-13T08:38:36.993462+0000","last_active":"2024-09-13T08:38:38.309013+0000","last_peered":"2024-09-13T08:38:38.309013+0000","last_clean":"2024-09-13T08:38:38.309013+0000","last_became_active":"2024-09-13T08:38:36.992818+0000","last_became_peered":"2024-09-13T08:38:36.992818+0000","last_unstale":"2024-09-13T08:38:38.309013+0000","last_undegraded":"2024-09-13T08:38:38.309013+0000","last_fullsized":"2024-09-13T08:38:38.309013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:31.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978467+0000","last_change":"2024-09-13T08:38:35.977566+0000","last_active":"2024-09-13T08:38:37.978467+0000","last_peered":"2024-09-13T08:38:37.978467+0000","last_clean":"2024-09-13T08:38:37.978467+0000","last_became_active":"2024-09-13T08:38:35.977254+0000","last_became_peered":"2024-09-13T08:38:35.977254+0000","last_unstale":"2024-09-13T08:38:37.978467+0000","last_undegraded":"2024-09-13T08:38:37.978467+0000","last_fullsized":"2024-09-13T08:38:37.978467+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:14:20.559469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309573+0000","last_change":"2024-09-13T08:38:36.993737+0000","last_active":"2024-09-13T08:38:38.309573+0000","last_peered":"2024-09-13T08:38:38.309573+0000","last_clean":"2024-09-13T08:38:38.309573+0000","last_became_active":"2024-09-13T08:38:36.992965+0000","last_became_peered":"2024-09-13T08:38:36.992965+0000","last_unstale":"2024-09-13T08:38:38.309573+0000","last_undegraded":"2024-09-13T08:38:38.309573+0000","last_fullsized":"2024-09-13T08:38:38.309573+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:30.182371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977899+0000","last_change":"2024-09-13T08:38:35.986117+0000","last_active":"2024-09-13T08:38:37.977899+0000","last_peered":"2024-09-13T08:38:37.977899+0000","last_clean":"2024-09-13T08:38:37.977899+0000","last_became_active":"2024-09-13T08:38:35.985799+0000","last_became_peered":"2024-09-13T08:38:35.985799+0000","last_unstale":"2024-09-13T08:38:37.977899+0000","last_undegraded":"2024-09-13T08:38:37.977899+0000","last_fullsized":"2024-09-13T08:38:37.977899+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:44.280453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":154,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308359+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:38:38.308359+0000","last_peered":"2024-09-13T08:38:38.308359+0000","last_clean":"2024-09-13T08:38:38.308359+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:38:38.308359+0000","last_undegraded":"2024-09-13T08:38:38.308359+0000","last_fullsized":"2024-09-13T08:38:38.308359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":548864,"data_stored":218478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31732,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340108,"statfs":{"total":96636764160,"available":96604270592,"internally_reserved":0,"allocated":552960,"data_stored":219060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":808834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":557056,"data_stored":220056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":548864,"data_stored":218534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1142784,"data_stored":808882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509462,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":552960,"data_stored":219116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":532480,"data_stored":211669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:38:41.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:38:41.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3670418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd594005210 msgr2=0x7fd59400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3670418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd594005210 0x7fd59400b9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8001d90 tx=0x7fd594005830 comp rx=0 tx=0).stop 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3670418637 shutdown_connections 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3670418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd594005210 0x7fd59400b9c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3670418637 >> 172.21.15.31:0/3670418637 conn(0x7fd5b00cc910 msgr2=0x7fd5b00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3670418637 shutdown_connections 2024-09-13T08:38:41.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.416+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3670418637 wait complete. 2024-09-13T08:38:41.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.418+0000 7fd5b7a00640 1 Processor -- start 2024-09-13T08:38:41.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.418+0000 7fd5b7a00640 1 -- start start 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 0x7fd5b009ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b01595a0 0x7fd5b0159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00cb9d0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00cb850 con 0x7fd5b01595a0 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5b7a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b00cf850 con 0x7fd594005210 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.419+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5b5000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 0x7fd5b009ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44758/0 (socket says 172.21.15.31:44758) 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 -- 172.21.15.31:0/3150272348 learned_addr learned my addr 172.21.15.31:0/3150272348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5b5a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b01595a0 0x7fd5b0159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 -- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 msgr2=0x7fd5b009ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 0x7fd5b009ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 -- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b01595a0 msgr2=0x7fd5b0159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b01595a0 0x7fd5b0159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 -- 172.21.15.31:0/3150272348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b0157070 con 0x7fd5b01f57e0 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5b5000640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 0x7fd5b009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.420+0000 7fd5a7e00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd59800e510 tx=0x7fd59800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.421+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598015070 con 0x7fd5b01f57e0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.421+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd598010980 con 0x7fd5b01f57e0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.421+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b0157350 con 0x7fd5b01f57e0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.421+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598010dd0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.421+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b0158dc0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.422+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b00ce3c0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.423+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fd598018070 con 0x7fd5b01f57e0 2024-09-13T08:38:41.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.424+0000 7fd5ad600640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 0x7fd58c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.424+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd59809afc0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.425+0000 7fd5b5000640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 0x7fd58c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.425+0000 7fd5b5000640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 0x7fd58c07aec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00b37b0 tx=0x7fd5a0005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:41.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.428+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd598062cf0 con 0x7fd5b01f57e0 2024-09-13T08:38:41.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.619+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd5b00b14d0 con 0x7fd58c078a00 2024-09-13T08:38:41.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.624+0000 7fd5ad600640 1 -- 172.21.15.31:0/3150272348 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7fd5b00b14d0 con 0x7fd58c078a00 2024-09-13T08:38:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:38:41.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 msgr2=0x7fd58c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 0x7fd58c07aec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00b37b0 tx=0x7fd5a0005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 msgr2=0x7fd5b01f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd59800e510 tx=0x7fd59800e9d0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 shutdown_connections 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b01595a0 0x7fd5b0159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5b01f57e0 0x7fd5b01f5c60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd58c078a00 0x7fd58c07aec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 --2- 172.21.15.31:0/3150272348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd594005210 0x7fd5b009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 >> 172.21.15.31:0/3150272348 conn(0x7fd5b00cc910 msgr2=0x7fd5b00adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 shutdown_connections 2024-09-13T08:38:41.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.628+0000 7fd5b7a00640 1 -- 172.21.15.31:0/3150272348 wait complete. 2024-09-13T08:38:41.644 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-13T08:38:41.057239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238336,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736384,"statfs":{"total":773094113280,"available":772850057216,"internally_reserved":0,"allocated":5578752,"data_stored":2924629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001459"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310452+0000","last_change":"2024-09-13T08:38:37.001717+0000","last_active":"2024-09-13T08:38:38.310452+0000","last_peered":"2024-09-13T08:38:38.310452+0000","last_clean":"2024-09-13T08:38:38.310452+0000","last_became_active":"2024-09-13T08:38:37.001417+0000","last_became_peered":"2024-09-13T08:38:37.001417+0000","last_unstale":"2024-09-13T08:38:38.310452+0000","last_undegraded":"2024-09-13T08:38:38.310452+0000","last_fullsized":"2024-09-13T08:38:38.310452+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:10.615827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310140+0000","last_change":"2024-09-13T08:38:36.982983+0000","last_active":"2024-09-13T08:38:38.310140+0000","last_peered":"2024-09-13T08:38:38.310140+0000","last_clean":"2024-09-13T08:38:38.310140+0000","last_became_active":"2024-09-13T08:38:36.982706+0000","last_became_peered":"2024-09-13T08:38:36.982706+0000","last_unstale":"2024-09-13T08:38:38.310140+0000","last_undegraded":"2024-09-13T08:38:38.310140+0000","last_fullsized":"2024-09-13T08:38:38.310140+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309206+0000","last_change":"2024-09-13T08:38:36.980093+0000","last_active":"2024-09-13T08:38:38.309206+0000","last_peered":"2024-09-13T08:38:38.309206+0000","last_clean":"2024-09-13T08:38:38.309206+0000","last_became_active":"2024-09-13T08:38:36.976661+0000","last_became_peered":"2024-09-13T08:38:36.976661+0000","last_unstale":"2024-09-13T08:38:38.309206+0000","last_undegraded":"2024-09-13T08:38:38.309206+0000","last_fullsized":"2024-09-13T08:38:38.309206+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:30.442843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309369+0000","last_change":"2024-09-13T08:38:36.991627+0000","last_active":"2024-09-13T08:38:38.309369+0000","last_peered":"2024-09-13T08:38:38.309369+0000","last_clean":"2024-09-13T08:38:38.309369+0000","last_became_active":"2024-09-13T08:38:36.991331+0000","last_became_peered":"2024-09-13T08:38:36.991331+0000","last_unstale":"2024-09-13T08:38:38.309369+0000","last_undegraded":"2024-09-13T08:38:38.309369+0000","last_fullsized":"2024-09-13T08:38:38.309369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:29.953483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309637+0000","last_change":"2024-09-13T08:38:36.994148+0000","last_active":"2024-09-13T08:38:38.309637+0000","last_peered":"2024-09-13T08:38:38.309637+0000","last_clean":"2024-09-13T08:38:38.309637+0000","last_became_active":"2024-09-13T08:38:36.993838+0000","last_became_peered":"2024-09-13T08:38:36.993838+0000","last_unstale":"2024-09-13T08:38:38.309637+0000","last_undegraded":"2024-09-13T08:38:38.309637+0000","last_fullsized":"2024-09-13T08:38:38.309637+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:16.831423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277595+0000","last_change":"2024-09-13T08:38:36.994944+0000","last_active":"2024-09-13T08:38:38.277595+0000","last_peered":"2024-09-13T08:38:38.277595+0000","last_clean":"2024-09-13T08:38:38.277595+0000","last_became_active":"2024-09-13T08:38:36.994637+0000","last_became_peered":"2024-09-13T08:38:36.994637+0000","last_unstale":"2024-09-13T08:38:38.277595+0000","last_undegraded":"2024-09-13T08:38:38.277595+0000","last_fullsized":"2024-09-13T08:38:38.277595+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:31.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977522+0000","last_change":"2024-09-13T08:38:36.992617+0000","last_active":"2024-09-13T08:38:37.977522+0000","last_peered":"2024-09-13T08:38:37.977522+0000","last_clean":"2024-09-13T08:38:37.977522+0000","last_became_active":"2024-09-13T08:38:36.991986+0000","last_became_peered":"2024-09-13T08:38:36.991986+0000","last_unstale":"2024-09-13T08:38:37.977522+0000","last_undegraded":"2024-09-13T08:38:37.977522+0000","last_fullsized":"2024-09-13T08:38:37.977522+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:59:54.439228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308970+0000","last_change":"2024-09-13T08:38:36.987688+0000","last_active":"2024-09-13T08:38:38.308970+0000","last_peered":"2024-09-13T08:38:38.308970+0000","last_clean":"2024-09-13T08:38:38.308970+0000","last_became_active":"2024-09-13T08:38:36.986338+0000","last_became_peered":"2024-09-13T08:38:36.986338+0000","last_unstale":"2024-09-13T08:38:38.308970+0000","last_undegraded":"2024-09-13T08:38:38.308970+0000","last_fullsized":"2024-09-13T08:38:38.308970+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:33.875455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277819+0000","last_change":"2024-09-13T08:38:36.984422+0000","last_active":"2024-09-13T08:38:38.277819+0000","last_peered":"2024-09-13T08:38:38.277819+0000","last_clean":"2024-09-13T08:38:38.277819+0000","last_became_active":"2024-09-13T08:38:36.984123+0000","last_became_peered":"2024-09-13T08:38:36.984123+0000","last_unstale":"2024-09-13T08:38:38.277819+0000","last_undegraded":"2024-09-13T08:38:38.277819+0000","last_fullsized":"2024-09-13T08:38:38.277819+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:27.393269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.982556+0000","last_change":"2024-09-13T08:38:36.980412+0000","last_active":"2024-09-13T08:38:36.982556+0000","last_peered":"2024-09-13T08:38:36.982556+0000","last_clean":"2024-09-13T08:38:36.982556+0000","last_became_active":"2024-09-13T08:38:36.980088+0000","last_became_peered":"2024-09-13T08:38:36.980088+0000","last_unstale":"2024-09-13T08:38:36.982556+0000","last_undegraded":"2024-09-13T08:38:36.982556+0000","last_fullsized":"2024-09-13T08:38:36.982556+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:02.764241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309445+0000","last_change":"2024-09-13T08:38:36.991525+0000","last_active":"2024-09-13T08:38:38.309445+0000","last_peered":"2024-09-13T08:38:38.309445+0000","last_clean":"2024-09-13T08:38:38.309445+0000","last_became_active":"2024-09-13T08:38:36.990939+0000","last_became_peered":"2024-09-13T08:38:36.990939+0000","last_unstale":"2024-09-13T08:38:38.309445+0000","last_undegraded":"2024-09-13T08:38:38.309445+0000","last_fullsized":"2024-09-13T08:38:38.309445+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:54.057235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309706+0000","last_change":"2024-09-13T08:38:36.979115+0000","last_active":"2024-09-13T08:38:38.309706+0000","last_peered":"2024-09-13T08:38:38.309706+0000","last_clean":"2024-09-13T08:38:38.309706+0000","last_became_active":"2024-09-13T08:38:36.978828+0000","last_became_peered":"2024-09-13T08:38:36.978828+0000","last_unstale":"2024-09-13T08:38:38.309706+0000","last_undegraded":"2024-09-13T08:38:38.309706+0000","last_fullsized":"2024-09-13T08:38:38.309706+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:16.831423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277485+0000","last_change":"2024-09-13T08:38:36.989066+0000","last_active":"2024-09-13T08:38:38.277485+0000","last_peered":"2024-09-13T08:38:38.277485+0000","last_clean":"2024-09-13T08:38:38.277485+0000","last_became_active":"2024-09-13T08:38:36.988720+0000","last_became_peered":"2024-09-13T08:38:36.988720+0000","last_unstale":"2024-09-13T08:38:38.277485+0000","last_undegraded":"2024-09-13T08:38:38.277485+0000","last_fullsized":"2024-09-13T08:38:38.277485+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:34.821199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309077+0000","last_change":"2024-09-13T08:38:36.986743+0000","last_active":"2024-09-13T08:38:38.309077+0000","last_peered":"2024-09-13T08:38:38.309077+0000","last_clean":"2024-09-13T08:38:38.309077+0000","last_became_active":"2024-09-13T08:38:36.986307+0000","last_became_peered":"2024-09-13T08:38:36.986307+0000","last_unstale":"2024-09-13T08:38:38.309077+0000","last_undegraded":"2024-09-13T08:38:38.309077+0000","last_fullsized":"2024-09-13T08:38:38.309077+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309002+0000","last_change":"2024-09-13T08:38:36.993262+0000","last_active":"2024-09-13T08:38:38.309002+0000","last_peered":"2024-09-13T08:38:38.309002+0000","last_clean":"2024-09-13T08:38:38.309002+0000","last_became_active":"2024-09-13T08:38:36.992919+0000","last_became_peered":"2024-09-13T08:38:36.992919+0000","last_unstale":"2024-09-13T08:38:38.309002+0000","last_undegraded":"2024-09-13T08:38:38.309002+0000","last_fullsized":"2024-09-13T08:38:38.309002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:52.991190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309351+0000","last_change":"2024-09-13T08:38:36.995098+0000","last_active":"2024-09-13T08:38:38.309351+0000","last_peered":"2024-09-13T08:38:38.309351+0000","last_clean":"2024-09-13T08:38:38.309351+0000","last_became_active":"2024-09-13T08:38:36.992649+0000","last_became_peered":"2024-09-13T08:38:36.992649+0000","last_unstale":"2024-09-13T08:38:38.309351+0000","last_undegraded":"2024-09-13T08:38:38.309351+0000","last_fullsized":"2024-09-13T08:38:38.309351+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:12.768061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310264+0000","last_change":"2024-09-13T08:38:36.991658+0000","last_active":"2024-09-13T08:38:38.310264+0000","last_peered":"2024-09-13T08:38:38.310264+0000","last_clean":"2024-09-13T08:38:38.310264+0000","last_became_active":"2024-09-13T08:38:36.991322+0000","last_became_peered":"2024-09-13T08:38:36.991322+0000","last_unstale":"2024-09-13T08:38:38.310264+0000","last_undegraded":"2024-09-13T08:38:38.310264+0000","last_fullsized":"2024-09-13T08:38:38.310264+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:10.615827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309262+0000","last_change":"2024-09-13T08:38:36.987380+0000","last_active":"2024-09-13T08:38:38.309262+0000","last_peered":"2024-09-13T08:38:38.309262+0000","last_clean":"2024-09-13T08:38:38.309262+0000","last_became_active":"2024-09-13T08:38:36.987139+0000","last_became_peered":"2024-09-13T08:38:36.987139+0000","last_unstale":"2024-09-13T08:38:38.309262+0000","last_undegraded":"2024-09-13T08:38:38.309262+0000","last_fullsized":"2024-09-13T08:38:38.309262+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:45.406855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310298+0000","last_change":"2024-09-13T08:38:36.992766+0000","last_active":"2024-09-13T08:38:38.310298+0000","last_peered":"2024-09-13T08:38:38.310298+0000","last_clean":"2024-09-13T08:38:38.310298+0000","last_became_active":"2024-09-13T08:38:36.989989+0000","last_became_peered":"2024-09-13T08:38:36.989989+0000","last_unstale":"2024-09-13T08:38:38.310298+0000","last_undegraded":"2024-09-13T08:38:38.310298+0000","last_fullsized":"2024-09-13T08:38:38.310298+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:53.221145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.981634+0000","last_change":"2024-09-13T08:38:36.972726+0000","last_active":"2024-09-13T08:38:36.981634+0000","last_peered":"2024-09-13T08:38:36.981634+0000","last_clean":"2024-09-13T08:38:36.981634+0000","last_became_active":"2024-09-13T08:38:36.972471+0000","last_became_peered":"2024-09-13T08:38:36.972471+0000","last_unstale":"2024-09-13T08:38:36.981634+0000","last_undegraded":"2024-09-13T08:38:36.981634+0000","last_fullsized":"2024-09-13T08:38:36.981634+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:14.854884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309683+0000","last_change":"2024-09-13T08:38:36.983169+0000","last_active":"2024-09-13T08:38:38.309683+0000","last_peered":"2024-09-13T08:38:38.309683+0000","last_clean":"2024-09-13T08:38:38.309683+0000","last_became_active":"2024-09-13T08:38:36.982575+0000","last_became_peered":"2024-09-13T08:38:36.982575+0000","last_unstale":"2024-09-13T08:38:38.309683+0000","last_undegraded":"2024-09-13T08:38:38.309683+0000","last_fullsized":"2024-09-13T08:38:38.309683+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:45.289837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978446+0000","last_change":"2024-09-13T08:38:36.969942+0000","last_active":"2024-09-13T08:38:37.978446+0000","last_peered":"2024-09-13T08:38:37.978446+0000","last_clean":"2024-09-13T08:38:37.978446+0000","last_became_active":"2024-09-13T08:38:36.969649+0000","last_became_peered":"2024-09-13T08:38:36.969649+0000","last_unstale":"2024-09-13T08:38:37.978446+0000","last_undegraded":"2024-09-13T08:38:37.978446+0000","last_fullsized":"2024-09-13T08:38:37.978446+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:01.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977554+0000","last_change":"2024-09-13T08:38:36.992980+0000","last_active":"2024-09-13T08:38:37.977554+0000","last_peered":"2024-09-13T08:38:37.977554+0000","last_clean":"2024-09-13T08:38:37.977554+0000","last_became_active":"2024-09-13T08:38:36.992339+0000","last_became_peered":"2024-09-13T08:38:36.992339+0000","last_unstale":"2024-09-13T08:38:37.977554+0000","last_undegraded":"2024-09-13T08:38:37.977554+0000","last_fullsized":"2024-09-13T08:38:37.977554+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:56.392500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309013+0000","last_change":"2024-09-13T08:38:36.991547+0000","last_active":"2024-09-13T08:38:38.309013+0000","last_peered":"2024-09-13T08:38:38.309013+0000","last_clean":"2024-09-13T08:38:38.309013+0000","last_became_active":"2024-09-13T08:38:36.991253+0000","last_became_peered":"2024-09-13T08:38:36.991253+0000","last_unstale":"2024-09-13T08:38:38.309013+0000","last_undegraded":"2024-09-13T08:38:38.309013+0000","last_fullsized":"2024-09-13T08:38:38.309013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:17.801741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310427+0000","last_change":"2024-09-13T08:38:37.001718+0000","last_active":"2024-09-13T08:38:38.310427+0000","last_peered":"2024-09-13T08:38:38.310427+0000","last_clean":"2024-09-13T08:38:38.310427+0000","last_became_active":"2024-09-13T08:38:37.001378+0000","last_became_peered":"2024-09-13T08:38:37.001378+0000","last_unstale":"2024-09-13T08:38:38.310427+0000","last_undegraded":"2024-09-13T08:38:38.310427+0000","last_fullsized":"2024-09-13T08:38:38.310427+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:53:37.359065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309350+0000","last_change":"2024-09-13T08:38:36.990170+0000","last_active":"2024-09-13T08:38:38.309350+0000","last_peered":"2024-09-13T08:38:38.309350+0000","last_clean":"2024-09-13T08:38:38.309350+0000","last_became_active":"2024-09-13T08:38:36.989751+0000","last_became_peered":"2024-09-13T08:38:36.989751+0000","last_unstale":"2024-09-13T08:38:38.309350+0000","last_undegraded":"2024-09-13T08:38:38.309350+0000","last_fullsized":"2024-09-13T08:38:38.309350+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:54.329849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978009+0000","last_change":"2024-09-13T08:38:37.002661+0000","last_active":"2024-09-13T08:38:37.978009+0000","last_peered":"2024-09-13T08:38:37.978009+0000","last_clean":"2024-09-13T08:38:37.978009+0000","last_became_active":"2024-09-13T08:38:37.002380+0000","last_became_peered":"2024-09-13T08:38:37.002380+0000","last_unstale":"2024-09-13T08:38:37.978009+0000","last_undegraded":"2024-09-13T08:38:37.978009+0000","last_fullsized":"2024-09-13T08:38:37.978009+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:09.877487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977949+0000","last_change":"2024-09-13T08:38:36.987371+0000","last_active":"2024-09-13T08:38:37.977949+0000","last_peered":"2024-09-13T08:38:37.977949+0000","last_clean":"2024-09-13T08:38:37.977949+0000","last_became_active":"2024-09-13T08:38:36.987046+0000","last_became_peered":"2024-09-13T08:38:36.987046+0000","last_unstale":"2024-09-13T08:38:37.977949+0000","last_undegraded":"2024-09-13T08:38:37.977949+0000","last_fullsized":"2024-09-13T08:38:37.977949+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:04.546377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309829+0000","last_change":"2024-09-13T08:38:36.996341+0000","last_active":"2024-09-13T08:38:38.309829+0000","last_peered":"2024-09-13T08:38:38.309829+0000","last_clean":"2024-09-13T08:38:38.309829+0000","last_became_active":"2024-09-13T08:38:36.990607+0000","last_became_peered":"2024-09-13T08:38:36.990607+0000","last_unstale":"2024-09-13T08:38:38.309829+0000","last_undegraded":"2024-09-13T08:38:38.309829+0000","last_fullsized":"2024-09-13T08:38:38.309829+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309239+0000","last_change":"2024-09-13T08:38:36.989533+0000","last_active":"2024-09-13T08:38:38.309239+0000","last_peered":"2024-09-13T08:38:38.309239+0000","last_clean":"2024-09-13T08:38:38.309239+0000","last_became_active":"2024-09-13T08:38:36.989291+0000","last_became_peered":"2024-09-13T08:38:36.989291+0000","last_unstale":"2024-09-13T08:38:38.309239+0000","last_undegraded":"2024-09-13T08:38:38.309239+0000","last_fullsized":"2024-09-13T08:38:38.309239+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:12.768061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277465+0000","last_change":"2024-09-13T08:38:36.991591+0000","last_active":"2024-09-13T08:38:38.277465+0000","last_peered":"2024-09-13T08:38:38.277465+0000","last_clean":"2024-09-13T08:38:38.277465+0000","last_became_active":"2024-09-13T08:38:36.991318+0000","last_became_peered":"2024-09-13T08:38:36.991318+0000","last_unstale":"2024-09-13T08:38:38.277465+0000","last_undegraded":"2024-09-13T08:38:38.277465+0000","last_fullsized":"2024-09-13T08:38:38.277465+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:21.688201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309010+0000","last_change":"2024-09-13T08:38:36.986408+0000","last_active":"2024-09-13T08:38:38.309010+0000","last_peered":"2024-09-13T08:38:38.309010+0000","last_clean":"2024-09-13T08:38:38.309010+0000","last_became_active":"2024-09-13T08:38:36.983578+0000","last_became_peered":"2024-09-13T08:38:36.983578+0000","last_unstale":"2024-09-13T08:38:38.309010+0000","last_undegraded":"2024-09-13T08:38:38.309010+0000","last_fullsized":"2024-09-13T08:38:38.309010+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979280+0000","last_change":"2024-09-13T08:38:36.972913+0000","last_active":"2024-09-13T08:38:37.979280+0000","last_peered":"2024-09-13T08:38:37.979280+0000","last_clean":"2024-09-13T08:38:37.979280+0000","last_became_active":"2024-09-13T08:38:36.972673+0000","last_became_peered":"2024-09-13T08:38:36.972673+0000","last_unstale":"2024-09-13T08:38:37.979280+0000","last_undegraded":"2024-09-13T08:38:37.979280+0000","last_fullsized":"2024-09-13T08:38:37.979280+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:39.966579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277800+0000","last_change":"2024-09-13T08:38:35.976185+0000","last_active":"2024-09-13T08:38:38.277800+0000","last_peered":"2024-09-13T08:38:38.277800+0000","last_clean":"2024-09-13T08:38:38.277800+0000","last_became_active":"2024-09-13T08:38:35.975896+0000","last_became_peered":"2024-09-13T08:38:35.975896+0000","last_unstale":"2024-09-13T08:38:38.277800+0000","last_undegraded":"2024-09-13T08:38:38.277800+0000","last_fullsized":"2024-09-13T08:38:38.277800+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:35.170683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310531+0000","last_change":"2024-09-13T08:38:36.990308+0000","last_active":"2024-09-13T08:38:38.310531+0000","last_peered":"2024-09-13T08:38:38.310531+0000","last_clean":"2024-09-13T08:38:38.310531+0000","last_became_active":"2024-09-13T08:38:36.989885+0000","last_became_peered":"2024-09-13T08:38:36.989885+0000","last_unstale":"2024-09-13T08:38:38.310531+0000","last_undegraded":"2024-09-13T08:38:38.310531+0000","last_fullsized":"2024-09-13T08:38:38.310531+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:32:11.072827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977587+0000","last_change":"2024-09-13T08:38:35.982559+0000","last_active":"2024-09-13T08:38:37.977587+0000","last_peered":"2024-09-13T08:38:37.977587+0000","last_clean":"2024-09-13T08:38:37.977587+0000","last_became_active":"2024-09-13T08:38:35.980833+0000","last_became_peered":"2024-09-13T08:38:35.980833+0000","last_unstale":"2024-09-13T08:38:37.977587+0000","last_undegraded":"2024-09-13T08:38:37.977587+0000","last_fullsized":"2024-09-13T08:38:37.977587+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:41.245329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309951+0000","last_change":"2024-09-13T08:38:36.983729+0000","last_active":"2024-09-13T08:38:38.309951+0000","last_peered":"2024-09-13T08:38:38.309951+0000","last_clean":"2024-09-13T08:38:38.309951+0000","last_became_active":"2024-09-13T08:38:36.983394+0000","last_became_peered":"2024-09-13T08:38:36.983394+0000","last_unstale":"2024-09-13T08:38:38.309951+0000","last_undegraded":"2024-09-13T08:38:38.309951+0000","last_fullsized":"2024-09-13T08:38:38.309951+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:00.854735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277577+0000","last_change":"2024-09-13T08:38:35.972025+0000","last_active":"2024-09-13T08:38:38.277577+0000","last_peered":"2024-09-13T08:38:38.277577+0000","last_clean":"2024-09-13T08:38:38.277577+0000","last_became_active":"2024-09-13T08:38:35.971648+0000","last_became_peered":"2024-09-13T08:38:35.971648+0000","last_unstale":"2024-09-13T08:38:38.277577+0000","last_undegraded":"2024-09-13T08:38:38.277577+0000","last_fullsized":"2024-09-13T08:38:38.277577+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:10.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309663+0000","last_change":"2024-09-13T08:38:36.991221+0000","last_active":"2024-09-13T08:38:38.309663+0000","last_peered":"2024-09-13T08:38:38.309663+0000","last_clean":"2024-09-13T08:38:38.309663+0000","last_became_active":"2024-09-13T08:38:36.990544+0000","last_became_peered":"2024-09-13T08:38:36.990544+0000","last_unstale":"2024-09-13T08:38:38.309663+0000","last_undegraded":"2024-09-13T08:38:38.309663+0000","last_fullsized":"2024-09-13T08:38:38.309663+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:20.944856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978065+0000","last_change":"2024-09-13T08:38:35.980735+0000","last_active":"2024-09-13T08:38:37.978065+0000","last_peered":"2024-09-13T08:38:37.978065+0000","last_clean":"2024-09-13T08:38:37.978065+0000","last_became_active":"2024-09-13T08:38:35.980417+0000","last_became_peered":"2024-09-13T08:38:35.980417+0000","last_unstale":"2024-09-13T08:38:37.978065+0000","last_undegraded":"2024-09-13T08:38:37.978065+0000","last_fullsized":"2024-09-13T08:38:37.978065+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:19.935473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977824+0000","last_change":"2024-09-13T08:38:36.998184+0000","last_active":"2024-09-13T08:38:37.977824+0000","last_peered":"2024-09-13T08:38:37.977824+0000","last_clean":"2024-09-13T08:38:37.977824+0000","last_became_active":"2024-09-13T08:38:36.997768+0000","last_became_peered":"2024-09-13T08:38:36.997768+0000","last_unstale":"2024-09-13T08:38:37.977824+0000","last_undegraded":"2024-09-13T08:38:37.977824+0000","last_fullsized":"2024-09-13T08:38:37.977824+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:31.035669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277680+0000","last_change":"2024-09-13T08:38:35.971969+0000","last_active":"2024-09-13T08:38:38.277680+0000","last_peered":"2024-09-13T08:38:38.277680+0000","last_clean":"2024-09-13T08:38:38.277680+0000","last_became_active":"2024-09-13T08:38:35.971551+0000","last_became_peered":"2024-09-13T08:38:35.971551+0000","last_unstale":"2024-09-13T08:38:38.277680+0000","last_undegraded":"2024-09-13T08:38:38.277680+0000","last_fullsized":"2024-09-13T08:38:38.277680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:01.920028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.992719+0000","last_change":"2024-09-13T08:38:36.985088+0000","last_active":"2024-09-13T08:38:36.992719+0000","last_peered":"2024-09-13T08:38:36.992719+0000","last_clean":"2024-09-13T08:38:36.992719+0000","last_became_active":"2024-09-13T08:38:36.984815+0000","last_became_peered":"2024-09-13T08:38:36.984815+0000","last_unstale":"2024-09-13T08:38:36.992719+0000","last_undegraded":"2024-09-13T08:38:36.992719+0000","last_fullsized":"2024-09-13T08:38:36.992719+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:39.810743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977550+0000","last_change":"2024-09-13T08:38:35.982396+0000","last_active":"2024-09-13T08:38:37.977550+0000","last_peered":"2024-09-13T08:38:37.977550+0000","last_clean":"2024-09-13T08:38:37.977550+0000","last_became_active":"2024-09-13T08:38:35.981259+0000","last_became_peered":"2024-09-13T08:38:35.981259+0000","last_unstale":"2024-09-13T08:38:37.977550+0000","last_undegraded":"2024-09-13T08:38:37.977550+0000","last_fullsized":"2024-09-13T08:38:37.977550+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:53.047851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.995997+0000","last_change":"2024-09-13T08:38:36.987818+0000","last_active":"2024-09-13T08:38:36.995997+0000","last_peered":"2024-09-13T08:38:36.995997+0000","last_clean":"2024-09-13T08:38:36.995997+0000","last_became_active":"2024-09-13T08:38:36.987556+0000","last_became_peered":"2024-09-13T08:38:36.987556+0000","last_unstale":"2024-09-13T08:38:36.995997+0000","last_undegraded":"2024-09-13T08:38:36.995997+0000","last_fullsized":"2024-09-13T08:38:36.995997+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:12:22.223773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309220+0000","last_change":"2024-09-13T08:38:35.980406+0000","last_active":"2024-09-13T08:38:38.309220+0000","last_peered":"2024-09-13T08:38:38.309220+0000","last_clean":"2024-09-13T08:38:38.309220+0000","last_became_active":"2024-09-13T08:38:35.973870+0000","last_became_peered":"2024-09-13T08:38:35.973870+0000","last_unstale":"2024-09-13T08:38:38.309220+0000","last_undegraded":"2024-09-13T08:38:38.309220+0000","last_fullsized":"2024-09-13T08:38:38.309220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977761+0000","last_change":"2024-09-13T08:38:36.989200+0000","last_active":"2024-09-13T08:38:37.977761+0000","last_peered":"2024-09-13T08:38:37.977761+0000","last_clean":"2024-09-13T08:38:37.977761+0000","last_became_active":"2024-09-13T08:38:36.987601+0000","last_became_peered":"2024-09-13T08:38:36.987601+0000","last_unstale":"2024-09-13T08:38:37.977761+0000","last_undegraded":"2024-09-13T08:38:37.977761+0000","last_fullsized":"2024-09-13T08:38:37.977761+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:25.936306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.965600+0000","last_change":"2024-09-13T08:38:35.982758+0000","last_active":"2024-09-13T08:38:36.965600+0000","last_peered":"2024-09-13T08:38:36.965600+0000","last_clean":"2024-09-13T08:38:36.965600+0000","last_became_active":"2024-09-13T08:38:35.982164+0000","last_became_peered":"2024-09-13T08:38:35.982164+0000","last_unstale":"2024-09-13T08:38:36.965600+0000","last_undegraded":"2024-09-13T08:38:36.965600+0000","last_fullsized":"2024-09-13T08:38:36.965600+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:57:04.832009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309335+0000","last_change":"2024-09-13T08:38:36.969592+0000","last_active":"2024-09-13T08:38:38.309335+0000","last_peered":"2024-09-13T08:38:38.309335+0000","last_clean":"2024-09-13T08:38:38.309335+0000","last_became_active":"2024-09-13T08:38:36.969303+0000","last_became_peered":"2024-09-13T08:38:36.969303+0000","last_unstale":"2024-09-13T08:38:38.309335+0000","last_undegraded":"2024-09-13T08:38:38.309335+0000","last_fullsized":"2024-09-13T08:38:38.309335+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:18.930654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309142+0000","last_change":"2024-09-13T08:38:35.972396+0000","last_active":"2024-09-13T08:38:38.309142+0000","last_peered":"2024-09-13T08:38:38.309142+0000","last_clean":"2024-09-13T08:38:38.309142+0000","last_became_active":"2024-09-13T08:38:35.972103+0000","last_became_peered":"2024-09-13T08:38:35.972103+0000","last_unstale":"2024-09-13T08:38:38.309142+0000","last_undegraded":"2024-09-13T08:38:38.309142+0000","last_fullsized":"2024-09-13T08:38:38.309142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:17.203949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309450+0000","last_change":"2024-09-13T08:38:36.992997+0000","last_active":"2024-09-13T08:38:38.309450+0000","last_peered":"2024-09-13T08:38:38.309450+0000","last_clean":"2024-09-13T08:38:38.309450+0000","last_became_active":"2024-09-13T08:38:36.992729+0000","last_became_peered":"2024-09-13T08:38:36.992729+0000","last_unstale":"2024-09-13T08:38:38.309450+0000","last_undegraded":"2024-09-13T08:38:38.309450+0000","last_fullsized":"2024-09-13T08:38:38.309450+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:18.297013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.965473+0000","last_change":"2024-09-13T08:38:35.982428+0000","last_active":"2024-09-13T08:38:36.965473+0000","last_peered":"2024-09-13T08:38:36.965473+0000","last_clean":"2024-09-13T08:38:36.965473+0000","last_became_active":"2024-09-13T08:38:35.981773+0000","last_became_peered":"2024-09-13T08:38:35.981773+0000","last_unstale":"2024-09-13T08:38:36.965473+0000","last_undegraded":"2024-09-13T08:38:36.965473+0000","last_fullsized":"2024-09-13T08:38:36.965473+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:12.594169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309354+0000","last_change":"2024-09-13T08:38:36.989498+0000","last_active":"2024-09-13T08:38:38.309354+0000","last_peered":"2024-09-13T08:38:38.309354+0000","last_clean":"2024-09-13T08:38:38.309354+0000","last_became_active":"2024-09-13T08:38:36.985877+0000","last_became_peered":"2024-09-13T08:38:36.985877+0000","last_unstale":"2024-09-13T08:38:38.309354+0000","last_undegraded":"2024-09-13T08:38:38.309354+0000","last_fullsized":"2024-09-13T08:38:38.309354+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:20.539807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":18,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309873+0000","last_change":"2024-09-13T08:38:35.971378+0000","last_active":"2024-09-13T08:38:38.309873+0000","last_peered":"2024-09-13T08:38:38.309873+0000","last_clean":"2024-09-13T08:38:38.309873+0000","last_became_active":"2024-09-13T08:38:35.971078+0000","last_became_peered":"2024-09-13T08:38:35.971078+0000","last_unstale":"2024-09-13T08:38:38.309873+0000","last_undegraded":"2024-09-13T08:38:38.309873+0000","last_fullsized":"2024-09-13T08:38:38.309873+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:19.825456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977964+0000","last_change":"2024-09-13T08:38:36.995852+0000","last_active":"2024-09-13T08:38:37.977964+0000","last_peered":"2024-09-13T08:38:37.977964+0000","last_clean":"2024-09-13T08:38:37.977964+0000","last_became_active":"2024-09-13T08:38:36.995581+0000","last_became_peered":"2024-09-13T08:38:36.995581+0000","last_unstale":"2024-09-13T08:38:37.977964+0000","last_undegraded":"2024-09-13T08:38:37.977964+0000","last_fullsized":"2024-09-13T08:38:37.977964+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:34:20.397099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.006895+0000","last_change":"2024-09-13T08:38:35.986508+0000","last_active":"2024-09-13T08:38:37.006895+0000","last_peered":"2024-09-13T08:38:37.006895+0000","last_clean":"2024-09-13T08:38:37.006895+0000","last_became_active":"2024-09-13T08:38:35.986183+0000","last_became_peered":"2024-09-13T08:38:35.986183+0000","last_unstale":"2024-09-13T08:38:37.006895+0000","last_undegraded":"2024-09-13T08:38:37.006895+0000","last_fullsized":"2024-09-13T08:38:37.006895+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:47:18.762193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309265+0000","last_change":"2024-09-13T08:38:36.986089+0000","last_active":"2024-09-13T08:38:38.309265+0000","last_peered":"2024-09-13T08:38:38.309265+0000","last_clean":"2024-09-13T08:38:38.309265+0000","last_became_active":"2024-09-13T08:38:36.985807+0000","last_became_peered":"2024-09-13T08:38:36.985807+0000","last_unstale":"2024-09-13T08:38:38.309265+0000","last_undegraded":"2024-09-13T08:38:38.309265+0000","last_fullsized":"2024-09-13T08:38:38.309265+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:03:21.391056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978783+0000","last_change":"2024-09-13T08:38:35.976190+0000","last_active":"2024-09-13T08:38:37.978783+0000","last_peered":"2024-09-13T08:38:37.978783+0000","last_clean":"2024-09-13T08:38:37.978783+0000","last_became_active":"2024-09-13T08:38:35.975860+0000","last_became_peered":"2024-09-13T08:38:35.975860+0000","last_unstale":"2024-09-13T08:38:37.978783+0000","last_undegraded":"2024-09-13T08:38:37.978783+0000","last_fullsized":"2024-09-13T08:38:37.978783+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:01.551693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978678+0000","last_change":"2024-09-13T08:38:36.977472+0000","last_active":"2024-09-13T08:38:37.978678+0000","last_peered":"2024-09-13T08:38:37.978678+0000","last_clean":"2024-09-13T08:38:37.978678+0000","last_became_active":"2024-09-13T08:38:36.977126+0000","last_became_peered":"2024-09-13T08:38:36.977126+0000","last_unstale":"2024-09-13T08:38:37.978678+0000","last_undegraded":"2024-09-13T08:38:37.978678+0000","last_fullsized":"2024-09-13T08:38:37.978678+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:30.442843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310193+0000","last_change":"2024-09-13T08:38:35.970380+0000","last_active":"2024-09-13T08:38:38.310193+0000","last_peered":"2024-09-13T08:38:38.310193+0000","last_clean":"2024-09-13T08:38:38.310193+0000","last_became_active":"2024-09-13T08:38:35.969233+0000","last_became_peered":"2024-09-13T08:38:35.969233+0000","last_unstale":"2024-09-13T08:38:38.310193+0000","last_undegraded":"2024-09-13T08:38:38.310193+0000","last_fullsized":"2024-09-13T08:38:38.310193+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:46.376760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978369+0000","last_change":"2024-09-13T08:38:36.972042+0000","last_active":"2024-09-13T08:38:37.978369+0000","last_peered":"2024-09-13T08:38:37.978369+0000","last_clean":"2024-09-13T08:38:37.978369+0000","last_became_active":"2024-09-13T08:38:36.971784+0000","last_became_peered":"2024-09-13T08:38:36.971784+0000","last_unstale":"2024-09-13T08:38:37.978369+0000","last_undegraded":"2024-09-13T08:38:37.978369+0000","last_fullsized":"2024-09-13T08:38:37.978369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:06:15.641433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277323+0000","last_change":"2024-09-13T08:38:35.976982+0000","last_active":"2024-09-13T08:38:38.277323+0000","last_peered":"2024-09-13T08:38:38.277323+0000","last_clean":"2024-09-13T08:38:38.277323+0000","last_became_active":"2024-09-13T08:38:35.976727+0000","last_became_peered":"2024-09-13T08:38:35.976727+0000","last_unstale":"2024-09-13T08:38:38.277323+0000","last_undegraded":"2024-09-13T08:38:38.277323+0000","last_fullsized":"2024-09-13T08:38:38.277323+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:45:44.532236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309010+0000","last_change":"2024-09-13T08:38:36.997329+0000","last_active":"2024-09-13T08:38:38.309010+0000","last_peered":"2024-09-13T08:38:38.309010+0000","last_clean":"2024-09-13T08:38:38.309010+0000","last_became_active":"2024-09-13T08:38:36.997059+0000","last_became_peered":"2024-09-13T08:38:36.997059+0000","last_unstale":"2024-09-13T08:38:38.309010+0000","last_undegraded":"2024-09-13T08:38:38.309010+0000","last_fullsized":"2024-09-13T08:38:38.309010+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.455558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":21,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.000752+0000","last_change":"2024-09-13T08:38:35.974864+0000","last_active":"2024-09-13T08:38:37.000752+0000","last_peered":"2024-09-13T08:38:37.000752+0000","last_clean":"2024-09-13T08:38:37.000752+0000","last_became_active":"2024-09-13T08:38:35.974501+0000","last_became_peered":"2024-09-13T08:38:35.974501+0000","last_unstale":"2024-09-13T08:38:37.000752+0000","last_undegraded":"2024-09-13T08:38:37.000752+0000","last_fullsized":"2024-09-13T08:38:37.000752+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:46.238519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979319+0000","last_change":"2024-09-13T08:38:36.970506+0000","last_active":"2024-09-13T08:38:37.979319+0000","last_peered":"2024-09-13T08:38:37.979319+0000","last_clean":"2024-09-13T08:38:37.979319+0000","last_became_active":"2024-09-13T08:38:36.970223+0000","last_became_peered":"2024-09-13T08:38:36.970223+0000","last_unstale":"2024-09-13T08:38:37.979319+0000","last_undegraded":"2024-09-13T08:38:37.979319+0000","last_fullsized":"2024-09-13T08:38:37.979319+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:45.406855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309747+0000","last_change":"2024-09-13T08:38:35.981247+0000","last_active":"2024-09-13T08:38:38.309747+0000","last_peered":"2024-09-13T08:38:38.309747+0000","last_clean":"2024-09-13T08:38:38.309747+0000","last_became_active":"2024-09-13T08:38:35.980838+0000","last_became_peered":"2024-09-13T08:38:35.980838+0000","last_unstale":"2024-09-13T08:38:38.309747+0000","last_undegraded":"2024-09-13T08:38:38.309747+0000","last_fullsized":"2024-09-13T08:38:38.309747+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:02.412059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.979678+0000","last_change":"2024-09-13T08:38:36.969498+0000","last_active":"2024-09-13T08:38:37.979678+0000","last_peered":"2024-09-13T08:38:37.979678+0000","last_clean":"2024-09-13T08:38:37.979678+0000","last_became_active":"2024-09-13T08:38:36.969200+0000","last_became_peered":"2024-09-13T08:38:36.969200+0000","last_unstale":"2024-09-13T08:38:37.979678+0000","last_undegraded":"2024-09-13T08:38:37.979678+0000","last_fullsized":"2024-09-13T08:38:37.979678+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:30:09.901419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309374+0000","last_change":"2024-09-13T08:38:35.972332+0000","last_active":"2024-09-13T08:38:38.309374+0000","last_peered":"2024-09-13T08:38:38.309374+0000","last_clean":"2024-09-13T08:38:38.309374+0000","last_became_active":"2024-09-13T08:38:35.971977+0000","last_became_peered":"2024-09-13T08:38:35.971977+0000","last_unstale":"2024-09-13T08:38:38.309374+0000","last_undegraded":"2024-09-13T08:38:38.309374+0000","last_fullsized":"2024-09-13T08:38:38.309374+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:51.164377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309964+0000","last_change":"2024-09-13T08:38:36.994845+0000","last_active":"2024-09-13T08:38:38.309964+0000","last_peered":"2024-09-13T08:38:38.309964+0000","last_clean":"2024-09-13T08:38:38.309964+0000","last_became_active":"2024-09-13T08:38:36.994549+0000","last_became_peered":"2024-09-13T08:38:36.994549+0000","last_unstale":"2024-09-13T08:38:38.309964+0000","last_undegraded":"2024-09-13T08:38:38.309964+0000","last_fullsized":"2024-09-13T08:38:38.309964+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:53.221145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308964+0000","last_change":"2024-09-13T08:38:35.983889+0000","last_active":"2024-09-13T08:38:38.308964+0000","last_peered":"2024-09-13T08:38:38.308964+0000","last_clean":"2024-09-13T08:38:38.308964+0000","last_became_active":"2024-09-13T08:38:35.982289+0000","last_became_peered":"2024-09-13T08:38:35.982289+0000","last_unstale":"2024-09-13T08:38:38.308964+0000","last_undegraded":"2024-09-13T08:38:38.308964+0000","last_fullsized":"2024-09-13T08:38:38.308964+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:32.480346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308892+0000","last_change":"2024-09-13T08:38:36.977348+0000","last_active":"2024-09-13T08:38:38.308892+0000","last_peered":"2024-09-13T08:38:38.308892+0000","last_clean":"2024-09-13T08:38:38.308892+0000","last_became_active":"2024-09-13T08:38:36.977084+0000","last_became_peered":"2024-09-13T08:38:36.977084+0000","last_unstale":"2024-09-13T08:38:38.308892+0000","last_undegraded":"2024-09-13T08:38:38.308892+0000","last_fullsized":"2024-09-13T08:38:38.308892+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:39.966579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277502+0000","last_change":"2024-09-13T08:38:35.978119+0000","last_active":"2024-09-13T08:38:38.277502+0000","last_peered":"2024-09-13T08:38:38.277502+0000","last_clean":"2024-09-13T08:38:38.277502+0000","last_became_active":"2024-09-13T08:38:35.976443+0000","last_became_peered":"2024-09-13T08:38:35.976443+0000","last_unstale":"2024-09-13T08:38:38.277502+0000","last_undegraded":"2024-09-13T08:38:38.277502+0000","last_fullsized":"2024-09-13T08:38:38.277502+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:54:02.906350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.983829+0000","last_change":"2024-09-13T08:38:36.981947+0000","last_active":"2024-09-13T08:38:36.983829+0000","last_peered":"2024-09-13T08:38:36.983829+0000","last_clean":"2024-09-13T08:38:36.983829+0000","last_became_active":"2024-09-13T08:38:36.980929+0000","last_became_peered":"2024-09-13T08:38:36.980929+0000","last_unstale":"2024-09-13T08:38:36.983829+0000","last_undegraded":"2024-09-13T08:38:36.983829+0000","last_fullsized":"2024-09-13T08:38:36.983829+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:45.228779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.002522+0000","last_change":"2024-09-13T08:38:35.982084+0000","last_active":"2024-09-13T08:38:37.002522+0000","last_peered":"2024-09-13T08:38:37.002522+0000","last_clean":"2024-09-13T08:38:37.002522+0000","last_became_active":"2024-09-13T08:38:35.981666+0000","last_became_peered":"2024-09-13T08:38:35.981666+0000","last_unstale":"2024-09-13T08:38:37.002522+0000","last_undegraded":"2024-09-13T08:38:37.002522+0000","last_fullsized":"2024-09-13T08:38:37.002522+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:50.240478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277502+0000","last_change":"2024-09-13T08:38:36.978758+0000","last_active":"2024-09-13T08:38:38.277502+0000","last_peered":"2024-09-13T08:38:38.277502+0000","last_clean":"2024-09-13T08:38:38.277502+0000","last_became_active":"2024-09-13T08:38:36.978436+0000","last_became_peered":"2024-09-13T08:38:36.978436+0000","last_unstale":"2024-09-13T08:38:38.277502+0000","last_undegraded":"2024-09-13T08:38:38.277502+0000","last_fullsized":"2024-09-13T08:38:38.277502+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.455558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310099+0000","last_change":"2024-09-13T08:38:35.979632+0000","last_active":"2024-09-13T08:38:38.310099+0000","last_peered":"2024-09-13T08:38:38.310099+0000","last_clean":"2024-09-13T08:38:38.310099+0000","last_became_active":"2024-09-13T08:38:35.979339+0000","last_became_peered":"2024-09-13T08:38:35.979339+0000","last_unstale":"2024-09-13T08:38:38.310099+0000","last_undegraded":"2024-09-13T08:38:38.310099+0000","last_fullsized":"2024-09-13T08:38:38.310099+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:35.170683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309220+0000","last_change":"2024-09-13T08:38:36.980687+0000","last_active":"2024-09-13T08:38:38.309220+0000","last_peered":"2024-09-13T08:38:38.309220+0000","last_clean":"2024-09-13T08:38:38.309220+0000","last_became_active":"2024-09-13T08:38:36.980408+0000","last_became_peered":"2024-09-13T08:38:36.980408+0000","last_unstale":"2024-09-13T08:38:38.309220+0000","last_undegraded":"2024-09-13T08:38:38.309220+0000","last_fullsized":"2024-09-13T08:38:38.309220+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:35:09.103775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.963487+0000","last_change":"2024-09-13T08:38:35.976033+0000","last_active":"2024-09-13T08:38:36.963487+0000","last_peered":"2024-09-13T08:38:36.963487+0000","last_clean":"2024-09-13T08:38:36.963487+0000","last_became_active":"2024-09-13T08:38:35.975464+0000","last_became_peered":"2024-09-13T08:38:35.975464+0000","last_unstale":"2024-09-13T08:38:36.963487+0000","last_undegraded":"2024-09-13T08:38:36.963487+0000","last_fullsized":"2024-09-13T08:38:36.963487+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:01.658298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309071+0000","last_change":"2024-09-13T08:38:36.983639+0000","last_active":"2024-09-13T08:38:38.309071+0000","last_peered":"2024-09-13T08:38:38.309071+0000","last_clean":"2024-09-13T08:38:38.309071+0000","last_became_active":"2024-09-13T08:38:36.983327+0000","last_became_peered":"2024-09-13T08:38:36.983327+0000","last_unstale":"2024-09-13T08:38:38.309071+0000","last_undegraded":"2024-09-13T08:38:38.309071+0000","last_fullsized":"2024-09-13T08:38:38.309071+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:31.782847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977738+0000","last_change":"2024-09-13T08:38:35.977525+0000","last_active":"2024-09-13T08:38:37.977738+0000","last_peered":"2024-09-13T08:38:37.977738+0000","last_clean":"2024-09-13T08:38:37.977738+0000","last_became_active":"2024-09-13T08:38:35.977346+0000","last_became_peered":"2024-09-13T08:38:35.977346+0000","last_unstale":"2024-09-13T08:38:37.977738+0000","last_undegraded":"2024-09-13T08:38:37.977738+0000","last_fullsized":"2024-09-13T08:38:37.977738+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:28.944099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978039+0000","last_change":"2024-09-13T08:38:36.988070+0000","last_active":"2024-09-13T08:38:37.978039+0000","last_peered":"2024-09-13T08:38:37.978039+0000","last_clean":"2024-09-13T08:38:37.978039+0000","last_became_active":"2024-09-13T08:38:36.987808+0000","last_became_peered":"2024-09-13T08:38:36.987808+0000","last_unstale":"2024-09-13T08:38:37.978039+0000","last_undegraded":"2024-09-13T08:38:37.978039+0000","last_fullsized":"2024-09-13T08:38:37.978039+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:36.635788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309479+0000","last_change":"2024-09-13T08:38:35.986462+0000","last_active":"2024-09-13T08:38:38.309479+0000","last_peered":"2024-09-13T08:38:38.309479+0000","last_clean":"2024-09-13T08:38:38.309479+0000","last_became_active":"2024-09-13T08:38:35.986216+0000","last_became_peered":"2024-09-13T08:38:35.986216+0000","last_unstale":"2024-09-13T08:38:38.309479+0000","last_undegraded":"2024-09-13T08:38:38.309479+0000","last_fullsized":"2024-09-13T08:38:38.309479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:54:02.906350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.997393+0000","last_change":"2024-09-13T08:38:36.994000+0000","last_active":"2024-09-13T08:38:36.997393+0000","last_peered":"2024-09-13T08:38:36.997393+0000","last_clean":"2024-09-13T08:38:36.997393+0000","last_became_active":"2024-09-13T08:38:36.992137+0000","last_became_peered":"2024-09-13T08:38:36.992137+0000","last_unstale":"2024-09-13T08:38:36.997393+0000","last_undegraded":"2024-09-13T08:38:36.997393+0000","last_fullsized":"2024-09-13T08:38:36.997393+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:38.542302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.310261+0000","last_change":"2024-09-13T08:38:35.976885+0000","last_active":"2024-09-13T08:38:38.310261+0000","last_peered":"2024-09-13T08:38:38.310261+0000","last_clean":"2024-09-13T08:38:38.310261+0000","last_became_active":"2024-09-13T08:38:35.976270+0000","last_became_peered":"2024-09-13T08:38:35.976270+0000","last_unstale":"2024-09-13T08:38:38.310261+0000","last_undegraded":"2024-09-13T08:38:38.310261+0000","last_fullsized":"2024-09-13T08:38:38.310261+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:13.773728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978146+0000","last_change":"2024-09-13T08:38:36.995799+0000","last_active":"2024-09-13T08:38:37.978146+0000","last_peered":"2024-09-13T08:38:37.978146+0000","last_clean":"2024-09-13T08:38:37.978146+0000","last_became_active":"2024-09-13T08:38:36.995276+0000","last_became_peered":"2024-09-13T08:38:36.995276+0000","last_unstale":"2024-09-13T08:38:37.978146+0000","last_undegraded":"2024-09-13T08:38:37.978146+0000","last_fullsized":"2024-09-13T08:38:37.978146+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:37.521180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978991+0000","last_change":"2024-09-13T08:38:35.972242+0000","last_active":"2024-09-13T08:38:37.978991+0000","last_peered":"2024-09-13T08:38:37.978991+0000","last_clean":"2024-09-13T08:38:37.978991+0000","last_became_active":"2024-09-13T08:38:35.971769+0000","last_became_peered":"2024-09-13T08:38:35.971769+0000","last_unstale":"2024-09-13T08:38:37.978991+0000","last_undegraded":"2024-09-13T08:38:37.978991+0000","last_fullsized":"2024-09-13T08:38:37.978991+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:52:36.718111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T08:38:36.983456+0000","last_change":"2024-09-13T08:38:36.981500+0000","last_active":"2024-09-13T08:38:36.983456+0000","last_peered":"2024-09-13T08:38:36.983456+0000","last_clean":"2024-09-13T08:38:36.983456+0000","last_became_active":"2024-09-13T08:38:36.980588+0000","last_became_peered":"2024-09-13T08:38:36.980588+0000","last_unstale":"2024-09-13T08:38:36.983456+0000","last_undegraded":"2024-09-13T08:38:36.983456+0000","last_fullsized":"2024-09-13T08:38:36.983456+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:21.041457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309712+0000","last_change":"2024-09-13T08:38:35.973799+0000","last_active":"2024-09-13T08:38:38.309712+0000","last_peered":"2024-09-13T08:38:38.309712+0000","last_clean":"2024-09-13T08:38:38.309712+0000","last_became_active":"2024-09-13T08:38:35.973523+0000","last_became_peered":"2024-09-13T08:38:35.973523+0000","last_unstale":"2024-09-13T08:38:38.309712+0000","last_undegraded":"2024-09-13T08:38:38.309712+0000","last_fullsized":"2024-09-13T08:38:38.309712+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:10.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309594+0000","last_change":"2024-09-13T08:38:36.982987+0000","last_active":"2024-09-13T08:38:38.309594+0000","last_peered":"2024-09-13T08:38:38.309594+0000","last_clean":"2024-09-13T08:38:38.309594+0000","last_became_active":"2024-09-13T08:38:36.982676+0000","last_became_peered":"2024-09-13T08:38:36.982676+0000","last_unstale":"2024-09-13T08:38:38.309594+0000","last_undegraded":"2024-09-13T08:38:38.309594+0000","last_fullsized":"2024-09-13T08:38:38.309594+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:42.254713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977667+0000","last_change":"2024-09-13T08:38:35.986366+0000","last_active":"2024-09-13T08:38:37.977667+0000","last_peered":"2024-09-13T08:38:37.977667+0000","last_clean":"2024-09-13T08:38:37.977667+0000","last_became_active":"2024-09-13T08:38:35.986083+0000","last_became_peered":"2024-09-13T08:38:35.986083+0000","last_unstale":"2024-09-13T08:38:37.977667+0000","last_undegraded":"2024-09-13T08:38:37.977667+0000","last_fullsized":"2024-09-13T08:38:37.977667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:29.172988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308695+0000","last_change":"2024-09-13T08:38:36.986721+0000","last_active":"2024-09-13T08:38:38.308695+0000","last_peered":"2024-09-13T08:38:38.308695+0000","last_clean":"2024-09-13T08:38:38.308695+0000","last_became_active":"2024-09-13T08:38:36.986450+0000","last_became_peered":"2024-09-13T08:38:36.986450+0000","last_unstale":"2024-09-13T08:38:38.308695+0000","last_undegraded":"2024-09-13T08:38:38.308695+0000","last_fullsized":"2024-09-13T08:38:38.308695+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:01.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.277422+0000","last_change":"2024-09-13T08:38:35.982648+0000","last_active":"2024-09-13T08:38:38.277422+0000","last_peered":"2024-09-13T08:38:38.277422+0000","last_clean":"2024-09-13T08:38:38.277422+0000","last_became_active":"2024-09-13T08:38:35.982370+0000","last_became_peered":"2024-09-13T08:38:35.982370+0000","last_unstale":"2024-09-13T08:38:38.277422+0000","last_undegraded":"2024-09-13T08:38:38.277422+0000","last_fullsized":"2024-09-13T08:38:38.277422+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:02.412059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309013+0000","last_change":"2024-09-13T08:38:36.993462+0000","last_active":"2024-09-13T08:38:38.309013+0000","last_peered":"2024-09-13T08:38:38.309013+0000","last_clean":"2024-09-13T08:38:38.309013+0000","last_became_active":"2024-09-13T08:38:36.992818+0000","last_became_peered":"2024-09-13T08:38:36.992818+0000","last_unstale":"2024-09-13T08:38:38.309013+0000","last_undegraded":"2024-09-13T08:38:38.309013+0000","last_fullsized":"2024-09-13T08:38:38.309013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:31.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.978467+0000","last_change":"2024-09-13T08:38:35.977566+0000","last_active":"2024-09-13T08:38:37.978467+0000","last_peered":"2024-09-13T08:38:37.978467+0000","last_clean":"2024-09-13T08:38:37.978467+0000","last_became_active":"2024-09-13T08:38:35.977254+0000","last_became_peered":"2024-09-13T08:38:35.977254+0000","last_unstale":"2024-09-13T08:38:37.978467+0000","last_undegraded":"2024-09-13T08:38:37.978467+0000","last_fullsized":"2024-09-13T08:38:37.978467+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:14:20.559469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.309573+0000","last_change":"2024-09-13T08:38:36.993737+0000","last_active":"2024-09-13T08:38:38.309573+0000","last_peered":"2024-09-13T08:38:38.309573+0000","last_clean":"2024-09-13T08:38:38.309573+0000","last_became_active":"2024-09-13T08:38:36.992965+0000","last_became_peered":"2024-09-13T08:38:36.992965+0000","last_unstale":"2024-09-13T08:38:38.309573+0000","last_undegraded":"2024-09-13T08:38:38.309573+0000","last_fullsized":"2024-09-13T08:38:38.309573+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:35.950111+0000","last_clean_scrub_stamp":"2024-09-13T08:38:35.950111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:30.182371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:37.977899+0000","last_change":"2024-09-13T08:38:35.986117+0000","last_active":"2024-09-13T08:38:37.977899+0000","last_peered":"2024-09-13T08:38:37.977899+0000","last_clean":"2024-09-13T08:38:37.977899+0000","last_became_active":"2024-09-13T08:38:35.985799+0000","last_became_peered":"2024-09-13T08:38:35.985799+0000","last_unstale":"2024-09-13T08:38:37.977899+0000","last_undegraded":"2024-09-13T08:38:37.977899+0000","last_fullsized":"2024-09-13T08:38:37.977899+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:38:34.940728+0000","last_clean_scrub_stamp":"2024-09-13T08:38:34.940728+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:44.280453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":154,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T08:38:38.308359+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:38:38.308359+0000","last_peered":"2024-09-13T08:38:38.308359+0000","last_clean":"2024-09-13T08:38:38.308359+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:38:38.308359+0000","last_undegraded":"2024-09-13T08:38:38.308359+0000","last_fullsized":"2024-09-13T08:38:38.308359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":548864,"data_stored":218478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31732,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340108,"statfs":{"total":96636764160,"available":96604270592,"internally_reserved":0,"allocated":552960,"data_stored":219060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":808834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":557056,"data_stored":220056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":548864,"data_stored":218534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1142784,"data_stored":808882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509462,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":552960,"data_stored":219116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":532480,"data_stored":211669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:38:41.648 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:38:41.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.851+0000 7ffa56c00640 1 -- 172.21.15.31:0/252825237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa34003a50 msgr2=0x7ffa34003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.851+0000 7ffa56c00640 1 --2- 172.21.15.31:0/252825237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa34003a50 0x7ffa34003e70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ffa40001690 tx=0x7ffa34005830 comp rx=0 tx=0).stop 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.851+0000 7ffa56c00640 1 -- 172.21.15.31:0/252825237 shutdown_connections 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.851+0000 7ffa56c00640 1 --2- 172.21.15.31:0/252825237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa34003a50 0x7ffa34003e70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.851+0000 7ffa56c00640 1 -- 172.21.15.31:0/252825237 >> 172.21.15.31:0/252825237 conn(0x7ffa500cfbb0 msgr2=0x7ffa500cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.852+0000 7ffa56c00640 1 -- 172.21.15.31:0/252825237 shutdown_connections 2024-09-13T08:38:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.852+0000 7ffa56c00640 1 -- 172.21.15.31:0/252825237 wait complete. 2024-09-13T08:38:41.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.853+0000 7ffa56c00640 1 Processor -- start 2024-09-13T08:38:41.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.853+0000 7ffa56c00640 1 -- start start 2024-09-13T08:38:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa34003a50 0x7ffa500b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa501503b0 0x7ffa501f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa500cbfd0 con 0x7ffa501f7ea0 2024-09-13T08:38:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa500cbe50 con 0x7ffa34003a50 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa56c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffa50070010 con 0x7ffa501503b0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa54c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa54c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44778/0 (socket says 172.21.15.31:44778) 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.854+0000 7ffa54c00640 1 -- 172.21.15.31:0/4046705365 learned_addr learned my addr 172.21.15.31:0/4046705365 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa4fe00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa34003a50 0x7ffa500b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa47400640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa501503b0 0x7ffa501f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 -- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa501503b0 msgr2=0x7ffa501f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa501503b0 0x7ffa501f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 -- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa34003a50 msgr2=0x7ffa500b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa34003a50 0x7ffa500b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 -- 172.21.15.31:0/4046705365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa500c63b0 con 0x7ffa501f7ea0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa54c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffa500c6c10 tx=0x7ffa48010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:41.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48003e40 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa48015070 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800ed10 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.855+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa500b4500 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.856+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa500b4ac0 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.857+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa500cdb60 con 0x7ffa501f7ea0 2024-09-13T08:38:41.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.858+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7ffa48004050 con 0x7ffa501f7ea0 2024-09-13T08:38:41.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.859+0000 7ffa4cc00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 0x7ffa2c07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:41.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.859+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7ffa4809b680 con 0x7ffa501f7ea0 2024-09-13T08:38:41.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.859+0000 7ffa4fe00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 0x7ffa2c07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:41.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.860+0000 7ffa4fe00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 0x7ffa2c07aef0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ffa40032000 tx=0x7ffa4002c040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:41.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:41.862+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffa48063320 con 0x7ffa501f7ea0 2024-09-13T08:38:41.934 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:38:41.933+0000 7fb0f5800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:38:42.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.131+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ffa500b4c90 con 0x7ffa501f7ea0 2024-09-13T08:38:42.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.135+0000 7ffa4cc00640 1 -- 172.21.15.31:0/4046705365 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) ==== 190+0+147 (secure 0 0 0) 0x7ffa48067d50 con 0x7ffa501f7ea0 2024-09-13T08:38:42.136 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:38:42.137 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:38:42.137 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:38:42.137 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:38:42.137 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:38:42.137 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:38:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 msgr2=0x7ffa2c07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 0x7ffa2c07aef0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ffa40032000 tx=0x7ffa4002c040 comp rx=0 tx=0).stop 2024-09-13T08:38:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 msgr2=0x7ffa501f8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffa500c6c10 tx=0x7ffa48010f40 comp rx=0 tx=0).stop 2024-09-13T08:38:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 shutdown_connections 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa501f7ea0 0x7ffa501f8320 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa2c078a30 0x7ffa2c07aef0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa501503b0 0x7ffa501f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 --2- 172.21.15.31:0/4046705365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa34003a50 0x7ffa500b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 >> 172.21.15.31:0/4046705365 conn(0x7ffa500cfbb0 msgr2=0x7ffa50072cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 shutdown_connections 2024-09-13T08:38:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.138+0000 7ffa56c00640 1 -- 172.21.15.31:0/4046705365 wait complete. 2024-09-13T08:38:42.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 -- 172.21.15.31:0/3557950803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd590003a50 msgr2=0x7fd590003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/3557950803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd590003a50 0x7fd590003e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009d00 tx=0x7fd590005830 comp rx=0 tx=0).stop 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 -- 172.21.15.31:0/3557950803 shutdown_connections 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/3557950803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd590003a50 0x7fd590003e70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 -- 172.21.15.31:0/3557950803 >> 172.21.15.31:0/3557950803 conn(0x7fd5ac0cc910 msgr2=0x7fd5ac0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 -- 172.21.15.31:0/3557950803 shutdown_connections 2024-09-13T08:38:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.315+0000 7fd5b1c00640 1 -- 172.21.15.31:0/3557950803 wait complete. 2024-09-13T08:38:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.317+0000 7fd5b1c00640 1 Processor -- start 2024-09-13T08:38:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.317+0000 7fd5b1c00640 1 -- start start 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd590003a50 0x7fd5ac1f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5ac09d1b0 0x7fd5ac0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ac0cb9d0 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ac0cb850 con 0x7fd5ac09d1b0 2024-09-13T08:38:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5b1c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ac0cf850 con 0x7fd590003a50 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44790/0 (socket says 172.21.15.31:44790) 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.318+0000 7fd5a2000640 1 -- 172.21.15.31:0/1511780820 learned_addr learned my addr 172.21.15.31:0/1511780820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5aaa00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd590003a50 0x7fd5ac1f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5ab400640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5ac09d1b0 0x7fd5ac0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 -- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd590003a50 msgr2=0x7fd5ac1f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd590003a50 0x7fd5ac1f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 -- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5ac09d1b0 msgr2=0x7fd5ac0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5ac09d1b0 0x7fd5ac0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 -- 172.21.15.31:0/1511780820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ac1f8350 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a2000640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd594003fb0 tx=0x7fd594001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594015070 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd594011840 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594011c90 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.319+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ac0b3ff0 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.320+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ac1f85e0 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.320+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ac0ce3c0 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.322+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fd594018070 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.323+0000 7fd5a3400640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 0x7fd58807aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.323+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd59409b880 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.324+0000 7fd5aaa00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 0x7fd58807aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:42.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.324+0000 7fd5aaa00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 0x7fd58807aef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd59c035000 tx=0x7fd59c0046c0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.328+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd594063500 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.560+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5ac0c36e0 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.561+0000 7fd5a3400640 1 -- 172.21.15.31:0/1511780820 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5223 (secure 0 0 0) 0x7fd594067f30 con 0x7fd5ac1f90c0 2024-09-13T08:38:42.562 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:42.562 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"btime":"2024-09-13T08:38:39:983238+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:42.562 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 msgr2=0x7fd58807aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 0x7fd58807aef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd59c035000 tx=0x7fd59c0046c0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 msgr2=0x7fd5ac156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd594003fb0 tx=0x7fd594001630 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 shutdown_connections 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5ac09d1b0 0x7fd5ac0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5ac1f90c0 0x7fd5ac156240 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd588078a30 0x7fd58807aef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 --2- 172.21.15.31:0/1511780820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd590003a50 0x7fd5ac1f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 >> 172.21.15.31:0/1511780820 conn(0x7fd5ac0cc910 msgr2=0x7fd5ac0cbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:42.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 shutdown_connections 2024-09-13T08:38:42.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:42.564+0000 7fd5b1c00640 1 -- 172.21.15.31:0/1511780820 wait complete. 2024-09-13T08:38:42.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:38:36.443046+0000', 'modified': '2024-09-13T08:38:39.419176+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5286': {'gid': 5286, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6834/1342811284', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 1342811284}, {'type': 'v1', 'addr': '172.21.15.31:6834', 'nonce': 1342811284}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5286, 'qdb_cluster': [5286]} 2024-09-13T08:38:42.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:38:42.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:38:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 -- 172.21.15.31:0/2330744569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500d0cd0 msgr2=0x7f5b500d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/2330744569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500d0cd0 0x7f5b500d1110 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5b5006d3f0 tx=0x7f5b3c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 -- 172.21.15.31:0/2330744569 shutdown_connections 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/2330744569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500d0cd0 0x7f5b500d1110 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/2330744569 >> v2:172.21.15.45:3300/0 conn(0x7f5b5015e160 0x7f5b500d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 -- 172.21.15.31:0/2330744569 >> 172.21.15.31:0/2330744569 conn(0x7f5b500d3f50 msgr2=0x7f5b500d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 -- 172.21.15.31:0/2330744569 shutdown_connections 2024-09-13T08:38:43.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.147+0000 7f5b4fe00640 1 -- 172.21.15.31:0/2330744569 wait complete. 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.149+0000 7f5b4fe00640 1 Processor -- start 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.149+0000 7f5b4fe00640 1 -- start start 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b500c8340 0x7f5b50154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b500d0cd0 0x7f5b500b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b500cfe00 con 0x7f5b5015e160 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b500cfc80 con 0x7f5b500d0cd0 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b500c8ee0 con 0x7f5b500c8340 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b500c8340 0x7f5b50154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44808/0 (socket says 172.21.15.31:44808) 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 -- 172.21.15.31:0/3702973916 learned_addr learned my addr 172.21.15.31:0/3702973916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4e000640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b500d0cd0 0x7f5b500b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 -- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b500c8340 msgr2=0x7f5b50154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b500c8340 0x7f5b50154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 -- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b500d0cd0 msgr2=0x7f5b500b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b500d0cd0 0x7f5b500b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.150+0000 7f5b4f400640 1 -- 172.21.15.31:0/3702973916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b50161010 con 0x7f5b5015e160 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b4f400640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800c7d0 tx=0x7f5b4800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b46a00640 1 -- 172.21.15.31:0/3702973916 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48003b70 con 0x7f5b5015e160 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b46a00640 1 -- 172.21.15.31:0/3702973916 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b48003d30 con 0x7f5b5015e160 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b46a00640 1 -- 172.21.15.31:0/3702973916 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800aab0 con 0x7f5b5015e160 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b501612f0 con 0x7f5b5015e160 2024-09-13T08:38:43.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.151+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b500b81e0 con 0x7f5b5015e160 2024-09-13T08:38:43.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.153+0000 7f5b46a00640 1 -- 172.21.15.31:0/3702973916 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f5b4800ac50 con 0x7f5b5015e160 2024-09-13T08:38:43.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.154+0000 7f5b46a00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 0x7f5b3007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.154+0000 7f5b46a00640 1 -- 172.21.15.31:0/3702973916 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5b4809a9a0 con 0x7f5b5015e160 2024-09-13T08:38:43.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.155+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 0x7f5b3007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.155+0000 7f5b58837740 1 Processor -- start 2024-09-13T08:38:43.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.155+0000 7f5b58837740 1 -- start start 2024-09-13T08:38:43.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.155+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 0x7f5b3007aec0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009b30 tx=0x7f5b40005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 0x55cfa2498580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa2498ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cfa23d7430 con 0x55cfa2499aa0 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cfa23c75d0 con 0x55cfa2455190 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55cfa23b6a00 con 0x55cfa2454c10 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44814/0 (socket says 172.21.15.31:44814) 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2510193932 learned_addr learned my addr 172.21.15.31:0/2510193932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4e000640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 0x55cfa2498580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 msgr2=0x55cfa2498580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 0x55cfa2498580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4f400640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa2498ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.156+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 msgr2=0x55cfa2498ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa2498ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2510193932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa237f520 con 0x55cfa2499aa0 2024-09-13T08:38:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b4e000640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 0x55cfa2498580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b4f400640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa2498ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5b400058c0 tx=0x7f5b40014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bcb5d5599a2ec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b3b400640 1 -- 172.21.15.31:0/2510193932 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40003d70 con 0x55cfa2499aa0 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b3b400640 1 -- 172.21.15.31:0/2510193932 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b400044d0 con 0x55cfa2499aa0 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.157+0000 7f5b3b400640 1 -- 172.21.15.31:0/2510193932 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40005280 con 0x55cfa2499aa0 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 -- 172.21.15.31:0/2510193932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 msgr2=0x55cfa2499e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5b400058c0 tx=0x7f5b40014000 comp rx=0 tx=0).stop 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 -- 172.21.15.31:0/2510193932 shutdown_connections 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa2498ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2454c10 0x55cfa2498580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 --2- 172.21.15.31:0/2510193932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa2499e80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 -- 172.21.15.31:0/2510193932 >> 172.21.15.31:0/2510193932 conn(0x55cfa24501e0 msgr2=0x55cfa2453c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 -- 172.21.15.31:0/2510193932 shutdown_connections 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.158+0000 7f5b58837740 1 -- 172.21.15.31:0/2510193932 wait complete. 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.159+0000 7f5b58837740 1 -- start start 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.159+0000 7f5b58837740 1 Processor -- start 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa24a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cfa235d330 con 0x55cfa2499aa0 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cfa23ba810 con 0x55cfa2455190 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55cfa2127620 con 0x55cfa24ac070 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b58837740 10 client.0 _ll_register_callbacks cb 0x55cfa2362420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:38:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b4e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b4e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44820/0 (socket says 172.21.15.31:44820) 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b4e000640 1 -- 172.21.15.31:0/1135851351 learned_addr learned my addr 172.21.15.31:0/1135851351 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa24a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.160+0000 7f5b4f400640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 -- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 msgr2=0x55cfa24ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24ac4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 -- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 msgr2=0x55cfa24a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa24a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 -- 172.21.15.31:0/1135851351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa2118850 con 0x55cfa2499aa0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b4e000640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0099c0 tx=0x7f5b3c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ab74a3404122b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c035070 con 0x55cfa2499aa0 2024-09-13T08:38:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5b3c0037f0 con 0x55cfa2499aa0 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b400044d0 con 0x55cfa2499aa0 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b3c0039f0 con 0x55cfa2499aa0 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b58837740 10 client.5412 fetch_fsmap learned FSMap version 27 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.161+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x55cfa218f6b0 con 0x55cfa2499aa0 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b400044d0 con 0x55cfa2499aa0 2024-09-13T08:38:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 3502+0+0 (secure 0 0 0) 0x7f5b3c003c80 con 0x55cfa2499aa0 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 10 client.5412 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 10 client.5412 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 20 client.5412 populate_metadata read hostname 'smithi031' 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa24914a0 0x55cfa2491880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 4 client.5412 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 --> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfa23c9520 con 0x55cfa24914a0 2024-09-13T08:38:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.162+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa24914a0 0x55cfa2491880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.163+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa24914a0 0x55cfa2491880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=774b5d878ca72247 server_cookie=82633e29798185c9 in_seq=0 out_seq=0 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.163+0000 7f5b2fe00640 10 client.5412 ms_handle_connect on v2:172.21.15.31:6832/1342811284 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b2fe00640 1 -- 172.21.15.31:0/1135851351 <== mds.0 v2:172.21.15.31:6832/1342811284 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cfa23c9520 con 0x55cfa24914a0 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b2fe00640 10 client.5412 handle_command_reply: tid=0 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 client.5412 shutdown 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 msgr2=0x55cfa248f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0099c0 tx=0x7f5b3c0043b0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 shutdown_connections 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa24914a0 0x55cfa2491880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2499aa0 0x55cfa248f0e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 --2- 172.21.15.31:0/1135851351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2455190 0x55cfa24a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b2fe00640 0 client.5412 ms_handle_reset on v2:172.21.15.31:6832/1342811284 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 >> 172.21.15.31:0/1135851351 conn(0x55cfa2454c10 msgr2=0x55cfa244edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 shutdown_connections 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 1 -- 172.21.15.31:0/1135851351 wait complete. 2024-09-13T08:38:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.164+0000 7f5b58837740 20 client.5412 trim_cache size 0 max 16384 2024-09-13T08:38:43.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.183+0000 7f5b58837740 1 Processor -- start 2024-09-13T08:38:43.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.183+0000 7f5b58837740 1 -- start start 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2499aa0 0x55cfa2498370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501f8e20 con 0x55cfa2455190 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501f8180 con 0x55cfa2499aa0 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b501f87d0 con 0x55cfa24ac070 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.184+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44832/0 (socket says 172.21.15.31:44832) 2024-09-13T08:38:43.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 -- 172.21.15.31:0/1607996082 learned_addr learned my addr 172.21.15.31:0/1607996082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 -- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 msgr2=0x55cfa24988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 -- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2499aa0 msgr2=0x55cfa2498370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2499aa0 0x55cfa2498370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 -- 172.21.15.31:0/1607996082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa24ac820 con 0x55cfa2455190 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5b40011810 tx=0x7f5b400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e377b506027ef6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b3aa00640 1 -- 172.21.15.31:0/1607996082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4001e4e0 con 0x55cfa2455190 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b3aa00640 1 -- 172.21.15.31:0/1607996082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b4001e680 con 0x55cfa2455190 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.185+0000 7f5b3aa00640 1 -- 172.21.15.31:0/1607996082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4001e830 con 0x55cfa2455190 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 -- 172.21.15.31:0/1607996082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 msgr2=0x55cfa248f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5b40011810 tx=0x7f5b400046c0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 -- 172.21.15.31:0/1607996082 shutdown_connections 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa24ac070 0x55cfa24988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa2499aa0 0x55cfa2498370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 --2- 172.21.15.31:0/1607996082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa248f1c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 -- 172.21.15.31:0/1607996082 >> 172.21.15.31:0/1607996082 conn(0x55cfa2454c10 msgr2=0x55cfa244d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 -- 172.21.15.31:0/1607996082 shutdown_connections 2024-09-13T08:38:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.186+0000 7f5b58837740 1 -- 172.21.15.31:0/1607996082 wait complete. 2024-09-13T08:38:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.187+0000 7f5b58837740 1 -- start start 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.187+0000 7f5b58837740 1 Processor -- start 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2499aa0 0x55cfa24ff020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa24ac070 0x55cfa2498370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cfa24920a0 con 0x55cfa2455190 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cfa24a9930 con 0x55cfa24ac070 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55cfa2468170 con 0x55cfa2499aa0 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b58837740 10 client.0 _ll_register_callbacks cb 0x7f5b3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44844/0 (socket says 172.21.15.31:44844) 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2301904587 learned_addr learned my addr 172.21.15.31:0/2301904587 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4e000640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2499aa0 0x55cfa24ff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4f400640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa24ac070 0x55cfa2498370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.188+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2499aa0 msgr2=0x55cfa24ff020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2499aa0 0x55cfa24ff020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa24ac070 msgr2=0x55cfa2498370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa24ac070 0x55cfa2498370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b4ea00640 1 -- 172.21.15.31:0/2301904587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa2479df0 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b4ea00640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b40022000 tx=0x7f5b40004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be571439948f856 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4001e830 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5b40009040 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b4001e680 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b40042050 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b58837740 10 client.5418 fetch_fsmap learned FSMap version 27 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.189+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x55cfa2479430 con 0x55cfa2455190 2024-09-13T08:38:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b4001e680 con 0x55cfa2455190 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 3502+0+0 (secure 0 0 0) 0x7f5b40045020 con 0x55cfa2455190 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 10 client.5418 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 10 client.5418 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 20 client.5418 populate_metadata read hostname 'smithi031' 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa2519800 0x55cfa2519be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 4 client.5418 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 --> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] -- command(tid 0: {"prefix": "status"}) -- 0x55cfa23c4fd0 con 0x55cfa2519800 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.190+0000 7f5b4e000640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa2519800 0x55cfa2519be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b4e000640 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa2519800 0x55cfa2519be0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c48da3694baae92 server_cookie=689678b9bb4785a5 in_seq=0 out_seq=0 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b2fe00640 10 client.5418 ms_handle_connect on v2:172.21.15.31:6832/1342811284 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b2fe00640 1 -- 172.21.15.31:0/2301904587 <== mds.0 v2:172.21.15.31:6832/1342811284 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55cfa23c4fd0 con 0x55cfa2519800 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b2fe00640 10 client.5418 handle_command_reply: tid=0 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 client.5418 shutdown 2024-09-13T08:38:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 msgr2=0x55cfa2452070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b40022000 tx=0x7f5b40004570 comp rx=0 tx=0).stop 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 shutdown_connections 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:6832/1342811284,v1:172.21.15.31:6834/1342811284] conn(0x55cfa2519800 0x55cfa2519be0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cfa24ac070 0x55cfa2498370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55cfa2499aa0 0x55cfa24ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b2fe00640 0 client.5418 ms_handle_reset on v2:172.21.15.31:6832/1342811284 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 --2- 172.21.15.31:0/2301904587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cfa2455190 0x55cfa2452070 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 >> 172.21.15.31:0/2301904587 conn(0x55cfa2454c10 msgr2=0x55cfa24bc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 shutdown_connections 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 1 -- 172.21.15.31:0/2301904587 wait complete. 2024-09-13T08:38:43.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.191+0000 7f5b58837740 20 client.5418 trim_cache size 0 max 16384 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 5286, 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.233377773, 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 27, 2024-09-13T08:38:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 48, 2024-09-13T08:38:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 48, 2024-09-13T08:38:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.502852418 2024-09-13T08:38:43.198 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 msgr2=0x7f5b3007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 0x7f5b3007aec0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009b30 tx=0x7f5b40005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 msgr2=0x7f5b50160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800c7d0 tx=0x7f5b4800cc90 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 shutdown_connections 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b5015e160 0x7f5b50160630 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b500d0cd0 0x7f5b500b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b30078a00 0x7f5b3007aec0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 --2- 172.21.15.31:0/3702973916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b500c8340 0x7f5b50154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 >> 172.21.15.31:0/3702973916 conn(0x7f5b500d3f50 msgr2=0x7f5b5006d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 shutdown_connections 2024-09-13T08:38:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.192+0000 7f5b4fe00640 1 -- 172.21.15.31:0/3702973916 wait complete. 2024-09-13T08:38:43.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 -- 172.21.15.31:0/102024436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c0ca5c0 msgr2=0x7f0a8c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 --2- 172.21.15.31:0/102024436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c0cc300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005fd0 tx=0x7f0a7402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 -- 172.21.15.31:0/102024436 shutdown_connections 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 --2- 172.21.15.31:0/102024436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 0x7f0a8c0b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 --2- 172.21.15.31:0/102024436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c0cc300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 -- 172.21.15.31:0/102024436 >> 172.21.15.31:0/102024436 conn(0x7f0a8c0cc910 msgr2=0x7f0a8c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 -- 172.21.15.31:0/102024436 shutdown_connections 2024-09-13T08:38:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.370+0000 7f0a91200640 1 -- 172.21.15.31:0/102024436 wait complete. 2024-09-13T08:38:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.371+0000 7f0a91200640 1 Processor -- start 2024-09-13T08:38:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.371+0000 7f0a91200640 1 -- start start 2024-09-13T08:38:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 0x7f0a8c0b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 0x7f0a8c1ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c073b10 con 0x7f0a8c0ca5c0 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c073990 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.372+0000 7f0a91200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c073c90 con 0x7f0a8c0b1910 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 0x7f0a8c1ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a89e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 0x7f0a8c0b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44846/0 (socket says 172.21.15.31:44846) 2024-09-13T08:38:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a81400640 1 -- 172.21.15.31:0/171672487 learned_addr learned my addr 172.21.15.31:0/171672487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 -- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 msgr2=0x7f0a8c0b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 0x7f0a8c0b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 -- 172.21.15.31:0/171672487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 msgr2=0x7f0a8c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a8a800640 1 -- 172.21.15.31:0/171672487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0c7170 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.373+0000 7f0a81400640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a8a800640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 0x7f0a8c1ea990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009fd0 tx=0x7f0a8400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8400ce20 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a84004c10 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84005020 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c0c7450 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.374+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c15c8a0 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.375+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c0b12c0 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.377+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f0a840051c0 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.377+0000 7f0a82a00640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 0x7f0a6807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.378+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f0a8409bf80 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.378+0000 7f0a89e00640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 0x7f0a6807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.379+0000 7f0a89e00640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 0x7f0a6807aec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c001720 tx=0x7f0a7c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:43.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.381+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0a8c0b12c0 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.614+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0a8c0b1d60 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.615+0000 7f0a82a00640 1 -- 172.21.15.31:0/171672487 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f0a84063bf0 con 0x7f0a8c15dfc0 2024-09-13T08:38:43.616 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 msgr2=0x7f0a6807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 0x7f0a6807aec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c001720 tx=0x7f0a7c009040 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 msgr2=0x7f0a8c1ea990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 0x7f0a8c1ea990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009fd0 tx=0x7f0a8400ef10 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 shutdown_connections 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a8c15dfc0 0x7f0a8c1ea990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a8c0ca5c0 0x7f0a8c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0a68078a00 0x7f0a6807aec0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 --2- 172.21.15.31:0/171672487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a8c0b1910 0x7f0a8c0b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.617+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 >> 172.21.15.31:0/171672487 conn(0x7f0a8c0cc910 msgr2=0x7f0a8c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.618+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 shutdown_connections 2024-09-13T08:38:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43.618+0000 7f0a91200640 1 -- 172.21.15.31:0/171672487 wait complete. 2024-09-13T08:38:43.635 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:38:43.636 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:43.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:43.637 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:38:43.672 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:38:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42608sec preferred_lft 42608sec 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:38:43.674 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:43.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:43.675 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:38:43.766 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:38:43.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:43.769 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:38:43.805 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:38:43.806 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:38:43.806 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:38:43.806 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:38:43.806 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:38:43.806 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:38:43.806 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:38:43.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:38:43.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:38:43.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:38:43.807 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:38:43.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:38:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:44.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:44.006 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:38:44.042 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:38:44.042 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:38:44.042 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:44.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:44.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:38:44.139 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:44.139 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:44.175 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:38:44.175 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:44.175 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:44.250 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:38:44.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:44.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:44.379 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:44.379 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:44.407 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:44.407 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:44.407 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:38:44.407 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:38:44.407 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:38:44.410 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:38:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:38:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:38:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:38:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:38:44.413 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:38:44.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:44.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:44.470 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:38:44.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:38:44.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:44.567 INFO:teuthology.orchestra.run.smithi031.stdout:client5424 2024-09-13T08:38:44.568 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:44.568 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:44.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:38:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 -- 172.21.15.31:0/4043607205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa728004d40 msgr2=0x7fa7280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 --2- 172.21.15.31:0/4043607205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa728004d40 0x7fa7280051c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa738009ed0 tx=0x7fa728005830 comp rx=0 tx=0).stop 2024-09-13T08:38:44.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa746600640 1 -- 172.21.15.31:0/4043607205 reap_dead start 2024-09-13T08:38:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 -- 172.21.15.31:0/4043607205 shutdown_connections 2024-09-13T08:38:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 -- 172.21.15.31:0/4043607205 >> 172.21.15.31:0/4043607205 conn(0x7fa7400cc910 msgr2=0x7fa7400cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 -- 172.21.15.31:0/4043607205 shutdown_connections 2024-09-13T08:38:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.805+0000 7fa748600640 1 -- 172.21.15.31:0/4043607205 wait complete. 2024-09-13T08:38:44.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.807+0000 7fa748600640 1 Processor -- start 2024-09-13T08:38:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.807+0000 7fa748600640 1 -- start start 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa728004d40 0x7fa7401f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7401f5080 0x7fa7400c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7400cead0 con 0x7fa74009c630 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7400b1a20 con 0x7fa7401f5080 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa748600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa7400cec50 con 0x7fa728004d40 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa745200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa745200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44862/0 (socket says 172.21.15.31:44862) 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.808+0000 7fa745200640 1 -- 172.21.15.31:0/1128835767 learned_addr learned my addr 172.21.15.31:0/1128835767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745c00640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa728004d40 0x7fa7401f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa746600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7401f5080 0x7fa7400c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:44.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 -- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa728004d40 msgr2=0x7fa7401f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa728004d40 0x7fa7401f7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 -- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7401f5080 msgr2=0x7fa7400c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7401f5080 0x7fa7400c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 -- 172.21.15.31:0/1128835767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74015aa10 con 0x7fa74009c630 2024-09-13T08:38:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa745200640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa72c00e610 tx=0x7fa72c00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c015070 con 0x7fa74009c630 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa72c00ca60 con 0x7fa74009c630 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c011460 con 0x7fa74009c630 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.809+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7401f85d0 con 0x7fa74009c630 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.810+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7401f8b90 con 0x7fa74009c630 2024-09-13T08:38:44.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.811+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7400cf610 con 0x7fa74009c630 2024-09-13T08:38:44.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.812+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fa72c018070 con 0x7fa74009c630 2024-09-13T08:38:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.813+0000 7fa73d600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 0x7fa72007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.813+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fa72c09b210 con 0x7fa74009c630 2024-09-13T08:38:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.813+0000 7fa745c00640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 0x7fa72007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.814+0000 7fa745c00640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 0x7fa72007aec0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa734009b20 tx=0x7fa734005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:44.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:44.818+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa7400cf610 con 0x7fa74009c630 2024-09-13T08:38:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.055+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa7400b0d40 con 0x7fa74009c630 2024-09-13T08:38:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.056+0000 7fa73d600640 1 -- 172.21.15.31:0/1128835767 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7fa72c0634e0 con 0x7fa74009c630 2024-09-13T08:38:45.057 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:38:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.058+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 msgr2=0x7fa72007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.058+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 0x7fa72007aec0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa734009b20 tx=0x7fa734005f90 comp rx=0 tx=0).stop 2024-09-13T08:38:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.058+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 msgr2=0x7fa740159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.058+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa72c00e610 tx=0x7fa72c00ead0 comp rx=0 tx=0).stop 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 shutdown_connections 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7401f5080 0x7fa7400c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74009c630 0x7fa740159f80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa720078a00 0x7fa72007aec0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 --2- 172.21.15.31:0/1128835767 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa728004d40 0x7fa7401f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 >> 172.21.15.31:0/1128835767 conn(0x7fa7400cc910 msgr2=0x7fa740074160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 shutdown_connections 2024-09-13T08:38:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:45.059+0000 7fa748600640 1 -- 172.21.15.31:0/1128835767 wait complete. 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:38:45.076 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:45.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.077 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:38:45.120 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:45.121 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42611sec preferred_lft 42611sec 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:38:45.122 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:38:45.123 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:38:45.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.124 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:38:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:38:45.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.222 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:38:45.259 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:38:45.259 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:38:45.259 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:38:45.259 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:38:45.259 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:38:45.260 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:38:45.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:38:45.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:45.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.457 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:38:45.493 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:38:45.494 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:38:45.494 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:45.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:38:45.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:45.590 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:45.624 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:38:45.625 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:38:45.625 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:45.699 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:38:45.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:45.828 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:45.829 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:45.857 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:38:45.857 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:38:45.857 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:38:45.858 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:38:45.858 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:38:45.860 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:38:45.860 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:38:45.860 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:38:45.860 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:38:45.860 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:38:45.863 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:38:45.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:45.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:45.919 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:38:45.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:38:46.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:38:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:38:46.008 INFO:teuthology.orchestra.run.smithi045.stdout:client5296 2024-09-13T08:38:46.009 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:46.009 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:46.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:38:46.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 -- 172.21.15.31:0/3033386298 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c0cd780 msgr2=0x7f0e5c06f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:46.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 --2- 172.21.15.31:0/3033386298 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c0cd780 0x7f0e5c06f500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0e54005fd0 tx=0x7f0e5402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:46.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 -- 172.21.15.31:0/3033386298 shutdown_connections 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 --2- 172.21.15.31:0/3033386298 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c0cd780 0x7f0e5c06f500 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 --2- 172.21.15.31:0/3033386298 >> v2:172.21.15.31:3300/0 conn(0x7f0e5c15d5e0 0x7f0e5c0ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 -- 172.21.15.31:0/3033386298 >> 172.21.15.31:0/3033386298 conn(0x7f0e5c0cfbb0 msgr2=0x7f0e5c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 -- 172.21.15.31:0/3033386298 shutdown_connections 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.221+0000 7f0e61a00640 1 -- 172.21.15.31:0/3033386298 wait complete. 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.223+0000 7f0e61a00640 1 Processor -- start 2024-09-13T08:38:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.223+0000 7f0e61a00640 1 -- start start 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 0x7f0e5c14ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 0x7f0e5c1f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c06ff40 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c06fdc0 con 0x7f0e5c0cd780 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e61a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0e5c0700c0 con 0x7f0e5c070650 2024-09-13T08:38:46.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57442/0 (socket says 172.21.15.31:57442) 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 -- 172.21.15.31:0/1569017119 learned_addr learned my addr 172.21.15.31:0/1569017119 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5aa00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 0x7f0e5c14ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5a000640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 0x7f0e5c1f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 -- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 msgr2=0x7f0e5c14ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 0x7f0e5c14ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 -- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 msgr2=0x7f0e5c1f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 0x7f0e5c1f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5b400640 1 -- 172.21.15.31:0/1569017119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c0c93b0 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5aa00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 0x7f0e5c14ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.224+0000 7f0e5a000640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 0x7f0e5c1f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:46.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e5b400640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0e54006010 tx=0x7f0e54031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54040070 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e54004040 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54004450 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c0b1660 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.225+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c0b1910 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.226+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5c06f500 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.227+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f0e54039040 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.228+0000 7f0e52a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 0x7f0e3c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:46.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.228+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f0e540bc320 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.229+0000 7f0e5aa00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 0x7f0e3c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:46.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.229+0000 7f0e5aa00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 0x7f0e3c07aec0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c007a50 tx=0x7f0e4c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:46.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.232+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0e540845f0 con 0x7f0e5c15d5e0 2024-09-13T08:38:46.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:46.469+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f0e5c0739a0 con 0x7f0e5c15d5e0 2024-09-13T08:38:47.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.120+0000 7f0e52a00640 1 -- 172.21.15.31:0/1569017119 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v28) ==== 136+0+0 (secure 0 0 0) 0x7f0e54083ef0 con 0x7f0e5c15d5e0 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.123+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 msgr2=0x7f0e3c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.123+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 0x7f0e3c07aec0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c007a50 tx=0x7f0e4c009040 comp rx=0 tx=0).stop 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.123+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 msgr2=0x7f0e5c1f81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.123+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0e54006010 tx=0x7f0e54031670 comp rx=0 tx=0).stop 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 shutdown_connections 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e5c15d5e0 0x7f0e5c1f81d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e5c0cd780 0x7f0e5c1f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0e3c078a00 0x7f0e3c07aec0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 --2- 172.21.15.31:0/1569017119 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0e5c070650 0x7f0e5c14ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 >> 172.21.15.31:0/1569017119 conn(0x7f0e5c0cfbb0 msgr2=0x7f0e5c072c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 shutdown_connections 2024-09-13T08:38:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.124+0000 7f0e61a00640 1 -- 172.21.15.31:0/1569017119 wait complete. 2024-09-13T08:38:47.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:38:47.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.308+0000 7f0605e00640 1 -- 172.21.15.31:0/3380617362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 msgr2=0x7f06000ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.308+0000 7f0605e00640 1 --2- 172.21.15.31:0/3380617362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f06000ce290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009830 tx=0x7f05e802a5c0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.308+0000 7f0605e00640 1 -- 172.21.15.31:0/3380617362 shutdown_connections 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.308+0000 7f0605e00640 1 --2- 172.21.15.31:0/3380617362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f06000ce290 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.308+0000 7f0605e00640 1 -- 172.21.15.31:0/3380617362 >> 172.21.15.31:0/3380617362 conn(0x7f06000b18f0 msgr2=0x7f06000b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.309+0000 7f0605e00640 1 -- 172.21.15.31:0/3380617362 shutdown_connections 2024-09-13T08:38:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.309+0000 7f0605e00640 1 -- 172.21.15.31:0/3380617362 wait complete. 2024-09-13T08:38:47.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.310+0000 7f0605e00640 1 Processor -- start 2024-09-13T08:38:47.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.310+0000 7f0605e00640 1 -- start start 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0600073860 0x7f06001f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f0600157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06000733e0 con 0x7f060015be30 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06000ce9e0 con 0x7f06000ced20 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.311+0000 7f0605e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f06000743c0 con 0x7f0600073860 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57454/0 (socket says 172.21.15.31:57454) 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 -- 172.21.15.31:0/4171375590 learned_addr learned my addr 172.21.15.31:0/4171375590 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05fea00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0600073860 0x7f06001f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05f6000640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f0600157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 -- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0600073860 msgr2=0x7f06001f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0600073860 0x7f06001f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 -- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 msgr2=0x7f0600157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f0600157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 -- 172.21.15.31:0/4171375590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06000b2d90 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.312+0000 7f05ff400640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05f800cbb0 tx=0x7f05f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.313+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8016020 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.313+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05f800ae30 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.313+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8012660 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.313+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06000c9dc0 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.313+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06000c92f0 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.314+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06000b2f60 con 0x7f060015be30 2024-09-13T08:38:47.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.315+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f05f8019020 con 0x7f060015be30 2024-09-13T08:38:47.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.316+0000 7f05f7e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 0x7f05dc07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.316+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f05f809b000 con 0x7f060015be30 2024-09-13T08:38:47.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.316+0000 7f05fea00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 0x7f05dc07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.317+0000 7f05fea00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 0x7f05dc07aef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000f60 tx=0x7f05f0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:47.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.320+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f05f8020030 con 0x7f060015be30 2024-09-13T08:38:47.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.539+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f0600074840 con 0x7f05dc078a30 2024-09-13T08:38:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.541+0000 7f05f7e00640 1 -- 172.21.15.31:0/4171375590 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f0600074840 con 0x7f05dc078a30 2024-09-13T08:38:47.542 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:38:47.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:38:47.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:38:47.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:38:47.542 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 msgr2=0x7f05dc07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 0x7f05dc07aef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000f60 tx=0x7f05f0009040 comp rx=0 tx=0).stop 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 msgr2=0x7f06000cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05f800cbb0 tx=0x7f05f800a430 comp rx=0 tx=0).stop 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 shutdown_connections 2024-09-13T08:38:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060015be30 0x7f06000cb350 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06000ced20 0x7f0600157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05dc078a30 0x7f05dc07aef0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 --2- 172.21.15.31:0/4171375590 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0600073860 0x7f06001f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 >> 172.21.15.31:0/4171375590 conn(0x7f06000b18f0 msgr2=0x7f06000af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 shutdown_connections 2024-09-13T08:38:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.543+0000 7f0605e00640 1 -- 172.21.15.31:0/4171375590 wait complete. 2024-09-13T08:38:47.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 -- 172.21.15.31:0/667065803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 msgr2=0x7f7fbc15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 --2- 172.21.15.31:0/667065803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc15b0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0724f0 tx=0x7f7fac02a560 comp rx=0 tx=0).stop 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 -- 172.21.15.31:0/667065803 shutdown_connections 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 --2- 172.21.15.31:0/667065803 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc06ee50 0x7f7fbc0ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 --2- 172.21.15.31:0/667065803 >> v2:172.21.15.45:3300/0 conn(0x7f7fbc15b620 0x7f7fbc0ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 --2- 172.21.15.31:0/667065803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc15b0e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.722+0000 7f7fc1000640 1 -- 172.21.15.31:0/667065803 >> 172.21.15.31:0/667065803 conn(0x7f7fbc0cfbb0 msgr2=0x7f7fbc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.723+0000 7f7fc1000640 1 -- 172.21.15.31:0/667065803 shutdown_connections 2024-09-13T08:38:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.723+0000 7f7fc1000640 1 -- 172.21.15.31:0/667065803 wait complete. 2024-09-13T08:38:47.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.724+0000 7f7fc1000640 1 Processor -- start 2024-09-13T08:38:47.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.724+0000 7f7fc1000640 1 -- start start 2024-09-13T08:38:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fbc06ee50 0x7f7fbc14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc15b620 0x7f7fbc166d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc15df90 con 0x7f7fbc15ad00 2024-09-13T08:38:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc06ec30 con 0x7f7fbc06ee50 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fc1000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc15e110 con 0x7f7fbc15b620 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fb1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fb1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57464/0 (socket says 172.21.15.31:57464) 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fb1400640 1 -- 172.21.15.31:0/955738330 learned_addr learned my addr 172.21.15.31:0/955738330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.725+0000 7f7fb9e00640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fbc06ee50 0x7f7fbc14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fba800640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc15b620 0x7f7fbc166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 -- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc15b620 msgr2=0x7f7fbc166d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc15b620 0x7f7fbc166d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 -- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fbc06ee50 msgr2=0x7f7fbc14f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fbc06ee50 0x7f7fbc14f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 -- 172.21.15.31:0/955738330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc14de10 con 0x7f7fbc15ad00 2024-09-13T08:38:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb1400640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b410 tx=0x7f7fa400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4015020 con 0x7f7fbc15ad00 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fa4011940 con 0x7f7fbc15ad00 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4011d90 con 0x7f7fbc15ad00 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.726+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc1654a0 con 0x7f7fbc15ad00 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.727+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc165760 con 0x7f7fbc15ad00 2024-09-13T08:38:47.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.728+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fbc15b0e0 con 0x7f7fbc15ad00 2024-09-13T08:38:47.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.729+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f7fa4018020 con 0x7f7fbc15ad00 2024-09-13T08:38:47.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.730+0000 7f7fb2a00640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 0x7f7f9807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:47.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.730+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f7fa409aaa0 con 0x7f7fbc15ad00 2024-09-13T08:38:47.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.730+0000 7f7fb9e00640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 0x7f7f9807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:47.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.731+0000 7f7fb9e00640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 0x7f7f9807aec0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7fac033000 tx=0x7f7fac004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:47.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.734+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7fa401f020 con 0x7f7fbc15ad00 2024-09-13T08:38:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.932+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7fbc06ac30 con 0x7f7fbc15ad00 2024-09-13T08:38:47.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.932+0000 7f7fb2a00640 1 -- 172.21.15.31:0/955738330 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f7fbc14de10 con 0x7f7fbc15ad00 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.934+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 msgr2=0x7f7f9807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.934+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 0x7f7f9807aec0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7fac033000 tx=0x7f7fac004730 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.934+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 msgr2=0x7f7fbc14d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.934+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b410 tx=0x7f7fa400b8d0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 shutdown_connections 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7f98078a00 0x7f7f9807aec0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7fbc15b620 0x7f7fbc166d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7fbc15ad00 0x7f7fbc14d8d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 --2- 172.21.15.31:0/955738330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fbc06ee50 0x7f7fbc14f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 >> 172.21.15.31:0/955738330 conn(0x7f7fbc0cfbb0 msgr2=0x7f7fbc156720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 shutdown_connections 2024-09-13T08:38:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:47.935+0000 7f7fc1000640 1 -- 172.21.15.31:0/955738330 wait complete. 2024-09-13T08:38:47.952 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:47.953 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:47.988 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:38:47.988 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:48.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.266+0000 7f1572400640 1 -- 172.21.15.31:0/3792479507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1550003a00 msgr2=0x7f1550003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.266+0000 7f1572400640 1 --2- 172.21.15.31:0/3792479507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1550003a00 0x7f1550003e20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f155c002230 tx=0x7f1550005830 comp rx=0 tx=0).stop 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.266+0000 7f156be00640 1 -- 172.21.15.31:0/3792479507 reap_dead start 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.267+0000 7f1572400640 1 -- 172.21.15.31:0/3792479507 shutdown_connections 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.267+0000 7f1572400640 1 -- 172.21.15.31:0/3792479507 >> 172.21.15.31:0/3792479507 conn(0x7f156c0b18f0 msgr2=0x7f156c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.267+0000 7f1572400640 1 -- 172.21.15.31:0/3792479507 shutdown_connections 2024-09-13T08:38:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.267+0000 7f1572400640 1 -- 172.21.15.31:0/3792479507 wait complete. 2024-09-13T08:38:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.268+0000 7f1572400640 1 Processor -- start 2024-09-13T08:38:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.268+0000 7f1572400640 1 -- start start 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1550003a00 0x7f156c0c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f156c1f5590 0x7f156c0c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0ce950 con 0x7f156c154a00 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0ce7d0 con 0x7f1550003a00 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.269+0000 7f1572400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f156c073cf0 con 0x7f156c1f5590 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57486/0 (socket says 172.21.15.31:57486) 2024-09-13T08:38:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 -- 172.21.15.31:0/1535688365 learned_addr learned my addr 172.21.15.31:0/1535688365 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f156be00640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f156c1f5590 0x7f156c0c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f156b400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1550003a00 0x7f156c0c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 -- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f156c1f5590 msgr2=0x7f156c0c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f156c1f5590 0x7f156c0c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 -- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1550003a00 msgr2=0x7f156c0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1550003a00 0x7f156c0c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 -- 172.21.15.31:0/1535688365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0c9600 con 0x7f156c154a00 2024-09-13T08:38:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.270+0000 7f1562a00640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1554011ae0 tx=0x7f1554009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.271+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155400fc50 con 0x7f156c154a00 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.271+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f155400fdf0 con 0x7f156c154a00 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.271+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554004280 con 0x7f156c154a00 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.271+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c15da40 con 0x7f156c154a00 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.271+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c15e000 con 0x7f156c154a00 2024-09-13T08:38:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.272+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156c074280 con 0x7f156c154a00 2024-09-13T08:38:48.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.273+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f15540035f0 con 0x7f156c154a00 2024-09-13T08:38:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.274+0000 7f1563e00640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 0x7f154807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.274+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f155409b720 con 0x7f156c154a00 2024-09-13T08:38:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.274+0000 7f156b400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 0x7f154807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.275+0000 7f156b400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 0x7f154807aec0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f155c033000 tx=0x7f155c038040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.278+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f15540639f0 con 0x7f156c154a00 2024-09-13T08:38:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.483+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f156c0b25f0 con 0x7f156c154a00 2024-09-13T08:38:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.483+0000 7f1563e00640 1 -- 172.21.15.31:0/1535688365 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7f15540632f0 con 0x7f156c154a00 2024-09-13T08:38:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 msgr2=0x7f154807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 0x7f154807aec0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f155c033000 tx=0x7f155c038040 comp rx=0 tx=0).stop 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 msgr2=0x7f156c15e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1554011ae0 tx=0x7f1554009f10 comp rx=0 tx=0).stop 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 shutdown_connections 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1548078a00 0x7f154807aec0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f156c1f5590 0x7f156c0c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c154a00 0x7f156c15e800 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 --2- 172.21.15.31:0/1535688365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1550003a00 0x7f156c0c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 >> 172.21.15.31:0/1535688365 conn(0x7f156c0b18f0 msgr2=0x7f156c0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 shutdown_connections 2024-09-13T08:38:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.486+0000 7f1572400640 1 -- 172.21.15.31:0/1535688365 wait complete. 2024-09-13T08:38:48.503 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:38:48.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:48.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:38:48.565 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:38:48.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:48.566 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:38:48.602 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:48.602 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:48.642 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:38:48.642 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:38:48.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:38:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 -- 172.21.15.31:0/3380937364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c073860 msgr2=0x7fd65c073d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 --2- 172.21.15.31:0/3380937364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c073860 0x7fd65c073d10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd654005fd0 tx=0x7fd65402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 -- 172.21.15.31:0/3380937364 shutdown_connections 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 --2- 172.21.15.31:0/3380937364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c073860 0x7fd65c073d10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 --2- 172.21.15.31:0/3380937364 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c15be30 0x7fd65c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 -- 172.21.15.31:0/3380937364 >> 172.21.15.31:0/3380937364 conn(0x7fd65c0b18f0 msgr2=0x7fd65c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 -- 172.21.15.31:0/3380937364 shutdown_connections 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.883+0000 7fd661800640 1 -- 172.21.15.31:0/3380937364 wait complete. 2024-09-13T08:38:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.885+0000 7fd661800640 1 Processor -- start 2024-09-13T08:38:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.885+0000 7fd661800640 1 -- start start 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c073860 0x7fd65c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65c0ced20 0x7fd65c09d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c0733e0 con 0x7fd65c15be30 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c0ce9e0 con 0x7fd65c0ced20 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd661800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c074250 con 0x7fd65c073860 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57508/0 (socket says 172.21.15.31:57508) 2024-09-13T08:38:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 -- 172.21.15.31:0/2055590855 learned_addr learned my addr 172.21.15.31:0/2055590855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65aa00640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c073860 0x7fd65c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65a000640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65c0ced20 0x7fd65c09d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 -- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c073860 msgr2=0x7fd65c1583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c073860 0x7fd65c1583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 -- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65c0ced20 msgr2=0x7fd65c09d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65c0ced20 0x7fd65c09d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.886+0000 7fd65b400640 1 -- 172.21.15.31:0/2055590855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c15a930 con 0x7fd65c15be30 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd65b400640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd654009cd0 tx=0x7fd654003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654040070 con 0x7fd65c15be30 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd654004130 con 0x7fd65c15be30 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654004540 con 0x7fd65c15be30 2024-09-13T08:38:48.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c15ac10 con 0x7fd65c15be30 2024-09-13T08:38:48.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.887+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c09c710 con 0x7fd65c15be30 2024-09-13T08:38:48.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.888+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c074740 con 0x7fd65c15be30 2024-09-13T08:38:48.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.889+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fd654039040 con 0x7fd65c15be30 2024-09-13T08:38:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.890+0000 7fd652a00640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 0x7fd63c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.890+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd6540bc270 con 0x7fd65c15be30 2024-09-13T08:38:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.891+0000 7fd65aa00640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 0x7fd63c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.891+0000 7fd65aa00640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 0x7fd63c07aec0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00bb50 tx=0x7fd64c00b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:48.894+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd654083e60 con 0x7fd65c15be30 2024-09-13T08:38:49.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.105+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd65c0b25f0 con 0x7fd65c15be30 2024-09-13T08:38:49.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.106+0000 7fd652a00640 1 -- 172.21.15.31:0/2055590855 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7fd654088890 con 0x7fd65c15be30 2024-09-13T08:38:49.107 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:38:49.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 msgr2=0x7fd63c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 0x7fd63c07aec0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00bb50 tx=0x7fd64c00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 msgr2=0x7fd65c09c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd654009cd0 tx=0x7fd654003970 comp rx=0 tx=0).stop 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 shutdown_connections 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd65c15be30 0x7fd65c09c1d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65c0ced20 0x7fd65c09d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd63c078a00 0x7fd63c07aec0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 --2- 172.21.15.31:0/2055590855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd65c073860 0x7fd65c1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 >> 172.21.15.31:0/2055590855 conn(0x7fd65c0b18f0 msgr2=0x7fd65c0b3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 shutdown_connections 2024-09-13T08:38:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.108+0000 7fd661800640 1 -- 172.21.15.31:0/2055590855 wait complete. 2024-09-13T08:38:49.125 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:38:49.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:49.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:38:49.190 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:38:49.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:38:49.190 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:38:49.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:49.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 -- 172.21.15.31:0/764859375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28640746d0 msgr2=0x7f28640b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 --2- 172.21.15.31:0/764859375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28640746d0 0x7f28640b1910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f285c005fd0 tx=0x7f285c02ab00 comp rx=0 tx=0).stop 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 -- 172.21.15.31:0/764859375 shutdown_connections 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 --2- 172.21.15.31:0/764859375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28640746d0 0x7f28640b1910 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 --2- 172.21.15.31:0/764859375 >> v2:172.21.15.45:3300/0 conn(0x7f2864152f10 0x7f28640cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 -- 172.21.15.31:0/764859375 >> 172.21.15.31:0/764859375 conn(0x7f28640cc910 msgr2=0x7f28640cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 -- 172.21.15.31:0/764859375 shutdown_connections 2024-09-13T08:38:49.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.387+0000 7f286bc00640 1 -- 172.21.15.31:0/764859375 wait complete. 2024-09-13T08:38:49.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.389+0000 7f286bc00640 1 Processor -- start 2024-09-13T08:38:49.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.389+0000 7f286bc00640 1 -- start start 2024-09-13T08:38:49.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2864072850 0x7f28640b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:49.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 0x7f28640b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:49.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:49.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28640ca660 con 0x7f2864152f10 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28640ca4e0 con 0x7f28640746d0 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f286bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f28640b1e50 con 0x7f2864072850 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57528/0 (socket says 172.21.15.31:57528) 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 -- 172.21.15.31:0/2082105876 learned_addr learned my addr 172.21.15.31:0/2082105876 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f285be00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 0x7f28640b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869200640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2864072850 0x7f28640b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 -- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2864072850 msgr2=0x7f28640b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2864072850 0x7f28640b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 -- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 msgr2=0x7f28640b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 0x7f28640b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f2869c00640 1 -- 172.21.15.31:0/2082105876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28641f5170 con 0x7f2864152f10 2024-09-13T08:38:49.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.390+0000 7f285be00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 0x7f28640b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:38:49.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f2869c00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f285c006010 tx=0x7f285c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:49.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c044070 con 0x7f2864152f10 2024-09-13T08:38:49.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f285c02c7c0 con 0x7f2864152f10 2024-09-13T08:38:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c0044b0 con 0x7f2864152f10 2024-09-13T08:38:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28641f5450 con 0x7f2864152f10 2024-09-13T08:38:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.391+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28641f5a10 con 0x7f2864152f10 2024-09-13T08:38:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.392+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28640b2350 con 0x7f2864152f10 2024-09-13T08:38:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.393+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f285c004650 con 0x7f2864152f10 2024-09-13T08:38:49.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.394+0000 7f2861600640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 0x7f284007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:49.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.394+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f285c0bb3d0 con 0x7f2864152f10 2024-09-13T08:38:49.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.395+0000 7f2869200640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 0x7f284007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:49.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.395+0000 7f2869200640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 0x7f284007aec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2854005fd0 tx=0x7f2854005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:49.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.399+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f285c032030 con 0x7f2864152f10 2024-09-13T08:38:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.632+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28640ce730 con 0x7f2864152f10 2024-09-13T08:38:49.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.633+0000 7f2861600640 1 -- 172.21.15.31:0/2082105876 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f285c083610 con 0x7f2864152f10 2024-09-13T08:38:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:49.634 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:49.634 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:38:49.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 msgr2=0x7f284007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 0x7f284007aec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2854005fd0 tx=0x7f2854005850 comp rx=0 tx=0).stop 2024-09-13T08:38:49.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 msgr2=0x7f28640b1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f285c006010 tx=0x7f285c003f10 comp rx=0 tx=0).stop 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 shutdown_connections 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2864152f10 0x7f28640b1620 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28640746d0 0x7f28640b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2840078a00 0x7f284007aec0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 --2- 172.21.15.31:0/2082105876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2864072850 0x7f28640b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 >> 172.21.15.31:0/2082105876 conn(0x7f28640cc910 msgr2=0x7f286415b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 shutdown_connections 2024-09-13T08:38:49.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.635+0000 7f286bc00640 1 -- 172.21.15.31:0/2082105876 wait complete. 2024-09-13T08:38:49.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:38:49.831 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-09-13T08:38:49.832 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-13T08:38:49.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol15900 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 -- 172.21.15.31:0/3028180042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 msgr2=0x7fcf100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 --2- 172.21.15.31:0/3028180042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf100051c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400e3d0 tx=0x7fcf10005830 comp rx=0 tx=0).stop 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf2b400640 1 -- 172.21.15.31:0/3028180042 reap_dead start 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 -- 172.21.15.31:0/3028180042 shutdown_connections 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 -- 172.21.15.31:0/3028180042 >> 172.21.15.31:0/3028180042 conn(0x7fcf2c0cfbb0 msgr2=0x7fcf2c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 -- 172.21.15.31:0/3028180042 shutdown_connections 2024-09-13T08:38:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.997+0000 7fcf31a00640 1 -- 172.21.15.31:0/3028180042 wait complete. 2024-09-13T08:38:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.998+0000 7fcf31a00640 1 Processor -- start 2024-09-13T08:38:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.999+0000 7fcf31a00640 1 -- start start 2024-09-13T08:38:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.999+0000 7fcf31a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:49.999+0000 7fcf31a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf2c15f690 0x7fcf2c15d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf31a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf2c0b4bb0 0x7fcf2c0c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf31a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c0cbe50 con 0x7fcf10004d40 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf31a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c070040 con 0x7fcf2c0b4bb0 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf31a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcf2c0cbfd0 con 0x7fcf2c15f690 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57554/0 (socket says 172.21.15.31:57554) 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 -- 172.21.15.31:0/71217623 learned_addr learned my addr 172.21.15.31:0/71217623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 -- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf2c15f690 msgr2=0x7fcf2c15d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf2c15f690 0x7fcf2c15d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 -- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf2c0b4bb0 msgr2=0x7fcf2c0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf2c0b4bb0 0x7fcf2c0c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 -- 172.21.15.31:0/71217623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2c1f5810 con 0x7fcf10004d40 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.000+0000 7fcf2aa00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0066e0 tx=0x7fcf1c003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.001+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00f780 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.001+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf1c016020 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.001+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00fbc0 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.001+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2c0c4860 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.001+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf2c1f5250 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.002+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf2c0b5b30 con 0x7fcf10004d40 2024-09-13T08:38:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.003+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7fcf1c011430 con 0x7fcf10004d40 2024-09-13T08:38:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.004+0000 7fcf22a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 0x7fcf0c07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.004+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fcf1c09b870 con 0x7fcf10004d40 2024-09-13T08:38:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.004+0000 7fcf2a000640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 0x7fcf0c07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.005+0000 7fcf2a000640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 0x7fcf0c07aef0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0b5830 tx=0x7fcf180093b0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:50.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.008+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcf1c020020 con 0x7fcf10004d40 2024-09-13T08:38:50.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.235+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol15900", "target": ["mon-mgr", ""]}) -- 0x7fcf2c06ea10 con 0x7fcf0c078a30 2024-09-13T08:38:50.244 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:50.243+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:50.244 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:50.243+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:50.244 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:50.243+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:50.244 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:50.243+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:50.244 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:38:50.243+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:38:50.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.288+0000 7fcf22a00640 1 -- 172.21.15.31:0/71217623 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcf2c06ea10 con 0x7fcf0c078a30 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 msgr2=0x7fcf0c07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 0x7fcf0c07aef0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0b5830 tx=0x7fcf180093b0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 msgr2=0x7fcf2c1f4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0066e0 tx=0x7fcf1c003a80 comp rx=0 tx=0).stop 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 shutdown_connections 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf2c0b4bb0 0x7fcf2c0c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcf0c078a30 0x7fcf0c07aef0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcf2c15f690 0x7fcf2c15d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 --2- 172.21.15.31:0/71217623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf10004d40 0x7fcf2c1f4860 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 >> 172.21.15.31:0/71217623 conn(0x7fcf2c0cfbb0 msgr2=0x7fcf2c073da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 shutdown_connections 2024-09-13T08:38:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.291+0000 7fcf31a00640 1 -- 172.21.15.31:0/71217623 wait complete. 2024-09-13T08:38:50.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:50.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 -- 172.21.15.31:0/362026741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 msgr2=0x7f35700acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 --2- 172.21.15.31:0/362026741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f35700acd40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f35700c6ef0 tx=0x7f356802aac0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 -- 172.21.15.31:0/362026741 shutdown_connections 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 --2- 172.21.15.31:0/362026741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f35700acd40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 --2- 172.21.15.31:0/362026741 >> v2:172.21.15.45:3300/0 conn(0x7f35700cf280 0x7f35700ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 -- 172.21.15.31:0/362026741 >> 172.21.15.31:0/362026741 conn(0x7f35700cc910 msgr2=0x7f35700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 -- 172.21.15.31:0/362026741 shutdown_connections 2024-09-13T08:38:50.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.467+0000 7f3576a00640 1 -- 172.21.15.31:0/362026741 wait complete. 2024-09-13T08:38:50.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.469+0000 7f3576a00640 1 Processor -- start 2024-09-13T08:38:50.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.469+0000 7f3576a00640 1 -- start start 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35700cf280 0x7f357014add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f357015af30 0x7f357014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35700cdf10 con 0x7f35700caa20 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f35700cdd90 con 0x7f35700cf280 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3576a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f35700b1980 con 0x7f357015af30 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f356fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f356fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57572/0 (socket says 172.21.15.31:57572) 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f356fe00640 1 -- 172.21.15.31:0/3694680393 learned_addr learned my addr 172.21.15.31:0/3694680393 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3567400640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35700cf280 0x7f357014add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.470+0000 7f3574a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f357015af30 0x7f357014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 -- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f357015af30 msgr2=0x7f357014cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f357015af30 0x7f357014cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 -- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35700cf280 msgr2=0x7f357014add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35700cf280 0x7f357014add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 -- 172.21.15.31:0/3694680393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f357009aa10 con 0x7f35700caa20 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356fe00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f356000c3a0 tx=0x7f356000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560017070 con 0x7f35700caa20 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35600053c0 con 0x7f35700caa20 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560003850 con 0x7f35700caa20 2024-09-13T08:38:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.471+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f357014b620 con 0x7f35700caa20 2024-09-13T08:38:50.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.472+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f357014bbe0 con 0x7f35700caa20 2024-09-13T08:38:50.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.473+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35700b1dd0 con 0x7f35700caa20 2024-09-13T08:38:50.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.474+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f356001a070 con 0x7f35700caa20 2024-09-13T08:38:50.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.475+0000 7f356cc00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 0x7f354c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:50.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.476+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5854+0+0 (secure 0 0 0) 0x7f356009a080 con 0x7f35700caa20 2024-09-13T08:38:50.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.476+0000 7f3567400640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 0x7f354c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:50.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.477+0000 7f3567400640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 0x7f354c07aec0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f35700cd320 tx=0x7f355800b390 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:50.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.480+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3560061c90 con 0x7f35700caa20 2024-09-13T08:38:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.711+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35700b5e10 con 0x7f35700caa20 2024-09-13T08:38:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.711+0000 7f356cc00640 1 -- 172.21.15.31:0/3694680393 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f35600666c0 con 0x7f35700caa20 2024-09-13T08:38:50.712 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:50.713 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 msgr2=0x7f354c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 0x7f354c07aec0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f35700cd320 tx=0x7f355800b390 comp rx=0 tx=0).stop 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 msgr2=0x7f3570146e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f356000c3a0 tx=0x7f356000c860 comp rx=0 tx=0).stop 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 shutdown_connections 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f354c078a00 0x7f354c07aec0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f357015af30 0x7f357014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35700cf280 0x7f357014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 --2- 172.21.15.31:0/3694680393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35700caa20 0x7f3570146e50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 >> 172.21.15.31:0/3694680393 conn(0x7f35700cc910 msgr2=0x7f357015b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 shutdown_connections 2024-09-13T08:38:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:50.714+0000 7f3576a00640 1 -- 172.21.15.31:0/3694680393 wait complete. 2024-09-13T08:38:50.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:38:50.910 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":0.64898918100000003,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.967841246293684,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:38:50.910 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.967841246293684 }, "state": "open", "uptime": 0.648989181 } ] 2024-09-13T08:38:50.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:51.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 -- 172.21.15.31:0/3362368127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478003aa0 msgr2=0x7f1478003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 --2- 172.21.15.31:0/3362368127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478003aa0 0x7f1478003ee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1484006f80 tx=0x7f1478005830 comp rx=0 tx=0).stop 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 -- 172.21.15.31:0/3362368127 shutdown_connections 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 --2- 172.21.15.31:0/3362368127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478003aa0 0x7f1478003ee0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 -- 172.21.15.31:0/3362368127 >> 172.21.15.31:0/3362368127 conn(0x7f14940cc910 msgr2=0x7f14940cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 -- 172.21.15.31:0/3362368127 shutdown_connections 2024-09-13T08:38:51.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.071+0000 7f149a400640 1 -- 172.21.15.31:0/3362368127 wait complete. 2024-09-13T08:38:51.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.073+0000 7f149a400640 1 Processor -- start 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.073+0000 7f149a400640 1 -- start start 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1478003aa0 0x7f14941f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 0x7f14940b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14940cb850 con 0x7f14941f6550 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14940b1a20 con 0x7f14941f60d0 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f149a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f14940cb9d0 con 0x7f1478003aa0 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f1493e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f1493e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57592/0 (socket says 172.21.15.31:57592) 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f1493e00640 1 -- 172.21.15.31:0/3448999554 learned_addr learned my addr 172.21.15.31:0/3448999554 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f148aa00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 0x7f14940b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:51.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.074+0000 7f1493400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1478003aa0 0x7f14941f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 -- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1478003aa0 msgr2=0x7f14941f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1478003aa0 0x7f14941f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 -- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 msgr2=0x7f14940b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 0x7f14940b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 -- 172.21.15.31:0/3448999554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14940c3bf0 con 0x7f14941f6550 2024-09-13T08:38:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f148aa00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 0x7f14940b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:38:51.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f1493e00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f14940cf330 tx=0x7f148c0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c011870 con 0x7f14941f6550 2024-09-13T08:38:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f148c017070 con 0x7f14941f6550 2024-09-13T08:38:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14940c3ed0 con 0x7f14941f6550 2024-09-13T08:38:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c011d60 con 0x7f14941f6550 2024-09-13T08:38:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.075+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14940c4490 con 0x7f14941f6550 2024-09-13T08:38:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.076+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14940ce5d0 con 0x7f14941f6550 2024-09-13T08:38:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.077+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 101440+0+0 (secure 0 0 0) 0x7f148c01c460 con 0x7f14941f6550 2024-09-13T08:38:51.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.079+0000 7f148be00640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 0x7f147007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:51.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.079+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5854+0+0 (secure 0 0 0) 0x7f148c09b620 con 0x7f14941f6550 2024-09-13T08:38:51.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.079+0000 7f1493400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 0x7f147007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:51.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.080+0000 7f1493400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 0x7f147007aec0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1484033000 tx=0x7f1484001930 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:51.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.083+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f148c0637d0 con 0x7f14941f6550 2024-09-13T08:38:51.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.149+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f148c05f340 con 0x7f14941f6550 2024-09-13T08:38:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.324+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14940acde0 con 0x7f14941f6550 2024-09-13T08:38:51.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.325+0000 7f148be00640 1 -- 172.21.15.31:0/3448999554 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f148c06edf0 con 0x7f14941f6550 2024-09-13T08:38:51.326 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:51.326 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:51.326 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:38:51.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 msgr2=0x7f147007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:51.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 0x7f147007aec0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1484033000 tx=0x7f1484001930 comp rx=0 tx=0).stop 2024-09-13T08:38:51.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 msgr2=0x7f1494150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f14940cf330 tx=0x7f148c0019b0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 shutdown_connections 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14941f6550 0x7f1494150e30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14941f60d0 0x7f14940b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1470078a00 0x7f147007aec0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 --2- 172.21.15.31:0/3448999554 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1478003aa0 0x7f14941f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 >> 172.21.15.31:0/3448999554 conn(0x7f14940cc910 msgr2=0x7f14940ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 shutdown_connections 2024-09-13T08:38:51.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:51.327+0000 7f149a400640 1 -- 172.21.15.31:0/3448999554 wait complete. 2024-09-13T08:38:51.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:38:51.525 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":1.264978913,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.936598007727397,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:38:51.526 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.936598007727397 }, "state": "open", "uptime": 1.264978913 } ] 2024-09-13T08:38:51.526 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:38:56.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:38:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 -- 172.21.15.31:0/1509781768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 msgr2=0x7f823c0d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 --2- 172.21.15.31:0/1509781768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0d0a80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8228005fd0 tx=0x7f822802a670 comp rx=0 tx=0).stop 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 -- 172.21.15.31:0/1509781768 shutdown_connections 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 --2- 172.21.15.31:0/1509781768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0d0a80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 -- 172.21.15.31:0/1509781768 >> 172.21.15.31:0/1509781768 conn(0x7f823c0d4070 msgr2=0x7f823c0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 -- 172.21.15.31:0/1509781768 shutdown_connections 2024-09-13T08:38:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.698+0000 7f8241400640 1 -- 172.21.15.31:0/1509781768 wait complete. 2024-09-13T08:38:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.700+0000 7f8241400640 1 Processor -- start 2024-09-13T08:38:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.700+0000 7f8241400640 1 -- start start 2024-09-13T08:38:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f823c0d4c00 0x7f823c160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f823c15e310 0x7f823c161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f823c1696b0 con 0x7f823c0b9d30 2024-09-13T08:38:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0d13d0 con 0x7f823c15e310 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f8241400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f823c169830 con 0x7f823c0d4c00 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f823a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f823a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47668/0 (socket says 172.21.15.31:47668) 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.701+0000 7f823a800640 1 -- 172.21.15.31:0/4218924160 learned_addr learned my addr 172.21.15.31:0/4218924160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f8239e00640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f823c0d4c00 0x7f823c160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823b200640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f823c15e310 0x7f823c161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 -- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f823c0d4c00 msgr2=0x7f823c160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f823c0d4c00 0x7f823c160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 -- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f823c15e310 msgr2=0x7f823c161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f823c15e310 0x7f823c161350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 -- 172.21.15.31:0/4218924160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c0cdbc0 con 0x7f823c0b9d30 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f823a800640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c390 tx=0x7f822c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.702+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c017070 con 0x7f823c0b9d30 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.703+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f822c0053c0 con 0x7f823c0b9d30 2024-09-13T08:38:56.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.703+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c003880 con 0x7f823c0b9d30 2024-09-13T08:38:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.703+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c0cd600 con 0x7f823c0b9d30 2024-09-13T08:38:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.703+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c15ff40 con 0x7f823c0b9d30 2024-09-13T08:38:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.704+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823c0d2440 con 0x7f823c0b9d30 2024-09-13T08:38:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.705+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f822c01a070 con 0x7f823c0b9d30 2024-09-13T08:38:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.706+0000 7f8232a00640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 0x7f821c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:38:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.707+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f822c09a130 con 0x7f823c0b9d30 2024-09-13T08:38:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.707+0000 7f8239e00640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 0x7f821c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:38:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.708+0000 7f8239e00640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 0x7f821c07af50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8228033000 tx=0x7f8228004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:38:56.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.711+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f822c061d10 con 0x7f823c0b9d30 2024-09-13T08:38:56.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.948+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f823c06f180 con 0x7f823c0b9d30 2024-09-13T08:38:56.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.949+0000 7f8232a00640 1 -- 172.21.15.31:0/4218924160 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f822c066740 con 0x7f823c0b9d30 2024-09-13T08:38:56.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:38:56.951 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:38:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:38:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 msgr2=0x7f821c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 0x7f821c07af50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8228033000 tx=0x7f8228004730 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 msgr2=0x7f823c0cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c390 tx=0x7f822c00c850 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 shutdown_connections 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f823c15e310 0x7f823c161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f821c078a90 0x7f821c07af50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f823c0d4c00 0x7f823c160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 --2- 172.21.15.31:0/4218924160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823c0b9d30 0x7f823c0cd020 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 >> 172.21.15.31:0/4218924160 conn(0x7f823c0d4070 msgr2=0x7f823c0b7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 shutdown_connections 2024-09-13T08:38:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:38:56.951+0000 7f8241400640 1 -- 172.21.15.31:0/4218924160 wait complete. 2024-09-13T08:38:56.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:38:57.148 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":6.8878851819999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.653448196968739,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:38:57.149 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.65344819696874 }, "state": "open", "uptime": 6.887885182 } ] 2024-09-13T08:38:57.149 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:02.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 -- 172.21.15.31:0/3035153694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd270003a30 msgr2=0x7fd270003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 --2- 172.21.15.31:0/3035153694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd270003a30 0x7fd270003e50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c310 tx=0x7fd270005830 comp rx=0 tx=0).stop 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd28e400640 1 -- 172.21.15.31:0/3035153694 reap_dead start 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 -- 172.21.15.31:0/3035153694 shutdown_connections 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 -- 172.21.15.31:0/3035153694 >> 172.21.15.31:0/3035153694 conn(0x7fd2880cc910 msgr2=0x7fd2880cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 -- 172.21.15.31:0/3035153694 shutdown_connections 2024-09-13T08:39:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.322+0000 7fd290400640 1 -- 172.21.15.31:0/3035153694 wait complete. 2024-09-13T08:39:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.324+0000 7fd290400640 1 Processor -- start 2024-09-13T08:39:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.324+0000 7fd290400640 1 -- start start 2024-09-13T08:39:02.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 0x7fd28806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 0x7fd28815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881600f0 con 0x7fd2880b53b0 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2880cead0 con 0x7fd270003a30 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd290400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd2880b3f30 con 0x7fd28809aa10 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd28d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd28d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47676/0 (socket says 172.21.15.31:47676) 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.325+0000 7fd28d000640 1 -- 172.21.15.31:0/3312818793 learned_addr learned my addr 172.21.15.31:0/3312818793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28da00640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 0x7fd28806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 -- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 msgr2=0x7fd28815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28e400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 0x7fd28815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 0x7fd28815d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 -- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 msgr2=0x7fd28806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 0x7fd28806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 -- 172.21.15.31:0/3312818793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2880b36b0 con 0x7fd2880b53b0 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28da00640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 0x7fd28806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28e400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 0x7fd28815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd28d000640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd274004e20 tx=0x7fd274001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2740107d0 con 0x7fd2880b53b0 2024-09-13T08:39:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd274016070 con 0x7fd2880b53b0 2024-09-13T08:39:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274010c60 con 0x7fd2880b53b0 2024-09-13T08:39:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2880b59c0 con 0x7fd2880b53b0 2024-09-13T08:39:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.326+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2880b3c50 con 0x7fd2880b53b0 2024-09-13T08:39:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.327+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2880cb9c0 con 0x7fd2880b53b0 2024-09-13T08:39:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.328+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7fd27401b460 con 0x7fd2880b53b0 2024-09-13T08:39:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.330+0000 7fd285600640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 0x7fd26807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.330+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fd27409b730 con 0x7fd2880b53b0 2024-09-13T08:39:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.330+0000 7fd28da00640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 0x7fd26807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:02.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.331+0000 7fd28da00640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 0x7fd26807af50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd27c033000 tx=0x7fd27c0019f0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:02.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.335+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd274063890 con 0x7fd2880b53b0 2024-09-13T08:39:02.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.569+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2880b0270 con 0x7fd2880b53b0 2024-09-13T08:39:02.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.570+0000 7fd285600640 1 -- 172.21.15.31:0/3312818793 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fd274063190 con 0x7fd2880b53b0 2024-09-13T08:39:02.572 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:02.572 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:02.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 msgr2=0x7fd26807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 0x7fd26807af50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd27c033000 tx=0x7fd27c0019f0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 msgr2=0x7fd2880c2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd274004e20 tx=0x7fd274001630 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 shutdown_connections 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd268078a90 0x7fd26807af50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28809aa10 0x7fd28815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2880b53b0 0x7fd2880c2d00 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 --2- 172.21.15.31:0/3312818793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd270003a30 0x7fd28806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 >> 172.21.15.31:0/3312818793 conn(0x7fd2880cc910 msgr2=0x7fd2880af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 shutdown_connections 2024-09-13T08:39:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:02.573+0000 7fd290400640 1 -- 172.21.15.31:0/3312818793 wait complete. 2024-09-13T08:39:02.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:02.769 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":12.507791501,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.37410133433325,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:02.769 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.37410133433325 }, "state": "open", "uptime": 12.507791501 } ] 2024-09-13T08:39:02.770 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:05.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:39:05.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:39:07.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:07.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 -- 172.21.15.31:0/1093154721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 msgr2=0x7f19bc0b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:07.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 --2- 172.21.15.31:0/1093154721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc0b0f80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f19bc08d300 tx=0x7f19b402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:39:07.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 -- 172.21.15.31:0/1093154721 shutdown_connections 2024-09-13T08:39:07.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 --2- 172.21.15.31:0/1093154721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc0b0f80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 --2- 172.21.15.31:0/1093154721 >> v2:172.21.15.45:3300/0 conn(0x7f19bc0b47f0 0x7f19bc0d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 -- 172.21.15.31:0/1093154721 >> 172.21.15.31:0/1093154721 conn(0x7f19bc08d8b0 msgr2=0x7f19bc09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 -- 172.21.15.31:0/1093154721 shutdown_connections 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.942+0000 7f19c0a00640 1 -- 172.21.15.31:0/1093154721 wait complete. 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.944+0000 7f19c0a00640 1 Processor -- start 2024-09-13T08:39:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.944+0000 7f19c0a00640 1 -- start start 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19bc0b47f0 0x7f19bc161de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19bc15dfa0 0x7f19bc1f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc0b1850 con 0x7f19bc0d4110 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc0b16d0 con 0x7f19bc0b47f0 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19c0a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc0b19d0 con 0x7f19bc15dfa0 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33354/0 (socket says 172.21.15.31:33354) 2024-09-13T08:39:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 -- 172.21.15.31:0/3562709764 learned_addr learned my addr 172.21.15.31:0/3562709764 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19bb400640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19bc0b47f0 0x7f19bc161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19bbe00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19bc15dfa0 0x7f19bc1f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 -- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19bc15dfa0 msgr2=0x7f19bc1f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19bc15dfa0 0x7f19bc1f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 -- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19bc0b47f0 msgr2=0x7f19bc161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19bc0b47f0 0x7f19bc161de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.945+0000 7f19b3e00640 1 -- 172.21.15.31:0/3562709764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc0cde10 con 0x7f19bc0d4110 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19b3e00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f19a400ea20 tx=0x7f19a400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:07.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a400cd80 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19a4010430 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc15f150 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a4010840 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.946+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc15f710 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.947+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19bc09d2d0 con 0x7f19bc0d4110 2024-09-13T08:39:07.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.948+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f19a40109e0 con 0x7f19bc0d4110 2024-09-13T08:39:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.949+0000 7f19b3400640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 0x7f199807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:07.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.950+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f19a409bb10 con 0x7f19bc0d4110 2024-09-13T08:39:07.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.950+0000 7f19bb400640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 0x7f199807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:07.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.951+0000 7f19bb400640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 0x7f199807af50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f19ac007fd0 tx=0x7f19ac009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:07.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:07.954+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f19bc09d2d0 con 0x7f19bc0d4110 2024-09-13T08:39:08.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.190+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19bc0d4cf0 con 0x7f19bc0d4110 2024-09-13T08:39:08.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.191+0000 7f19b3400640 1 -- 172.21.15.31:0/3562709764 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f19a4063be0 con 0x7f19bc0d4110 2024-09-13T08:39:08.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:08.193 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:08.193 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 msgr2=0x7f199807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 0x7f199807af50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f19ac007fd0 tx=0x7f19ac009040 comp rx=0 tx=0).stop 2024-09-13T08:39:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 msgr2=0x7f19bc1f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f19a400ea20 tx=0x7f19a400eee0 comp rx=0 tx=0).stop 2024-09-13T08:39:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 shutdown_connections 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1998078a90 0x7f199807af50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19bc15dfa0 0x7f19bc1f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19bc0d4110 0x7f19bc1f3840 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 --2- 172.21.15.31:0/3562709764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19bc0b47f0 0x7f19bc161de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 >> 172.21.15.31:0/3562709764 conn(0x7f19bc08d8b0 msgr2=0x7f19bc0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 shutdown_connections 2024-09-13T08:39:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:08.193+0000 7f19c0a00640 1 -- 172.21.15.31:0/3562709764 wait complete. 2024-09-13T08:39:08.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:08.390 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":18.129697785000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.098260777267672,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:08.391 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.098260777267672 }, "state": "open", "uptime": 18.129697785 } ] 2024-09-13T08:39:08.391 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:13.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.563+0000 7f6997800640 1 -- 172.21.15.31:0/723851672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900cf850 msgr2=0x7f69900ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.563+0000 7f6997800640 1 --2- 172.21.15.31:0/723851672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900cf850 0x7f69900ce3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69900cfe60 tx=0x7f697802a5c0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.564+0000 7f6997800640 1 -- 172.21.15.31:0/723851672 shutdown_connections 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.564+0000 7f6997800640 1 --2- 172.21.15.31:0/723851672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900cf850 0x7f69900ce3c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.564+0000 7f6997800640 1 -- 172.21.15.31:0/723851672 >> 172.21.15.31:0/723851672 conn(0x7f69900cc910 msgr2=0x7f69900cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.564+0000 7f6997800640 1 -- 172.21.15.31:0/723851672 shutdown_connections 2024-09-13T08:39:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.564+0000 7f6997800640 1 -- 172.21.15.31:0/723851672 wait complete. 2024-09-13T08:39:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.565+0000 7f6997800640 1 Processor -- start 2024-09-13T08:39:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.565+0000 7f6997800640 1 -- start start 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 0x7f699014d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f699015bd70 0x7f69900c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69901600f0 con 0x7f69900cf850 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69900cb850 con 0x7f69900ce900 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.566+0000 7f6997800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f699009d2e0 con 0x7f699015bd70 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33370/0 (socket says 172.21.15.31:33370) 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 -- 172.21.15.31:0/568007285 learned_addr learned my addr 172.21.15.31:0/568007285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6995800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f699015bd70 0x7f69900c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6994e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 0x7f699014d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 -- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f699015bd70 msgr2=0x7f69900c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f699015bd70 0x7f69900c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 -- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 msgr2=0x7f699014d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 0x7f699014d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 -- 172.21.15.31:0/568007285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699009cd20 con 0x7f69900cf850 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6994e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 0x7f699014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:39:13.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.567+0000 7f6987e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f697802abb0 tx=0x7f6978032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.568+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978040070 con 0x7f69900cf850 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.568+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6978005310 con 0x7f69900cf850 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.568+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69780047a0 con 0x7f69900cf850 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.568+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699009d000 con 0x7f69900cf850 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.568+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69900c6f60 con 0x7f69900cf850 2024-09-13T08:39:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.569+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69900cc3b0 con 0x7f69900cf850 2024-09-13T08:39:13.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.570+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f6978039040 con 0x7f69900cf850 2024-09-13T08:39:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.571+0000 7f698d600640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 0x7f696c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.571+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f69780bbfe0 con 0x7f69900cf850 2024-09-13T08:39:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.571+0000 7f6994e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 0x7f696c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:13.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.572+0000 7f6994e00640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 0x7f696c07af50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6980007fd0 tx=0x7f6980009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:13.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.575+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6978084250 con 0x7f69900cf850 2024-09-13T08:39:13.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.813+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f699015fd80 con 0x7f69900cf850 2024-09-13T08:39:13.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.813+0000 7f698d600640 1 -- 172.21.15.31:0/568007285 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f6978083b50 con 0x7f69900cf850 2024-09-13T08:39:13.815 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:13.815 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:13.816 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 msgr2=0x7f696c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 0x7f696c07af50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6980007fd0 tx=0x7f6980009040 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 msgr2=0x7f69900c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f697802abb0 tx=0x7f6978032c10 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 shutdown_connections 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f696c078a90 0x7f696c07af50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f699015bd70 0x7f69900c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69900cf850 0x7f69900c9300 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.815+0000 7f6997800640 1 --2- 172.21.15.31:0/568007285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69900ce900 0x7f699014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.816+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 >> 172.21.15.31:0/568007285 conn(0x7f69900cc910 msgr2=0x7f699015e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.816+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 shutdown_connections 2024-09-13T08:39:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:13.816+0000 7f6997800640 1 -- 172.21.15.31:0/568007285 wait complete. 2024-09-13T08:39:13.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:14.012 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":23.750604086999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.826028160406938,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:14.012 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.82602816040694 }, "state": "open", "uptime": 23.750604087 } ] 2024-09-13T08:39:14.012 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:19.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:19.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 -- 172.21.15.31:0/826972236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 msgr2=0x7f3438152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 --2- 172.21.15.31:0/826972236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438152680 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3430005fd0 tx=0x7f343002aa90 comp rx=0 tx=0).stop 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 -- 172.21.15.31:0/826972236 shutdown_connections 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 --2- 172.21.15.31:0/826972236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438152680 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 --2- 172.21.15.31:0/826972236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f343806f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 -- 172.21.15.31:0/826972236 >> 172.21.15.31:0/826972236 conn(0x7f343806cb00 msgr2=0x7f343806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 -- 172.21.15.31:0/826972236 shutdown_connections 2024-09-13T08:39:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.185+0000 7f343ca00640 1 -- 172.21.15.31:0/826972236 wait complete. 2024-09-13T08:39:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.187+0000 7f343ca00640 1 Processor -- start 2024-09-13T08:39:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.187+0000 7f343ca00640 1 -- start start 2024-09-13T08:39:19.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:19.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f34381f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:19.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 0x7f343814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3438152bc0 con 0x7f34380b52d0 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3438070460 con 0x7f34380ccb00 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f343ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3438152d40 con 0x7f34380cc720 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f3437400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f3437400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59532/0 (socket says 172.21.15.31:59532) 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f3437400640 1 -- 172.21.15.31:0/3773099155 learned_addr learned my addr 172.21.15.31:0/3773099155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437e00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 0x7f343814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.188+0000 7f342ea00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f34381f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 -- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 msgr2=0x7f34381f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f34381f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 -- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 msgr2=0x7f343814fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 0x7f343814fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 -- 172.21.15.31:0/3773099155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34381f3540 con 0x7f34380b52d0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437e00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 0x7f343814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f342ea00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f34381f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f3437400640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f342800e9f0 tx=0x7f342800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800cd80 con 0x7f34380b52d0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3428010430 con 0x7f34380b52d0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343815cf20 con 0x7f34380b52d0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.189+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428010840 con 0x7f34380b52d0 2024-09-13T08:39:19.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.190+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343815d4e0 con 0x7f34380b52d0 2024-09-13T08:39:19.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.190+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438153260 con 0x7f34380b52d0 2024-09-13T08:39:19.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.192+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f34280109e0 con 0x7f34380b52d0 2024-09-13T08:39:19.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.193+0000 7f342fe00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 0x7f341407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:19.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.193+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f342809ba90 con 0x7f34380b52d0 2024-09-13T08:39:19.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.193+0000 7f342ea00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 0x7f341407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:19.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.194+0000 7f342ea00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 0x7f341407af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3420007fd0 tx=0x7f3420009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:19.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.198+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f34381f3540 con 0x7f34380b52d0 2024-09-13T08:39:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.430+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34380c9730 con 0x7f34380b52d0 2024-09-13T08:39:19.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.431+0000 7f342fe00640 1 -- 172.21.15.31:0/3773099155 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f3428063cb0 con 0x7f34380b52d0 2024-09-13T08:39:19.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:19.432 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 msgr2=0x7f341407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 0x7f341407af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3420007fd0 tx=0x7f3420009040 comp rx=0 tx=0).stop 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 msgr2=0x7f3438154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f342800e9f0 tx=0x7f342800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 shutdown_connections 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34380ccb00 0x7f343814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3414078a90 0x7f341407af50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34380cc720 0x7f34381f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 --2- 172.21.15.31:0/3773099155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34380b52d0 0x7f3438154f10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:19.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 >> 172.21.15.31:0/3773099155 conn(0x7f343806cb00 msgr2=0x7f343806f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:19.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 shutdown_connections 2024-09-13T08:39:19.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:19.433+0000 7f343ca00640 1 -- 172.21.15.31:0/3773099155 wait complete. 2024-09-13T08:39:19.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:19.628 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":29.368510442000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.557450674064377,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:19.629 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.557450674064377 }, "state": "open", "uptime": 29.368510442 } ] 2024-09-13T08:39:19.629 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:24.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.802+0000 7f69dd600640 1 -- 172.21.15.31:0/2096993237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d815dfc0 msgr2=0x7f69d80cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.802+0000 7f69dd600640 1 --2- 172.21.15.31:0/2096993237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d815dfc0 0x7f69d80cc300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f69d809de00 tx=0x7f69c802a560 comp rx=0 tx=0).stop 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 -- 172.21.15.31:0/2096993237 shutdown_connections 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 --2- 172.21.15.31:0/2096993237 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69d80ca5c0 0x7f69d80ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 --2- 172.21.15.31:0/2096993237 >> v2:172.21.15.45:3300/0 conn(0x7f69d80b1910 0x7f69d80b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 --2- 172.21.15.31:0/2096993237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d815dfc0 0x7f69d80cc300 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 -- 172.21.15.31:0/2096993237 >> 172.21.15.31:0/2096993237 conn(0x7f69d80cc910 msgr2=0x7f69d80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 -- 172.21.15.31:0/2096993237 shutdown_connections 2024-09-13T08:39:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.803+0000 7f69dd600640 1 -- 172.21.15.31:0/2096993237 wait complete. 2024-09-13T08:39:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.804+0000 7f69dd600640 1 Processor -- start 2024-09-13T08:39:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.804+0000 7f69dd600640 1 -- start start 2024-09-13T08:39:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69d80ca5c0 0x7f69d8158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d815dfc0 0x7f69d80b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8073900 con 0x7f69d80b1910 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8073780 con 0x7f69d815dfc0 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.805+0000 7f69dd600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f69d8073a80 con 0x7f69d80ca5c0 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59538/0 (socket says 172.21.15.31:59538) 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 -- 172.21.15.31:0/1993159999 learned_addr learned my addr 172.21.15.31:0/1993159999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 -- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69d80ca5c0 msgr2=0x7f69d8158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69d80ca5c0 0x7f69d8158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 -- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d815dfc0 msgr2=0x7f69d80b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d815dfc0 0x7f69d80b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 -- 172.21.15.31:0/1993159999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d81f8ca0 con 0x7f69d80b1910 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.806+0000 7f69d6a00640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f69c8034040 tx=0x7f69c80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.807+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c803e030 con 0x7f69d80b1910 2024-09-13T08:39:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.807+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69c80047a0 con 0x7f69d80b1910 2024-09-13T08:39:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.807+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8035040 con 0x7f69d80b1910 2024-09-13T08:39:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.807+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d81f8f80 con 0x7f69d80b1910 2024-09-13T08:39:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.807+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d80b43c0 con 0x7f69d80b1910 2024-09-13T08:39:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.808+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d8073fa0 con 0x7f69d80b1910 2024-09-13T08:39:24.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.809+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f69c8041030 con 0x7f69d80b1910 2024-09-13T08:39:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.810+0000 7f69cf400640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 0x7f69b407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.810+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f69c80bb630 con 0x7f69d80b1910 2024-09-13T08:39:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.810+0000 7f69ce000640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 0x7f69b407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.811+0000 7f69ce000640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 0x7f69b407af50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f69d80b31b0 tx=0x7f69c0005c70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:24.814+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f69c80837d0 con 0x7f69d80b1910 2024-09-13T08:39:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.050+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69d80af110 con 0x7f69d80b1910 2024-09-13T08:39:25.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.050+0000 7f69cf400640 1 -- 172.21.15.31:0/1993159999 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f69c80830d0 con 0x7f69d80b1910 2024-09-13T08:39:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:25.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:25.052 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 msgr2=0x7f69b407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 0x7f69b407af50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f69d80b31b0 tx=0x7f69c0005c70 comp rx=0 tx=0).stop 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 msgr2=0x7f69d815bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f69c8034040 tx=0x7f69c80039e0 comp rx=0 tx=0).stop 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 shutdown_connections 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d815dfc0 0x7f69d80b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69b4078a90 0x7f69b407af50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69d80ca5c0 0x7f69d8158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 --2- 172.21.15.31:0/1993159999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69d80b1910 0x7f69d815bc50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 >> 172.21.15.31:0/1993159999 conn(0x7f69d80cc910 msgr2=0x7f69d80ad4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 shutdown_connections 2024-09-13T08:39:25.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:25.053+0000 7f69dd600640 1 -- 172.21.15.31:0/1993159999 wait complete. 2024-09-13T08:39:25.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:25.249 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":34.989416744000003,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.292196175029275,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:25.250 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.292196175029275 }, "state": "open", "uptime": 34.989416744 } ] 2024-09-13T08:39:25.250 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:30.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:30.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 -- 172.21.15.31:0/118383394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 msgr2=0x7f62a806f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 --2- 172.21.15.31:0/118383394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a806f9b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6290009830 tx=0x7f629002a510 comp rx=0 tx=0).stop 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 -- 172.21.15.31:0/118383394 shutdown_connections 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 --2- 172.21.15.31:0/118383394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80ccb00 0x7f62a8152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 --2- 172.21.15.31:0/118383394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a806f9b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.422+0000 7f62a7e00640 1 -- 172.21.15.31:0/118383394 >> 172.21.15.31:0/118383394 conn(0x7f62a806cb00 msgr2=0x7f62a806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.423+0000 7f62a7e00640 1 -- 172.21.15.31:0/118383394 shutdown_connections 2024-09-13T08:39:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.423+0000 7f62a7e00640 1 -- 172.21.15.31:0/118383394 wait complete. 2024-09-13T08:39:30.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.424+0000 7f62a7e00640 1 Processor -- start 2024-09-13T08:39:30.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.424+0000 7f62a7e00640 1 -- start start 2024-09-13T08:39:30.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:30.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80cc720 0x7f62a81f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a80ccb00 0x7f62a81f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8152bc0 con 0x7f62a80b52d0 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62a80703e0 con 0x7f62a80ccb00 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f62a8152d40 con 0x7f62a80cc720 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.425+0000 7f62a6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39962/0 (socket says 172.21.15.31:39962) 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 -- 172.21.15.31:0/1691970066 learned_addr learned my addr 172.21.15.31:0/1691970066 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a7400640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a80ccb00 0x7f62a81f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f629e000640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80cc720 0x7f62a81f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 -- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80cc720 msgr2=0x7f62a81f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80cc720 0x7f62a81f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 -- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a80ccb00 msgr2=0x7f62a81f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:30.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a80ccb00 0x7f62a81f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 -- 172.21.15.31:0/1691970066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a8146fd0 con 0x7f62a80b52d0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.426+0000 7f62a6a00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f629800c530 tx=0x7f629800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.427+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298017070 con 0x7f62a80b52d0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.427+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62980035f0 con 0x7f62a80b52d0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.427+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a81472b0 con 0x7f62a80b52d0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.427+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298003a40 con 0x7f62a80b52d0 2024-09-13T08:39:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.427+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81f55f0 con 0x7f62a80b52d0 2024-09-13T08:39:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.428+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a8147a80 con 0x7f62a80b52d0 2024-09-13T08:39:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.429+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f629801a070 con 0x7f62a80b52d0 2024-09-13T08:39:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.430+0000 7f629f400640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 0x7f628407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.431+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f629809a3a0 con 0x7f62a80b52d0 2024-09-13T08:39:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.431+0000 7f629e000640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 0x7f628407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:30.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.432+0000 7f629e000640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 0x7f628407af50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6290034000 tx=0x7f6290004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.435+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6298062500 con 0x7f62a80b52d0 2024-09-13T08:39:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.666+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62a806f9b0 con 0x7f62a80b52d0 2024-09-13T08:39:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.667+0000 7f629f400640 1 -- 172.21.15.31:0/1691970066 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f6298061e00 con 0x7f62a80b52d0 2024-09-13T08:39:30.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:30.668 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 msgr2=0x7f628407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 0x7f628407af50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6290034000 tx=0x7f6290004730 comp rx=0 tx=0).stop 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 msgr2=0x7f62a815e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f629800c530 tx=0x7f629800c9f0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 shutdown_connections 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a80ccb00 0x7f62a81f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6284078a90 0x7f628407af50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a80cc720 0x7f62a81f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 --2- 172.21.15.31:0/1691970066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a80b52d0 0x7f62a815e640 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 >> 172.21.15.31:0/1691970066 conn(0x7f62a806cb00 msgr2=0x7f62a80b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 shutdown_connections 2024-09-13T08:39:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:30.669+0000 7f62a7e00640 1 -- 172.21.15.31:0/1691970066 wait complete. 2024-09-13T08:39:30.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:30.864 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":40.603323011999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.03068824087028,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:30.864 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.03068824087028 }, "state": "open", "uptime": 40.603323012 } ] 2024-09-13T08:39:30.864 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:35.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:36.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:39:36.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 -- 172.21.15.31:0/2289375102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c003fd0 msgr2=0x7fb92c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 --2- 172.21.15.31:0/2289375102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c003fd0 0x7fb92c004800 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb930009b50 tx=0x7fb92c005830 comp rx=0 tx=0).stop 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94d000640 1 -- 172.21.15.31:0/2289375102 reap_dead start 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 -- 172.21.15.31:0/2289375102 shutdown_connections 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 -- 172.21.15.31:0/2289375102 >> 172.21.15.31:0/2289375102 conn(0x7fb9480cc910 msgr2=0x7fb9480cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 -- 172.21.15.31:0/2289375102 shutdown_connections 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.039+0000 7fb94f000640 1 -- 172.21.15.31:0/2289375102 wait complete. 2024-09-13T08:39:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.041+0000 7fb94f000640 1 Processor -- start 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.041+0000 7fb94f000640 1 -- start start 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb92c003fd0 0x7fb9481f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb94814b7b0 0x7fb9480b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481600f0 con 0x7fb9480b0d20 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9480cb850 con 0x7fb94814b7b0 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb9480b3370 con 0x7fb92c003fd0 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35286/0 (socket says 172.21.15.31:35286) 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 -- 172.21.15.31:0/1357861462 learned_addr learned my addr 172.21.15.31:0/1357861462 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb93f400640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb94814b7b0 0x7fb9480b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:36.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb947e00640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb92c003fd0 0x7fb9481f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 -- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb92c003fd0 msgr2=0x7fb9481f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb92c003fd0 0x7fb9481f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 -- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb94814b7b0 msgr2=0x7fb9480b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.042+0000 7fb94d000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb94814b7b0 0x7fb9480b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb94d000640 1 -- 172.21.15.31:0/1357861462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9480b2d60 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb94d000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb940005fd0 tx=0x7fb940010e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94000ee20 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb940015070 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940003d00 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9480c9c50 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.043+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9480c9f30 con 0x7fb9480b0d20 2024-09-13T08:39:36.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.044+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9480ce3c0 con 0x7fb9480b0d20 2024-09-13T08:39:36.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.046+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7fb940004030 con 0x7fb9480b0d20 2024-09-13T08:39:36.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.047+0000 7fb944c00640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 0x7fb92407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:36.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.047+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fb94009b490 con 0x7fb9480b0d20 2024-09-13T08:39:36.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.047+0000 7fb947e00640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 0x7fb92407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:36.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.048+0000 7fb947e00640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 0x7fb92407af50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb938006c50 tx=0x7fb9380067a0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:36.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.051+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb940062fa0 con 0x7fb9480b0d20 2024-09-13T08:39:36.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.287+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb94806fa30 con 0x7fb9480b0d20 2024-09-13T08:39:36.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.288+0000 7fb944c00640 1 -- 172.21.15.31:0/1357861462 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fb9400679d0 con 0x7fb9480b0d20 2024-09-13T08:39:36.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:36.289 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:36.290 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 msgr2=0x7fb92407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 0x7fb92407af50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb938006c50 tx=0x7fb9380067a0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 msgr2=0x7fb9480b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb940005fd0 tx=0x7fb940010e80 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 shutdown_connections 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9480b0d20 0x7fb9480b3800 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb94814b7b0 0x7fb9480b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb924078a90 0x7fb92407af50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 --2- 172.21.15.31:0/1357861462 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb92c003fd0 0x7fb9481f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 >> 172.21.15.31:0/1357861462 conn(0x7fb9480cc910 msgr2=0x7fb94815b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:36.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 shutdown_connections 2024-09-13T08:39:36.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:36.291+0000 7fb94f000640 1 -- 172.21.15.31:0/1357861462 wait complete. 2024-09-13T08:39:36.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:36.487 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":46.226229105000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.772139244057101,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:36.488 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.7721392440571 }, "state": "open", "uptime": 46.226229105 } ] 2024-09-13T08:39:36.488 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:41.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:41.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 -- 172.21.15.31:0/3479011420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 msgr2=0x7f17ec1625a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:41.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 --2- 172.21.15.31:0/3479011420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1625a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0d1870 tx=0x7f17d802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 -- 172.21.15.31:0/3479011420 shutdown_connections 2024-09-13T08:39:41.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 --2- 172.21.15.31:0/3479011420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1625a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 --2- 172.21.15.31:0/3479011420 >> v2:172.21.15.45:3300/0 conn(0x7f17ec0d20c0 0x7f17ec162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 -- 172.21.15.31:0/3479011420 >> 172.21.15.31:0/3479011420 conn(0x7f17ec09d1c0 msgr2=0x7f17ec0d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 -- 172.21.15.31:0/3479011420 shutdown_connections 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.659+0000 7f17f2400640 1 -- 172.21.15.31:0/3479011420 wait complete. 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.661+0000 7f17f2400640 1 Processor -- start 2024-09-13T08:39:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.661+0000 7f17f2400640 1 -- start start 2024-09-13T08:39:41.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:41.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ec0d20c0 0x7f17ec14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:41.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 0x7f17ec1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:41.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec0d24a0 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec0b7eb0 con 0x7f17ec0d20c0 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f2400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec0d2620 con 0x7f17ec0d44f0 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35304/0 (socket says 172.21.15.31:35304) 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 -- 172.21.15.31:0/359229795 learned_addr learned my addr 172.21.15.31:0/359229795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17ebe00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ec0d20c0 0x7f17ec14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1a00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 0x7f17ec1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 -- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 msgr2=0x7f17ec1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 0x7f17ec1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 -- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ec0d20c0 msgr2=0x7f17ec14f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ec0d20c0 0x7f17ec14f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1000640 1 -- 172.21.15.31:0/359229795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec1f87a0 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.662+0000 7f17f1a00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 0x7f17ec1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17f1000640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17e000d770 tx=0x7f17e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0017070 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17e0009d20 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0005280 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec152c00 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.663+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1f7a60 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.664+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ec0b57b0 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.665+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f17e0005420 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.666+0000 7f17e8c00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 0x7f17cc07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.666+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f17e009b080 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.667+0000 7f17ebe00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 0x7f17cc07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.667+0000 7f17ebe00640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 0x7f17cc07af50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f17d8033000 tx=0x7f17d8004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.670+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f17e0062b10 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.903+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17ec08df70 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.904+0000 7f17e8c00640 1 -- 172.21.15.31:0/359229795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f17e0067540 con 0x7f17ec0cf2d0 2024-09-13T08:39:41.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:41.905 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:41.905 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.906+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 msgr2=0x7f17cc07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.906+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 0x7f17cc07af50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f17d8033000 tx=0x7f17d8004730 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 msgr2=0x7f17ec1f8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17e000d770 tx=0x7f17e000dc30 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 shutdown_connections 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f17cc078a90 0x7f17cc07af50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17ec0d44f0 0x7f17ec1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ec0d20c0 0x7f17ec14f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 --2- 172.21.15.31:0/359229795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17ec0cf2d0 0x7f17ec1f8260 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 >> 172.21.15.31:0/359229795 conn(0x7f17ec09d1c0 msgr2=0x7f17ec0d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 shutdown_connections 2024-09-13T08:39:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:41.907+0000 7f17f2400640 1 -- 172.21.15.31:0/359229795 wait complete. 2024-09-13T08:39:41.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:42.102 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":51.841135328,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.517288252594408,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:42.102 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.51728825259441 }, "state": "open", "uptime": 51.841135328 } ] 2024-09-13T08:39:42.102 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:47.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:47.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 -- 172.21.15.31:0/2438752078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64640b1910 msgr2=0x7f64640b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 --2- 172.21.15.31:0/2438752078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64640b1910 0x7f64640b1da0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f64640cd8d0 tx=0x7f645c02aa10 comp rx=0 tx=0).stop 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 -- 172.21.15.31:0/2438752078 shutdown_connections 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 --2- 172.21.15.31:0/2438752078 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640ca5c0 0x7f64640ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 --2- 172.21.15.31:0/2438752078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64640b1910 0x7f64640b1da0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 --2- 172.21.15.31:0/2438752078 >> v2:172.21.15.45:3300/0 conn(0x7f646415dfc0 0x7f64640cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 -- 172.21.15.31:0/2438752078 >> 172.21.15.31:0/2438752078 conn(0x7f64640cc910 msgr2=0x7f64640cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 -- 172.21.15.31:0/2438752078 shutdown_connections 2024-09-13T08:39:47.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.274+0000 7f646a800640 1 -- 172.21.15.31:0/2438752078 wait complete. 2024-09-13T08:39:47.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.276+0000 7f646a800640 1 Processor -- start 2024-09-13T08:39:47.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.276+0000 7f646a800640 1 -- start start 2024-09-13T08:39:47.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640b1910 0x7f64640c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:47.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64640ca5c0 0x7f646414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:47.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:47.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6464073900 con 0x7f646415dfc0 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6464073780 con 0x7f64640ca5c0 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f646a800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6464073a80 con 0x7f64640b1910 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.277+0000 7f6463e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36994/0 (socket says 172.21.15.31:36994) 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 -- 172.21.15.31:0/2876207017 learned_addr learned my addr 172.21.15.31:0/2876207017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463400640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640b1910 0x7f64640c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f645aa00640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64640ca5c0 0x7f646414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 -- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640b1910 msgr2=0x7f64640c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640b1910 0x7f64640c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 -- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64640ca5c0 msgr2=0x7f646414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64640ca5c0 0x7f646414e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 -- 172.21.15.31:0/2876207017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64640c7000 con 0x7f646415dfc0 2024-09-13T08:39:47.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.278+0000 7f6463e00640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f645c009d60 tx=0x7f645c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:47.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.281+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c047070 con 0x7f646415dfc0 2024-09-13T08:39:47.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.281+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f645c004250 con 0x7f646415dfc0 2024-09-13T08:39:47.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.281+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c033050 con 0x7f646415dfc0 2024-09-13T08:39:47.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.281+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64640c7270 con 0x7f646415dfc0 2024-09-13T08:39:47.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.281+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646414e7c0 con 0x7f646415dfc0 2024-09-13T08:39:47.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.282+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6464073fa0 con 0x7f646415dfc0 2024-09-13T08:39:47.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.283+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f645c04a070 con 0x7f646415dfc0 2024-09-13T08:39:47.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.284+0000 7f645be00640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 0x7f644007af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:47.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.284+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f645c0bc220 con 0x7f646415dfc0 2024-09-13T08:39:47.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.284+0000 7f6463400640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 0x7f644007af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:47.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.285+0000 7f6463400640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 0x7f644007af50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f646415b1e0 tx=0x7f6454005e30 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:47.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.288+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f645c0842f0 con 0x7f646415dfc0 2024-09-13T08:39:47.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.522+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64640af5d0 con 0x7f646415dfc0 2024-09-13T08:39:47.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.522+0000 7f645be00640 1 -- 172.21.15.31:0/2876207017 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f645c083bf0 con 0x7f646415dfc0 2024-09-13T08:39:47.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:47.523 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:47.524 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 msgr2=0x7f644007af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 0x7f644007af50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f646415b1e0 tx=0x7f6454005e30 comp rx=0 tx=0).stop 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 msgr2=0x7f646415d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f645c009d60 tx=0x7f645c003f10 comp rx=0 tx=0).stop 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 shutdown_connections 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f646415dfc0 0x7f646415d180 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64640ca5c0 0x7f646414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6440078a90 0x7f644007af50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 --2- 172.21.15.31:0/2876207017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64640b1910 0x7f64640c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 >> 172.21.15.31:0/2876207017 conn(0x7f64640cc910 msgr2=0x7f64640b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 shutdown_connections 2024-09-13T08:39:47.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:47.525+0000 7f646a800640 1 -- 172.21.15.31:0/2876207017 wait complete. 2024-09-13T08:39:47.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:47.720 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":57.460041484999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.265544087922759,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:47.721 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.26554408792276 }, "state": "open", "uptime": 57.460041485 } ] 2024-09-13T08:39:47.722 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:52.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 -- 172.21.15.31:0/2446662297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0ca5c0 msgr2=0x7fd98c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 --2- 172.21.15.31:0/2446662297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0ca5c0 0x7fd98c0cc300 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd98c0b27a0 tx=0x7fd97402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 -- 172.21.15.31:0/2446662297 shutdown_connections 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 --2- 172.21.15.31:0/2446662297 >> v2:172.21.15.45:3300/0 conn(0x7fd98c0b1910 0x7fd98c0b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 --2- 172.21.15.31:0/2446662297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0ca5c0 0x7fd98c0cc300 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.893+0000 7fd993000640 1 -- 172.21.15.31:0/2446662297 >> 172.21.15.31:0/2446662297 conn(0x7fd98c0cc910 msgr2=0x7fd98c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.894+0000 7fd993000640 1 -- 172.21.15.31:0/2446662297 shutdown_connections 2024-09-13T08:39:52.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.894+0000 7fd993000640 1 -- 172.21.15.31:0/2446662297 wait complete. 2024-09-13T08:39:52.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.895+0000 7fd993000640 1 Processor -- start 2024-09-13T08:39:52.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.895+0000 7fd993000640 1 -- start start 2024-09-13T08:39:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0ca5c0 0x7fd98c149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 0x7fd98c0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c073900 con 0x7fd98c0b1910 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c073780 con 0x7fd98c15dfc0 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd993000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c073a80 con 0x7fd98c0ca5c0 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.896+0000 7fd98be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37010/0 (socket says 172.21.15.31:37010) 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 -- 172.21.15.31:0/3588599497 learned_addr learned my addr 172.21.15.31:0/3588599497 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd983400640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0ca5c0 0x7fd98c149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd991000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 0x7fd98c0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 -- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0ca5c0 msgr2=0x7fd98c149520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0ca5c0 0x7fd98c149520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 -- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 msgr2=0x7fd98c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:52.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 0x7fd98c0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 -- 172.21.15.31:0/3588599497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0b2e60 con 0x7fd98c0b1910 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd991000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 0x7fd98c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.897+0000 7fd98be00640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00c3d0 tx=0x7fd97c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.898+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c017070 con 0x7fd98c0b1910 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.898+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd97c0053c0 con 0x7fd98c0b1910 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.898+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c003850 con 0x7fd98c0b1910 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.898+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c0b3100 con 0x7fd98c0b1910 2024-09-13T08:39:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.898+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c0b36c0 con 0x7fd98c0b1910 2024-09-13T08:39:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.899+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c073f80 con 0x7fd98c0b1910 2024-09-13T08:39:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.900+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7fd97c01a070 con 0x7fd98c0b1910 2024-09-13T08:39:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.901+0000 7fd988c00640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 0x7fd96807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.901+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fd97c09a180 con 0x7fd98c0b1910 2024-09-13T08:39:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.901+0000 7fd983400640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 0x7fd96807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:52.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.902+0000 7fd983400640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 0x7fd96807af50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:52.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:52.905+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd98c0cbe40 con 0x7fd98c0b1910 2024-09-13T08:39:53.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.140+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd98c06ac50 con 0x7fd98c0b1910 2024-09-13T08:39:53.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.141+0000 7fd988c00640 1 -- 172.21.15.31:0/3588599497 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fd97c061cd0 con 0x7fd98c0b1910 2024-09-13T08:39:53.143 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:53.143 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:53.143 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 msgr2=0x7fd96807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 0x7fd96807af50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974004730 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 msgr2=0x7fd98c1f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00c3d0 tx=0x7fd97c00c890 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 shutdown_connections 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c15dfc0 0x7fd98c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd968078a90 0x7fd96807af50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0ca5c0 0x7fd98c149520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 --2- 172.21.15.31:0/3588599497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c0b1910 0x7fd98c1f5980 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 >> 172.21.15.31:0/3588599497 conn(0x7fd98c0cc910 msgr2=0x7fd98c0ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:53.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 shutdown_connections 2024-09-13T08:39:53.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:53.143+0000 7fd993000640 1 -- 172.21.15.31:0/3588599497 wait complete. 2024-09-13T08:39:53.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:53.341 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":63.079947625999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.017003112849469,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:53.342 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.01700311284947 }, "state": "open", "uptime": 63.079947626 } ] 2024-09-13T08:39:53.342 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:39:58.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:39:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 -- 172.21.15.31:0/1686872890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15bd70 msgr2=0x7f4c1c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 --2- 172.21.15.31:0/1686872890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0cf850 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c005fd0 tx=0x7f4c0c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 -- 172.21.15.31:0/1686872890 shutdown_connections 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 --2- 172.21.15.31:0/1686872890 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15fe20 0x7f4c1c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 --2- 172.21.15.31:0/1686872890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0cf850 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 -- 172.21.15.31:0/1686872890 >> 172.21.15.31:0/1686872890 conn(0x7f4c1c0cc910 msgr2=0x7f4c1c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 -- 172.21.15.31:0/1686872890 shutdown_connections 2024-09-13T08:39:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.513+0000 7f4c23400640 1 -- 172.21.15.31:0/1686872890 wait complete. 2024-09-13T08:39:58.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.515+0000 7f4c23400640 1 Processor -- start 2024-09-13T08:39:58.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.515+0000 7f4c23400640 1 -- start start 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 0x7f4c1c1f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c0cec50 con 0x7f4c1c15fe20 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c0cead0 con 0x7f4c1c15bd70 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c23400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4c1c0cb850 con 0x7f4c1c15f9e0 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c21400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c21400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46972/0 (socket says 172.21.15.31:46972) 2024-09-13T08:39:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.516+0000 7f4c21400640 1 -- 172.21.15.31:0/4283078028 learned_addr learned my addr 172.21.15.31:0/4283078028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c20a00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 -- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 msgr2=0x7f4c1c1f7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c13e00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 0x7f4c1c1f7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 0x7f4c1c1f7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 -- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 msgr2=0x7f4c1c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 -- 172.21.15.31:0/4283078028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c0c3f10 con 0x7f4c1c15fe20 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c20a00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c13e00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 0x7f4c1c1f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c21400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e960 tx=0x7f4c1400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14004d20 con 0x7f4c1c15fe20 2024-09-13T08:39:58.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c1400cb20 con 0x7f4c1c15fe20 2024-09-13T08:39:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1c0c6070 con 0x7f4c1c15fe20 2024-09-13T08:39:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14005020 con 0x7f4c1c15fe20 2024-09-13T08:39:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.517+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c1c0c6630 con 0x7f4c1c15fe20 2024-09-13T08:39:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.518+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c1c0cf850 con 0x7f4c1c15fe20 2024-09-13T08:39:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.519+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f4c140051c0 con 0x7f4c1c15fe20 2024-09-13T08:39:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.520+0000 7f4c19600640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 0x7f4bf807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:39:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.520+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f4c1409b820 con 0x7f4c1c15fe20 2024-09-13T08:39:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.521+0000 7f4c20a00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 0x7f4bf807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:39:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.522+0000 7f4c20a00640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 0x7f4bf807af50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c033000 tx=0x7f4c0c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:39:58.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.525+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4c14016030 con 0x7f4c1c15fe20 2024-09-13T08:39:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.762+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c1c0ca9f0 con 0x7f4c1c15fe20 2024-09-13T08:39:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.763+0000 7f4c19600640 1 -- 172.21.15.31:0/4283078028 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f4c14063980 con 0x7f4c1c15fe20 2024-09-13T08:39:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:39:58.765 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:39:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:39:58.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 msgr2=0x7f4bf807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:58.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 0x7f4bf807af50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c033000 tx=0x7f4c0c004730 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 msgr2=0x7f4c1c1f8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e960 tx=0x7f4c1400ee20 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 shutdown_connections 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c1c15fe20 0x7f4c1c1f8400 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4bf8078a90 0x7f4bf807af50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c1c15f9e0 0x7f4c1c1f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 --2- 172.21.15.31:0/4283078028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c1c15bd70 0x7f4c1c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 >> 172.21.15.31:0/4283078028 conn(0x7f4c1c0cc910 msgr2=0x7f4c1c0acf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.765+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 shutdown_connections 2024-09-13T08:39:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:39:58.766+0000 7f4c23400640 1 -- 172.21.15.31:0/4283078028 wait complete. 2024-09-13T08:39:58.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:39:58.963 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":68.702853716000007,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.771538403125625,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:39:58.964 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.771538403125625 }, "state": "open", "uptime": 68.702853716 } ] 2024-09-13T08:39:58.964 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:40:03.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 -- 172.21.15.31:0/3924947384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 msgr2=0x7fb37015fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 --2- 172.21.15.31:0/3924947384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb37015fe20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb3700cc8c0 tx=0x7fb35802a560 comp rx=0 tx=0).stop 2024-09-13T08:40:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 -- 172.21.15.31:0/3924947384 shutdown_connections 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 --2- 172.21.15.31:0/3924947384 >> v2:172.21.15.45:3300/0 conn(0x7fb3700ce3c0 0x7fb3700ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 --2- 172.21.15.31:0/3924947384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb37015fe20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 --2- 172.21.15.31:0/3924947384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 0x7fb3700cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 -- 172.21.15.31:0/3924947384 >> 172.21.15.31:0/3924947384 conn(0x7fb3700cc910 msgr2=0x7fb3700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 -- 172.21.15.31:0/3924947384 shutdown_connections 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.138+0000 7fb376600640 1 -- 172.21.15.31:0/3924947384 wait complete. 2024-09-13T08:40:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.140+0000 7fb376600640 1 Processor -- start 2024-09-13T08:40:04.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.140+0000 7fb376600640 1 -- start start 2024-09-13T08:40:04.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3700ce3c0 0x7fb37006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 0x7fb3700c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3700cb9d0 con 0x7fb37015f9e0 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3700cb850 con 0x7fb3700ce3c0 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb376600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb3700cbc50 con 0x7fb37015bd70 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb36fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb36f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3700ce3c0 0x7fb37006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb36fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46984/0 (socket says 172.21.15.31:46984) 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb36fe00640 1 -- 172.21.15.31:0/803619015 learned_addr learned my addr 172.21.15.31:0/803619015 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.141+0000 7fb366a00640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 0x7fb3700c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 -- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 msgr2=0x7fb3700c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:04.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 0x7fb3700c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 -- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3700ce3c0 msgr2=0x7fb37006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:04.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3700ce3c0 0x7fb37006ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 -- 172.21.15.31:0/803619015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3700c5990 con 0x7fb37015f9e0 2024-09-13T08:40:04.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb36fe00640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb36800eaa0 tx=0x7fb36800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:04.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36800cdb0 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb368004c10 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3700c5c30 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb368005020 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.142+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3700c61f0 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.143+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3700cf850 con 0x7fb37015f9e0 2024-09-13T08:40:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.145+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7fb3680051c0 con 0x7fb37015f9e0 2024-09-13T08:40:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.146+0000 7fb367e00640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 0x7fb34c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:04.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.146+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fb36809b800 con 0x7fb37015f9e0 2024-09-13T08:40:04.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.146+0000 7fb36f400640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 0x7fb34c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:04.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.147+0000 7fb36f400640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 0x7fb34c07af50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb36000b390 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:04.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.150+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb368016030 con 0x7fb37015f9e0 2024-09-13T08:40:04.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.382+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb37015ec40 con 0x7fb37015f9e0 2024-09-13T08:40:04.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.382+0000 7fb367e00640 1 -- 172.21.15.31:0/803619015 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fb368063310 con 0x7fb37015f9e0 2024-09-13T08:40:04.384 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:04.384 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:04.385 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.384+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 msgr2=0x7fb34c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.384+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 0x7fb34c07af50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb36000b390 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.384+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 msgr2=0x7fb3700c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.384+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb36800eaa0 tx=0x7fb36800ef60 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 shutdown_connections 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37015f9e0 0x7fb3700c3ee0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb34c078a90 0x7fb34c07af50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb37015bd70 0x7fb3700c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 --2- 172.21.15.31:0/803619015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3700ce3c0 0x7fb37006ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 >> 172.21.15.31:0/803619015 conn(0x7fb3700cc910 msgr2=0x7fb3700acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 shutdown_connections 2024-09-13T08:40:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:04.385+0000 7fb376600640 1 -- 172.21.15.31:0/803619015 wait complete. 2024-09-13T08:40:04.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:40:04.581 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":74.319759908999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.529498931744609,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:40:04.581 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.52949893174461 }, "state": "open", "uptime": 74.319759909 } ] 2024-09-13T08:40:04.582 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:40:06.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:40:06.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:40:09.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:09.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 -- 172.21.15.31:0/1298774747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15bd70 msgr2=0x7ff9fc0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:09.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 --2- 172.21.15.31:0/1298774747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc0cf850 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec005fd0 tx=0x7ff9ec02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 -- 172.21.15.31:0/1298774747 shutdown_connections 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 --2- 172.21.15.31:0/1298774747 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15fe20 0x7ff9fc0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 --2- 172.21.15.31:0/1298774747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc0cf850 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 -- 172.21.15.31:0/1298774747 >> 172.21.15.31:0/1298774747 conn(0x7ff9fc0cc910 msgr2=0x7ff9fc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 -- 172.21.15.31:0/1298774747 shutdown_connections 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.752+0000 7ffa01200640 1 -- 172.21.15.31:0/1298774747 wait complete. 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.754+0000 7ffa01200640 1 Processor -- start 2024-09-13T08:40:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.754+0000 7ffa01200640 1 -- start start 2024-09-13T08:40:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15f9e0 0x7ff9fc0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cec50 con 0x7ff9fc15fe20 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cead0 con 0x7ff9fc15bd70 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ffa01200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc0cb850 con 0x7ff9fc15f9e0 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44278/0 (socket says 172.21.15.31:44278) 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 -- 172.21.15.31:0/3028171938 learned_addr learned my addr 172.21.15.31:0/3028171938 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9f9e00640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9f1400640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15f9e0 0x7ff9fc0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 -- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15f9e0 msgr2=0x7ff9fc0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15f9e0 0x7ff9fc0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 -- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9fc15bd70 msgr2=0x7ff9fc159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.755+0000 7ff9fa800640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ff9fa800640 1 -- 172.21.15.31:0/3028171938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc09bcb0 con 0x7ff9fc15fe20 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ff9fa800640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400e960 tx=0x7ff9f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4004d20 con 0x7ff9fc15fe20 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9f400cb20 con 0x7ff9fc15fe20 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4005020 con 0x7ff9fc15fe20 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc09bf90 con 0x7ff9fc15fe20 2024-09-13T08:40:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.756+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc09c550 con 0x7ff9fc15fe20 2024-09-13T08:40:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.758+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc0cf850 con 0x7ff9fc15fe20 2024-09-13T08:40:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.759+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7ff9f40051c0 con 0x7ff9fc15fe20 2024-09-13T08:40:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.760+0000 7ff9f2a00640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 0x7ff9d807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.760+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7ff9f409b6a0 con 0x7ff9fc15fe20 2024-09-13T08:40:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.760+0000 7ff9f9e00640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 0x7ff9d807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.761+0000 7ff9f9e00640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 0x7ff9d807af50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec033000 tx=0x7ff9ec004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.764+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff9f4016030 con 0x7ff9fc15fe20 2024-09-13T08:40:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.994+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9fc09d820 con 0x7ff9fc15fe20 2024-09-13T08:40:09.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.994+0000 7ff9f2a00640 1 -- 172.21.15.31:0/3028171938 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7ff9f4063250 con 0x7ff9fc15fe20 2024-09-13T08:40:09.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:09.997 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:09.997 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:09.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.996+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 msgr2=0x7ff9d807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.996+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 0x7ff9d807af50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec033000 tx=0x7ff9ec004730 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.996+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 msgr2=0x7ff9fc0b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.996+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400e960 tx=0x7ff9f400ee20 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 shutdown_connections 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc15fe20 0x7ff9fc0b4ce0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9d8078a90 0x7ff9d807af50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9fc15f9e0 0x7ff9fc0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 --2- 172.21.15.31:0/3028171938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9fc15bd70 0x7ff9fc159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 >> 172.21.15.31:0/3028171938 conn(0x7ff9fc0cc910 msgr2=0x7ff9fc0ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 shutdown_connections 2024-09-13T08:40:09.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:09.997+0000 7ffa01200640 1 -- 172.21.15.31:0/3028171938 wait complete. 2024-09-13T08:40:10.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:40:10.194 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":79.932666166999994,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.290749346594119,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:40:10.194 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.29074934659412 }, "state": "open", "uptime": 79.932666167 } ] 2024-09-13T08:40:10.194 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:40:15.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.367+0000 7fd527200640 1 -- 172.21.15.31:0/716372354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 msgr2=0x7fd5200d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.367+0000 7fd527200640 1 --2- 172.21.15.31:0/716372354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200d0c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd514005fd0 tx=0x7fd51402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 -- 172.21.15.31:0/716372354 shutdown_connections 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 --2- 172.21.15.31:0/716372354 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200d0240 0x7fd5200d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 --2- 172.21.15.31:0/716372354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200d0c80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 -- 172.21.15.31:0/716372354 >> 172.21.15.31:0/716372354 conn(0x7fd5200d3e90 msgr2=0x7fd5200d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 -- 172.21.15.31:0/716372354 shutdown_connections 2024-09-13T08:40:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.368+0000 7fd527200640 1 -- 172.21.15.31:0/716372354 wait complete. 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.369+0000 7fd527200640 1 Processor -- start 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.369+0000 7fd527200640 1 -- start start 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200b2f80 0x7fd5200ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5200ca7e0 0x7fd5201f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200cf5d0 con 0x7fd5200d08a0 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200b46f0 con 0x7fd5200ca7e0 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.370+0000 7fd527200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5200d24a0 con 0x7fd5200b2f80 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40882/0 (socket says 172.21.15.31:40882) 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 -- 172.21.15.31:0/16157619 learned_addr learned my addr 172.21.15.31:0/16157619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525e00640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200b2f80 0x7fd5200ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd526800640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5200ca7e0 0x7fd5201f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 -- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200b2f80 msgr2=0x7fd5200ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200b2f80 0x7fd5200ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 -- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5200ca7e0 msgr2=0x7fd5201f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5200ca7e0 0x7fd5201f7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 -- 172.21.15.31:0/16157619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5200cd3e0 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.371+0000 7fd525400640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd51000d770 tx=0x7fd51000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.372+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510017070 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.372+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd510009d20 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.372+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5100106c0 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.372+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5200cd700 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.372+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5200cdf40 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.373+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5200d29d0 con 0x7fd5200d08a0 2024-09-13T08:40:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.374+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7fd51001a070 con 0x7fd5200d08a0 2024-09-13T08:40:15.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.375+0000 7fd51a000640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 0x7fd50407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:15.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.375+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fd51009b100 con 0x7fd5200d08a0 2024-09-13T08:40:15.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.375+0000 7fd525e00640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 0x7fd50407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:15.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.376+0000 7fd525e00640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 0x7fd50407af50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd514033000 tx=0x7fd514004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:15.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.378+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd5200cd700 con 0x7fd5200d08a0 2024-09-13T08:40:15.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.612+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5200ceef0 con 0x7fd5200d08a0 2024-09-13T08:40:15.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.613+0000 7fd51a000640 1 -- 172.21.15.31:0/16157619 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fd510062b80 con 0x7fd5200d08a0 2024-09-13T08:40:15.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:15.614 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:15.614 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:15.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 msgr2=0x7fd50407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 0x7fd50407af50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd514033000 tx=0x7fd514004730 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 msgr2=0x7fd5200ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd51000d770 tx=0x7fd51000dc30 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 shutdown_connections 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5200ca7e0 0x7fd5201f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5200d08a0 0x7fd5200ca2a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd504078a90 0x7fd50407af50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 --2- 172.21.15.31:0/16157619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5200b2f80 0x7fd5200ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.615+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 >> 172.21.15.31:0/16157619 conn(0x7fd5200d3e90 msgr2=0x7fd52006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.616+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 shutdown_connections 2024-09-13T08:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:15.616+0000 7fd527200640 1 -- 172.21.15.31:0/16157619 wait complete. 2024-09-13T08:40:15.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:40:15.811 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":3411920279}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":5,"uptime":85.551572321999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.054825720614136,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5335 172.21.15.45:0/3411920279","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:40:15.811 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 3411920279, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 172.21.15.45:0/3411920279", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 5, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.054825720614136 }, "state": "open", "uptime": 85.551572322 } ] 2024-09-13T08:40:15.812 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-13T08:40:20.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:20.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 -- 172.21.15.31:0/1036756953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 msgr2=0x7f4dd80d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1036756953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80d0a80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4dd806a4f0 tx=0x7f4dc402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 -- 172.21.15.31:0/1036756953 shutdown_connections 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1036756953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80d0a80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1036756953 >> v2:172.21.15.45:3300/0 conn(0x7f4dd815e310 0x7f4dd80cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 -- 172.21.15.31:0/1036756953 >> 172.21.15.31:0/1036756953 conn(0x7f4dd80d4070 msgr2=0x7f4dd80d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 -- 172.21.15.31:0/1036756953 shutdown_connections 2024-09-13T08:40:20.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.985+0000 7f4ddd600640 1 -- 172.21.15.31:0/1036756953 wait complete. 2024-09-13T08:40:20.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.987+0000 7f4ddd600640 1 Processor -- start 2024-09-13T08:40:20.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.987+0000 7f4ddd600640 1 -- start start 2024-09-13T08:40:20.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:20.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80d4c00 0x7f4dd80a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 0x7f4dd80a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd80d2800 con 0x7f4dd80b9d30 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd80d13c0 con 0x7f4dd80d4c00 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4ddd600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd80d2980 con 0x7f4dd815e310 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4dd6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4dd6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40890/0 (socket says 172.21.15.31:40890) 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4dd6a00640 1 -- 172.21.15.31:0/1999374730 learned_addr learned my addr 172.21.15.31:0/1999374730 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.988+0000 7f4dd6000640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80d4c00 0x7f4dd80a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd7400640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 0x7f4dd80a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 -- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 msgr2=0x7f4dd80a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 0x7f4dd80a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:20.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 -- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80d4c00 msgr2=0x7f4dd80a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:20.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80d4c00 0x7f4dd80a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:20.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 -- 172.21.15.31:0/1999374730 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd80cc190 con 0x7f4dd80b9d30 2024-09-13T08:40:20.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd7400640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 0x7f4dd80a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dd6a00640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e7c0 tx=0x7f4dc800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800ccc0 con 0x7f4dd80b9d30 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4dc8010430 con 0x7f4dd80b9d30 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.989+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8010840 con 0x7f4dd80b9d30 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.990+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd80cc430 con 0x7f4dd80b9d30 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.990+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd80cc9f0 con 0x7f4dd80b9d30 2024-09-13T08:40:20.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.991+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd80d2e30 con 0x7f4dd80b9d30 2024-09-13T08:40:20.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.992+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 101491+0+0 (secure 0 0 0) 0x7f4dc80109e0 con 0x7f4dd80b9d30 2024-09-13T08:40:20.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.993+0000 7f4dcea00640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 0x7f4db807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:20.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.993+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f4dc809b960 con 0x7f4dd80b9d30 2024-09-13T08:40:20.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.994+0000 7f4dd6000640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 0x7f4db807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:20.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.995+0000 7f4dd6000640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 0x7f4db807af50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4033000 tx=0x7f4dc4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:20.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:20.998+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4dc80633e0 con 0x7f4dd80b9d30 2024-09-13T08:40:21.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.231+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dd8161a60 con 0x7f4dd80b9d30 2024-09-13T08:40:21.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.231+0000 7f4dcea00640 1 -- 172.21.15.31:0/1999374730 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f4dc8067e10 con 0x7f4dd80b9d30 2024-09-13T08:40:21.233 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:21.233 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:21.233 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:21.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 msgr2=0x7f4db807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 0x7f4db807af50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4033000 tx=0x7f4dc4004730 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 msgr2=0x7f4dd80cd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e7c0 tx=0x7f4dc800ec80 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 shutdown_connections 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4db8078a90 0x7f4db807af50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4dd815e310 0x7f4dd80a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80d4c00 0x7f4dd80a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 --2- 172.21.15.31:0/1999374730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4dd80b9d30 0x7f4dd80cd4d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.233+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 >> 172.21.15.31:0/1999374730 conn(0x7f4dd80d4070 msgr2=0x7f4dd80b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.234+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 shutdown_connections 2024-09-13T08:40:21.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.234+0000 7f4ddd600640 1 -- 172.21.15.31:0/1999374730 wait complete. 2024-09-13T08:40:21.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:40:21.432 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-09-13T08:40:21.432 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-13T08:40:21.432 DEBUG:tasks.ceph_test_case:wait_until_true: success in 80s and 0 retries 2024-09-13T08:40:21.432 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:40:21.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:40:21.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:40:21.624 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:40:21.625 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:40:21.625 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:40:21.625 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:21.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:40:21.661 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:40:21.662 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:40:21.662 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:21.699 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:40:21.699 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:40:21.700 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:40:21.700 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:21.736 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:40:21.737 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:40:21.737 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:40:21.737 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:21.777 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:40:21.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:21.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 -- 172.21.15.31:0/2055364975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f00746d0 msgr2=0x7fa1f00b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:21.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 --2- 172.21.15.31:0/2055364975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f00746d0 0x7fa1f00b1910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005fd0 tx=0x7fa1e802a810 comp rx=0 tx=0).stop 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 -- 172.21.15.31:0/2055364975 shutdown_connections 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 --2- 172.21.15.31:0/2055364975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f00746d0 0x7fa1f00b1910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 --2- 172.21.15.31:0/2055364975 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0152f10 0x7fa1f00cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 -- 172.21.15.31:0/2055364975 >> 172.21.15.31:0/2055364975 conn(0x7fa1f00cc910 msgr2=0x7fa1f00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 -- 172.21.15.31:0/2055364975 shutdown_connections 2024-09-13T08:40:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.937+0000 7fa1f7800640 1 -- 172.21.15.31:0/2055364975 wait complete. 2024-09-13T08:40:21.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.939+0000 7fa1f7800640 1 Processor -- start 2024-09-13T08:40:21.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.939+0000 7fa1f7800640 1 -- start start 2024-09-13T08:40:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 0x7fa1f0155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f0152f10 0x7fa1f00b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f00cc640 con 0x7fa1f00746d0 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f00b1e50 con 0x7fa1f0152f10 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f7800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f00b1fd0 con 0x7fa1f0072850 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f4e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 0x7fa1f0155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40892/0 (socket says 172.21.15.31:40892) 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1efe00640 1 -- 172.21.15.31:0/3499602665 learned_addr learned my addr 172.21.15.31:0/3499602665 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.940+0000 7fa1f5800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f0152f10 0x7fa1f00b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 -- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f0152f10 msgr2=0x7fa1f00b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f0152f10 0x7fa1f00b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 -- 172.21.15.31:0/3499602665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 msgr2=0x7fa1f01522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 -- 172.21.15.31:0/3499602665 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f006e8f0 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1efe00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1f4e00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 0x7fa1f0155e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400e9c0 tx=0x7fa1e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.941+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400ccc0 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.942+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e400ce60 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.942+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4004220 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.942+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f0070220 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.942+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f006ebe0 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.943+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f006f300 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.944+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fa1e40035f0 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.945+0000 7fa1ecc00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 0x7fa1d007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.945+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fa1e409b6d0 con 0x7fa1f0072850 2024-09-13T08:40:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.945+0000 7fa1efe00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 0x7fa1d007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.946+0000 7fa1efe00640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 0x7fa1d007aec0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa1f009e130 tx=0x7fa1dc009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:21.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:21.950+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa1e40631b0 con 0x7fa1f0072850 2024-09-13T08:40:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.181+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1f00b1910 con 0x7fa1f0072850 2024-09-13T08:40:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.182+0000 7fa1ecc00640 1 -- 172.21.15.31:0/3499602665 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fa1e4067be0 con 0x7fa1f0072850 2024-09-13T08:40:22.182 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:22.183 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 msgr2=0x7fa1d007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 0x7fa1d007aec0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa1f009e130 tx=0x7fa1dc009040 comp rx=0 tx=0).stop 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 msgr2=0x7fa1f0155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 0x7fa1f0155e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400e9c0 tx=0x7fa1e400ee80 comp rx=0 tx=0).stop 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 shutdown_connections 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f0152f10 0x7fa1f00b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00746d0 0x7fa1f01522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa1d0078a00 0x7fa1d007aec0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 --2- 172.21.15.31:0/3499602665 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0072850 0x7fa1f0155e50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 >> 172.21.15.31:0/3499602665 conn(0x7fa1f00cc910 msgr2=0x7fa1f00ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 shutdown_connections 2024-09-13T08:40:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.184+0000 7fa1f7800640 1 -- 172.21.15.31:0/3499602665 wait complete. 2024-09-13T08:40:22.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:22.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 -- 172.21.15.31:0/1230586257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 msgr2=0x7f318415f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 --2- 172.21.15.31:0/1230586257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f318415f1e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f316c005fd0 tx=0x7f316c02a4b0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 -- 172.21.15.31:0/1230586257 shutdown_connections 2024-09-13T08:40:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 --2- 172.21.15.31:0/1230586257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f318415f1e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 --2- 172.21.15.31:0/1230586257 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f318415e3e0 0x7f318415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 -- 172.21.15.31:0/1230586257 >> 172.21.15.31:0/1230586257 conn(0x7f31840cda20 msgr2=0x7f31840cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 -- 172.21.15.31:0/1230586257 shutdown_connections 2024-09-13T08:40:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.362+0000 7f3189e00640 1 -- 172.21.15.31:0/1230586257 wait complete. 2024-09-13T08:40:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.365+0000 7f3189e00640 1 Processor -- start 2024-09-13T08:40:22.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.365+0000 7f3189e00640 1 -- start start 2024-09-13T08:40:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31840d0040 0x7f31840cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318415e3e0 0x7f31840b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31840b0660 con 0x7f318415e7c0 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31840b04e0 con 0x7f318415e3e0 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3189e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f31840b07e0 con 0x7f31840d0040 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.366+0000 7f3183400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40914/0 (socket says 172.21.15.31:40914) 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 -- 172.21.15.31:0/3598394534 learned_addr learned my addr 172.21.15.31:0/3598394534 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f317a000640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318415e3e0 0x7f31840b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3182a00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31840d0040 0x7f31840cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 -- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31840d0040 msgr2=0x7f31840cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31840d0040 0x7f31840cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 -- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318415e3e0 msgr2=0x7f31840b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318415e3e0 0x7f31840b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 -- 172.21.15.31:0/3598394534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3184152080 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f3183400640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f317c00c800 tx=0x7f317c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.367+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c003da0 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.368+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f317c005070 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.368+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31840b5200 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.368+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c005500 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.368+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31840b57c0 con 0x7f318415e7c0 2024-09-13T08:40:22.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.369+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31840ceea0 con 0x7f318415e7c0 2024-09-13T08:40:22.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.370+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f317c0056a0 con 0x7f318415e7c0 2024-09-13T08:40:22.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.371+0000 7f317be00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 0x7f316007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.371+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f317c09aaf0 con 0x7f318415e7c0 2024-09-13T08:40:22.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.371+0000 7f3182a00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 0x7f316007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.372+0000 7f3182a00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 0x7f316007aef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3174005fd0 tx=0x7f317400b390 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:22.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.375+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f317c062720 con 0x7f318415e7c0 2024-09-13T08:40:22.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.604+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3184074b30 con 0x7f318415e7c0 2024-09-13T08:40:22.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.605+0000 7f317be00640 1 -- 172.21.15.31:0/3598394534 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f317c067150 con 0x7f318415e7c0 2024-09-13T08:40:22.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:22.606 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:22.607 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:22.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 msgr2=0x7f316007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 0x7f316007aef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3174005fd0 tx=0x7f317400b390 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 msgr2=0x7f3184151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f317c00c800 tx=0x7f317c00ccc0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 shutdown_connections 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.607+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f318415e7c0 0x7f3184151b40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318415e3e0 0x7f31840b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3160078a30 0x7f316007aef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 --2- 172.21.15.31:0/3598394534 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31840d0040 0x7f31840cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 >> 172.21.15.31:0/3598394534 conn(0x7f31840cda20 msgr2=0x7f31840aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 shutdown_connections 2024-09-13T08:40:22.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.608+0000 7f3189e00640 1 -- 172.21.15.31:0/3598394534 wait complete. 2024-09-13T08:40:22.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.785+0000 7f8cbe800640 1 -- 172.21.15.31:0/3355106819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c9c003c10 msgr2=0x7f8c9c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.785+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3355106819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c9c003c10 0x7f8c9c004010 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8001830 tx=0x7f8c9c005830 comp rx=0 tx=0).stop 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.785+0000 7f8cb7e00640 1 -- 172.21.15.31:0/3355106819 reap_dead start 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.786+0000 7f8cbe800640 1 -- 172.21.15.31:0/3355106819 shutdown_connections 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.786+0000 7f8cbe800640 1 -- 172.21.15.31:0/3355106819 >> 172.21.15.31:0/3355106819 conn(0x7f8cb80cc910 msgr2=0x7f8cb80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.786+0000 7f8cbe800640 1 -- 172.21.15.31:0/3355106819 shutdown_connections 2024-09-13T08:40:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.786+0000 7f8cbe800640 1 -- 172.21.15.31:0/3355106819 wait complete. 2024-09-13T08:40:22.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.787+0000 7f8cbe800640 1 Processor -- start 2024-09-13T08:40:22.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.787+0000 7f8cbe800640 1 -- start start 2024-09-13T08:40:22.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9c003c10 0x7f8cb80b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cb815a440 0x7f8cb80c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb80742b0 con 0x7f8cb80b1a00 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8074130 con 0x7f8c9c003c10 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.788+0000 7f8cbe800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb80cbb40 con 0x7f8cb815a440 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40938/0 (socket says 172.21.15.31:40938) 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 -- 172.21.15.31:0/3559646319 learned_addr learned my addr 172.21.15.31:0/3559646319 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8cb7400640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9c003c10 0x7f8cb80b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8cb7e00640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cb815a440 0x7f8cb80c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 -- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cb815a440 msgr2=0x7f8cb80c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cb815a440 0x7f8cb80c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 -- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9c003c10 msgr2=0x7f8cb80b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9c003c10 0x7f8cb80b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 -- 172.21.15.31:0/3559646319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb80c43a0 con 0x7f8cb80b1a00 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.789+0000 7f8caea00640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0011a20 tx=0x7f8ca0011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.790+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000fd30 con 0x7f8cb80b1a00 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.790+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ca0004030 con 0x7f8cb80b1a00 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.790+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0004480 con 0x7f8cb80b1a00 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.790+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb80b26a0 con 0x7f8cb80b1a00 2024-09-13T08:40:22.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.790+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb80b0730 con 0x7f8cb80b1a00 2024-09-13T08:40:22.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.791+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb815f480 con 0x7f8cb80b1a00 2024-09-13T08:40:22.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.792+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f8ca00035f0 con 0x7f8cb80b1a00 2024-09-13T08:40:22.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.793+0000 7f8cafe00640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 0x7f8c9407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:22.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.793+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f8ca009ba20 con 0x7f8cb80b1a00 2024-09-13T08:40:22.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.793+0000 7f8cb7400640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 0x7f8c9407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:22.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.794+0000 7f8cb7400640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 0x7f8c9407aec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8032000 tx=0x7f8ca8001360 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:22.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.797+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8ca0063610 con 0x7f8cb80b1a00 2024-09-13T08:40:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.995+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8cb80b5e10 con 0x7f8cb80b1a00 2024-09-13T08:40:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.996+0000 7f8cafe00640 1 -- 172.21.15.31:0/3559646319 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16912 (secure 0 0 0) 0x7f8ca0068040 con 0x7f8cb80b1a00 2024-09-13T08:40:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:22.998 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:51.136435+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:38:34.198287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T08:38:35.374739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.6","mappings":[{"from":5,"to":7}]},{"pgid":"7.28","mappings":[{"from":5,"to":7}]},{"pgid":"7.33","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 msgr2=0x7f8c9407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 0x7f8c9407aec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8032000 tx=0x7f8ca8001360 comp rx=0 tx=0).stop 2024-09-13T08:40:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 msgr2=0x7f8cb80b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0011a20 tx=0x7f8ca0011ee0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 shutdown_connections 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8c94078a00 0x7f8c9407aec0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cb815a440 0x7f8cb80c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb80b1a00 0x7f8cb80b0f30 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 --2- 172.21.15.31:0/3559646319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9c003c10 0x7f8cb80b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 >> 172.21.15.31:0/3559646319 conn(0x7f8cb80cc910 msgr2=0x7f8cb815c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 shutdown_connections 2024-09-13T08:40:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:22.999+0000 7f8cbe800640 1 -- 172.21.15.31:0/3559646319 wait complete. 2024-09-13T08:40:23.016 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:40:23.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 -- 172.21.15.31:0/3310023668 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd4151d80 msgr2=0x7f8cd415b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/3310023668 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd4151d80 0x7f8cd415b390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc005fd0 tx=0x7f8ccc02aa90 comp rx=0 tx=0).stop 2024-09-13T08:40:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 -- 172.21.15.31:0/3310023668 shutdown_connections 2024-09-13T08:40:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/3310023668 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd4151d80 0x7f8cd415b390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/3310023668 >> v2:172.21.15.45:3300/0 conn(0x7f8cd40ced20 0x7f8cd40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 -- 172.21.15.31:0/3310023668 >> 172.21.15.31:0/3310023668 conn(0x7f8cd406cb00 msgr2=0x7f8cd406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 -- 172.21.15.31:0/3310023668 shutdown_connections 2024-09-13T08:40:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.178+0000 7f8cd3e00640 1 -- 172.21.15.31:0/3310023668 wait complete. 2024-09-13T08:40:23.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.180+0000 7f8cd3e00640 1 Processor -- start 2024-09-13T08:40:23.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.180+0000 7f8cd3e00640 1 -- start start 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd40ced20 0x7f8cd4154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd4151d80 0x7f8cd415a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd415bc60 con 0x7f8cd40cf100 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd415bae0 con 0x7f8cd4151d80 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8cd415bde0 con 0x7f8cd40ced20 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40946/0 (socket says 172.21.15.31:40946) 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 -- 172.21.15.31:0/1858668001 learned_addr learned my addr 172.21.15.31:0/1858668001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd2a00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd40ced20 0x7f8cd4154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 -- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd40ced20 msgr2=0x7f8cd4154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cd3400640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd4151d80 0x7f8cd415a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd40ced20 0x7f8cd4154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 -- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd4151d80 msgr2=0x7f8cd415a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd4151d80 0x7f8cd415a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.181+0000 7f8cca000640 1 -- 172.21.15.31:0/1858668001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd40c9b90 con 0x7f8cd40cf100 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8cca000640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00c450 tx=0x7f8cbc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0052d0 con 0x7f8cd40cf100 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cbc005470 con 0x7f8cd40cf100 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc012870 con 0x7f8cd40cf100 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd41f4b50 con 0x7f8cd40cf100 2024-09-13T08:40:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.182+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd41547e0 con 0x7f8cd40cf100 2024-09-13T08:40:23.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.183+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd415ef10 con 0x7f8cd40cf100 2024-09-13T08:40:23.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.184+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f8cbc0035f0 con 0x7f8cd40cf100 2024-09-13T08:40:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.186+0000 7f8ccbe00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 0x7f8cb007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.186+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f8cbc09a9e0 con 0x7f8cd40cf100 2024-09-13T08:40:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.186+0000 7f8cd2a00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 0x7f8cb007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.187+0000 7f8cd2a00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 0x7f8cb007aef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b60 tx=0x7f8cc4005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.190+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8cd415ef10 con 0x7f8cd40cf100 2024-09-13T08:40:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.422+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cd40c92b0 con 0x7f8cd40cf100 2024-09-13T08:40:23.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.423+0000 7f8ccbe00640 1 -- 172.21.15.31:0/1858668001 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f8cd40c92b0 con 0x7f8cd40cf100 2024-09-13T08:40:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:23.424 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:23.424 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:23.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 msgr2=0x7f8cb007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 0x7f8cb007aef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b60 tx=0x7f8cc4005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:23.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 msgr2=0x7f8cd4159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00c450 tx=0x7f8cbc00c910 comp rx=0 tx=0).stop 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 shutdown_connections 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd4151d80 0x7f8cd415a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cd40cf100 0x7f8cd4159ca0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8cb0078a30 0x7f8cb007aef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 --2- 172.21.15.31:0/1858668001 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8cd40ced20 0x7f8cd4154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 >> 172.21.15.31:0/1858668001 conn(0x7f8cd406cb00 msgr2=0x7f8cd415e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 shutdown_connections 2024-09-13T08:40:23.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.425+0000 7f8cd3e00640 1 -- 172.21.15.31:0/1858668001 wait complete. 2024-09-13T08:40:23.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 -- 172.21.15.31:0/2471730619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c15e440 msgr2=0x7fe14c0b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 --2- 172.21.15.31:0/2471730619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c15e440 0x7fe14c0b1b30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0c40b0 tx=0x7fe13c02a4a0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 -- 172.21.15.31:0/2471730619 shutdown_connections 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 --2- 172.21.15.31:0/2471730619 >> v2:172.21.15.45:3300/0 conn(0x7fe14c15e820 0x7fe14c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 --2- 172.21.15.31:0/2471730619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c15e440 0x7fe14c0b1b30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 -- 172.21.15.31:0/2471730619 >> 172.21.15.31:0/2471730619 conn(0x7fe14c074260 msgr2=0x7fe14c074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 -- 172.21.15.31:0/2471730619 shutdown_connections 2024-09-13T08:40:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.604+0000 7fe151400640 1 -- 172.21.15.31:0/2471730619 wait complete. 2024-09-13T08:40:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.605+0000 7fe151400640 1 Processor -- start 2024-09-13T08:40:23.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.606+0000 7fe151400640 1 -- start start 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.606+0000 7fe151400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.606+0000 7fe151400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 0x7fe14c09cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe151400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe14c15e820 0x7fe14c1f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe151400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c0ceb70 con 0x7fe14c0b1600 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe151400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c0ce9f0 con 0x7fe14c15e440 2024-09-13T08:40:23.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe151400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe14c0cecf0 con 0x7fe14c15e820 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40972/0 (socket says 172.21.15.31:40972) 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 -- 172.21.15.31:0/413971929 learned_addr learned my addr 172.21.15.31:0/413971929 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe143e00640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 0x7fe14c09cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14b200640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe14c15e820 0x7fe14c1f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 -- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe14c15e820 msgr2=0x7fe14c1f7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe14c15e820 0x7fe14c1f7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 -- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 msgr2=0x7fe14c09cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 0x7fe14c09cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe14a800640 1 -- 172.21.15.31:0/413971929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14c09d560 con 0x7fe14c0b1600 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.607+0000 7fe143e00640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 0x7fe14c09cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:23.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe14a800640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009780 tx=0x7fe13c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:23.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c040070 con 0x7fe14c0b1600 2024-09-13T08:40:23.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe13c005310 con 0x7fe14c0b1600 2024-09-13T08:40:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe14c09d840 con 0x7fe14c0b1600 2024-09-13T08:40:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c0047a0 con 0x7fe14c0b1600 2024-09-13T08:40:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.608+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe14c0b6170 con 0x7fe14c0b1600 2024-09-13T08:40:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.609+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe14c09dc30 con 0x7fe14c0b1600 2024-09-13T08:40:23.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.610+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fe13c039040 con 0x7fe14c0b1600 2024-09-13T08:40:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.612+0000 7fe142a00640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 0x7fe12807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.612+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fe13c0bc070 con 0x7fe14c0b1600 2024-09-13T08:40:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.612+0000 7fe143e00640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 0x7fe12807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.613+0000 7fe143e00640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 0x7fe12807aec0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0d0a90 tx=0x7fe134005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.616+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe13c0842b0 con 0x7fe14c0b1600 2024-09-13T08:40:23.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.852+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe14c09da10 con 0x7fe14c0b1600 2024-09-13T08:40:23.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.853+0000 7fe142a00640 1 -- 172.21.15.31:0/413971929 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fe13c083bb0 con 0x7fe14c0b1600 2024-09-13T08:40:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:23.854 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"btime":"2024-09-13T08:38:47:114948+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:38:36.443046+0000","modified":"2024-09-13T08:38:39.419176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1342811284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1342811284},{"type":"v1","addr":"172.21.15.31:6834","nonce":1342811284}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-13T08:40:23.854 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 msgr2=0x7fe12807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 0x7fe12807aec0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0d0a90 tx=0x7fe134005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 msgr2=0x7fe14c09c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009780 tx=0x7fe13c032c10 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 shutdown_connections 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe128078a00 0x7fe12807aec0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe14c15e820 0x7fe14c1f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe14c15e440 0x7fe14c09cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 --2- 172.21.15.31:0/413971929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe14c0b1600 0x7fe14c09c9a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:23.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 >> 172.21.15.31:0/413971929 conn(0x7fe14c074260 msgr2=0x7fe14c06fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:23.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 shutdown_connections 2024-09-13T08:40:23.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:23.856+0000 7fe151400640 1 -- 172.21.15.31:0/413971929 wait complete. 2024-09-13T08:40:23.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 -- 172.21.15.31:0/2398961132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 msgr2=0x7f4ab406f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 --2- 172.21.15.31:0/2398961132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab406f9b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009830 tx=0x7f4aa402a5b0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 -- 172.21.15.31:0/2398961132 shutdown_connections 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 --2- 172.21.15.31:0/2398961132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab406f9b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 -- 172.21.15.31:0/2398961132 >> 172.21.15.31:0/2398961132 conn(0x7f4ab406cb00 msgr2=0x7f4ab406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 -- 172.21.15.31:0/2398961132 shutdown_connections 2024-09-13T08:40:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.033+0000 7f4ab3200640 1 -- 172.21.15.31:0/2398961132 wait complete. 2024-09-13T08:40:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.035+0000 7f4ab3200640 1 Processor -- start 2024-09-13T08:40:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.035+0000 7f4ab3200640 1 -- start start 2024-09-13T08:40:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab4153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ab40ccb00 0x7f4ab41f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab4070320 con 0x7f4ab40b52d0 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab40701a0 con 0x7f4ab40cc720 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab40704a0 con 0x7f4ab40ccb00 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40980/0 (socket says 172.21.15.31:40980) 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab1e00640 1 -- 172.21.15.31:0/3316981135 learned_addr learned my addr 172.21.15.31:0/3316981135 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4aa9400640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab4153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.036+0000 7f4ab2800640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ab40ccb00 0x7f4ab41f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 -- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ab40ccb00 msgr2=0x7f4ab41f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ab40ccb00 0x7f4ab41f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 -- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 msgr2=0x7f4ab4153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab4153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 -- 172.21.15.31:0/3316981135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab41f5670 con 0x7f4ab40b52d0 2024-09-13T08:40:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4aa9400640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab1e00640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4aa402aba0 tx=0x7f4aa4032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4040070 con 0x7f4ab40b52d0 2024-09-13T08:40:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4aa4005310 con 0x7f4ab40b52d0 2024-09-13T08:40:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40047a0 con 0x7f4ab40b52d0 2024-09-13T08:40:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab40c7860 con 0x7f4ab40b52d0 2024-09-13T08:40:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.037+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab4153500 con 0x7f4ab40b52d0 2024-09-13T08:40:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.039+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab415f1c0 con 0x7f4ab40b52d0 2024-09-13T08:40:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.040+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f4aa4039040 con 0x7f4ab40b52d0 2024-09-13T08:40:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.040+0000 7f4aaaa00640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 0x7f4a9007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.041+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7f4aa40bbe60 con 0x7f4ab40b52d0 2024-09-13T08:40:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.041+0000 7f4aa9400640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 0x7f4a9007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.042+0000 7f4aa9400640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 0x7f4a9007aec0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007fd0 tx=0x7f4a9c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.045+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4ab41f5670 con 0x7f4ab40b52d0 2024-09-13T08:40:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.239+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ab415b400 con 0x7f4ab40b52d0 2024-09-13T08:40:24.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.240+0000 7f4aaaa00640 1 -- 172.21.15.31:0/3316981135 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16912 (secure 0 0 0) 0x7f4aa4084010 con 0x7f4ab40b52d0 2024-09-13T08:40:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:24.242 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:38:51.136435+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:38:34.198287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T08:38:35.374739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.6","mappings":[{"from":5,"to":7}]},{"pgid":"7.28","mappings":[{"from":5,"to":7}]},{"pgid":"7.33","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.242+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 msgr2=0x7f4a9007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.242+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 0x7f4a9007aec0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007fd0 tx=0x7f4a9c009040 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 msgr2=0x7f4ab4152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4aa402aba0 tx=0x7f4aa4032b90 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 shutdown_connections 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4a90078a00 0x7f4a9007aec0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ab40ccb00 0x7f4ab41f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab40cc720 0x7f4ab4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 --2- 172.21.15.31:0/3316981135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40b52d0 0x7f4ab4152180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 >> 172.21.15.31:0/3316981135 conn(0x7f4ab406cb00 msgr2=0x7f4ab415bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 shutdown_connections 2024-09-13T08:40:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.243+0000 7f4ab3200640 1 -- 172.21.15.31:0/3316981135 wait complete. 2024-09-13T08:40:24.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:40:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 -- 172.21.15.31:0/608385895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 msgr2=0x7fe34c0d0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 --2- 172.21.15.31:0/608385895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0d0ea0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0c79d0 tx=0x7fe33802a540 comp rx=0 tx=0).stop 2024-09-13T08:40:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 -- 172.21.15.31:0/608385895 shutdown_connections 2024-09-13T08:40:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 --2- 172.21.15.31:0/608385895 >> v2:172.21.15.45:3300/0 conn(0x7fe34c0cea60 0x7fe34c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 --2- 172.21.15.31:0/608385895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0d0ea0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 -- 172.21.15.31:0/608385895 >> 172.21.15.31:0/608385895 conn(0x7fe34c0d3f50 msgr2=0x7fe34c0d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 -- 172.21.15.31:0/608385895 shutdown_connections 2024-09-13T08:40:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.421+0000 7fe351800640 1 -- 172.21.15.31:0/608385895 wait complete. 2024-09-13T08:40:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.422+0000 7fe351800640 1 Processor -- start 2024-09-13T08:40:24.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.423+0000 7fe351800640 1 -- start start 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c0cea60 0x7fe34c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe34c15e160 0x7fe34c1f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c0cf430 con 0x7fe34c0c9350 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c0d15f0 con 0x7fe34c0cea60 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe351800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe34c0cf5b0 con 0x7fe34c15e160 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41010/0 (socket says 172.21.15.31:41010) 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 -- 172.21.15.31:0/1357025299 learned_addr learned my addr 172.21.15.31:0/1357025299 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34b400640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c0cea60 0x7fe34c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe350e00640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe34c15e160 0x7fe34c1f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 -- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe34c15e160 msgr2=0x7fe34c1f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe34c15e160 0x7fe34c1f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 -- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c0cea60 msgr2=0x7fe34c14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c0cea60 0x7fe34c14c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.424+0000 7fe34be00640 1 -- 172.21.15.31:0/1357025299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c158450 con 0x7fe34c0c9350 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe34be00640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00c3a0 tx=0x7fe33c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c011790 con 0x7fe34c0c9350 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe33c017070 con 0x7fe34c0c9350 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c011cc0 con 0x7fe34c0c9350 2024-09-13T08:40:24.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c158730 con 0x7fe34c0c9350 2024-09-13T08:40:24.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.425+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c15b3d0 con 0x7fe34c0c9350 2024-09-13T08:40:24.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.426+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34c15cbc0 con 0x7fe34c0c9350 2024-09-13T08:40:24.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.427+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fe33c01c460 con 0x7fe34c0c9350 2024-09-13T08:40:24.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.428+0000 7fe343e00640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 0x7fe32c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.428+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5854+0+0 (secure 0 0 0) 0x7fe33c09ab90 con 0x7fe34c0c9350 2024-09-13T08:40:24.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.429+0000 7fe34b400640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 0x7fe32c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.429+0000 7fe34b400640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 0x7fe32c07aec0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe338034000 tx=0x7fe338004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.432+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe34c15cbc0 con 0x7fe34c0c9350 2024-09-13T08:40:24.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.662+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe34c0ceef0 con 0x7fe34c0c9350 2024-09-13T08:40:24.663 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:40:24.663+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:40:24.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.665+0000 7fe343e00640 1 -- 172.21.15.31:0/1357025299 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7fe34c15e540 con 0x7fe34c0c9350 2024-09-13T08:40:24.666 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:40:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 msgr2=0x7fe32c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 0x7fe32c07aec0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe338034000 tx=0x7fe338004730 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 msgr2=0x7fe34c0b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00c3a0 tx=0x7fe33c00c860 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 shutdown_connections 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.668+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe32c078a00 0x7fe32c07aec0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe34c15e160 0x7fe34c1f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c0cea60 0x7fe34c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 --2- 172.21.15.31:0/1357025299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c9350 0x7fe34c0b99b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 >> 172.21.15.31:0/1357025299 conn(0x7fe34c0d3f50 msgr2=0x7fe34c0d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 shutdown_connections 2024-09-13T08:40:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.669+0000 7fe351800640 1 -- 172.21.15.31:0/1357025299 wait complete. 2024-09-13T08:40:24.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:40:24.735 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 -- 172.21.15.31:0/3626022012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad8072560 msgr2=0x7fbad80b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 --2- 172.21.15.31:0/3626022012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad8072560 0x7fbad80b29a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbad806de70 tx=0x7fbac002a5f0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 -- 172.21.15.31:0/3626022012 shutdown_connections 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 --2- 172.21.15.31:0/3626022012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad8072560 0x7fbad80b29a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 --2- 172.21.15.31:0/3626022012 >> v2:172.21.15.45:3300/0 conn(0x7fbad815c040 0x7fbad80d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.850+0000 7fbade400640 1 -- 172.21.15.31:0/3626022012 >> 172.21.15.31:0/3626022012 conn(0x7fbad80cfb70 msgr2=0x7fbad80cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.851+0000 7fbade400640 1 -- 172.21.15.31:0/3626022012 shutdown_connections 2024-09-13T08:40:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.851+0000 7fbade400640 1 -- 172.21.15.31:0/3626022012 wait complete. 2024-09-13T08:40:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.852+0000 7fbade400640 1 Processor -- start 2024-09-13T08:40:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.852+0000 7fbade400640 1 -- start start 2024-09-13T08:40:24.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad8072560 0x7fbad80c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad80cb570 0x7fbad80c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbad806ff20 con 0x7fbad815c040 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbad806fda0 con 0x7fbad80cb570 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.853+0000 7fbade400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbad80700a0 con 0x7fbad8072560 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46208/0 (socket says 172.21.15.31:46208) 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 -- 172.21.15.31:0/3427982572 learned_addr learned my addr 172.21.15.31:0/3427982572 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad8072560 0x7fbad80c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbacea00640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad80cb570 0x7fbad80c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 -- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad8072560 msgr2=0x7fbad80c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad8072560 0x7fbad80c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 -- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad80cb570 msgr2=0x7fbad80c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad80cb570 0x7fbad80c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 -- 172.21.15.31:0/3427982572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad80c8a60 con 0x7fbad815c040 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.854+0000 7fbad7e00640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c7a0 tx=0x7fbad000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.855+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0003b70 con 0x7fbad815c040 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.855+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbad0003d30 con 0x7fbad815c040 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.855+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad000aab0 con 0x7fbad815c040 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.855+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad80c8d40 con 0x7fbad815c040 2024-09-13T08:40:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.855+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad80c9300 con 0x7fbad815c040 2024-09-13T08:40:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.856+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad814d890 con 0x7fbad815c040 2024-09-13T08:40:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.857+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fbad000ac50 con 0x7fbad815c040 2024-09-13T08:40:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.858+0000 7fbacfe00640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.858+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5940+0+0 (secure 0 0 0) 0x7fbad009acc0 con 0x7fbad815c040 2024-09-13T08:40:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.858+0000 7fbad7400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.859+0000 7fbad7400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005d70 tx=0x7fbac8005d00 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:24.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:24.862+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbad814d890 con 0x7fbad815c040 2024-09-13T08:40:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.099+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbad80cbbd0 con 0x7fbad815c040 2024-09-13T08:40:25.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.671+0000 7fbacfe00640 1 -- 172.21.15.31:0/3427982572 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fbad00a1080 con 0x7fbad815c040 2024-09-13T08:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 msgr2=0x7fbab407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005d70 tx=0x7fbac8005d00 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 msgr2=0x7fbad8150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c7a0 tx=0x7fbad000cc60 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 shutdown_connections 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad815c040 0x7fbad8150610 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad80cb570 0x7fbad80c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 --2- 172.21.15.31:0/3427982572 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad8072560 0x7fbad80c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.674+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 >> 172.21.15.31:0/3427982572 conn(0x7fbad80cfb70 msgr2=0x7fbad80ceeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:25.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.675+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 shutdown_connections 2024-09-13T08:40:25.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.675+0000 7fbade400640 1 -- 172.21.15.31:0/3427982572 wait complete. 2024-09-13T08:40:25.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.867+0000 7f0333e00640 1 -- 172.21.15.31:0/702932050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0318005340 msgr2=0x7f031800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.867+0000 7f0333e00640 1 --2- 172.21.15.31:0/702932050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0318005340 0x7f031800b9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f03340b54d0 tx=0x7f0318005830 comp rx=0 tx=0).stop 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.867+0000 7f0333400640 1 -- 172.21.15.31:0/702932050 reap_dead start 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.867+0000 7f0333e00640 1 -- 172.21.15.31:0/702932050 shutdown_connections 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.867+0000 7f0333e00640 1 -- 172.21.15.31:0/702932050 >> 172.21.15.31:0/702932050 conn(0x7f03340b1520 msgr2=0x7f03340b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.868+0000 7f0333e00640 1 -- 172.21.15.31:0/702932050 shutdown_connections 2024-09-13T08:40:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.868+0000 7f0333e00640 1 -- 172.21.15.31:0/702932050 wait complete. 2024-09-13T08:40:25.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.869+0000 7f0333e00640 1 Processor -- start 2024-09-13T08:40:25.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.869+0000 7f0333e00640 1 -- start start 2024-09-13T08:40:25.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0318005340 0x7f03341f5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f033415a030 0x7f03341e8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03340caad0 con 0x7f03340cf2c0 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03340ca950 con 0x7f033415a030 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.870+0000 7f0333e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03340cac50 con 0x7f0318005340 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46228/0 (socket says 172.21.15.31:46228) 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 -- 172.21.15.31:0/3457995346 learned_addr learned my addr 172.21.15.31:0/3457995346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f0332a00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0318005340 0x7f03341f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f0333400640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f033415a030 0x7f03341e8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 -- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0318005340 msgr2=0x7f03341f5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0318005340 0x7f03341f5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 -- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f033415a030 msgr2=0x7f03341e8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f033415a030 0x7f03341e8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 -- 172.21.15.31:0/3457995346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03341f4980 con 0x7f03340cf2c0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.871+0000 7f032a000640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f031c003d80 tx=0x7f031c001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.872+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c015070 con 0x7f03340cf2c0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.872+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f031c011840 con 0x7f03340cf2c0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.872+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c011c90 con 0x7f03340cf2c0 2024-09-13T08:40:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.872+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03340a96d0 con 0x7f03340cf2c0 2024-09-13T08:40:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.872+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03340a9c90 con 0x7f03340cf2c0 2024-09-13T08:40:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.873+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03341f47a0 con 0x7f03340cf2c0 2024-09-13T08:40:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.874+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f031c018070 con 0x7f03340cf2c0 2024-09-13T08:40:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.875+0000 7f032b400640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 0x7f031007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.875+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5940+0+0 (secure 0 0 0) 0x7f031c09b890 con 0x7f03340cf2c0 2024-09-13T08:40:25.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.876+0000 7f0332a00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 0x7f031007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:25.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.877+0000 7f0332a00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 0x7f031007aec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c00 tx=0x7f0324000fb0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:25.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:25.879+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f03341f47a0 con 0x7f03340cf2c0 2024-09-13T08:40:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.076+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f03340cb1e0 con 0x7f03340cf2c0 2024-09-13T08:40:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.673+0000 7f032b400640 1 -- 172.21.15.31:0/3457995346 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) ==== 181+0+0 (secure 0 0 0) 0x7f031c0634c0 con 0x7f03340cf2c0 2024-09-13T08:40:26.675 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:40:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.677+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 msgr2=0x7f031007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.677+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 0x7f031007aec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c00 tx=0x7f0324000fb0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.677+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 msgr2=0x7f0334159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.677+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f031c003d80 tx=0x7f031c001800 comp rx=0 tx=0).stop 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 shutdown_connections 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f033415a030 0x7f03341e8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cf2c0 0x7f0334159800 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0310078a00 0x7f031007aec0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 --2- 172.21.15.31:0/3457995346 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0318005340 0x7f03341f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 >> 172.21.15.31:0/3457995346 conn(0x7f03340b1520 msgr2=0x7f03340af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 shutdown_connections 2024-09-13T08:40:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.678+0000 7f0333e00640 1 -- 172.21.15.31:0/3457995346 wait complete. 2024-09-13T08:40:26.699 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.868+0000 7f6777a00640 1 -- 172.21.15.31:0/4034324709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f677015acb0 msgr2=0x7f67700cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.868+0000 7f6777a00640 1 --2- 172.21.15.31:0/4034324709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f677015acb0 0x7f67700cf4d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6770074de0 tx=0x7f676402a560 comp rx=0 tx=0).stop 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 -- 172.21.15.31:0/4034324709 shutdown_connections 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 --2- 172.21.15.31:0/4034324709 >> v2:172.21.15.45:3300/0 conn(0x7f67700741b0 0x7f67700cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 --2- 172.21.15.31:0/4034324709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f677015acb0 0x7f67700cf4d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 -- 172.21.15.31:0/4034324709 >> 172.21.15.31:0/4034324709 conn(0x7f67700cfbb0 msgr2=0x7f67700cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 -- 172.21.15.31:0/4034324709 shutdown_connections 2024-09-13T08:40:26.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.869+0000 7f6777a00640 1 -- 172.21.15.31:0/4034324709 wait complete. 2024-09-13T08:40:26.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.870+0000 7f6777a00640 1 Processor -- start 2024-09-13T08:40:26.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.870+0000 7f6777a00640 1 -- start start 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67700741b0 0x7f67701574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677015acb0 0x7f677015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f67700ce160 con 0x7f67700745f0 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67700cdfe0 con 0x7f677015acb0 2024-09-13T08:40:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.871+0000 7f6777a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f67700cb0f0 con 0x7f67700741b0 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46248/0 (socket says 172.21.15.31:46248) 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 -- 172.21.15.31:0/583407150 learned_addr learned my addr 172.21.15.31:0/583407150 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f6775000640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67700741b0 0x7f67701574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f6775a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677015acb0 0x7f677015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 -- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67700741b0 msgr2=0x7f67701574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67700741b0 0x7f67701574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 -- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677015acb0 msgr2=0x7f677015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677015acb0 0x7f677015d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 -- 172.21.15.31:0/583407150 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67701f8d90 con 0x7f67700745f0 2024-09-13T08:40:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.872+0000 7f676fe00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f675c00d770 tx=0x7f675c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:26.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.873+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c017070 con 0x7f67700745f0 2024-09-13T08:40:26.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.873+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f675c009d20 con 0x7f67700745f0 2024-09-13T08:40:26.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.873+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c005280 con 0x7f67700745f0 2024-09-13T08:40:26.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.873+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6770157a20 con 0x7f67700745f0 2024-09-13T08:40:26.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.873+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67701f84f0 con 0x7f67700745f0 2024-09-13T08:40:26.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.874+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67700cb5c0 con 0x7f67700745f0 2024-09-13T08:40:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.875+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f675c005420 con 0x7f67700745f0 2024-09-13T08:40:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.876+0000 7f676cc00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 0x7f675007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.876+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f675c09ace0 con 0x7f67700745f0 2024-09-13T08:40:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.876+0000 7f6775000640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 0x7f675007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:26.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.877+0000 7f6775000640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 0x7f675007aec0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6764033000 tx=0x7f6764004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:26.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:26.880+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f675c062ae0 con 0x7f67700745f0 2024-09-13T08:40:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.074+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f677015a7d0 con 0x7f67700745f0 2024-09-13T08:40:27.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.681+0000 7f676cc00640 1 -- 172.21.15.31:0/583407150 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) ==== 169+0+0 (secure 0 0 0) 0x7f675c067510 con 0x7f67700745f0 2024-09-13T08:40:27.682 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 msgr2=0x7f675007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 0x7f675007aec0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6764033000 tx=0x7f6764004730 comp rx=0 tx=0).stop 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 msgr2=0x7f677015d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f675c00d770 tx=0x7f675c00dc30 comp rx=0 tx=0).stop 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 shutdown_connections 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677015acb0 0x7f677015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67700745f0 0x7f677015d400 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6750078a00 0x7f675007aec0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 --2- 172.21.15.31:0/583407150 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f67700741b0 0x7f67701574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 >> 172.21.15.31:0/583407150 conn(0x7f67700cfbb0 msgr2=0x7f67700b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 shutdown_connections 2024-09-13T08:40:27.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.684+0000 7f6777a00640 1 -- 172.21.15.31:0/583407150 wait complete. 2024-09-13T08:40:27.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.871+0000 7fba3b600640 1 -- 172.21.15.31:0/3441639601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 msgr2=0x7fba18004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.871+0000 7fba3b600640 1 --2- 172.21.15.31:0/3441639601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba18004800 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009b50 tx=0x7fba18005830 comp rx=0 tx=0).stop 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.871+0000 7fba39600640 1 -- 172.21.15.31:0/3441639601 reap_dead start 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.872+0000 7fba3b600640 1 -- 172.21.15.31:0/3441639601 shutdown_connections 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.872+0000 7fba3b600640 1 -- 172.21.15.31:0/3441639601 >> 172.21.15.31:0/3441639601 conn(0x7fba340b18f0 msgr2=0x7fba340b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.872+0000 7fba3b600640 1 -- 172.21.15.31:0/3441639601 shutdown_connections 2024-09-13T08:40:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.872+0000 7fba3b600640 1 -- 172.21.15.31:0/3441639601 wait complete. 2024-09-13T08:40:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.873+0000 7fba3b600640 1 Processor -- start 2024-09-13T08:40:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.873+0000 7fba3b600640 1 -- start start 2024-09-13T08:40:27.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 0x7fba3409b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 0x7fba341ec250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba3406f710 con 0x7fba18004210 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba3406f590 con 0x7fba341535c0 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.874+0000 7fba3b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fba340b40c0 con 0x7fba3409cd50 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46256/0 (socket says 172.21.15.31:46256) 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 -- 172.21.15.31:0/562686684 learned_addr learned my addr 172.21.15.31:0/562686684 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba39600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 0x7fba341ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba2be00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 0x7fba3409b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 -- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 msgr2=0x7fba341ec250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 0x7fba341ec250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 -- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 msgr2=0x7fba3409b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 0x7fba3409b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 -- 172.21.15.31:0/562686684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba341f8b30 con 0x7fba18004210 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba39600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 0x7fba341ec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba2be00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 0x7fba3409b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba38c00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fba24003f80 tx=0x7fba24001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.875+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2400f760 con 0x7fba18004210 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.876+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba24016020 con 0x7fba18004210 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.876+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2400fba0 con 0x7fba18004210 2024-09-13T08:40:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.876+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba341f8e10 con 0x7fba18004210 2024-09-13T08:40:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.876+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba341f93d0 con 0x7fba18004210 2024-09-13T08:40:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.877+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba340b39b0 con 0x7fba18004210 2024-09-13T08:40:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.878+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fba2400fd40 con 0x7fba18004210 2024-09-13T08:40:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.879+0000 7fba31600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 0x7fba1007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.879+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fba2409b570 con 0x7fba18004210 2024-09-13T08:40:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.879+0000 7fba2be00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 0x7fba1007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:27.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.880+0000 7fba2be00640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 0x7fba1007aec0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba1c035000 tx=0x7fba1c0046c0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:27.883+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fba24020020 con 0x7fba18004210 2024-09-13T08:40:28.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.083+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fba340cfa40 con 0x7fba18004210 2024-09-13T08:40:28.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.087+0000 7fba31600640 1 -- 172.21.15.31:0/562686684 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7fba24063ab0 con 0x7fba18004210 2024-09-13T08:40:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 msgr2=0x7fba1007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 0x7fba1007aec0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba1c035000 tx=0x7fba1c0046c0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 msgr2=0x7fba3409d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fba24003f80 tx=0x7fba24001860 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 shutdown_connections 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba10078a00 0x7fba1007aec0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.089+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba3409cd50 0x7fba341ec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.090+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba341535c0 0x7fba3409b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.090+0000 7fba3b600640 1 --2- 172.21.15.31:0/562686684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba18004210 0x7fba3409d9e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.090+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 >> 172.21.15.31:0/562686684 conn(0x7fba340b18f0 msgr2=0x7fba34160690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.090+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 shutdown_connections 2024-09-13T08:40:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.090+0000 7fba3b600640 1 -- 172.21.15.31:0/562686684 wait complete. 2024-09-13T08:40:28.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-13T08:40:28.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 -- 172.21.15.31:0/4075145183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee340cd870 msgr2=0x7fee340cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 --2- 172.21.15.31:0/4075145183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee340cd870 0x7fee340cea40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fee2c005fd0 tx=0x7fee2c02aa10 comp rx=0 tx=0).stop 2024-09-13T08:40:28.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 -- 172.21.15.31:0/4075145183 shutdown_connections 2024-09-13T08:40:28.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 --2- 172.21.15.31:0/4075145183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee340cd870 0x7fee340cea40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 --2- 172.21.15.31:0/4075145183 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 0x7fee340ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 -- 172.21.15.31:0/4075145183 >> 172.21.15.31:0/4075145183 conn(0x7fee340d1c50 msgr2=0x7fee340d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:28.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 -- 172.21.15.31:0/4075145183 shutdown_connections 2024-09-13T08:40:28.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.273+0000 7fee3ae00640 1 -- 172.21.15.31:0/4075145183 wait complete. 2024-09-13T08:40:28.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.275+0000 7fee3ae00640 1 Processor -- start 2024-09-13T08:40:28.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.275+0000 7fee3ae00640 1 -- start start 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340cd870 0x7fee3415daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 0x7fee340b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee3406e3d0 con 0x7fee34161850 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fee340b32e0 con 0x7fee340cd870 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3ae00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fee3406e550 con 0x7fee34161470 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46272/0 (socket says 172.21.15.31:46272) 2024-09-13T08:40:28.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee3a400640 1 -- 172.21.15.31:0/3599489664 learned_addr learned my addr 172.21.15.31:0/3599489664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:28.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee39000640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 0x7fee340b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.276+0000 7fee39a00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340cd870 0x7fee3415daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 -- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 msgr2=0x7fee340b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 0x7fee340b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 -- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340cd870 msgr2=0x7fee3415daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340cd870 0x7fee3415daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 -- 172.21.15.31:0/3599489664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee341f8050 con 0x7fee34161850 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3a400640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006010 tx=0x7fee2c003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c047070 con 0x7fee34161850 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee2c004250 con 0x7fee34161850 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c033050 con 0x7fee34161850 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee341f8dd0 con 0x7fee34161850 2024-09-13T08:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.277+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee341f82f0 con 0x7fee34161850 2024-09-13T08:40:28.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.278+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3406f5e0 con 0x7fee34161850 2024-09-13T08:40:28.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.279+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fee2c04a070 con 0x7fee34161850 2024-09-13T08:40:28.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.280+0000 7fee31600640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 0x7fee1407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.281+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fee2c0bbd40 con 0x7fee34161850 2024-09-13T08:40:28.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.281+0000 7fee39a00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 0x7fee1407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.282+0000 7fee39a00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 0x7fee1407aec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee28005fd0 tx=0x7fee28005850 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:28.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.284+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fee2c084290 con 0x7fee34161850 2024-09-13T08:40:28.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.558+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7fee3406f300 con 0x7fee34161850 2024-09-13T08:40:28.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.682+0000 7fee31600640 1 -- 172.21.15.31:0/3599489664 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v183) ==== 140+0+0 (secure 0 0 0) 0x7fee2c083b90 con 0x7fee34161850 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 msgr2=0x7fee1407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 0x7fee1407aec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee28005fd0 tx=0x7fee28005850 comp rx=0 tx=0).stop 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 msgr2=0x7fee340b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006010 tx=0x7fee2c003f30 comp rx=0 tx=0).stop 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 shutdown_connections 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee34161850 0x7fee340b67b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fee14078a00 0x7fee1407aec0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee34161470 0x7fee340b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 --2- 172.21.15.31:0/3599489664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee340cd870 0x7fee3415daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 >> 172.21.15.31:0/3599489664 conn(0x7fee340d1c50 msgr2=0x7fee340d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:28.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 shutdown_connections 2024-09-13T08:40:28.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.685+0000 7fee3ae00640 1 -- 172.21.15.31:0/3599489664 wait complete. 2024-09-13T08:40:28.703 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-13T08:40:28.703 INFO:tasks.cephfs_test_runner:Starting test: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate handcrafted .meta file on legacy subvol root doesn't break the system 2024-09-13T08:40:28.704 INFO:tasks.cephfs_test_runner:test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:40:28.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 -- 172.21.15.31:0/1003846452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 msgr2=0x7f5844003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 --2- 172.21.15.31:0/1003846452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5844003ee0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f58500016c0 tx=0x7f5844005830 comp rx=0 tx=0).stop 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 -- 172.21.15.31:0/1003846452 shutdown_connections 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 --2- 172.21.15.31:0/1003846452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5844003ee0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 -- 172.21.15.31:0/1003846452 >> 172.21.15.31:0/1003846452 conn(0x7f58600cc910 msgr2=0x7f58600cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 -- 172.21.15.31:0/1003846452 shutdown_connections 2024-09-13T08:40:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.866+0000 7f5865800640 1 -- 172.21.15.31:0/1003846452 wait complete. 2024-09-13T08:40:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.868+0000 7f5865800640 1 Processor -- start 2024-09-13T08:40:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.868+0000 7f5865800640 1 -- start start 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 0x7f58600b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 0x7f58600b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58600cb850 con 0x7f5844003aa0 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58600b1a20 con 0x7f58600b4130 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f5865800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f58600cb9d0 con 0x7f58600b0900 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46292/0 (socket says 172.21.15.31:46292) 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 -- 172.21.15.31:0/1092132239 learned_addr learned my addr 172.21.15.31:0/1092132239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 -- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 msgr2=0x7f58600b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585e000640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 0x7f58600b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585f400640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 0x7f58600b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 0x7f58600b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 -- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 msgr2=0x7f58600b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 0x7f58600b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585ea00640 1 -- 172.21.15.31:0/1092132239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58600b4df0 con 0x7f5844003aa0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585e000640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 0x7f58600b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.869+0000 7f585f400640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 0x7f58600b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f585ea00640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5850001720 tx=0x7f5850006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500035f0 con 0x7f5844003aa0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5850039070 con 0x7f5844003aa0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850003b20 con 0x7f5844003aa0 2024-09-13T08:40:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58600b50d0 con 0x7f5844003aa0 2024-09-13T08:40:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.870+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58601f5b40 con 0x7f5844003aa0 2024-09-13T08:40:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.871+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58600cc270 con 0x7f5844003aa0 2024-09-13T08:40:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.872+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f585003b030 con 0x7f5844003aa0 2024-09-13T08:40:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.873+0000 7f5856a00640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 0x7f584007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.873+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f58500b9930 con 0x7f5844003aa0 2024-09-13T08:40:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.874+0000 7f585e000640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 0x7f584007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.874+0000 7f585e000640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 0x7f584007aef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f584c001cc0 tx=0x7f584c001e50 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:28.877+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f58600cc5a0 con 0x7f5844003aa0 2024-09-13T08:40:29.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.077+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58600ce990 con 0x7f5844003aa0 2024-09-13T08:40:29.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.078+0000 7f5856a00640 1 -- 172.21.15.31:0/1092132239 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+12984 (secure 0 0 0) 0x7f5850081e70 con 0x7f5844003aa0 2024-09-13T08:40:29.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:29.079 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:27.674706+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6834/1342811284":"2024-09-14T08:40:24.663346+0000","172.21.15.31:6832/1342811284":"2024-09-14T08:40:24.663346+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.080+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 msgr2=0x7f584007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.080+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 0x7f584007aef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f584c001cc0 tx=0x7f584c001e50 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.080+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 msgr2=0x7f5860154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.080+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5850001720 tx=0x7f5850006060 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 shutdown_connections 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600b4130 0x7f58600b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5840078a30 0x7f584007aef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f58600b0900 0x7f58600b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 --2- 172.21.15.31:0/1092132239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5844003aa0 0x7f5860154270 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 >> 172.21.15.31:0/1092132239 conn(0x7f58600cc910 msgr2=0x7f58600ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:29.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 shutdown_connections 2024-09-13T08:40:29.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.081+0000 7f5865800640 1 -- 172.21.15.31:0/1092132239 wait complete. 2024-09-13T08:40:29.098 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:40:29.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:40:29.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 -- 172.21.15.31:0/732116676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a40051c0 msgr2=0x7f39a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 --2- 172.21.15.31:0/732116676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f39c00b5a90 tx=0x7f39a4005830 comp rx=0 tx=0).stop 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 -- 172.21.15.31:0/732116676 shutdown_connections 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 --2- 172.21.15.31:0/732116676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 -- 172.21.15.31:0/732116676 >> 172.21.15.31:0/732116676 conn(0x7f39c00cfbb0 msgr2=0x7f39c00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 -- 172.21.15.31:0/732116676 shutdown_connections 2024-09-13T08:40:29.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.263+0000 7f39c7200640 1 -- 172.21.15.31:0/732116676 wait complete. 2024-09-13T08:40:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.265+0000 7f39c7200640 1 Processor -- start 2024-09-13T08:40:29.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.265+0000 7f39c7200640 1 -- start start 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a40051c0 0x7f39c00c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39c01e5d80 0x7f39c01e61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39c00726e0 con 0x7f39c01e4440 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0072560 con 0x7f39a40051c0 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c7200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39c0072860 con 0x7f39c01e5d80 2024-09-13T08:40:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46314/0 (socket says 172.21.15.31:46314) 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 -- 172.21.15.31:0/1172912307 learned_addr learned my addr 172.21.15.31:0/1172912307 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39c5200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39c01e5d80 0x7f39c01e61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bfe00640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a40051c0 0x7f39c00c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 -- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39c01e5d80 msgr2=0x7f39c01e61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39c01e5d80 0x7f39c01e61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 -- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a40051c0 msgr2=0x7f39c00c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a40051c0 0x7f39c00c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.266+0000 7f39bf400640 1 -- 172.21.15.31:0/1172912307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c01e4b00 con 0x7f39c01e4440 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39bf400640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f39ac003f00 tx=0x7f39ac001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac011640 con 0x7f39c01e4440 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39ac016020 con 0x7f39c01e4440 2024-09-13T08:40:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c01e4de0 con 0x7f39c01e4440 2024-09-13T08:40:29.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac011a80 con 0x7f39c01e4440 2024-09-13T08:40:29.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.267+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c01e53a0 con 0x7f39c01e4440 2024-09-13T08:40:29.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.268+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c01e55a0 con 0x7f39c01e4440 2024-09-13T08:40:29.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.269+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f39ac011c20 con 0x7f39c01e4440 2024-09-13T08:40:29.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.270+0000 7f39b7e00640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 0x7f39a007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.270+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f39ac09b680 con 0x7f39c01e4440 2024-09-13T08:40:29.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.271+0000 7f39bfe00640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 0x7f39a007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.272+0000 7f39bfe00640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 0x7f39a007aef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009b10 tx=0x7f39b0005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:29.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.275+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f39ac020020 con 0x7f39c01e4440 2024-09-13T08:40:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.469+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f39c00cb9c0 con 0x7f39c01e4440 2024-09-13T08:40:29.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.469+0000 7f39b7e00640 1 -- 172.21.15.31:0/1172912307 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) ==== 93+0+10 (secure 0 0 0) 0x7f39ac063bc0 con 0x7f39c01e4440 2024-09-13T08:40:29.470 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.471+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 msgr2=0x7f39a007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.471+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 0x7f39a007aef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009b10 tx=0x7f39b0005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.471+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 msgr2=0x7f39c014ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.471+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f39ac003f00 tx=0x7f39ac001630 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 shutdown_connections 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f39a0078a30 0x7f39a007aef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39c01e5d80 0x7f39c01e61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39c01e4440 0x7f39c014ae40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 --2- 172.21.15.31:0/1172912307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a40051c0 0x7f39c00c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 >> 172.21.15.31:0/1172912307 conn(0x7f39c00cfbb0 msgr2=0x7f39c015e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:29.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 shutdown_connections 2024-09-13T08:40:29.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.472+0000 7f39c7200640 1 -- 172.21.15.31:0/1172912307 wait complete. 2024-09-13T08:40:29.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.649+0000 7f060d600640 1 -- 172.21.15.31:0/2849186840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080b1f90 msgr2=0x7f0608168cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.649+0000 7f060d600640 1 --2- 172.21.15.31:0/2849186840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080b1f90 0x7f0608168cd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0600005fd0 tx=0x7f060002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.650+0000 7f060d600640 1 -- 172.21.15.31:0/2849186840 shutdown_connections 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.650+0000 7f060d600640 1 --2- 172.21.15.31:0/2849186840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080b1f90 0x7f0608168cd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.650+0000 7f060d600640 1 -- 172.21.15.31:0/2849186840 >> 172.21.15.31:0/2849186840 conn(0x7f060808d8b0 msgr2=0x7f060809d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.650+0000 7f060d600640 1 -- 172.21.15.31:0/2849186840 shutdown_connections 2024-09-13T08:40:29.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.650+0000 7f060d600640 1 -- 172.21.15.31:0/2849186840 wait complete. 2024-09-13T08:40:29.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.651+0000 7f060d600640 1 Processor -- start 2024-09-13T08:40:29.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.651+0000 7f060d600640 1 -- start start 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b47f0 0x7f06080cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f060815dfa0 0x7f06080cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0608169390 con 0x7f06080b1f90 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0608169210 con 0x7f060815dfa0 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f060d600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0608169510 con 0x7f06080b47f0 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.652+0000 7f0607e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46322/0 (socket says 172.21.15.31:46322) 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 -- 172.21.15.31:0/2626342490 learned_addr learned my addr 172.21.15.31:0/2626342490 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f060cc00640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f060815dfa0 0x7f06080cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f05ff400640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b47f0 0x7f06080cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 -- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b47f0 msgr2=0x7f06080cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b47f0 0x7f06080cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 -- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f060815dfa0 msgr2=0x7f06080cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:29.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f060815dfa0 0x7f06080cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:29.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 -- 172.21.15.31:0/2626342490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06081f5220 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.653+0000 7f0607e00640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05f800c3a0 tx=0x7f05f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.654+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8017070 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.654+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05f80053c0 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.654+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06081f5500 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.654+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8003850 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.654+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06081f5ac0 con 0x7f06080b1f90 2024-09-13T08:40:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.655+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06080d3bd0 con 0x7f06080b1f90 2024-09-13T08:40:29.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.656+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f05f801a070 con 0x7f06080b1f90 2024-09-13T08:40:29.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.657+0000 7f0604c00640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 0x7f05e407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:29.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.657+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f05f8099cd0 con 0x7f06080b1f90 2024-09-13T08:40:29.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.657+0000 7f05ff400640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 0x7f05e407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:29.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.658+0000 7f05ff400640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 0x7f05e407aec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007fd0 tx=0x7f05f0001410 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:29.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.662+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f05f8061c70 con 0x7f06080b1f90 2024-09-13T08:40:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:29.929+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f06080d4cf0 con 0x7f06080b1f90 2024-09-13T08:40:30.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.690+0000 7f0604c00640 1 -- 172.21.15.31:0/2626342490 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v185) ==== 164+0+0 (secure 0 0 0) 0x7f05f80666a0 con 0x7f06080b1f90 2024-09-13T08:40:30.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 msgr2=0x7f05e407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:30.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 0x7f05e407aec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007fd0 tx=0x7f05f0001410 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 msgr2=0x7f06081f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05f800c3a0 tx=0x7f05f800c860 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 shutdown_connections 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f060815dfa0 0x7f06080cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f05e4078a00 0x7f05e407aec0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b47f0 0x7f06080cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 --2- 172.21.15.31:0/2626342490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b1f90 0x7f06081f49d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 >> 172.21.15.31:0/2626342490 conn(0x7f060808d8b0 msgr2=0x7f0608157c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 shutdown_connections 2024-09-13T08:40:30.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.693+0000 7f060d600640 1 -- 172.21.15.31:0/2626342490 wait complete. 2024-09-13T08:40:30.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:40:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.876+0000 7f6828a00640 1 -- 172.21.15.31:0/1613589446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 msgr2=0x7f6824162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.876+0000 7f6828a00640 1 --2- 172.21.15.31:0/1613589446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f6824162060 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f68240d0350 tx=0x7f681402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 -- 172.21.15.31:0/1613589446 shutdown_connections 2024-09-13T08:40:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 --2- 172.21.15.31:0/1613589446 >> v2:172.21.15.45:3300/0 conn(0x7f68240d44f0 0x7f68241625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 --2- 172.21.15.31:0/1613589446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f6824162060 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 -- 172.21.15.31:0/1613589446 >> 172.21.15.31:0/1613589446 conn(0x7f682409d1c0 msgr2=0x7f68240d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 -- 172.21.15.31:0/1613589446 shutdown_connections 2024-09-13T08:40:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.877+0000 7f6828a00640 1 -- 172.21.15.31:0/1613589446 wait complete. 2024-09-13T08:40:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.878+0000 7f6828a00640 1 Processor -- start 2024-09-13T08:40:30.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.878+0000 7f6828a00640 1 -- start start 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 0x7f68240b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 0x7f68240b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f68240d24a0 con 0x7f68240d20c0 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68240b7eb0 con 0x7f68240d44f0 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.879+0000 7f6828a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f68240d2620 con 0x7f68240cf2d0 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46334/0 (socket says 172.21.15.31:46334) 2024-09-13T08:40:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 -- 172.21.15.31:0/1598520525 learned_addr learned my addr 172.21.15.31:0/1598520525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6822800640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 0x7f68240b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6823200640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 0x7f68240b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 -- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 msgr2=0x7f68240b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 0x7f68240b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 -- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 msgr2=0x7f68240b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 0x7f68240b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 -- 172.21.15.31:0/1598520525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6824152390 con 0x7f68240d20c0 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6822800640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 0x7f68240b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6823200640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 0x7f68240b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f6819e00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c400 tx=0x7f680c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.880+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c011790 con 0x7f68240d20c0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.881+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f680c017070 con 0x7f68240d20c0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.881+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c011d00 con 0x7f68240d20c0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.881+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6824152630 con 0x7f68240d20c0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.881+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6824152bf0 con 0x7f68240d20c0 2024-09-13T08:40:30.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.882+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68240b57b0 con 0x7f68240d20c0 2024-09-13T08:40:30.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.883+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f680c01c460 con 0x7f68240d20c0 2024-09-13T08:40:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.884+0000 7f681b400640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 0x7f680007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.884+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f680c09a950 con 0x7f68240d20c0 2024-09-13T08:40:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.884+0000 7f6822800640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 0x7f680007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:30.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.885+0000 7f6822800640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 0x7f680007aef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6814033000 tx=0x7f6814004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:30.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:30.888+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f680c062e90 con 0x7f68240d20c0 2024-09-13T08:40:31.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.088+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f682408df70 con 0x7f68240d20c0 2024-09-13T08:40:31.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.088+0000 7f681b400640 1 -- 172.21.15.31:0/1598520525 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7f680c062790 con 0x7f68240d20c0 2024-09-13T08:40:31.089 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:40:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:40:31.091 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:40:31.092 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:40:31.092 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:40:31.092 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:40:31.092 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:40:31.092 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 msgr2=0x7f680007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 0x7f680007aef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6814033000 tx=0x7f6814004730 comp rx=0 tx=0).stop 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 msgr2=0x7f68240a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c400 tx=0x7f680c00c8c0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 shutdown_connections 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68240d44f0 0x7f68240b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68240d20c0 0x7f68240a0910 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6800078a30 0x7f680007aef0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 --2- 172.21.15.31:0/1598520525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68240cf2d0 0x7f68240b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 >> 172.21.15.31:0/1598520525 conn(0x7f682409d1c0 msgr2=0x7f68240b8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:31.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 shutdown_connections 2024-09-13T08:40:31.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.091+0000 7f6828a00640 1 -- 172.21.15.31:0/1598520525 wait complete. 2024-09-13T08:40:31.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:40:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.271+0000 7f3359c00640 1 -- 172.21.15.31:0/2745802744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f335415bfc0 msgr2=0x7f3354072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.271+0000 7f3359c00640 1 --2- 172.21.15.31:0/2745802744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f335415bfc0 0x7f3354072560 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f33540cfab0 tx=0x7f334402a560 comp rx=0 tx=0).stop 2024-09-13T08:40:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 -- 172.21.15.31:0/2745802744 shutdown_connections 2024-09-13T08:40:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 --2- 172.21.15.31:0/2745802744 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fec0 0x7f33540cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 --2- 172.21.15.31:0/2745802744 >> v2:172.21.15.45:3300/0 conn(0x7f335406fa60 0x7f33540cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 --2- 172.21.15.31:0/2745802744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f335415bfc0 0x7f3354072560 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 -- 172.21.15.31:0/2745802744 >> 172.21.15.31:0/2745802744 conn(0x7f33540cfbb0 msgr2=0x7f33540cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 -- 172.21.15.31:0/2745802744 shutdown_connections 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.272+0000 7f3359c00640 1 -- 172.21.15.31:0/2745802744 wait complete. 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.273+0000 7f3359c00640 1 Processor -- start 2024-09-13T08:40:31.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.273+0000 7f3359c00640 1 -- start start 2024-09-13T08:40:31.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fa60 0x7f33540cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335415bfc0 0x7f33540c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33540b2b60 con 0x7f33541483c0 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33540b29e0 con 0x7f335415bfc0 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3359c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f33540b2ce0 con 0x7f335406fa60 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.274+0000 7f3353400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46354/0 (socket says 172.21.15.31:46354) 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 -- 172.21.15.31:0/89560913 learned_addr learned my addr 172.21.15.31:0/89560913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3352a00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fa60 0x7f33540cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f334a000640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335415bfc0 0x7f33540c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 -- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fa60 msgr2=0x7f33540cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fa60 0x7f33540cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 -- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335415bfc0 msgr2=0x7f33540c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335415bfc0 0x7f33540c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 -- 172.21.15.31:0/89560913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33540cf730 con 0x7f33541483c0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f3353400640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f334c00eaa0 tx=0x7f334c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.275+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c00cdb0 con 0x7f33541483c0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.276+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f334c004c10 con 0x7f33541483c0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.276+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c005020 con 0x7f33541483c0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.276+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33541f86f0 con 0x7f33541483c0 2024-09-13T08:40:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.276+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33541f8cb0 con 0x7f33541483c0 2024-09-13T08:40:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.277+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33540b3200 con 0x7f33541483c0 2024-09-13T08:40:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.278+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f334c0051c0 con 0x7f33541483c0 2024-09-13T08:40:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.279+0000 7f334b400640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 0x7f333007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.279+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f334c09b340 con 0x7f33541483c0 2024-09-13T08:40:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.279+0000 7f3352a00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 0x7f333007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.280+0000 7f3352a00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 0x7f333007aec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3344033000 tx=0x7f3344004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:31.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.283+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f334c016030 con 0x7f33541483c0 2024-09-13T08:40:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.479+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f33540b4610 con 0x7f33541483c0 2024-09-13T08:40:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.483+0000 7f334b400640 1 -- 172.21.15.31:0/89560913 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f334c063910 con 0x7f33541483c0 2024-09-13T08:40:31.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 msgr2=0x7f333007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 0x7f333007aec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3344033000 tx=0x7f3344004730 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 msgr2=0x7f33541e8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f334c00eaa0 tx=0x7f334c00ef60 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 shutdown_connections 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33541483c0 0x7f33541e8850 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335415bfc0 0x7f33540c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3330078a00 0x7f333007aec0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 --2- 172.21.15.31:0/89560913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f335406fa60 0x7f33540cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.485+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 >> 172.21.15.31:0/89560913 conn(0x7f33540cfbb0 msgr2=0x7f3354074080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.486+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 shutdown_connections 2024-09-13T08:40:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:31.486+0000 7f3359c00640 1 -- 172.21.15.31:0/89560913 wait complete. 2024-09-13T08:40:31.501 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:40:31.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:40:31.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:40:31.688 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:40:31.689 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:40:31.689 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:31.726 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:40:31.726 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:40:31.726 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:31.764 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:40:31.764 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:40:31.764 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:31.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:40:31.802 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:40:31.802 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:31.840 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:40:31.840 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:40:31.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:32.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 -- 172.21.15.31:0/3332113079 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 msgr2=0x7f9f680cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 --2- 172.21.15.31:0/3332113079 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680cf850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009830 tx=0x7f9f5802a520 comp rx=0 tx=0).stop 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 -- 172.21.15.31:0/3332113079 shutdown_connections 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 --2- 172.21.15.31:0/3332113079 >> v2:172.21.15.45:3300/0 conn(0x7f9f6815f9e0 0x7f9f6815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 --2- 172.21.15.31:0/3332113079 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680cf850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 -- 172.21.15.31:0/3332113079 >> 172.21.15.31:0/3332113079 conn(0x7f9f680cc910 msgr2=0x7f9f680cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 -- 172.21.15.31:0/3332113079 shutdown_connections 2024-09-13T08:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.002+0000 7f9f6d400640 1 -- 172.21.15.31:0/3332113079 wait complete. 2024-09-13T08:40:32.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.004+0000 7f9f6d400640 1 Processor -- start 2024-09-13T08:40:32.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.004+0000 7f9f6d400640 1 -- start start 2024-09-13T08:40:32.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 0x7f9f681479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680cb9d0 con 0x7f9f6815f9e0 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680cb850 con 0x7f9f680ce3c0 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f6d400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9f680cc360 con 0x7f9f6815bd70 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f67200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 0x7f9f681479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f66800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f66800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:34754/0 (socket says 172.21.15.31:34754) 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f67200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 0x7f9f681479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46370/0 (socket says 172.21.15.31:46370) 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f66800640 1 -- 172.21.15.31:0/2674659673 learned_addr learned my addr 172.21.15.31:0/2674659673 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.005+0000 7f9f5fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 -- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 msgr2=0x7f9f680ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 -- 172.21.15.31:0/2674659673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 msgr2=0x7f9f681479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 0x7f9f681479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 -- 172.21.15.31:0/2674659673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6814b460 con 0x7f9f680ce3c0 2024-09-13T08:40:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f66800640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9f58000c00 tx=0x7f9f58032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58040070 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f580038e0 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f680b2f10 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.006+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58003d30 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.007+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f680b34d0 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.008+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f680cf850 con 0x7f9f680ce3c0 2024-09-13T08:40:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.009+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f9f58039040 con 0x7f9f680ce3c0 2024-09-13T08:40:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.010+0000 7f9f5ea00640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 0x7f9f4407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.010+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f9f580bc470 con 0x7f9f680ce3c0 2024-09-13T08:40:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.010+0000 7f9f5fe00640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 0x7f9f4407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.011+0000 7f9f5fe00640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 0x7f9f4407aec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9f6814ed20 tx=0x7f9f50009260 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.014+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9f58084470 con 0x7f9f680ce3c0 2024-09-13T08:40:32.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.249+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f680cd0e0 con 0x7f9f680ce3c0 2024-09-13T08:40:32.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.250+0000 7f9f5ea00640 1 -- 172.21.15.31:0/2674659673 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+3911 (secure 0 0 0) 0x7f9f58088ea0 con 0x7f9f680ce3c0 2024-09-13T08:40:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:32.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"btime":"2024-09-13T08:40:25:666524+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2525416150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2525416150},{"type":"v1","addr":"172.21.15.31:6835","nonce":2525416150}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":4931,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/3432577045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":3432577045},{"type":"v1","addr":"172.21.15.45:6836","nonce":3432577045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/2495537779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2495537779},{"type":"v1","addr":"172.21.15.45:6837","nonce":2495537779}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5667,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/4109042648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":4109042648},{"type":"v1","addr":"172.21.15.31:6834","nonce":4109042648}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[]} 2024-09-13T08:40:32.251 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-09-13T08:40:32.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 msgr2=0x7f9f4407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 0x7f9f4407aec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9f6814ed20 tx=0x7f9f50009260 comp rx=0 tx=0).stop 2024-09-13T08:40:32.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 msgr2=0x7f9f680adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9f58000c00 tx=0x7f9f58032c10 comp rx=0 tx=0).stop 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 shutdown_connections 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f6815f9e0 0x7f9f681479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f9f44078a00 0x7f9f4407aec0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9f6815bd70 0x7f9f680ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 --2- 172.21.15.31:0/2674659673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f680ce3c0 0x7f9f680adf10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 >> 172.21.15.31:0/2674659673 conn(0x7f9f680cc910 msgr2=0x7f9f680cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 shutdown_connections 2024-09-13T08:40:32.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.252+0000 7f9f6d400640 1 -- 172.21.15.31:0/2674659673 wait complete. 2024-09-13T08:40:32.269 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:40:32.269 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:40:32.270 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:40:32.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:40:32.270 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:40:32.270 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:40:32.270 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:40:32.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:40:32.271 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:40:32.271 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:40:32.271 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:40:32.271 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:40:32.271 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:40:32.271 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:40:32.271 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:40:32.272 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:40:32.284 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:40:32.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:40:32.286 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:40:32.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:40:32.291 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:40:32.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:40:32.292 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:40:32.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:40:32.330 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:40:32.331 DEBUG:teuthology.parallel:result is None 2024-09-13T08:40:32.334 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:40:32.334 DEBUG:teuthology.parallel:result is None 2024-09-13T08:40:32.338 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:40:32.338 DEBUG:teuthology.parallel:result is None 2024-09-13T08:40:32.340 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:40:32.340 DEBUG:teuthology.parallel:result is None 2024-09-13T08:40:32.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:40:32.462 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:40:32.462 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:40:32.475 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:40:32.475 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:40:32.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 -- 172.21.15.31:0/2592035493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc8815e160 msgr2=0x7fcc880d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 --2- 172.21.15.31:0/2592035493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc8815e160 0x7fcc880d0790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009830 tx=0x7fcc7c02a510 comp rx=0 tx=0).stop 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 -- 172.21.15.31:0/2592035493 shutdown_connections 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 --2- 172.21.15.31:0/2592035493 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880cfa70 0x7fcc880d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 --2- 172.21.15.31:0/2592035493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc8815e160 0x7fcc880d0790 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 -- 172.21.15.31:0/2592035493 >> 172.21.15.31:0/2592035493 conn(0x7fcc880d3f50 msgr2=0x7fcc880d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 -- 172.21.15.31:0/2592035493 shutdown_connections 2024-09-13T08:40:32.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.509+0000 7fcc8de00640 1 -- 172.21.15.31:0/2592035493 wait complete. 2024-09-13T08:40:32.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.510+0000 7fcc8de00640 1 Processor -- start 2024-09-13T08:40:32.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.511+0000 7fcc8de00640 1 -- start start 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.511+0000 7fcc8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.511+0000 7fcc8de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880d0cd0 0x7fcc881f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8815e160 0x7fcc881f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc880c8bd0 con 0x7fcc880cfa70 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc880c8a50 con 0x7fcc8815e160 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcc880c8d50 con 0x7fcc880d0cd0 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46408/0 (socket says 172.21.15.31:46408) 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 -- 172.21.15.31:0/1207768067 learned_addr learned my addr 172.21.15.31:0/1207768067 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc87e00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880d0cd0 0x7fcc881f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8d400640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8815e160 0x7fcc881f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 -- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880d0cd0 msgr2=0x7fcc881f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880d0cd0 0x7fcc881f84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 -- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8815e160 msgr2=0x7fcc881f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8815e160 0x7fcc881f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 -- 172.21.15.31:0/1207768067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc880b9100 con 0x7fcc880cfa70 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.512+0000 7fcc8ca00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.513+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c005c90 con 0x7fcc880cfa70 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.513+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc7c031e50 con 0x7fcc880cfa70 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.513+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0332f0 con 0x7fcc880cfa70 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.513+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc880b93e0 con 0x7fcc880cfa70 2024-09-13T08:40:32.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.513+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc880b99a0 con 0x7fcc880cfa70 2024-09-13T08:40:32.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.514+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc880d1110 con 0x7fcc880cfa70 2024-09-13T08:40:32.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.515+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fcc7c033490 con 0x7fcc880cfa70 2024-09-13T08:40:32.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.516+0000 7fcc84c00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 0x7fcc6807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:32.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.516+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fcc7c0bbe50 con 0x7fcc880cfa70 2024-09-13T08:40:32.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.516+0000 7fcc87e00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 0x7fcc6807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:32.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.517+0000 7fcc87e00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 0x7fcc6807aec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcc880c9fc0 tx=0x7fcc74009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:32.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.520+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcc7c03f030 con 0x7fcc880cfa70 2024-09-13T08:40:32.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:32.733+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fcc881ea160 con 0x7fcc880cfa70 2024-09-13T08:40:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.723+0000 7fcc84c00640 1 -- 172.21.15.31:0/1207768067 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v58) ==== 97+0+0 (secure 0 0 0) 0x7fcc7c084390 con 0x7fcc880cfa70 2024-09-13T08:40:33.725 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.727+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 msgr2=0x7fcc6807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.727+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 0x7fcc6807aec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcc880c9fc0 tx=0x7fcc74009040 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.727+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 msgr2=0x7fcc880b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.727+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c033c90 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 shutdown_connections 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8815e160 0x7fcc881f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcc68078a00 0x7fcc6807aec0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc880d0cd0 0x7fcc881f84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 --2- 172.21.15.31:0/1207768067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc880cfa70 0x7fcc880b8790 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 >> 172.21.15.31:0/1207768067 conn(0x7fcc880d3f50 msgr2=0x7fcc880c8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 shutdown_connections 2024-09-13T08:40:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.728+0000 7fcc8de00640 1 -- 172.21.15.31:0/1207768067 wait complete. 2024-09-13T08:40:33.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:40:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 -- 172.21.15.31:0/1847703835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15f3d0 msgr2=0x7fabec0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 --2- 172.21.15.31:0/1847703835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15f3d0 0x7fabec0ceea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009830 tx=0x7fabd402a490 comp rx=0 tx=0).stop 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 -- 172.21.15.31:0/1847703835 shutdown_connections 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 --2- 172.21.15.31:0/1847703835 >> v2:172.21.15.45:3300/0 conn(0x7fabec0cf3e0 0x7fabec06f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 --2- 172.21.15.31:0/1847703835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15f3d0 0x7fabec0ceea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 -- 172.21.15.31:0/1847703835 >> 172.21.15.31:0/1847703835 conn(0x7fabec0cc910 msgr2=0x7fabec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 -- 172.21.15.31:0/1847703835 shutdown_connections 2024-09-13T08:40:33.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.913+0000 7fabf2c00640 1 -- 172.21.15.31:0/1847703835 wait complete. 2024-09-13T08:40:33.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.915+0000 7fabf2c00640 1 Processor -- start 2024-09-13T08:40:33.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.915+0000 7fabf2c00640 1 -- start start 2024-09-13T08:40:33.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec0cf3e0 0x7fabec1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:33.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:33.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec15f3d0 0x7fabec1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:33.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabec06fe90 con 0x7fabec15ad60 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fabec06fd10 con 0x7fabec15f3d0 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf2c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fabec0ca4e0 con 0x7fabec0cf3e0 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46424/0 (socket says 172.21.15.31:46424) 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabe3e00640 1 -- 172.21.15.31:0/546566740 learned_addr learned my addr 172.21.15.31:0/546566740 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabebe00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec0cf3e0 0x7fabec1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.916+0000 7fabf0c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec15f3d0 0x7fabec1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 -- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec0cf3e0 msgr2=0x7fabec1f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec0cf3e0 0x7fabec1f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 -- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec15f3d0 msgr2=0x7fabec1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec15f3d0 0x7fabec1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 -- 172.21.15.31:0/546566740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec1f7030 con 0x7fabec15ad60 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe3e00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fabd402aa80 tx=0x7fabd4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4031d60 con 0x7fabec15ad60 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabd4033e10 con 0x7fabec15ad60 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4005260 con 0x7fabec15ad60 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.917+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec1f7310 con 0x7fabec15ad60 2024-09-13T08:40:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.918+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec1f2f40 con 0x7fabec15ad60 2024-09-13T08:40:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.919+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec06f4e0 con 0x7fabec15ad60 2024-09-13T08:40:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.920+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fabd4005400 con 0x7fabec15ad60 2024-09-13T08:40:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.921+0000 7fabe8c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 0x7fabc807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.921+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 4857+0+0 (secure 0 0 0) 0x7fabd40bb890 con 0x7fabec15ad60 2024-09-13T08:40:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.922+0000 7fabebe00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 0x7fabc807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.922+0000 7fabebe00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 0x7fabc807aec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0017c0 tx=0x7fabdc009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:33.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:33.925+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fabd403f030 con 0x7fabec15ad60 2024-09-13T08:40:34.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.261+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fabec070610 con 0x7fabec15ad60 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.262+0000 7fabe8c00640 1 -- 172.21.15.31:0/546566740 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+5304 (secure 0 0 0) 0x7fabd4083920 con 0x7fabec15ad60 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:40:34.267 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:40:34.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.270 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.271 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.272 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:40:34.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:40:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:40:34.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:40:34.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.279 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:40:34.280 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:40:34.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 msgr2=0x7fabc807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 0x7fabc807aec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0017c0 tx=0x7fabdc009040 comp rx=0 tx=0).stop 2024-09-13T08:40:34.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 msgr2=0x7fabec0b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fabd402aa80 tx=0x7fabd4033c20 comp rx=0 tx=0).stop 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 shutdown_connections 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec15f3d0 0x7fabec1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15ad60 0x7fabec0b1e70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fabc8078a00 0x7fabc807aec0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 --2- 172.21.15.31:0/546566740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec0cf3e0 0x7fabec1f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 >> 172.21.15.31:0/546566740 conn(0x7fabec0cc910 msgr2=0x7fabec15b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 shutdown_connections 2024-09-13T08:40:34.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.264+0000 7fabf2c00640 1 -- 172.21.15.31:0/546566740 wait complete. 2024-09-13T08:40:34.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:40:34.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 -- 172.21.15.31:0/557585550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad40cf850 msgr2=0x7fbad40ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 --2- 172.21.15.31:0/557585550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad40cf850 0x7fbad40ce3c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbad40ca470 tx=0x7fbac002a560 comp rx=0 tx=0).stop 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 -- 172.21.15.31:0/557585550 shutdown_connections 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 --2- 172.21.15.31:0/557585550 >> v2:172.21.15.45:3300/0 conn(0x7fbad40ce900 0x7fbad40cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 --2- 172.21.15.31:0/557585550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad40cf850 0x7fbad40ce3c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 -- 172.21.15.31:0/557585550 >> 172.21.15.31:0/557585550 conn(0x7fbad40cc910 msgr2=0x7fbad40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 -- 172.21.15.31:0/557585550 shutdown_connections 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.443+0000 7fbadc600640 1 -- 172.21.15.31:0/557585550 wait complete. 2024-09-13T08:40:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.444+0000 7fbadc600640 1 Processor -- start 2024-09-13T08:40:34.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 -- start start 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad40ce900 0x7fbad41f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad40cf850 0x7fbad40c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbad41600f0 con 0x7fbad415bd70 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40cb850 con 0x7fbad40cf850 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.445+0000 7fbadc600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbad4150940 con 0x7fbad40ce900 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46444/0 (socket says 172.21.15.31:46444) 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 -- 172.21.15.31:0/3013863355 learned_addr learned my addr 172.21.15.31:0/3013863355 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbad9c00640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad40ce900 0x7fbad41f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbad9200640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad40cf850 0x7fbad40c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 -- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad40ce900 msgr2=0x7fbad41f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad40ce900 0x7fbad41f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 -- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad40cf850 msgr2=0x7fbad40c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad40cf850 0x7fbad40c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 -- 172.21.15.31:0/3013863355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad40c9cf0 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.446+0000 7fbada600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00eaa0 tx=0x7fbacc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.447+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc00cdb0 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.447+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbacc004c10 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.447+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc005020 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.447+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad40b47d0 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.447+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad40b4d90 con 0x7fbad415bd70 2024-09-13T08:40:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.448+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad40ce3a0 con 0x7fbad415bd70 2024-09-13T08:40:34.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.449+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fbacc0051c0 con 0x7fbad415bd70 2024-09-13T08:40:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.450+0000 7fbad1600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.450+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 4857+0+0 (secure 0 0 0) 0x7fbacc09b380 con 0x7fbad415bd70 2024-09-13T08:40:34.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.450+0000 7fbad9c00640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.451+0000 7fbad9c00640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006fd0 tx=0x7fbac8008040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.454+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbacc016030 con 0x7fbad415bd70 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.652+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbad40afc30 con 0x7fbad415bd70 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.653+0000 7fbad1600640 1 -- 172.21.15.31:0/3013863355 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v58) ==== 84+0+65 (secure 0 0 0) 0x7fbacc0639a0 con 0x7fbad415bd70 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:40:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:34.655 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 msgr2=0x7fbab407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006fd0 tx=0x7fbac8008040 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 msgr2=0x7fbad4158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00eaa0 tx=0x7fbacc00ef60 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 shutdown_connections 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad415bd70 0x7fbad4158020 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbad40cf850 0x7fbad40c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbab4078a00 0x7fbab407aec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 --2- 172.21.15.31:0/3013863355 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbad40ce900 0x7fbad41f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 >> 172.21.15.31:0/3013863355 conn(0x7fbad40cc910 msgr2=0x7fbad40ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:34.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 shutdown_connections 2024-09-13T08:40:34.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.655+0000 7fbadc600640 1 -- 172.21.15.31:0/3013863355 wait complete. 2024-09-13T08:40:34.671 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:40:34.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:40:34.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 -- 172.21.15.31:0/3947878694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 msgr2=0x7f5b2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 --2- 172.21.15.31:0/3947878694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b2c004800 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806db40 tx=0x7f5b2c005830 comp rx=0 tx=0).stop 2024-09-13T08:40:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b47e00640 1 -- 172.21.15.31:0/3947878694 reap_dead start 2024-09-13T08:40:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 -- 172.21.15.31:0/3947878694 shutdown_connections 2024-09-13T08:40:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 -- 172.21.15.31:0/3947878694 >> 172.21.15.31:0/3947878694 conn(0x7f5b480cfbb0 msgr2=0x7f5b480cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 -- 172.21.15.31:0/3947878694 shutdown_connections 2024-09-13T08:40:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.840+0000 7f5b4e600640 1 -- 172.21.15.31:0/3947878694 wait complete. 2024-09-13T08:40:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.842+0000 7f5b4e600640 1 Processor -- start 2024-09-13T08:40:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.842+0000 7f5b4e600640 1 -- start start 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b480adfe0 0x7f5b48154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b480c9900 0x7f5b48156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b480cbfd0 con 0x7f5b2c003f70 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b480cbe50 con 0x7f5b480c9900 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b4e600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48072560 con 0x7f5b480adfe0 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b47400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b47400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45396/0 (socket says 172.21.15.31:45396) 2024-09-13T08:40:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.843+0000 7f5b47400640 1 -- 172.21.15.31:0/2791645052 learned_addr learned my addr 172.21.15.31:0/2791645052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47e00640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b480c9900 0x7f5b48156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 -- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b480adfe0 msgr2=0x7f5b48154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b480adfe0 0x7f5b48154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 -- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b480c9900 msgr2=0x7f5b48156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b480c9900 0x7f5b48156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 -- 172.21.15.31:0/2791645052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b480b4240 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b47400640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b38011b10 tx=0x7f5b38009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3801e070 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b38004050 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800fea0 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b480b4520 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.844+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b48155390 con 0x7f5b2c003f70 2024-09-13T08:40:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.845+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b480cb740 con 0x7f5b2c003f70 2024-09-13T08:40:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.847+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f5b38021070 con 0x7f5b2c003f70 2024-09-13T08:40:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.848+0000 7f5b3fe00640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 0x7f5b2407aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.848+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 4857+0+0 (secure 0 0 0) 0x7f5b3809b460 con 0x7f5b2c003f70 2024-09-13T08:40:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.848+0000 7f5b3ea00640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 0x7f5b2407aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.849+0000 7f5b3ea00640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 0x7f5b2407aef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0057e0 tx=0x7f5b30001110 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:34.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:34.852+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5b380044a0 con 0x7f5b2c003f70 2024-09-13T08:40:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.046+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5b480cb310 con 0x7f5b2c003f70 2024-09-13T08:40:35.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.714+0000 7f5b3fe00640 1 -- 172.21.15.31:0/2791645052 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v59) ==== 138+0+0 (secure 0 0 0) 0x7f5b38063460 con 0x7f5b2c003f70 2024-09-13T08:40:35.715 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:40:35.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.717+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 msgr2=0x7f5b2407aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:35.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.717+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 0x7f5b2407aef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0057e0 tx=0x7f5b30001110 comp rx=0 tx=0).stop 2024-09-13T08:40:35.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.717+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 msgr2=0x7f5b481f8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:35.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.717+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b38011b10 tx=0x7f5b38009e90 comp rx=0 tx=0).stop 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 shutdown_connections 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b480c9900 0x7f5b48156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5b24078a30 0x7f5b2407aef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b480adfe0 0x7f5b48154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 --2- 172.21.15.31:0/2791645052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c003f70 0x7f5b481f8970 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 >> 172.21.15.31:0/2791645052 conn(0x7f5b480cfbb0 msgr2=0x7f5b4815b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 shutdown_connections 2024-09-13T08:40:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.718+0000 7f5b4e600640 1 -- 172.21.15.31:0/2791645052 wait complete. 2024-09-13T08:40:35.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:40:35.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 -- 172.21.15.31:0/909758828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fec0 msgr2=0x7fd18c0b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:35.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 --2- 172.21.15.31:0/909758828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fec0 0x7fd18c0b29e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd184005fd0 tx=0x7fd18402aaa0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 -- 172.21.15.31:0/909758828 shutdown_connections 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 --2- 172.21.15.31:0/909758828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fec0 0x7fd18c0b29e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 --2- 172.21.15.31:0/909758828 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 0x7fd18c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 -- 172.21.15.31:0/909758828 >> 172.21.15.31:0/909758828 conn(0x7fd18c0cfbb0 msgr2=0x7fd18c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 -- 172.21.15.31:0/909758828 shutdown_connections 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.915+0000 7fd191400640 1 -- 172.21.15.31:0/909758828 wait complete. 2024-09-13T08:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.917+0000 7fd191400640 1 Processor -- start 2024-09-13T08:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.917+0000 7fd191400640 1 -- start start 2024-09-13T08:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fa60 0x7fd18c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 0x7fd18c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c0cbfd0 con 0x7fd18c13ffc0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c0cbe50 con 0x7fd18c06fa60 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd191400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd18c0b2f20 con 0x7fd18c15bfc0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fa60 0x7fd18c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45408/0 (socket says 172.21.15.31:45408) 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 -- 172.21.15.31:0/83874867 learned_addr learned my addr 172.21.15.31:0/83874867 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd189e00640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 0x7fd18c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 -- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 msgr2=0x7fd18c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 0x7fd18c15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 -- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fa60 msgr2=0x7fd18c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fa60 0x7fd18c15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.918+0000 7fd18b200640 1 -- 172.21.15.31:0/83874867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18c0b3680 con 0x7fd18c13ffc0 2024-09-13T08:40:35.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd18b200640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd184006010 tx=0x7fd184003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:35.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184046070 con 0x7fd18c13ffc0 2024-09-13T08:40:35.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd184004040 con 0x7fd18c13ffc0 2024-09-13T08:40:35.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184004430 con 0x7fd18c13ffc0 2024-09-13T08:40:35.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18c15f1a0 con 0x7fd18c13ffc0 2024-09-13T08:40:35.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.919+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18c0b3bd0 con 0x7fd18c13ffc0 2024-09-13T08:40:35.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.920+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd18c1535f0 con 0x7fd18c13ffc0 2024-09-13T08:40:35.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.921+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fd184049070 con 0x7fd18c13ffc0 2024-09-13T08:40:35.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.922+0000 7fd182a00640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 0x7fd16c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:35.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.922+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5227+0+0 (secure 0 0 0) 0x7fd1840bb8a0 con 0x7fd18c13ffc0 2024-09-13T08:40:35.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.923+0000 7fd18a800640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 0x7fd16c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:35.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.924+0000 7fd18a800640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 0x7fd16c07aec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0095f0 tx=0x7fd17c005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:35.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:35.927+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd184032030 con 0x7fd18c13ffc0 2024-09-13T08:40:36.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.127+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd18c159200 con 0x7fd18c13ffc0 2024-09-13T08:40:36.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:40:36.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:40:36.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.721+0000 7fd182a00640 1 -- 172.21.15.31:0/83874867 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v60) ==== 170+0+0 (secure 0 0 0) 0x7fd184083720 con 0x7fd18c13ffc0 2024-09-13T08:40:36.722 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:40:36.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.725+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 msgr2=0x7fd16c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:36.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.725+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 0x7fd16c07aec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0095f0 tx=0x7fd17c005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.725+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 msgr2=0x7fd18c15e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.725+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd184006010 tx=0x7fd184003ce0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 shutdown_connections 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c13ffc0 0x7fd18c15e4c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fd16c078a00 0x7fd16c07aec0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd18c15bfc0 0x7fd18c15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 --2- 172.21.15.31:0/83874867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c06fa60 0x7fd18c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 >> 172.21.15.31:0/83874867 conn(0x7fd18c0cfbb0 msgr2=0x7fd18c0b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:36.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 shutdown_connections 2024-09-13T08:40:36.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.726+0000 7fd191400640 1 -- 172.21.15.31:0/83874867 wait complete. 2024-09-13T08:40:36.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 -- 172.21.15.31:0/401116476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f0003fd0 msgr2=0x7f28f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 --2- 172.21.15.31:0/401116476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f0003fd0 0x7f28f0004800 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009b50 tx=0x7f28f0005830 comp rx=0 tx=0).stop 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2911600640 1 -- 172.21.15.31:0/401116476 reap_dead start 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 -- 172.21.15.31:0/401116476 shutdown_connections 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 -- 172.21.15.31:0/401116476 >> 172.21.15.31:0/401116476 conn(0x7f290c0cfbb0 msgr2=0x7f290c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 -- 172.21.15.31:0/401116476 shutdown_connections 2024-09-13T08:40:36.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.923+0000 7f2913600640 1 -- 172.21.15.31:0/401116476 wait complete. 2024-09-13T08:40:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.925+0000 7f2913600640 1 Processor -- start 2024-09-13T08:40:36.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.925+0000 7f2913600640 1 -- start start 2024-09-13T08:40:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f28f0003fd0 0x7f290c152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f290c0b17b0 0x7f290c0c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0cbfd0 con 0x7f290c0c9630 2024-09-13T08:40:36.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0cbe50 con 0x7f290c0b17b0 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2913600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f290c06fff0 con 0x7f28f0003fd0 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2911600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2911600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45426/0 (socket says 172.21.15.31:45426) 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.926+0000 7f2911600640 1 -- 172.21.15.31:0/3236777137 learned_addr learned my addr 172.21.15.31:0/3236777137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2910c00640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f28f0003fd0 0x7f290c152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2903e00640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f290c0b17b0 0x7f290c0c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 -- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f28f0003fd0 msgr2=0x7f290c152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f28f0003fd0 0x7f290c152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 -- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f290c0b17b0 msgr2=0x7f290c0c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f290c0b17b0 0x7f290c0c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 -- 172.21.15.31:0/3236777137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c0b09f0 con 0x7f290c0c9630 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2911600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f290c0cfb50 tx=0x7f290400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.927+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904004b20 con 0x7f290c0c9630 2024-09-13T08:40:36.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.928+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2904004cc0 con 0x7f290c0c9630 2024-09-13T08:40:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.928+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c0b0cd0 con 0x7f290c0c9630 2024-09-13T08:40:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.928+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904003bd0 con 0x7f290c0c9630 2024-09-13T08:40:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.928+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c0b1290 con 0x7f290c0c9630 2024-09-13T08:40:36.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.929+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c0cd780 con 0x7f290c0c9630 2024-09-13T08:40:36.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.931+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f2904004030 con 0x7f290c0c9630 2024-09-13T08:40:36.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.932+0000 7f2909600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 0x7f28e807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:36.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.932+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5609+0+0 (secure 0 0 0) 0x7f290409b560 con 0x7f290c0c9630 2024-09-13T08:40:36.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.933+0000 7f2910c00640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 0x7f28e807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.933+0000 7f2910c00640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 0x7f28e807aec0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f28fc006b70 tx=0x7f28fc0067a0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:36.936+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f29040632c0 con 0x7f290c0c9630 2024-09-13T08:40:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.169+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f290c0b1460 con 0x7f290c0c9630 2024-09-13T08:40:37.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.733+0000 7f2909600640 1 -- 172.21.15.31:0/3236777137 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v33) ==== 172+0+0 (secure 0 0 0) 0x7f2904067cf0 con 0x7f290c0c9630 2024-09-13T08:40:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 8 and data pool 9 2024-09-13T08:40:37.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.739+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 msgr2=0x7f28e807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:37.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.739+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 0x7f28e807aec0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f28fc006b70 tx=0x7f28fc0067a0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.739+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 msgr2=0x7f290c0af740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.739+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f290c0cfb50 tx=0x7f290400f3c0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 shutdown_connections 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f290c0c9630 0x7f290c0af740 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f290c0b17b0 0x7f290c0c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f28e8078a00 0x7f28e807aec0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 --2- 172.21.15.31:0/3236777137 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f28f0003fd0 0x7f290c152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.740+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 >> 172.21.15.31:0/3236777137 conn(0x7f290c0cfbb0 msgr2=0x7f290c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.741+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 shutdown_connections 2024-09-13T08:40:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.741+0000 7f2913600640 1 -- 172.21.15.31:0/3236777137 wait complete. 2024-09-13T08:40:37.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.940+0000 7f226a400640 1 -- 172.21.15.31:0/163140836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 msgr2=0x7f22640ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.940+0000 7f226a400640 1 --2- 172.21.15.31:0/163140836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f22640ceea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f22640d0f70 tx=0x7f225402aa30 comp rx=0 tx=0).stop 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 -- 172.21.15.31:0/163140836 shutdown_connections 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 --2- 172.21.15.31:0/163140836 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f226415c520 0x7f226406cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 --2- 172.21.15.31:0/163140836 >> v2:172.21.15.45:3300/0 conn(0x7f22640cb5f0 0x7f226415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 --2- 172.21.15.31:0/163140836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f22640ceea0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 -- 172.21.15.31:0/163140836 >> 172.21.15.31:0/163140836 conn(0x7f22640cda20 msgr2=0x7f22640cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 -- 172.21.15.31:0/163140836 shutdown_connections 2024-09-13T08:40:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.941+0000 7f226a400640 1 -- 172.21.15.31:0/163140836 wait complete. 2024-09-13T08:40:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.942+0000 7f226a400640 1 Processor -- start 2024-09-13T08:40:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 -- start start 2024-09-13T08:40:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 0x7f22640c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f226415c520 0x7f2264157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2264073d50 con 0x7f226415bc00 2024-09-13T08:40:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2264073bd0 con 0x7f22640cb5f0 2024-09-13T08:40:37.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.943+0000 7f226a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2264073ed0 con 0x7f226415c520 2024-09-13T08:40:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45430/0 (socket says 172.21.15.31:45430) 2024-09-13T08:40:37.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 -- 172.21.15.31:0/415198268 learned_addr learned my addr 172.21.15.31:0/415198268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 -- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f226415c520 msgr2=0x7f2264157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f226415c520 0x7f2264157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2263400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 0x7f22640c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 -- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 msgr2=0x7f22640c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 0x7f22640c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 -- 172.21.15.31:0/415198268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2264156e40 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2263400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 0x7f22640c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.944+0000 7f2262a00640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f225000d720 tx=0x7f225000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.945+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250017070 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.945+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2250009ab0 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.945+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250010460 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.945+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22641570e0 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.945+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22641576a0 con 0x7f226415bc00 2024-09-13T08:40:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.946+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22640ceea0 con 0x7f226415bc00 2024-09-13T08:40:37.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.947+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f225001a070 con 0x7f226415bc00 2024-09-13T08:40:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.948+0000 7f225b400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 0x7f224407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.948+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5709+0+0 (secure 0 0 0) 0x7f225009adc0 con 0x7f226415bc00 2024-09-13T08:40:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.949+0000 7f2263400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 0x7f224407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.949+0000 7f2263400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 0x7f224407aec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2254035000 tx=0x7f2254004ab0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:37.953+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2250063090 con 0x7f226415bc00 2024-09-13T08:40:38.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.150+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22640c85b0 con 0x7f226415bc00 2024-09-13T08:40:38.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.151+0000 7f225b400640 1 -- 172.21.15.31:0/415198268 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+16690 (secure 0 0 0) 0x7f2250062990 con 0x7f226415bc00 2024-09-13T08:40:38.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:38.153 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:37.170211+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:40:35.056540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T08:40:36.137508+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 msgr2=0x7f224407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 0x7f224407aec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2254035000 tx=0x7f2254004ab0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 msgr2=0x7f226414d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f225000d720 tx=0x7f225000dbe0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 shutdown_connections 2024-09-13T08:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2244078a00 0x7f224407aec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f226415c520 0x7f2264157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f226415bc00 0x7f226414d370 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 --2- 172.21.15.31:0/415198268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22640cb5f0 0x7f22640c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 >> 172.21.15.31:0/415198268 conn(0x7f22640cda20 msgr2=0x7f22640cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 shutdown_connections 2024-09-13T08:40:38.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.154+0000 7f226a400640 1 -- 172.21.15.31:0/415198268 wait complete. 2024-09-13T08:40:38.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.331+0000 7f95f9600640 1 -- 172.21.15.31:0/3014319795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40b47f0 msgr2=0x7f95f40d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.331+0000 7f95f9600640 1 --2- 172.21.15.31:0/3014319795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40b47f0 0x7f95f40d3bd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f95f40d19a0 tx=0x7f95e002a5f0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 -- 172.21.15.31:0/3014319795 shutdown_connections 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 --2- 172.21.15.31:0/3014319795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40b47f0 0x7f95f40d3bd0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 --2- 172.21.15.31:0/3014319795 >> v2:172.21.15.45:3300/0 conn(0x7f95f415dfa0 0x7f95f40d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 -- 172.21.15.31:0/3014319795 >> 172.21.15.31:0/3014319795 conn(0x7f95f408d8b0 msgr2=0x7f95f409d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 -- 172.21.15.31:0/3014319795 shutdown_connections 2024-09-13T08:40:38.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.332+0000 7f95f9600640 1 -- 172.21.15.31:0/3014319795 wait complete. 2024-09-13T08:40:38.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.333+0000 7f95f9600640 1 Processor -- start 2024-09-13T08:40:38.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.334+0000 7f95f9600640 1 -- start start 2024-09-13T08:40:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.334+0000 7f95f9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 0x7f95f415fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.334+0000 7f95f9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.334+0000 7f95f9600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 0x7f95f41f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95f40d2c60 con 0x7f95f40d4110 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f95f40b21a0 con 0x7f95f40b47f0 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f9600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f95f41620b0 con 0x7f95f415dfa0 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45454/0 (socket says 172.21.15.31:45454) 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 -- 172.21.15.31:0/3887281181 learned_addr learned my addr 172.21.15.31:0/3887281181 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3e00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 0x7f95f415fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 -- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 msgr2=0x7f95f41f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f8c00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 0x7f95f41f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 0x7f95f41f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 -- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 msgr2=0x7f95f415fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 0x7f95f415fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3400640 1 -- 172.21.15.31:0/3887281181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f415e9b0 con 0x7f95f40d4110 2024-09-13T08:40:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f3e00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 0x7f95f415fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.335+0000 7f95f8c00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 0x7f95f41f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95f3400640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009850 tx=0x7f95e0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e003f070 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95e00047a0 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0035030 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f415ec90 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.336+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f415f250 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.337+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f41625a0 con 0x7f95f40d4110 2024-09-13T08:40:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.338+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f95e0042070 con 0x7f95f40d4110 2024-09-13T08:40:38.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.339+0000 7f95ebe00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 0x7f95d407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.339+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5709+0+0 (secure 0 0 0) 0x7f95e00bbb90 con 0x7f95f40d4110 2024-09-13T08:40:38.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.340+0000 7f95f3e00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 0x7f95d407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.340+0000 7f95f3e00640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 0x7f95d407aec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009ab0 tx=0x7f95e4005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:38.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.344+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f95e0083dd0 con 0x7f95f40d4110 2024-09-13T08:40:38.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.539+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95f40d4cf0 con 0x7f95f40d4110 2024-09-13T08:40:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.541+0000 7f95ebe00640 1 -- 172.21.15.31:0/3887281181 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+16690 (secure 0 0 0) 0x7f95e00836d0 con 0x7f95f40d4110 2024-09-13T08:40:38.542 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:38.542 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:37.170211+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:40:35.056540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T08:40:36.137508+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 msgr2=0x7f95d407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 0x7f95d407aec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009ab0 tx=0x7f95e4005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 msgr2=0x7f95f41f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009850 tx=0x7f95e0003820 comp rx=0 tx=0).stop 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 shutdown_connections 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f95d4078a00 0x7f95d407aec0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f95f415dfa0 0x7f95f41f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95f40d4110 0x7f95f41f8030 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 --2- 172.21.15.31:0/3887281181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95f40b47f0 0x7f95f415fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 >> 172.21.15.31:0/3887281181 conn(0x7f95f408d8b0 msgr2=0x7f95f40b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 shutdown_connections 2024-09-13T08:40:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.543+0000 7f95f9600640 1 -- 172.21.15.31:0/3887281181 wait complete. 2024-09-13T08:40:38.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:40:38.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.723+0000 7f47a7c00640 1 -- 172.21.15.31:0/1656020761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 msgr2=0x7f47a00cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.723+0000 7f47a7c00640 1 --2- 172.21.15.31:0/1656020761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a00cdcc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f47a00d0a70 tx=0x7f479802aa10 comp rx=0 tx=0).stop 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 -- 172.21.15.31:0/1656020761 shutdown_connections 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 --2- 172.21.15.31:0/1656020761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a00cdcc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 --2- 172.21.15.31:0/1656020761 >> v2:172.21.15.45:3300/0 conn(0x7f47a00741b0 0x7f47a00cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 --2- 172.21.15.31:0/1656020761 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a00cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 -- 172.21.15.31:0/1656020761 >> 172.21.15.31:0/1656020761 conn(0x7f47a00cfbb0 msgr2=0x7f47a00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 -- 172.21.15.31:0/1656020761 shutdown_connections 2024-09-13T08:40:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.724+0000 7f47a7c00640 1 -- 172.21.15.31:0/1656020761 wait complete. 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.725+0000 7f47a7c00640 1 Processor -- start 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.725+0000 7f47a7c00640 1 -- start start 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47a00741b0 0x7f47a00c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47a00cad70 con 0x7f47a00745f0 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47a00cabf0 con 0x7f47a00741b0 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f47a7c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f47a00caef0 con 0x7f47a015acb0 2024-09-13T08:40:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.726+0000 7f4797e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45464/0 (socket says 172.21.15.31:45464) 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 -- 172.21.15.31:0/2359213958 learned_addr learned my addr 172.21.15.31:0/2359213958 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f47a5200640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47a00741b0 0x7f47a00c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f47a5c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 -- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 msgr2=0x7f47a014c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a014c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 -- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47a00741b0 msgr2=0x7f47a00c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47a00741b0 0x7f47a00c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 -- 172.21.15.31:0/2359213958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a00b0a90 con 0x7f47a00745f0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f47a5c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a014c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.727+0000 7f4797e00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f478800d780 tx=0x7f478800dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.728+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788017070 con 0x7f47a00745f0 2024-09-13T08:40:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.728+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f478800de90 con 0x7f47a00745f0 2024-09-13T08:40:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.728+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47880099d0 con 0x7f47a00745f0 2024-09-13T08:40:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.728+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a00b0d70 con 0x7f47a00745f0 2024-09-13T08:40:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.728+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a00c7430 con 0x7f47a00745f0 2024-09-13T08:40:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.729+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a00cf4d0 con 0x7f47a00745f0 2024-09-13T08:40:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.730+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f4788009b70 con 0x7f47a00745f0 2024-09-13T08:40:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.731+0000 7f479d600640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 0x7f477c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.732+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5709+0+0 (secure 0 0 0) 0x7f478809ad70 con 0x7f47a00745f0 2024-09-13T08:40:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.732+0000 7f47a5200640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 0x7f477c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.733+0000 7f47a5200640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 0x7f477c07aec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4790004630 tx=0x7f479000b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:38.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.736+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4788062960 con 0x7f47a00745f0 2024-09-13T08:40:38.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:38.974+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f47a00b2ca0 con 0x7f47a00745f0 2024-09-13T08:40:39.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.762+0000 7f479d600640 1 -- 172.21.15.31:0/2359213958 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) ==== 118+0+0 (secure 0 0 0) 0x7f4788067390 con 0x7f47a00745f0 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 msgr2=0x7f477c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 0x7f477c07aec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4790004630 tx=0x7f479000b2e0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 msgr2=0x7f47a014ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f478800d780 tx=0x7f478800dc40 comp rx=0 tx=0).stop 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 shutdown_connections 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f477c078a00 0x7f477c07aec0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47a015acb0 0x7f47a014c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a00745f0 0x7f47a014ee50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 --2- 172.21.15.31:0/2359213958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47a00741b0 0x7f47a00c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 >> 172.21.15.31:0/2359213958 conn(0x7f47a00cfbb0 msgr2=0x7f47a00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:39.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 shutdown_connections 2024-09-13T08:40:39.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.766+0000 7f47a7c00640 1 -- 172.21.15.31:0/2359213958 wait complete. 2024-09-13T08:40:39.786 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:40:39.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:40:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 -- 172.21.15.31:0/3752430673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700b4ae0 msgr2=0x7f3c700d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 --2- 172.21.15.31:0/3752430673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700b4ae0 0x7f3c700d1550 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3c700c7790 tx=0x7f3c5802a530 comp rx=0 tx=0).stop 2024-09-13T08:40:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 -- 172.21.15.31:0/3752430673 shutdown_connections 2024-09-13T08:40:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 --2- 172.21.15.31:0/3752430673 >> v2:172.21.15.45:3300/0 conn(0x7f3c700d2170 0x7f3c700b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 --2- 172.21.15.31:0/3752430673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700b4ae0 0x7f3c700d1550 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 -- 172.21.15.31:0/3752430673 >> 172.21.15.31:0/3752430673 conn(0x7f3c700d3f50 msgr2=0x7f3c700d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 -- 172.21.15.31:0/3752430673 shutdown_connections 2024-09-13T08:40:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.954+0000 7f3c76600640 1 -- 172.21.15.31:0/3752430673 wait complete. 2024-09-13T08:40:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.956+0000 7f3c76600640 1 Processor -- start 2024-09-13T08:40:39.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.956+0000 7f3c76600640 1 -- start start 2024-09-13T08:40:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3c700b4ae0 0x7f3c700cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c700d2170 0x7f3c701574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c700cedc0 con 0x7f3c700d1d90 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c700cec40 con 0x7f3c700d2170 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c76600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3c700cef40 con 0x7f3c700b4ae0 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45486/0 (socket says 172.21.15.31:45486) 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 -- 172.21.15.31:0/3954999337 learned_addr learned my addr 172.21.15.31:0/3954999337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c6f400640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3c700b4ae0 0x7f3c700cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c6fe00640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c700d2170 0x7f3c701574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 -- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3c700b4ae0 msgr2=0x7f3c700cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3c700b4ae0 0x7f3c700cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 -- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c700d2170 msgr2=0x7f3c701574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c700d2170 0x7f3c701574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:39.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.957+0000 7f3c66a00640 1 -- 172.21.15.31:0/3954999337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c700b6da0 con 0x7f3c700d1d90 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c66a00640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c5802ab20 tx=0x7f3c58032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58040070 con 0x7f3c700d1d90 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c58005310 con 0x7f3c700d1d90 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c700b7040 con 0x7f3c700d1d90 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580047a0 con 0x7f3c700d1d90 2024-09-13T08:40:39.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.958+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c700b7600 con 0x7f3c700d1d90 2024-09-13T08:40:39.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.959+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c700d1550 con 0x7f3c700d1d90 2024-09-13T08:40:39.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.960+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f3c58039040 con 0x7f3c700d1d90 2024-09-13T08:40:39.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.961+0000 7f3c67e00640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 0x7f3c4c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:39.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.961+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f3c580bc0a0 con 0x7f3c700d1d90 2024-09-13T08:40:39.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.962+0000 7f3c6f400640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 0x7f3c4c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:39.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.963+0000 7f3c6f400640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 0x7f3c4c07aec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3c600012e0 tx=0x7f3c60009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:39.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:39.966+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3c58084370 con 0x7f3c700d1d90 2024-09-13T08:40:40.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.199+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f3c700b1c20 con 0x7f3c700d1d90 2024-09-13T08:40:40.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.765+0000 7f3c67e00640 1 -- 172.21.15.31:0/3954999337 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7f3c58083c70 con 0x7f3c700d1d90 2024-09-13T08:40:40.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 msgr2=0x7f3c4c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:40.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 0x7f3c4c07aec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3c600012e0 tx=0x7f3c60009040 comp rx=0 tx=0).stop 2024-09-13T08:40:40.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 msgr2=0x7f3c700b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c5802ab20 tx=0x7f3c58032c10 comp rx=0 tx=0).stop 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 shutdown_connections 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c700d2170 0x7f3c701574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c700d1d90 0x7f3c700b3200 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3c4c078a00 0x7f3c4c07aec0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 --2- 172.21.15.31:0/3954999337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3c700b4ae0 0x7f3c700cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.768+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 >> 172.21.15.31:0/3954999337 conn(0x7f3c700d3f50 msgr2=0x7f3c70158670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.769+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 shutdown_connections 2024-09-13T08:40:40.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.769+0000 7f3c76600640 1 -- 172.21.15.31:0/3954999337 wait complete. 2024-09-13T08:40:40.788 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:40:40.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 -- 172.21.15.31:0/3678691680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 msgr2=0x7fafe80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 --2- 172.21.15.31:0/3678691680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80d3bd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafe8162a40 tx=0x7fafd402a560 comp rx=0 tx=0).stop 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 -- 172.21.15.31:0/3678691680 shutdown_connections 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 --2- 172.21.15.31:0/3678691680 >> v2:172.21.15.45:3300/0 conn(0x7fafe80d4110 0x7fafe81619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 --2- 172.21.15.31:0/3678691680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80d3bd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 --2- 172.21.15.31:0/3678691680 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe815dfa0 0x7fafe80d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 -- 172.21.15.31:0/3678691680 >> 172.21.15.31:0/3678691680 conn(0x7fafe808d8b0 msgr2=0x7fafe809d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 -- 172.21.15.31:0/3678691680 shutdown_connections 2024-09-13T08:40:40.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.954+0000 7fafedc00640 1 -- 172.21.15.31:0/3678691680 wait complete. 2024-09-13T08:40:40.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.955+0000 7fafedc00640 1 Processor -- start 2024-09-13T08:40:40.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 -- start start 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe80d4110 0x7fafe80cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe815dfa0 0x7fafe814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8162270 con 0x7fafe80b47f0 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafe81620f0 con 0x7fafe815dfa0 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.956+0000 7fafedc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fafe81623f0 con 0x7fafe80d4110 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45506/0 (socket says 172.21.15.31:45506) 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 -- 172.21.15.31:0/3460406682 learned_addr learned my addr 172.21.15.31:0/3460406682 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7400640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe80d4110 0x7fafe80cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafed200640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe815dfa0 0x7fafe814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 -- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe80d4110 msgr2=0x7fafe80cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe80d4110 0x7fafe80cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 -- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe815dfa0 msgr2=0x7fafe814f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe815dfa0 0x7fafe814f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 -- 172.21.15.31:0/3460406682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe8150860 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.957+0000 7fafe7e00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e7c0 tx=0x7fafd800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.958+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd800ccc0 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.958+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafd800ce80 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.958+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd8005280 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.958+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe8150b40 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.958+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe81f8070 con 0x7fafe80b47f0 2024-09-13T08:40:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.959+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe8150d80 con 0x7fafe80b47f0 2024-09-13T08:40:40.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.960+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fafd8005420 con 0x7fafe80b47f0 2024-09-13T08:40:40.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.961+0000 7fafdfe00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 0x7fafc807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:40.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.962+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fafd809b730 con 0x7fafe80b47f0 2024-09-13T08:40:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.962+0000 7fafe7400640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 0x7fafc807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.963+0000 7fafe7400640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 0x7fafc807aec0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fafd4033000 tx=0x7fafd4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:40.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:40.966+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fafd8063a00 con 0x7fafe80b47f0 2024-09-13T08:40:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.200+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafe81619a0 con 0x7fafe80b47f0 2024-09-13T08:40:41.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.201+0000 7fafdfe00640 1 -- 172.21.15.31:0/3460406682 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5217 (secure 0 0 0) 0x7fafd8063300 con 0x7fafe80b47f0 2024-09-13T08:40:41.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:41.202 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":37,"btime":"2024-09-13T08:40:40:762682+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:40:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 37 2024-09-13T08:40:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 msgr2=0x7fafc807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 0x7fafc807aec0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fafd4033000 tx=0x7fafd4004730 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 msgr2=0x7fafe80cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e7c0 tx=0x7fafd800ec80 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 shutdown_connections 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe815dfa0 0x7fafe814f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fafc8078a00 0x7fafc807aec0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafe80d4110 0x7fafe80cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 --2- 172.21.15.31:0/3460406682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafe80b47f0 0x7fafe80cd700 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.203+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 >> 172.21.15.31:0/3460406682 conn(0x7fafe808d8b0 msgr2=0x7fafe806ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.204+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 shutdown_connections 2024-09-13T08:40:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.204+0000 7fafedc00640 1 -- 172.21.15.31:0/3460406682 wait complete. 2024-09-13T08:40:41.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:40:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 -- 172.21.15.31:0/4063454315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 msgr2=0x7f34480ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 --2- 172.21.15.31:0/4063454315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34480ce3c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f34480ca470 tx=0x7f343402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 -- 172.21.15.31:0/4063454315 shutdown_connections 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 --2- 172.21.15.31:0/4063454315 >> v2:172.21.15.45:3300/0 conn(0x7f34480ce900 0x7f34480cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 --2- 172.21.15.31:0/4063454315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34480ce3c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 -- 172.21.15.31:0/4063454315 >> 172.21.15.31:0/4063454315 conn(0x7f34480cc910 msgr2=0x7f34480cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:41.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 -- 172.21.15.31:0/4063454315 shutdown_connections 2024-09-13T08:40:41.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.380+0000 7f344dc00640 1 -- 172.21.15.31:0/4063454315 wait complete. 2024-09-13T08:40:41.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.382+0000 7f344dc00640 1 Processor -- start 2024-09-13T08:40:41.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.382+0000 7f344dc00640 1 -- start start 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 0x7f3448155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f344815bd70 0x7f344806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f34481600f0 con 0x7f34480cf850 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34480cb850 con 0x7f344815bd70 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f344dc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3448156250 con 0x7f34480ce900 2024-09-13T08:40:41.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45512/0 (socket says 172.21.15.31:45512) 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 -- 172.21.15.31:0/1810563162 learned_addr learned my addr 172.21.15.31:0/1810563162 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3447400640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f344815bd70 0x7f344806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446a00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 0x7f3448155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 -- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 msgr2=0x7f3448155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 0x7f3448155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 -- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f344815bd70 msgr2=0x7f344806f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f344815bd70 0x7f344806f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446000640 1 -- 172.21.15.31:0/1810563162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34481f5940 con 0x7f34480cf850 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.383+0000 7f3446a00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 0x7f3448155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f3446000640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3434009850 tx=0x7f3434003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343403f070 con 0x7f34480cf850 2024-09-13T08:40:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34340047a0 con 0x7f34480cf850 2024-09-13T08:40:41.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344806f0f0 con 0x7f34480cf850 2024-09-13T08:40:41.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434035030 con 0x7f34480cf850 2024-09-13T08:40:41.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.384+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34480b4090 con 0x7f34480cf850 2024-09-13T08:40:41.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.385+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34480ce3c0 con 0x7f34480cf850 2024-09-13T08:40:41.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.386+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f3434042070 con 0x7f34480cf850 2024-09-13T08:40:41.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.387+0000 7f343ea00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 0x7f342807aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.387+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f34340bbbd0 con 0x7f34480cf850 2024-09-13T08:40:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.388+0000 7f3446a00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 0x7f342807aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.388+0000 7f3446a00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 0x7f342807aef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3438007fd0 tx=0x7f3438009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.392+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3434083850 con 0x7f34480cf850 2024-09-13T08:40:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.590+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34480cc400 con 0x7f34480cf850 2024-09-13T08:40:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.591+0000 7f343ea00640 1 -- 172.21.15.31:0/1810563162 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+16677 (secure 0 0 0) 0x7f3434088280 con 0x7f34480cf850 2024-09-13T08:40:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:41.592 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:38.736850+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:40:35.056540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T08:40:36.137508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:40:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 msgr2=0x7f342807aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 0x7f342807aef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3438007fd0 tx=0x7f3438009040 comp rx=0 tx=0).stop 2024-09-13T08:40:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 msgr2=0x7f34481f5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3434009850 tx=0x7f3434003910 comp rx=0 tx=0).stop 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 shutdown_connections 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f344815bd70 0x7f344806f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34480cf850 0x7f34481f5400 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3428078a30 0x7f342807aef0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 --2- 172.21.15.31:0/1810563162 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34480ce900 0x7f3448155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 >> 172.21.15.31:0/1810563162 conn(0x7f34480cc910 msgr2=0x7f34480cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 shutdown_connections 2024-09-13T08:40:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.593+0000 7f344dc00640 1 -- 172.21.15.31:0/1810563162 wait complete. 2024-09-13T08:40:41.611 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:40:41.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:40:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 -- 172.21.15.31:0/2684915268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd2015bf80 msgr2=0x7fdd200cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 --2- 172.21.15.31:0/2684915268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd2015bf80 0x7fdd200cfe10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdd140098e0 tx=0x7fdd1402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 -- 172.21.15.31:0/2684915268 shutdown_connections 2024-09-13T08:40:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 --2- 172.21.15.31:0/2684915268 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd200d2940 0x7fdd200d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 --2- 172.21.15.31:0/2684915268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd2015bf80 0x7fdd200cfe10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 -- 172.21.15.31:0/2684915268 >> 172.21.15.31:0/2684915268 conn(0x7fdd200d1e20 msgr2=0x7fdd200d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 -- 172.21.15.31:0/2684915268 shutdown_connections 2024-09-13T08:40:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.772+0000 7fdd27800640 1 -- 172.21.15.31:0/2684915268 wait complete. 2024-09-13T08:40:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.774+0000 7fdd27800640 1 Processor -- start 2024-09-13T08:40:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.774+0000 7fdd27800640 1 -- start start 2024-09-13T08:40:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 0x7fdd20148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 0x7fdd201f8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200cdc40 con 0x7fdd200d2940 2024-09-13T08:40:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200cdac0 con 0x7fdd200c70b0 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd27800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdd200c7c50 con 0x7fdd2015bf80 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45524/0 (socket says 172.21.15.31:45524) 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd1fe00640 1 -- 172.21.15.31:0/2407506768 learned_addr learned my addr 172.21.15.31:0/2407506768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd24e00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 0x7fdd20148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.776+0000 7fdd25800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 0x7fdd201f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 -- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 msgr2=0x7fdd201f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 0x7fdd201f8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 -- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 msgr2=0x7fdd20148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 0x7fdd20148ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 -- 172.21.15.31:0/2407506768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd201f9150 con 0x7fdd200d2940 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd25800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 0x7fdd201f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd24e00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 0x7fdd20148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1fe00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e9f0 tx=0x7fdd0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00cd80 con 0x7fdd200d2940 2024-09-13T08:40:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd0c010430 con 0x7fdd200d2940 2024-09-13T08:40:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c010840 con 0x7fdd200d2940 2024-09-13T08:40:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd201f9430 con 0x7fdd200d2940 2024-09-13T08:40:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.777+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd20148780 con 0x7fdd200d2940 2024-09-13T08:40:41.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.778+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd200c8110 con 0x7fdd200d2940 2024-09-13T08:40:41.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.780+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fdd0c0109e0 con 0x7fdd200d2940 2024-09-13T08:40:41.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.780+0000 7fdd1cc00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 0x7fdd0007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:41.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.781+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fdd0c09b950 con 0x7fdd200d2940 2024-09-13T08:40:41.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.781+0000 7fdd24e00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 0x7fdd0007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:41.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.782+0000 7fdd24e00640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 0x7fdd0007aec0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdd14034000 tx=0x7fdd14004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:41.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.785+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdd0c063c50 con 0x7fdd200d2940 2024-09-13T08:40:41.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.979+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd200d2700 con 0x7fdd00078a00 2024-09-13T08:40:41.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.985+0000 7fdd1cc00640 1 -- 172.21.15.31:0/2407506768 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251528 (secure 0 0 0) 0x7fdd200d2700 con 0x7fdd00078a00 2024-09-13T08:40:41.989 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:41.991 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:40:41.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 msgr2=0x7fdd0007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 0x7fdd0007aec0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdd14034000 tx=0x7fdd14004730 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 msgr2=0x7fdd2006f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e9f0 tx=0x7fdd0c00eeb0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 shutdown_connections 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdd00078a00 0x7fdd0007aec0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd2015bf80 0x7fdd201f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd200d2940 0x7fdd2006f5f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 --2- 172.21.15.31:0/2407506768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd200c70b0 0x7fdd20148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 >> 172.21.15.31:0/2407506768 conn(0x7fdd200d1e20 msgr2=0x7fdd200c5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.989+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 shutdown_connections 2024-09-13T08:40:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:41.990+0000 7fdd27800640 1 -- 172.21.15.31:0/2407506768 wait complete. 2024-09-13T08:40:42.008 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-13T08:40:41.090232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248052,"kb_used_data":6832,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726668,"statfs":{"total":773094113280,"available":772840108032,"internally_reserved":0,"allocated":6995968,"data_stored":3560893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.374100"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.761134+0000","last_change":"2024-09-13T08:40:37.757129+0000","last_active":"2024-09-13T08:40:37.761134+0000","last_peered":"2024-09-13T08:40:37.761134+0000","last_clean":"2024-09-13T08:40:37.761134+0000","last_became_active":"2024-09-13T08:40:37.756648+0000","last_became_peered":"2024-09-13T08:40:37.756648+0000","last_unstale":"2024-09-13T08:40:37.761134+0000","last_undegraded":"2024-09-13T08:40:37.761134+0000","last_fullsized":"2024-09-13T08:40:37.761134+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757494+0000","last_change":"2024-09-13T08:40:37.756234+0000","last_active":"2024-09-13T08:40:37.757494+0000","last_peered":"2024-09-13T08:40:37.757494+0000","last_clean":"2024-09-13T08:40:37.757494+0000","last_became_active":"2024-09-13T08:40:37.755925+0000","last_became_peered":"2024-09-13T08:40:37.755925+0000","last_unstale":"2024-09-13T08:40:37.757494+0000","last_undegraded":"2024-09-13T08:40:37.757494+0000","last_fullsized":"2024-09-13T08:40:37.757494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:24.421420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760116+0000","last_change":"2024-09-13T08:40:37.756481+0000","last_active":"2024-09-13T08:40:37.760116+0000","last_peered":"2024-09-13T08:40:37.760116+0000","last_clean":"2024-09-13T08:40:37.760116+0000","last_became_active":"2024-09-13T08:40:37.756190+0000","last_became_peered":"2024-09-13T08:40:37.756190+0000","last_unstale":"2024-09-13T08:40:37.760116+0000","last_undegraded":"2024-09-13T08:40:37.760116+0000","last_fullsized":"2024-09-13T08:40:37.760116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:45.588595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759390+0000","last_change":"2024-09-13T08:40:37.752214+0000","last_active":"2024-09-13T08:40:37.759390+0000","last_peered":"2024-09-13T08:40:37.759390+0000","last_clean":"2024-09-13T08:40:37.759390+0000","last_became_active":"2024-09-13T08:40:37.752015+0000","last_became_peered":"2024-09-13T08:40:37.752015+0000","last_unstale":"2024-09-13T08:40:37.759390+0000","last_undegraded":"2024-09-13T08:40:37.759390+0000","last_fullsized":"2024-09-13T08:40:37.759390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:19.281735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755493+0000","last_change":"2024-09-13T08:40:37.749549+0000","last_active":"2024-09-13T08:40:37.755493+0000","last_peered":"2024-09-13T08:40:37.755493+0000","last_clean":"2024-09-13T08:40:37.755493+0000","last_became_active":"2024-09-13T08:40:37.748944+0000","last_became_peered":"2024-09-13T08:40:37.748944+0000","last_unstale":"2024-09-13T08:40:37.755493+0000","last_undegraded":"2024-09-13T08:40:37.755493+0000","last_fullsized":"2024-09-13T08:40:37.755493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:05.311605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762020+0000","last_change":"2024-09-13T08:40:37.756210+0000","last_active":"2024-09-13T08:40:37.762020+0000","last_peered":"2024-09-13T08:40:37.762020+0000","last_clean":"2024-09-13T08:40:37.762020+0000","last_became_active":"2024-09-13T08:40:37.755876+0000","last_became_peered":"2024-09-13T08:40:37.755876+0000","last_unstale":"2024-09-13T08:40:37.762020+0000","last_undegraded":"2024-09-13T08:40:37.762020+0000","last_fullsized":"2024-09-13T08:40:37.762020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:37.401016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743263+0000","last_change":"2024-09-13T08:40:37.738548+0000","last_active":"2024-09-13T08:40:37.743263+0000","last_peered":"2024-09-13T08:40:37.743263+0000","last_clean":"2024-09-13T08:40:37.743263+0000","last_became_active":"2024-09-13T08:40:37.738294+0000","last_became_peered":"2024-09-13T08:40:37.738294+0000","last_unstale":"2024-09-13T08:40:37.743263+0000","last_undegraded":"2024-09-13T08:40:37.743263+0000","last_fullsized":"2024-09-13T08:40:37.743263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:11.567595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758242+0000","last_change":"2024-09-13T08:40:37.752144+0000","last_active":"2024-09-13T08:40:37.758242+0000","last_peered":"2024-09-13T08:40:37.758242+0000","last_clean":"2024-09-13T08:40:37.758242+0000","last_became_active":"2024-09-13T08:40:37.751574+0000","last_became_peered":"2024-09-13T08:40:37.751574+0000","last_unstale":"2024-09-13T08:40:37.758242+0000","last_undegraded":"2024-09-13T08:40:37.758242+0000","last_fullsized":"2024-09-13T08:40:37.758242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.900728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765529+0000","last_change":"2024-09-13T08:40:37.759047+0000","last_active":"2024-09-13T08:40:37.765529+0000","last_peered":"2024-09-13T08:40:37.765529+0000","last_clean":"2024-09-13T08:40:37.765529+0000","last_became_active":"2024-09-13T08:40:37.757616+0000","last_became_peered":"2024-09-13T08:40:37.757616+0000","last_unstale":"2024-09-13T08:40:37.765529+0000","last_undegraded":"2024-09-13T08:40:37.765529+0000","last_fullsized":"2024-09-13T08:40:37.765529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769285+0000","last_change":"2024-09-13T08:40:37.766302+0000","last_active":"2024-09-13T08:40:37.769285+0000","last_peered":"2024-09-13T08:40:37.769285+0000","last_clean":"2024-09-13T08:40:37.769285+0000","last_became_active":"2024-09-13T08:40:37.765985+0000","last_became_peered":"2024-09-13T08:40:37.765985+0000","last_unstale":"2024-09-13T08:40:37.769285+0000","last_undegraded":"2024-09-13T08:40:37.769285+0000","last_fullsized":"2024-09-13T08:40:37.769285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.726360+0000","last_change":"2024-09-13T08:40:36.726360+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.726360+0000","last_undegraded":"2024-09-13T08:40:36.726360+0000","last_fullsized":"2024-09-13T08:40:36.726360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767321+0000","last_change":"2024-09-13T08:40:37.762085+0000","last_active":"2024-09-13T08:40:37.767321+0000","last_peered":"2024-09-13T08:40:37.767321+0000","last_clean":"2024-09-13T08:40:37.767321+0000","last_became_active":"2024-09-13T08:40:37.761803+0000","last_became_peered":"2024-09-13T08:40:37.761803+0000","last_unstale":"2024-09-13T08:40:37.767321+0000","last_undegraded":"2024-09-13T08:40:37.767321+0000","last_fullsized":"2024-09-13T08:40:37.767321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760491+0000","last_change":"2024-09-13T08:40:37.742869+0000","last_active":"2024-09-13T08:40:37.760491+0000","last_peered":"2024-09-13T08:40:37.760491+0000","last_clean":"2024-09-13T08:40:37.760491+0000","last_became_active":"2024-09-13T08:40:37.742566+0000","last_became_peered":"2024-09-13T08:40:37.742566+0000","last_unstale":"2024-09-13T08:40:37.760491+0000","last_undegraded":"2024-09-13T08:40:37.760491+0000","last_fullsized":"2024-09-13T08:40:37.760491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:40.715704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755923+0000","last_change":"2024-09-13T08:40:37.750839+0000","last_active":"2024-09-13T08:40:37.755923+0000","last_peered":"2024-09-13T08:40:37.755923+0000","last_clean":"2024-09-13T08:40:37.755923+0000","last_became_active":"2024-09-13T08:40:37.750572+0000","last_became_peered":"2024-09-13T08:40:37.750572+0000","last_unstale":"2024-09-13T08:40:37.755923+0000","last_undegraded":"2024-09-13T08:40:37.755923+0000","last_fullsized":"2024-09-13T08:40:37.755923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764455+0000","last_change":"2024-09-13T08:40:37.755720+0000","last_active":"2024-09-13T08:40:37.764455+0000","last_peered":"2024-09-13T08:40:37.764455+0000","last_clean":"2024-09-13T08:40:37.764455+0000","last_became_active":"2024-09-13T08:40:37.755419+0000","last_became_peered":"2024-09-13T08:40:37.755419+0000","last_unstale":"2024-09-13T08:40:37.764455+0000","last_undegraded":"2024-09-13T08:40:37.764455+0000","last_fullsized":"2024-09-13T08:40:37.764455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:41.234982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756494+0000","last_change":"2024-09-13T08:40:37.748027+0000","last_active":"2024-09-13T08:40:37.756494+0000","last_peered":"2024-09-13T08:40:37.756494+0000","last_clean":"2024-09-13T08:40:37.756494+0000","last_became_active":"2024-09-13T08:40:37.747736+0000","last_became_peered":"2024-09-13T08:40:37.747736+0000","last_unstale":"2024-09-13T08:40:37.756494+0000","last_undegraded":"2024-09-13T08:40:37.756494+0000","last_fullsized":"2024-09-13T08:40:37.756494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:31.800897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763918+0000","last_change":"2024-09-13T08:40:37.759006+0000","last_active":"2024-09-13T08:40:37.763918+0000","last_peered":"2024-09-13T08:40:37.763918+0000","last_clean":"2024-09-13T08:40:37.763918+0000","last_became_active":"2024-09-13T08:40:37.758700+0000","last_became_peered":"2024-09-13T08:40:37.758700+0000","last_unstale":"2024-09-13T08:40:37.763918+0000","last_undegraded":"2024-09-13T08:40:37.763918+0000","last_fullsized":"2024-09-13T08:40:37.763918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.757199+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.756901+0000","last_became_peered":"2024-09-13T08:40:37.756901+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756708+0000","last_change":"2024-09-13T08:40:37.752015+0000","last_active":"2024-09-13T08:40:37.756708+0000","last_peered":"2024-09-13T08:40:37.756708+0000","last_clean":"2024-09-13T08:40:37.756708+0000","last_became_active":"2024-09-13T08:40:37.751710+0000","last_became_peered":"2024-09-13T08:40:37.751710+0000","last_unstale":"2024-09-13T08:40:37.756708+0000","last_undegraded":"2024-09-13T08:40:37.756708+0000","last_fullsized":"2024-09-13T08:40:37.756708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727096+0000","last_change":"2024-09-13T08:40:36.727096+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727096+0000","last_undegraded":"2024-09-13T08:40:36.727096+0000","last_fullsized":"2024-09-13T08:40:36.727096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764451+0000","last_change":"2024-09-13T08:40:37.751434+0000","last_active":"2024-09-13T08:40:37.764451+0000","last_peered":"2024-09-13T08:40:37.764451+0000","last_clean":"2024-09-13T08:40:37.764451+0000","last_became_active":"2024-09-13T08:40:37.745985+0000","last_became_peered":"2024-09-13T08:40:37.745985+0000","last_unstale":"2024-09-13T08:40:37.764451+0000","last_undegraded":"2024-09-13T08:40:37.764451+0000","last_fullsized":"2024-09-13T08:40:37.764451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:33.990893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727284+0000","last_change":"2024-09-13T08:40:36.727284+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727284+0000","last_undegraded":"2024-09-13T08:40:36.727284+0000","last_fullsized":"2024-09-13T08:40:36.727284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740716+0000","last_change":"2024-09-13T08:40:37.734978+0000","last_active":"2024-09-13T08:40:37.740716+0000","last_peered":"2024-09-13T08:40:37.740716+0000","last_clean":"2024-09-13T08:40:37.740716+0000","last_became_active":"2024-09-13T08:40:37.734070+0000","last_became_peered":"2024-09-13T08:40:37.734070+0000","last_unstale":"2024-09-13T08:40:37.740716+0000","last_undegraded":"2024-09-13T08:40:37.740716+0000","last_fullsized":"2024-09-13T08:40:37.740716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:54.518342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755838+0000","last_change":"2024-09-13T08:40:37.750441+0000","last_active":"2024-09-13T08:40:37.755838+0000","last_peered":"2024-09-13T08:40:37.755838+0000","last_clean":"2024-09-13T08:40:37.755838+0000","last_became_active":"2024-09-13T08:40:37.750132+0000","last_became_peered":"2024-09-13T08:40:37.750132+0000","last_unstale":"2024-09-13T08:40:37.755838+0000","last_undegraded":"2024-09-13T08:40:37.755838+0000","last_fullsized":"2024-09-13T08:40:37.755838+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755293+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.755293+0000","last_peered":"2024-09-13T08:40:37.755293+0000","last_clean":"2024-09-13T08:40:37.755293+0000","last_became_active":"2024-09-13T08:40:37.750728+0000","last_became_peered":"2024-09-13T08:40:37.750728+0000","last_unstale":"2024-09-13T08:40:37.755293+0000","last_undegraded":"2024-09-13T08:40:37.755293+0000","last_fullsized":"2024-09-13T08:40:37.755293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.741441+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.737794+0000","last_became_peered":"2024-09-13T08:40:37.737794+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:41.031994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755203+0000","last_change":"2024-09-13T08:40:37.748613+0000","last_active":"2024-09-13T08:40:37.755203+0000","last_peered":"2024-09-13T08:40:37.755203+0000","last_clean":"2024-09-13T08:40:37.755203+0000","last_became_active":"2024-09-13T08:40:37.745423+0000","last_became_peered":"2024-09-13T08:40:37.745423+0000","last_unstale":"2024-09-13T08:40:37.755203+0000","last_undegraded":"2024-09-13T08:40:37.755203+0000","last_fullsized":"2024-09-13T08:40:37.755203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765146+0000","last_change":"2024-09-13T08:40:37.762367+0000","last_active":"2024-09-13T08:40:37.765146+0000","last_peered":"2024-09-13T08:40:37.765146+0000","last_clean":"2024-09-13T08:40:37.765146+0000","last_became_active":"2024-09-13T08:40:37.762030+0000","last_became_peered":"2024-09-13T08:40:37.762030+0000","last_unstale":"2024-09-13T08:40:37.765146+0000","last_undegraded":"2024-09-13T08:40:37.765146+0000","last_fullsized":"2024-09-13T08:40:37.765146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:20.588520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758045+0000","last_change":"2024-09-13T08:40:37.750469+0000","last_active":"2024-09-13T08:40:37.758045+0000","last_peered":"2024-09-13T08:40:37.758045+0000","last_clean":"2024-09-13T08:40:37.758045+0000","last_became_active":"2024-09-13T08:40:37.748954+0000","last_became_peered":"2024-09-13T08:40:37.748954+0000","last_unstale":"2024-09-13T08:40:37.758045+0000","last_undegraded":"2024-09-13T08:40:37.758045+0000","last_fullsized":"2024-09-13T08:40:37.758045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:04.282317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747600+0000","last_change":"2024-09-13T08:40:37.744539+0000","last_active":"2024-09-13T08:40:37.747600+0000","last_peered":"2024-09-13T08:40:37.747600+0000","last_clean":"2024-09-13T08:40:37.747600+0000","last_became_active":"2024-09-13T08:40:37.744247+0000","last_became_peered":"2024-09-13T08:40:37.744247+0000","last_unstale":"2024-09-13T08:40:37.747600+0000","last_undegraded":"2024-09-13T08:40:37.747600+0000","last_fullsized":"2024-09-13T08:40:37.747600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:14.087231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756561+0000","last_change":"2024-09-13T08:40:37.750709+0000","last_active":"2024-09-13T08:40:37.756561+0000","last_peered":"2024-09-13T08:40:37.756561+0000","last_clean":"2024-09-13T08:40:37.756561+0000","last_became_active":"2024-09-13T08:40:37.749124+0000","last_became_peered":"2024-09-13T08:40:37.749124+0000","last_unstale":"2024-09-13T08:40:37.756561+0000","last_undegraded":"2024-09-13T08:40:37.756561+0000","last_fullsized":"2024-09-13T08:40:37.756561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.728732+0000","last_change":"2024-09-13T08:40:36.728732+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.728732+0000","last_undegraded":"2024-09-13T08:40:36.728732+0000","last_fullsized":"2024-09-13T08:40:36.728732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763275+0000","last_change":"2024-09-13T08:40:37.758027+0000","last_active":"2024-09-13T08:40:37.763275+0000","last_peered":"2024-09-13T08:40:37.763275+0000","last_clean":"2024-09-13T08:40:37.763275+0000","last_became_active":"2024-09-13T08:40:37.757576+0000","last_became_peered":"2024-09-13T08:40:37.757576+0000","last_unstale":"2024-09-13T08:40:37.763275+0000","last_undegraded":"2024-09-13T08:40:37.763275+0000","last_fullsized":"2024-09-13T08:40:37.763275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.744179+0000","last_change":"2024-09-13T08:40:36.740867+0000","last_active":"2024-09-13T08:40:36.744179+0000","last_peered":"2024-09-13T08:40:36.744179+0000","last_clean":"2024-09-13T08:40:36.744179+0000","last_became_active":"2024-09-13T08:40:36.740475+0000","last_became_peered":"2024-09-13T08:40:36.740475+0000","last_unstale":"2024-09-13T08:40:36.744179+0000","last_undegraded":"2024-09-13T08:40:36.744179+0000","last_fullsized":"2024-09-13T08:40:36.744179+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:00.821617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763350+0000","last_change":"2024-09-13T08:40:37.755669+0000","last_active":"2024-09-13T08:40:37.763350+0000","last_peered":"2024-09-13T08:40:37.763350+0000","last_clean":"2024-09-13T08:40:37.763350+0000","last_became_active":"2024-09-13T08:40:37.755390+0000","last_became_peered":"2024-09-13T08:40:37.755390+0000","last_unstale":"2024-09-13T08:40:37.763350+0000","last_undegraded":"2024-09-13T08:40:37.763350+0000","last_fullsized":"2024-09-13T08:40:37.763350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770118+0000","last_change":"2024-09-13T08:40:36.749525+0000","last_active":"2024-09-13T08:40:37.770118+0000","last_peered":"2024-09-13T08:40:37.770118+0000","last_clean":"2024-09-13T08:40:37.770118+0000","last_became_active":"2024-09-13T08:40:36.749236+0000","last_became_peered":"2024-09-13T08:40:36.749236+0000","last_unstale":"2024-09-13T08:40:37.770118+0000","last_undegraded":"2024-09-13T08:40:37.770118+0000","last_fullsized":"2024-09-13T08:40:37.770118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759658+0000","last_change":"2024-09-13T08:40:37.758005+0000","last_active":"2024-09-13T08:40:37.759658+0000","last_peered":"2024-09-13T08:40:37.759658+0000","last_clean":"2024-09-13T08:40:37.759658+0000","last_became_active":"2024-09-13T08:40:37.757312+0000","last_became_peered":"2024-09-13T08:40:37.757312+0000","last_unstale":"2024-09-13T08:40:37.759658+0000","last_undegraded":"2024-09-13T08:40:37.759658+0000","last_fullsized":"2024-09-13T08:40:37.759658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:18:18.952772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769133+0000","last_change":"2024-09-13T08:40:36.751851+0000","last_active":"2024-09-13T08:40:37.769133+0000","last_peered":"2024-09-13T08:40:37.769133+0000","last_clean":"2024-09-13T08:40:37.769133+0000","last_became_active":"2024-09-13T08:40:36.748632+0000","last_became_peered":"2024-09-13T08:40:36.748632+0000","last_unstale":"2024-09-13T08:40:37.769133+0000","last_undegraded":"2024-09-13T08:40:37.769133+0000","last_fullsized":"2024-09-13T08:40:37.769133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:56.150885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757513+0000","last_change":"2024-09-13T08:40:37.751225+0000","last_active":"2024-09-13T08:40:37.757513+0000","last_peered":"2024-09-13T08:40:37.757513+0000","last_clean":"2024-09-13T08:40:37.757513+0000","last_became_active":"2024-09-13T08:40:37.750923+0000","last_became_peered":"2024-09-13T08:40:37.750923+0000","last_unstale":"2024-09-13T08:40:37.757513+0000","last_undegraded":"2024-09-13T08:40:37.757513+0000","last_fullsized":"2024-09-13T08:40:37.757513+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:21.162327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772713+0000","last_change":"2024-09-13T08:40:36.746077+0000","last_active":"2024-09-13T08:40:37.772713+0000","last_peered":"2024-09-13T08:40:37.772713+0000","last_clean":"2024-09-13T08:40:37.772713+0000","last_became_active":"2024-09-13T08:40:36.745814+0000","last_became_peered":"2024-09-13T08:40:36.745814+0000","last_unstale":"2024-09-13T08:40:37.772713+0000","last_undegraded":"2024-09-13T08:40:37.772713+0000","last_fullsized":"2024-09-13T08:40:37.772713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:24.123182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757961+0000","last_change":"2024-09-13T08:40:37.749662+0000","last_active":"2024-09-13T08:40:37.757961+0000","last_peered":"2024-09-13T08:40:37.757961+0000","last_clean":"2024-09-13T08:40:37.757961+0000","last_became_active":"2024-09-13T08:40:37.749218+0000","last_became_peered":"2024-09-13T08:40:37.749218+0000","last_unstale":"2024-09-13T08:40:37.757961+0000","last_undegraded":"2024-09-13T08:40:37.757961+0000","last_fullsized":"2024-09-13T08:40:37.757961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:27.923545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769027+0000","last_change":"2024-09-13T08:40:36.738659+0000","last_active":"2024-09-13T08:40:37.769027+0000","last_peered":"2024-09-13T08:40:37.769027+0000","last_clean":"2024-09-13T08:40:37.769027+0000","last_became_active":"2024-09-13T08:40:36.738315+0000","last_became_peered":"2024-09-13T08:40:36.738315+0000","last_unstale":"2024-09-13T08:40:37.769027+0000","last_undegraded":"2024-09-13T08:40:37.769027+0000","last_fullsized":"2024-09-13T08:40:37.769027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:09.635872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727899+0000","last_change":"2024-09-13T08:40:36.727899+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727899+0000","last_undegraded":"2024-09-13T08:40:36.727899+0000","last_fullsized":"2024-09-13T08:40:36.727899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772875+0000","last_change":"2024-09-13T08:40:36.751002+0000","last_active":"2024-09-13T08:40:37.772875+0000","last_peered":"2024-09-13T08:40:37.772875+0000","last_clean":"2024-09-13T08:40:37.772875+0000","last_became_active":"2024-09-13T08:40:36.750731+0000","last_became_peered":"2024-09-13T08:40:36.750731+0000","last_unstale":"2024-09-13T08:40:37.772875+0000","last_undegraded":"2024-09-13T08:40:37.772875+0000","last_fullsized":"2024-09-13T08:40:37.772875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:24.257806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.742278+0000","last_change":"2024-09-13T08:40:37.736587+0000","last_active":"2024-09-13T08:40:37.742278+0000","last_peered":"2024-09-13T08:40:37.742278+0000","last_clean":"2024-09-13T08:40:37.742278+0000","last_became_active":"2024-09-13T08:40:37.734709+0000","last_became_peered":"2024-09-13T08:40:37.734709+0000","last_unstale":"2024-09-13T08:40:37.742278+0000","last_undegraded":"2024-09-13T08:40:37.742278+0000","last_fullsized":"2024-09-13T08:40:37.742278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730713+0000","last_change":"2024-09-13T08:40:36.740922+0000","last_active":"2024-09-13T08:40:37.730713+0000","last_peered":"2024-09-13T08:40:37.730713+0000","last_clean":"2024-09-13T08:40:37.730713+0000","last_became_active":"2024-09-13T08:40:36.740610+0000","last_became_peered":"2024-09-13T08:40:36.740610+0000","last_unstale":"2024-09-13T08:40:37.730713+0000","last_undegraded":"2024-09-13T08:40:37.730713+0000","last_fullsized":"2024-09-13T08:40:37.730713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758374+0000","last_change":"2024-09-13T08:40:37.753714+0000","last_active":"2024-09-13T08:40:37.758374+0000","last_peered":"2024-09-13T08:40:37.758374+0000","last_clean":"2024-09-13T08:40:37.758374+0000","last_became_active":"2024-09-13T08:40:37.753402+0000","last_became_peered":"2024-09-13T08:40:37.753402+0000","last_unstale":"2024-09-13T08:40:37.758374+0000","last_undegraded":"2024-09-13T08:40:37.758374+0000","last_fullsized":"2024-09-13T08:40:37.758374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:10.642715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731269+0000","last_change":"2024-09-13T08:40:36.737107+0000","last_active":"2024-09-13T08:40:37.731269+0000","last_peered":"2024-09-13T08:40:37.731269+0000","last_clean":"2024-09-13T08:40:37.731269+0000","last_became_active":"2024-09-13T08:40:36.736458+0000","last_became_peered":"2024-09-13T08:40:36.736458+0000","last_unstale":"2024-09-13T08:40:37.731269+0000","last_undegraded":"2024-09-13T08:40:37.731269+0000","last_fullsized":"2024-09-13T08:40:37.731269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:13.644488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.751369+0000","last_change":"2024-09-13T08:40:37.736394+0000","last_active":"2024-09-13T08:40:37.751369+0000","last_peered":"2024-09-13T08:40:37.751369+0000","last_clean":"2024-09-13T08:40:37.751369+0000","last_became_active":"2024-09-13T08:40:37.736142+0000","last_became_peered":"2024-09-13T08:40:37.736142+0000","last_unstale":"2024-09-13T08:40:37.751369+0000","last_undegraded":"2024-09-13T08:40:37.751369+0000","last_fullsized":"2024-09-13T08:40:37.751369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:00.761695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.733495+0000","last_change":"2024-09-13T08:40:36.740047+0000","last_active":"2024-09-13T08:40:37.733495+0000","last_peered":"2024-09-13T08:40:37.733495+0000","last_clean":"2024-09-13T08:40:37.733495+0000","last_became_active":"2024-09-13T08:40:36.739774+0000","last_became_peered":"2024-09-13T08:40:36.739774+0000","last_unstale":"2024-09-13T08:40:37.733495+0000","last_undegraded":"2024-09-13T08:40:37.733495+0000","last_fullsized":"2024-09-13T08:40:37.733495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760164+0000","last_change":"2024-09-13T08:40:37.757097+0000","last_active":"2024-09-13T08:40:37.760164+0000","last_peered":"2024-09-13T08:40:37.760164+0000","last_clean":"2024-09-13T08:40:37.760164+0000","last_became_active":"2024-09-13T08:40:37.756526+0000","last_became_peered":"2024-09-13T08:40:37.756526+0000","last_unstale":"2024-09-13T08:40:37.760164+0000","last_undegraded":"2024-09-13T08:40:37.760164+0000","last_fullsized":"2024-09-13T08:40:37.760164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735925+0000","last_change":"2024-09-13T08:40:36.743002+0000","last_active":"2024-09-13T08:40:37.735925+0000","last_peered":"2024-09-13T08:40:37.735925+0000","last_clean":"2024-09-13T08:40:37.735925+0000","last_became_active":"2024-09-13T08:40:36.740918+0000","last_became_peered":"2024-09-13T08:40:36.740918+0000","last_unstale":"2024-09-13T08:40:37.735925+0000","last_undegraded":"2024-09-13T08:40:37.735925+0000","last_fullsized":"2024-09-13T08:40:37.735925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:05.024141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759635+0000","last_change":"2024-09-13T08:40:37.756377+0000","last_active":"2024-09-13T08:40:37.759635+0000","last_peered":"2024-09-13T08:40:37.759635+0000","last_clean":"2024-09-13T08:40:37.759635+0000","last_became_active":"2024-09-13T08:40:37.756086+0000","last_became_peered":"2024-09-13T08:40:37.756086+0000","last_unstale":"2024-09-13T08:40:37.759635+0000","last_undegraded":"2024-09-13T08:40:37.759635+0000","last_fullsized":"2024-09-13T08:40:37.759635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:41.096839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772900+0000","last_change":"2024-09-13T08:40:36.745979+0000","last_active":"2024-09-13T08:40:37.772900+0000","last_peered":"2024-09-13T08:40:37.772900+0000","last_clean":"2024-09-13T08:40:37.772900+0000","last_became_active":"2024-09-13T08:40:36.745673+0000","last_became_peered":"2024-09-13T08:40:36.745673+0000","last_unstale":"2024-09-13T08:40:37.772900+0000","last_undegraded":"2024-09-13T08:40:37.772900+0000","last_fullsized":"2024-09-13T08:40:37.772900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:11.780765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759430+0000","last_change":"2024-09-13T08:40:37.752368+0000","last_active":"2024-09-13T08:40:37.759430+0000","last_peered":"2024-09-13T08:40:37.759430+0000","last_clean":"2024-09-13T08:40:37.759430+0000","last_became_active":"2024-09-13T08:40:37.752096+0000","last_became_peered":"2024-09-13T08:40:37.752096+0000","last_unstale":"2024-09-13T08:40:37.759430+0000","last_undegraded":"2024-09-13T08:40:37.759430+0000","last_fullsized":"2024-09-13T08:40:37.759430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:33.675957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776405+0000","last_change":"2024-09-13T08:40:36.745140+0000","last_active":"2024-09-13T08:40:37.776405+0000","last_peered":"2024-09-13T08:40:37.776405+0000","last_clean":"2024-09-13T08:40:37.776405+0000","last_became_active":"2024-09-13T08:40:36.743817+0000","last_became_peered":"2024-09-13T08:40:36.743817+0000","last_unstale":"2024-09-13T08:40:37.776405+0000","last_undegraded":"2024-09-13T08:40:37.776405+0000","last_fullsized":"2024-09-13T08:40:37.776405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:15.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745728+0000","last_change":"2024-09-13T08:40:37.740915+0000","last_active":"2024-09-13T08:40:37.745728+0000","last_peered":"2024-09-13T08:40:37.745728+0000","last_clean":"2024-09-13T08:40:37.745728+0000","last_became_active":"2024-09-13T08:40:37.740617+0000","last_became_peered":"2024-09-13T08:40:37.740617+0000","last_unstale":"2024-09-13T08:40:37.745728+0000","last_undegraded":"2024-09-13T08:40:37.745728+0000","last_fullsized":"2024-09-13T08:40:37.745728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770825+0000","last_change":"2024-09-13T08:40:36.740132+0000","last_active":"2024-09-13T08:40:37.770825+0000","last_peered":"2024-09-13T08:40:37.770825+0000","last_clean":"2024-09-13T08:40:37.770825+0000","last_became_active":"2024-09-13T08:40:36.739787+0000","last_became_peered":"2024-09-13T08:40:36.739787+0000","last_unstale":"2024-09-13T08:40:37.770825+0000","last_undegraded":"2024-09-13T08:40:37.770825+0000","last_fullsized":"2024-09-13T08:40:37.770825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.013701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759244+0000","last_change":"2024-09-13T08:40:37.754029+0000","last_active":"2024-09-13T08:40:37.759244+0000","last_peered":"2024-09-13T08:40:37.759244+0000","last_clean":"2024-09-13T08:40:37.759244+0000","last_became_active":"2024-09-13T08:40:37.753773+0000","last_became_peered":"2024-09-13T08:40:37.753773+0000","last_unstale":"2024-09-13T08:40:37.759244+0000","last_undegraded":"2024-09-13T08:40:37.759244+0000","last_fullsized":"2024-09-13T08:40:37.759244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727972+0000","last_change":"2024-09-13T08:40:36.746192+0000","last_active":"2024-09-13T08:40:37.727972+0000","last_peered":"2024-09-13T08:40:37.727972+0000","last_clean":"2024-09-13T08:40:37.727972+0000","last_became_active":"2024-09-13T08:40:36.745703+0000","last_became_peered":"2024-09-13T08:40:36.745703+0000","last_unstale":"2024-09-13T08:40:37.727972+0000","last_undegraded":"2024-09-13T08:40:37.727972+0000","last_fullsized":"2024-09-13T08:40:37.727972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:00.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755812+0000","last_change":"2024-09-13T08:40:37.750252+0000","last_active":"2024-09-13T08:40:37.755812+0000","last_peered":"2024-09-13T08:40:37.755812+0000","last_clean":"2024-09-13T08:40:37.755812+0000","last_became_active":"2024-09-13T08:40:37.749793+0000","last_became_peered":"2024-09-13T08:40:37.749793+0000","last_unstale":"2024-09-13T08:40:37.755812+0000","last_undegraded":"2024-09-13T08:40:37.755812+0000","last_fullsized":"2024-09-13T08:40:37.755812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727573+0000","last_change":"2024-09-13T08:40:36.748499+0000","last_active":"2024-09-13T08:40:37.727573+0000","last_peered":"2024-09-13T08:40:37.727573+0000","last_clean":"2024-09-13T08:40:37.727573+0000","last_became_active":"2024-09-13T08:40:36.745832+0000","last_became_peered":"2024-09-13T08:40:36.745832+0000","last_unstale":"2024-09-13T08:40:37.727573+0000","last_undegraded":"2024-09-13T08:40:37.727573+0000","last_fullsized":"2024-09-13T08:40:37.727573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:42.819908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740690+0000","last_change":"2024-09-13T08:40:37.734550+0000","last_active":"2024-09-13T08:40:37.740690+0000","last_peered":"2024-09-13T08:40:37.740690+0000","last_clean":"2024-09-13T08:40:37.740690+0000","last_became_active":"2024-09-13T08:40:37.734140+0000","last_became_peered":"2024-09-13T08:40:37.734140+0000","last_unstale":"2024-09-13T08:40:37.740690+0000","last_undegraded":"2024-09-13T08:40:37.740690+0000","last_fullsized":"2024-09-13T08:40:37.740690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:11.251715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":21,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776743+0000","last_change":"2024-09-13T08:40:36.746654+0000","last_active":"2024-09-13T08:40:37.776743+0000","last_peered":"2024-09-13T08:40:37.776743+0000","last_clean":"2024-09-13T08:40:37.776743+0000","last_became_active":"2024-09-13T08:40:36.746397+0000","last_became_peered":"2024-09-13T08:40:36.746397+0000","last_unstale":"2024-09-13T08:40:37.776743+0000","last_undegraded":"2024-09-13T08:40:37.776743+0000","last_fullsized":"2024-09-13T08:40:37.776743+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:15.203955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747535+0000","last_change":"2024-09-13T08:40:37.745284+0000","last_active":"2024-09-13T08:40:37.747535+0000","last_peered":"2024-09-13T08:40:37.747535+0000","last_clean":"2024-09-13T08:40:37.747535+0000","last_became_active":"2024-09-13T08:40:37.744957+0000","last_became_peered":"2024-09-13T08:40:37.744957+0000","last_unstale":"2024-09-13T08:40:37.747535+0000","last_undegraded":"2024-09-13T08:40:37.747535+0000","last_fullsized":"2024-09-13T08:40:37.747535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767493+0000","last_change":"2024-09-13T08:40:36.753006+0000","last_active":"2024-09-13T08:40:37.767493+0000","last_peered":"2024-09-13T08:40:37.767493+0000","last_clean":"2024-09-13T08:40:37.767493+0000","last_became_active":"2024-09-13T08:40:36.752498+0000","last_became_peered":"2024-09-13T08:40:36.752498+0000","last_unstale":"2024-09-13T08:40:37.767493+0000","last_undegraded":"2024-09-13T08:40:37.767493+0000","last_fullsized":"2024-09-13T08:40:37.767493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760187+0000","last_change":"2024-09-13T08:40:37.757310+0000","last_active":"2024-09-13T08:40:37.760187+0000","last_peered":"2024-09-13T08:40:37.760187+0000","last_clean":"2024-09-13T08:40:37.760187+0000","last_became_active":"2024-09-13T08:40:37.756852+0000","last_became_peered":"2024-09-13T08:40:37.756852+0000","last_unstale":"2024-09-13T08:40:37.760187+0000","last_undegraded":"2024-09-13T08:40:37.760187+0000","last_fullsized":"2024-09-13T08:40:37.760187+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770763+0000","last_change":"2024-09-13T08:40:36.741233+0000","last_active":"2024-09-13T08:40:37.770763+0000","last_peered":"2024-09-13T08:40:37.770763+0000","last_clean":"2024-09-13T08:40:37.770763+0000","last_became_active":"2024-09-13T08:40:36.740574+0000","last_became_peered":"2024-09-13T08:40:36.740574+0000","last_unstale":"2024-09-13T08:40:37.770763+0000","last_undegraded":"2024-09-13T08:40:37.770763+0000","last_fullsized":"2024-09-13T08:40:37.770763+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:52.377019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762842+0000","last_change":"2024-09-13T08:40:37.761727+0000","last_active":"2024-09-13T08:40:37.762842+0000","last_peered":"2024-09-13T08:40:37.762842+0000","last_clean":"2024-09-13T08:40:37.762842+0000","last_became_active":"2024-09-13T08:40:37.761016+0000","last_became_peered":"2024-09-13T08:40:37.761016+0000","last_unstale":"2024-09-13T08:40:37.762842+0000","last_undegraded":"2024-09-13T08:40:37.762842+0000","last_fullsized":"2024-09-13T08:40:37.762842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:58:06.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769868+0000","last_change":"2024-09-13T08:40:36.740608+0000","last_active":"2024-09-13T08:40:37.769868+0000","last_peered":"2024-09-13T08:40:37.769868+0000","last_clean":"2024-09-13T08:40:37.769868+0000","last_became_active":"2024-09-13T08:40:36.740349+0000","last_became_peered":"2024-09-13T08:40:36.740349+0000","last_unstale":"2024-09-13T08:40:37.769868+0000","last_undegraded":"2024-09-13T08:40:37.769868+0000","last_fullsized":"2024-09-13T08:40:37.769868+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.279565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762124+0000","last_change":"2024-09-13T08:40:37.758748+0000","last_active":"2024-09-13T08:40:37.762124+0000","last_peered":"2024-09-13T08:40:37.762124+0000","last_clean":"2024-09-13T08:40:37.762124+0000","last_became_active":"2024-09-13T08:40:37.756975+0000","last_became_peered":"2024-09-13T08:40:37.756975+0000","last_unstale":"2024-09-13T08:40:37.762124+0000","last_undegraded":"2024-09-13T08:40:37.762124+0000","last_fullsized":"2024-09-13T08:40:37.762124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:43.311241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736532+0000","last_change":"2024-09-13T08:40:36.748868+0000","last_active":"2024-09-13T08:40:37.736532+0000","last_peered":"2024-09-13T08:40:37.736532+0000","last_clean":"2024-09-13T08:40:37.736532+0000","last_became_active":"2024-09-13T08:40:36.748527+0000","last_became_peered":"2024-09-13T08:40:36.748527+0000","last_unstale":"2024-09-13T08:40:37.736532+0000","last_undegraded":"2024-09-13T08:40:37.736532+0000","last_fullsized":"2024-09-13T08:40:37.736532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:05.275938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759771+0000","last_change":"2024-09-13T08:40:37.756050+0000","last_active":"2024-09-13T08:40:37.759771+0000","last_peered":"2024-09-13T08:40:37.759771+0000","last_clean":"2024-09-13T08:40:37.759771+0000","last_became_active":"2024-09-13T08:40:37.755788+0000","last_became_peered":"2024-09-13T08:40:37.755788+0000","last_unstale":"2024-09-13T08:40:37.759771+0000","last_undegraded":"2024-09-13T08:40:37.759771+0000","last_fullsized":"2024-09-13T08:40:37.759771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.741357+0000","last_change":"2024-09-13T08:40:36.737698+0000","last_active":"2024-09-13T08:40:36.741357+0000","last_peered":"2024-09-13T08:40:36.741357+0000","last_clean":"2024-09-13T08:40:36.741357+0000","last_became_active":"2024-09-13T08:40:36.737306+0000","last_became_peered":"2024-09-13T08:40:36.737306+0000","last_unstale":"2024-09-13T08:40:36.741357+0000","last_undegraded":"2024-09-13T08:40:36.741357+0000","last_fullsized":"2024-09-13T08:40:36.741357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:51:28.448035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758021+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.758021+0000","last_peered":"2024-09-13T08:40:37.758021+0000","last_clean":"2024-09-13T08:40:37.758021+0000","last_became_active":"2024-09-13T08:40:37.750682+0000","last_became_peered":"2024-09-13T08:40:37.750682+0000","last_unstale":"2024-09-13T08:40:37.758021+0000","last_undegraded":"2024-09-13T08:40:37.758021+0000","last_fullsized":"2024-09-13T08:40:37.758021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:12.637948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736909+0000","last_change":"2024-09-13T08:40:36.749086+0000","last_active":"2024-09-13T08:40:37.736909+0000","last_peered":"2024-09-13T08:40:37.736909+0000","last_clean":"2024-09-13T08:40:37.736909+0000","last_became_active":"2024-09-13T08:40:36.747729+0000","last_became_peered":"2024-09-13T08:40:36.747729+0000","last_unstale":"2024-09-13T08:40:37.736909+0000","last_undegraded":"2024-09-13T08:40:37.736909+0000","last_fullsized":"2024-09-13T08:40:37.736909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:12:48.295035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767464+0000","last_change":"2024-09-13T08:40:37.765328+0000","last_active":"2024-09-13T08:40:37.767464+0000","last_peered":"2024-09-13T08:40:37.767464+0000","last_clean":"2024-09-13T08:40:37.767464+0000","last_became_active":"2024-09-13T08:40:37.762317+0000","last_became_peered":"2024-09-13T08:40:37.762317+0000","last_unstale":"2024-09-13T08:40:37.767464+0000","last_undegraded":"2024-09-13T08:40:37.767464+0000","last_fullsized":"2024-09-13T08:40:37.767464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:29:55.562604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732662+0000","last_change":"2024-09-13T08:40:36.747739+0000","last_active":"2024-09-13T08:40:37.732662+0000","last_peered":"2024-09-13T08:40:37.732662+0000","last_clean":"2024-09-13T08:40:37.732662+0000","last_became_active":"2024-09-13T08:40:36.747366+0000","last_became_peered":"2024-09-13T08:40:36.747366+0000","last_unstale":"2024-09-13T08:40:37.732662+0000","last_undegraded":"2024-09-13T08:40:37.732662+0000","last_fullsized":"2024-09-13T08:40:37.732662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.748364+0000","last_change":"2024-09-13T08:40:37.746566+0000","last_active":"2024-09-13T08:40:37.748364+0000","last_peered":"2024-09-13T08:40:37.748364+0000","last_clean":"2024-09-13T08:40:37.748364+0000","last_became_active":"2024-09-13T08:40:37.746382+0000","last_became_peered":"2024-09-13T08:40:37.746382+0000","last_unstale":"2024-09-13T08:40:37.748364+0000","last_undegraded":"2024-09-13T08:40:37.748364+0000","last_fullsized":"2024-09-13T08:40:37.748364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.768664+0000","last_change":"2024-09-13T08:40:36.743612+0000","last_active":"2024-09-13T08:40:37.768664+0000","last_peered":"2024-09-13T08:40:37.768664+0000","last_clean":"2024-09-13T08:40:37.768664+0000","last_became_active":"2024-09-13T08:40:36.743337+0000","last_became_peered":"2024-09-13T08:40:36.743337+0000","last_unstale":"2024-09-13T08:40:37.768664+0000","last_undegraded":"2024-09-13T08:40:37.768664+0000","last_fullsized":"2024-09-13T08:40:37.768664+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747771+0000","last_change":"2024-09-13T08:40:37.746100+0000","last_active":"2024-09-13T08:40:37.747771+0000","last_peered":"2024-09-13T08:40:37.747771+0000","last_clean":"2024-09-13T08:40:37.747771+0000","last_became_active":"2024-09-13T08:40:37.745801+0000","last_became_peered":"2024-09-13T08:40:37.745801+0000","last_unstale":"2024-09-13T08:40:37.747771+0000","last_undegraded":"2024-09-13T08:40:37.747771+0000","last_fullsized":"2024-09-13T08:40:37.747771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745934+0000","last_change":"2024-09-13T08:40:36.751068+0000","last_active":"2024-09-13T08:40:37.745934+0000","last_peered":"2024-09-13T08:40:37.745934+0000","last_clean":"2024-09-13T08:40:37.745934+0000","last_became_active":"2024-09-13T08:40:36.748029+0000","last_became_peered":"2024-09-13T08:40:36.748029+0000","last_unstale":"2024-09-13T08:40:37.745934+0000","last_undegraded":"2024-09-13T08:40:37.745934+0000","last_fullsized":"2024-09-13T08:40:37.745934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:25.694691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.749201+0000","last_change":"2024-09-13T08:40:37.743421+0000","last_active":"2024-09-13T08:40:37.749201+0000","last_peered":"2024-09-13T08:40:37.749201+0000","last_clean":"2024-09-13T08:40:37.749201+0000","last_became_active":"2024-09-13T08:40:37.743126+0000","last_became_peered":"2024-09-13T08:40:37.743126+0000","last_unstale":"2024-09-13T08:40:37.749201+0000","last_undegraded":"2024-09-13T08:40:37.749201+0000","last_fullsized":"2024-09-13T08:40:37.749201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731313+0000","last_change":"2024-09-13T08:40:36.742793+0000","last_active":"2024-09-13T08:40:37.731313+0000","last_peered":"2024-09-13T08:40:37.731313+0000","last_clean":"2024-09-13T08:40:37.731313+0000","last_became_active":"2024-09-13T08:40:36.742516+0000","last_became_peered":"2024-09-13T08:40:36.742516+0000","last_unstale":"2024-09-13T08:40:37.731313+0000","last_undegraded":"2024-09-13T08:40:37.731313+0000","last_fullsized":"2024-09-13T08:40:37.731313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767726+0000","last_change":"2024-09-13T08:40:37.763959+0000","last_active":"2024-09-13T08:40:37.767726+0000","last_peered":"2024-09-13T08:40:37.767726+0000","last_clean":"2024-09-13T08:40:37.767726+0000","last_became_active":"2024-09-13T08:40:37.763661+0000","last_became_peered":"2024-09-13T08:40:37.763661+0000","last_unstale":"2024-09-13T08:40:37.767726+0000","last_undegraded":"2024-09-13T08:40:37.767726+0000","last_fullsized":"2024-09-13T08:40:37.767726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:46.895173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735588+0000","last_change":"2024-09-13T08:40:36.750833+0000","last_active":"2024-09-13T08:40:37.735588+0000","last_peered":"2024-09-13T08:40:37.735588+0000","last_clean":"2024-09-13T08:40:37.735588+0000","last_became_active":"2024-09-13T08:40:36.750471+0000","last_became_peered":"2024-09-13T08:40:36.750471+0000","last_unstale":"2024-09-13T08:40:37.735588+0000","last_undegraded":"2024-09-13T08:40:37.735588+0000","last_fullsized":"2024-09-13T08:40:37.735588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:36.394174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762046+0000","last_change":"2024-09-13T08:40:37.756241+0000","last_active":"2024-09-13T08:40:37.762046+0000","last_peered":"2024-09-13T08:40:37.762046+0000","last_clean":"2024-09-13T08:40:37.762046+0000","last_became_active":"2024-09-13T08:40:37.755961+0000","last_became_peered":"2024-09-13T08:40:37.755961+0000","last_unstale":"2024-09-13T08:40:37.762046+0000","last_undegraded":"2024-09-13T08:40:37.762046+0000","last_fullsized":"2024-09-13T08:40:37.762046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:38.286408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730472+0000","last_change":"2024-09-13T08:40:36.750422+0000","last_active":"2024-09-13T08:40:37.730472+0000","last_peered":"2024-09-13T08:40:37.730472+0000","last_clean":"2024-09-13T08:40:37.730472+0000","last_became_active":"2024-09-13T08:40:36.749512+0000","last_became_peered":"2024-09-13T08:40:36.749512+0000","last_unstale":"2024-09-13T08:40:37.730472+0000","last_undegraded":"2024-09-13T08:40:37.730472+0000","last_fullsized":"2024-09-13T08:40:37.730472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764478+0000","last_change":"2024-09-13T08:40:37.756404+0000","last_active":"2024-09-13T08:40:37.764478+0000","last_peered":"2024-09-13T08:40:37.764478+0000","last_clean":"2024-09-13T08:40:37.764478+0000","last_became_active":"2024-09-13T08:40:37.755037+0000","last_became_peered":"2024-09-13T08:40:37.755037+0000","last_unstale":"2024-09-13T08:40:37.764478+0000","last_undegraded":"2024-09-13T08:40:37.764478+0000","last_fullsized":"2024-09-13T08:40:37.764478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:11.905503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732259+0000","last_change":"2024-09-13T08:40:36.740510+0000","last_active":"2024-09-13T08:40:37.732259+0000","last_peered":"2024-09-13T08:40:37.732259+0000","last_clean":"2024-09-13T08:40:37.732259+0000","last_became_active":"2024-09-13T08:40:36.740243+0000","last_became_peered":"2024-09-13T08:40:36.740243+0000","last_unstale":"2024-09-13T08:40:37.732259+0000","last_undegraded":"2024-09-13T08:40:37.732259+0000","last_fullsized":"2024-09-13T08:40:37.732259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:20.155484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758212+0000","last_change":"2024-09-13T08:40:37.751854+0000","last_active":"2024-09-13T08:40:37.758212+0000","last_peered":"2024-09-13T08:40:37.758212+0000","last_clean":"2024-09-13T08:40:37.758212+0000","last_became_active":"2024-09-13T08:40:37.749368+0000","last_became_peered":"2024-09-13T08:40:37.749368+0000","last_unstale":"2024-09-13T08:40:37.758212+0000","last_undegraded":"2024-09-13T08:40:37.758212+0000","last_fullsized":"2024-09-13T08:40:37.758212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:05.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743930+0000","last_change":"2024-09-13T08:40:36.743024+0000","last_active":"2024-09-13T08:40:37.743930+0000","last_peered":"2024-09-13T08:40:37.743930+0000","last_clean":"2024-09-13T08:40:37.743930+0000","last_became_active":"2024-09-13T08:40:36.742739+0000","last_became_peered":"2024-09-13T08:40:36.742739+0000","last_unstale":"2024-09-13T08:40:37.743930+0000","last_undegraded":"2024-09-13T08:40:37.743930+0000","last_fullsized":"2024-09-13T08:40:37.743930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:54.197613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.753493+0000","last_change":"2024-09-13T08:40:37.743948+0000","last_active":"2024-09-13T08:40:37.753493+0000","last_peered":"2024-09-13T08:40:37.753493+0000","last_clean":"2024-09-13T08:40:37.753493+0000","last_became_active":"2024-09-13T08:40:37.743656+0000","last_became_peered":"2024-09-13T08:40:37.743656+0000","last_unstale":"2024-09-13T08:40:37.753493+0000","last_undegraded":"2024-09-13T08:40:37.753493+0000","last_fullsized":"2024-09-13T08:40:37.753493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735241+0000","last_change":"2024-09-13T08:40:36.750854+0000","last_active":"2024-09-13T08:40:37.735241+0000","last_peered":"2024-09-13T08:40:37.735241+0000","last_clean":"2024-09-13T08:40:37.735241+0000","last_became_active":"2024-09-13T08:40:36.750177+0000","last_became_peered":"2024-09-13T08:40:36.750177+0000","last_unstale":"2024-09-13T08:40:37.735241+0000","last_undegraded":"2024-09-13T08:40:37.735241+0000","last_fullsized":"2024-09-13T08:40:37.735241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:20.695241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.754437+0000","last_change":"2024-09-13T08:40:37.746462+0000","last_active":"2024-09-13T08:40:37.754437+0000","last_peered":"2024-09-13T08:40:37.754437+0000","last_clean":"2024-09-13T08:40:37.754437+0000","last_became_active":"2024-09-13T08:40:37.745179+0000","last_became_peered":"2024-09-13T08:40:37.745179+0000","last_unstale":"2024-09-13T08:40:37.754437+0000","last_undegraded":"2024-09-13T08:40:37.754437+0000","last_fullsized":"2024-09-13T08:40:37.754437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:07.391500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.737843+0000","last_change":"2024-09-13T08:40:36.744466+0000","last_active":"2024-09-13T08:40:37.737843+0000","last_peered":"2024-09-13T08:40:37.737843+0000","last_clean":"2024-09-13T08:40:37.737843+0000","last_became_active":"2024-09-13T08:40:36.744206+0000","last_became_peered":"2024-09-13T08:40:36.744206+0000","last_unstale":"2024-09-13T08:40:37.737843+0000","last_undegraded":"2024-09-13T08:40:37.737843+0000","last_fullsized":"2024-09-13T08:40:37.737843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:40.300855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":179,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732740+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:40:37.732740+0000","last_peered":"2024-09-13T08:40:37.732740+0000","last_clean":"2024-09-13T08:40:37.732740+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:40:37.732740+0000","last_undegraded":"2024-09-13T08:40:37.732740+0000","last_fullsized":"2024-09-13T08:40:37.732740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":299641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509487,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1323008,"data_stored":890021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":729088,"data_stored":299709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":745472,"data_stored":301391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1306624,"data_stored":883212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509486,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":737280,"data_stored":301231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36048,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:40:42.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:40:42.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 -- 172.21.15.31:0/2545744555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80cc720 msgr2=0x7fbda806f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 --2- 172.21.15.31:0/2545744555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80cc720 0x7fbda806f9b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd98005fd0 tx=0x7fbd9802a8b0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 -- 172.21.15.31:0/2545744555 shutdown_connections 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 --2- 172.21.15.31:0/2545744555 >> v2:172.21.15.45:3300/0 conn(0x7fbda80ccb00 0x7fbda80b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 --2- 172.21.15.31:0/2545744555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80cc720 0x7fbda806f9b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 -- 172.21.15.31:0/2545744555 >> 172.21.15.31:0/2545744555 conn(0x7fbda806cb00 msgr2=0x7fbda806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.206+0000 7fbdacc00640 1 -- 172.21.15.31:0/2545744555 shutdown_connections 2024-09-13T08:40:42.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.207+0000 7fbdacc00640 1 -- 172.21.15.31:0/2545744555 wait complete. 2024-09-13T08:40:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.208+0000 7fbdacc00640 1 Processor -- start 2024-09-13T08:40:42.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.208+0000 7fbdacc00640 1 -- start start 2024-09-13T08:40:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80701a0 0x7fbda81f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbda80ccb00 0x7fbda81558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbda80b5490 con 0x7fbda80cc720 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbda80b5310 con 0x7fbda80ccb00 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbdacc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbda8152d90 con 0x7fbda80701a0 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbd9ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbd9ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45534/0 (socket says 172.21.15.31:45534) 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.209+0000 7fbd9ea00640 1 -- 172.21.15.31:0/3562434016 learned_addr learned my addr 172.21.15.31:0/3562434016 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbda7e00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbda80ccb00 0x7fbda81558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbda7400640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80701a0 0x7fbda81f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 -- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80701a0 msgr2=0x7fbda81f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80701a0 0x7fbda81f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 -- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbda80ccb00 msgr2=0x7fbda81558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbda80ccb00 0x7fbda81558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 -- 172.21.15.31:0/3562434016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda81f2870 con 0x7fbda80cc720 2024-09-13T08:40:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9ea00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000e7f0 tx=0x7fbd9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.210+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9000cc60 con 0x7fbda80cc720 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.211+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd9000ce00 con 0x7fbda80cc720 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.211+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda81f2b50 con 0x7fbda80cc720 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.211+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90004280 con 0x7fbda80cc720 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.211+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda81f3630 con 0x7fbda80cc720 2024-09-13T08:40:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.212+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda80b1420 con 0x7fbda80cc720 2024-09-13T08:40:42.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.213+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fbd900035f0 con 0x7fbda80cc720 2024-09-13T08:40:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.214+0000 7fbd9fe00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 0x7fbd8407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.214+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fbd9009b510 con 0x7fbda80cc720 2024-09-13T08:40:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.214+0000 7fbda7400640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 0x7fbd8407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.215+0000 7fbda7400640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 0x7fbd8407aec0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd98034000 tx=0x7fbd98004ab0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.218+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbd90063190 con 0x7fbda80cc720 2024-09-13T08:40:42.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.408+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbda806f090 con 0x7fbd84078a00 2024-09-13T08:40:42.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.414+0000 7fbd9fe00640 1 -- 172.21.15.31:0/3562434016 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251528 (secure 0 0 0) 0x7fbda806f090 con 0x7fbd84078a00 2024-09-13T08:40:42.418 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 msgr2=0x7fbd8407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 0x7fbd8407aec0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd98034000 tx=0x7fbd98004ab0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 msgr2=0x7fbda80aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000e7f0 tx=0x7fbd9000ecb0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 shutdown_connections 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbda80ccb00 0x7fbda81558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda80cc720 0x7fbda80aa500 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbd84078a00 0x7fbd8407aec0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 --2- 172.21.15.31:0/3562434016 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbda80701a0 0x7fbda81f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 >> 172.21.15.31:0/3562434016 conn(0x7fbda806cb00 msgr2=0x7fbda80c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 shutdown_connections 2024-09-13T08:40:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.418+0000 7fbdacc00640 1 -- 172.21.15.31:0/3562434016 wait complete. 2024-09-13T08:40:42.434 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-13T08:40:41.090232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248052,"kb_used_data":6832,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726668,"statfs":{"total":773094113280,"available":772840108032,"internally_reserved":0,"allocated":6995968,"data_stored":3560893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.374100"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.761134+0000","last_change":"2024-09-13T08:40:37.757129+0000","last_active":"2024-09-13T08:40:37.761134+0000","last_peered":"2024-09-13T08:40:37.761134+0000","last_clean":"2024-09-13T08:40:37.761134+0000","last_became_active":"2024-09-13T08:40:37.756648+0000","last_became_peered":"2024-09-13T08:40:37.756648+0000","last_unstale":"2024-09-13T08:40:37.761134+0000","last_undegraded":"2024-09-13T08:40:37.761134+0000","last_fullsized":"2024-09-13T08:40:37.761134+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757494+0000","last_change":"2024-09-13T08:40:37.756234+0000","last_active":"2024-09-13T08:40:37.757494+0000","last_peered":"2024-09-13T08:40:37.757494+0000","last_clean":"2024-09-13T08:40:37.757494+0000","last_became_active":"2024-09-13T08:40:37.755925+0000","last_became_peered":"2024-09-13T08:40:37.755925+0000","last_unstale":"2024-09-13T08:40:37.757494+0000","last_undegraded":"2024-09-13T08:40:37.757494+0000","last_fullsized":"2024-09-13T08:40:37.757494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:24.421420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760116+0000","last_change":"2024-09-13T08:40:37.756481+0000","last_active":"2024-09-13T08:40:37.760116+0000","last_peered":"2024-09-13T08:40:37.760116+0000","last_clean":"2024-09-13T08:40:37.760116+0000","last_became_active":"2024-09-13T08:40:37.756190+0000","last_became_peered":"2024-09-13T08:40:37.756190+0000","last_unstale":"2024-09-13T08:40:37.760116+0000","last_undegraded":"2024-09-13T08:40:37.760116+0000","last_fullsized":"2024-09-13T08:40:37.760116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:45.588595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759390+0000","last_change":"2024-09-13T08:40:37.752214+0000","last_active":"2024-09-13T08:40:37.759390+0000","last_peered":"2024-09-13T08:40:37.759390+0000","last_clean":"2024-09-13T08:40:37.759390+0000","last_became_active":"2024-09-13T08:40:37.752015+0000","last_became_peered":"2024-09-13T08:40:37.752015+0000","last_unstale":"2024-09-13T08:40:37.759390+0000","last_undegraded":"2024-09-13T08:40:37.759390+0000","last_fullsized":"2024-09-13T08:40:37.759390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:19.281735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755493+0000","last_change":"2024-09-13T08:40:37.749549+0000","last_active":"2024-09-13T08:40:37.755493+0000","last_peered":"2024-09-13T08:40:37.755493+0000","last_clean":"2024-09-13T08:40:37.755493+0000","last_became_active":"2024-09-13T08:40:37.748944+0000","last_became_peered":"2024-09-13T08:40:37.748944+0000","last_unstale":"2024-09-13T08:40:37.755493+0000","last_undegraded":"2024-09-13T08:40:37.755493+0000","last_fullsized":"2024-09-13T08:40:37.755493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:05.311605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762020+0000","last_change":"2024-09-13T08:40:37.756210+0000","last_active":"2024-09-13T08:40:37.762020+0000","last_peered":"2024-09-13T08:40:37.762020+0000","last_clean":"2024-09-13T08:40:37.762020+0000","last_became_active":"2024-09-13T08:40:37.755876+0000","last_became_peered":"2024-09-13T08:40:37.755876+0000","last_unstale":"2024-09-13T08:40:37.762020+0000","last_undegraded":"2024-09-13T08:40:37.762020+0000","last_fullsized":"2024-09-13T08:40:37.762020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:37.401016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743263+0000","last_change":"2024-09-13T08:40:37.738548+0000","last_active":"2024-09-13T08:40:37.743263+0000","last_peered":"2024-09-13T08:40:37.743263+0000","last_clean":"2024-09-13T08:40:37.743263+0000","last_became_active":"2024-09-13T08:40:37.738294+0000","last_became_peered":"2024-09-13T08:40:37.738294+0000","last_unstale":"2024-09-13T08:40:37.743263+0000","last_undegraded":"2024-09-13T08:40:37.743263+0000","last_fullsized":"2024-09-13T08:40:37.743263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:11.567595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758242+0000","last_change":"2024-09-13T08:40:37.752144+0000","last_active":"2024-09-13T08:40:37.758242+0000","last_peered":"2024-09-13T08:40:37.758242+0000","last_clean":"2024-09-13T08:40:37.758242+0000","last_became_active":"2024-09-13T08:40:37.751574+0000","last_became_peered":"2024-09-13T08:40:37.751574+0000","last_unstale":"2024-09-13T08:40:37.758242+0000","last_undegraded":"2024-09-13T08:40:37.758242+0000","last_fullsized":"2024-09-13T08:40:37.758242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.900728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765529+0000","last_change":"2024-09-13T08:40:37.759047+0000","last_active":"2024-09-13T08:40:37.765529+0000","last_peered":"2024-09-13T08:40:37.765529+0000","last_clean":"2024-09-13T08:40:37.765529+0000","last_became_active":"2024-09-13T08:40:37.757616+0000","last_became_peered":"2024-09-13T08:40:37.757616+0000","last_unstale":"2024-09-13T08:40:37.765529+0000","last_undegraded":"2024-09-13T08:40:37.765529+0000","last_fullsized":"2024-09-13T08:40:37.765529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769285+0000","last_change":"2024-09-13T08:40:37.766302+0000","last_active":"2024-09-13T08:40:37.769285+0000","last_peered":"2024-09-13T08:40:37.769285+0000","last_clean":"2024-09-13T08:40:37.769285+0000","last_became_active":"2024-09-13T08:40:37.765985+0000","last_became_peered":"2024-09-13T08:40:37.765985+0000","last_unstale":"2024-09-13T08:40:37.769285+0000","last_undegraded":"2024-09-13T08:40:37.769285+0000","last_fullsized":"2024-09-13T08:40:37.769285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.726360+0000","last_change":"2024-09-13T08:40:36.726360+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.726360+0000","last_undegraded":"2024-09-13T08:40:36.726360+0000","last_fullsized":"2024-09-13T08:40:36.726360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767321+0000","last_change":"2024-09-13T08:40:37.762085+0000","last_active":"2024-09-13T08:40:37.767321+0000","last_peered":"2024-09-13T08:40:37.767321+0000","last_clean":"2024-09-13T08:40:37.767321+0000","last_became_active":"2024-09-13T08:40:37.761803+0000","last_became_peered":"2024-09-13T08:40:37.761803+0000","last_unstale":"2024-09-13T08:40:37.767321+0000","last_undegraded":"2024-09-13T08:40:37.767321+0000","last_fullsized":"2024-09-13T08:40:37.767321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760491+0000","last_change":"2024-09-13T08:40:37.742869+0000","last_active":"2024-09-13T08:40:37.760491+0000","last_peered":"2024-09-13T08:40:37.760491+0000","last_clean":"2024-09-13T08:40:37.760491+0000","last_became_active":"2024-09-13T08:40:37.742566+0000","last_became_peered":"2024-09-13T08:40:37.742566+0000","last_unstale":"2024-09-13T08:40:37.760491+0000","last_undegraded":"2024-09-13T08:40:37.760491+0000","last_fullsized":"2024-09-13T08:40:37.760491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:40.715704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755923+0000","last_change":"2024-09-13T08:40:37.750839+0000","last_active":"2024-09-13T08:40:37.755923+0000","last_peered":"2024-09-13T08:40:37.755923+0000","last_clean":"2024-09-13T08:40:37.755923+0000","last_became_active":"2024-09-13T08:40:37.750572+0000","last_became_peered":"2024-09-13T08:40:37.750572+0000","last_unstale":"2024-09-13T08:40:37.755923+0000","last_undegraded":"2024-09-13T08:40:37.755923+0000","last_fullsized":"2024-09-13T08:40:37.755923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764455+0000","last_change":"2024-09-13T08:40:37.755720+0000","last_active":"2024-09-13T08:40:37.764455+0000","last_peered":"2024-09-13T08:40:37.764455+0000","last_clean":"2024-09-13T08:40:37.764455+0000","last_became_active":"2024-09-13T08:40:37.755419+0000","last_became_peered":"2024-09-13T08:40:37.755419+0000","last_unstale":"2024-09-13T08:40:37.764455+0000","last_undegraded":"2024-09-13T08:40:37.764455+0000","last_fullsized":"2024-09-13T08:40:37.764455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:41.234982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756494+0000","last_change":"2024-09-13T08:40:37.748027+0000","last_active":"2024-09-13T08:40:37.756494+0000","last_peered":"2024-09-13T08:40:37.756494+0000","last_clean":"2024-09-13T08:40:37.756494+0000","last_became_active":"2024-09-13T08:40:37.747736+0000","last_became_peered":"2024-09-13T08:40:37.747736+0000","last_unstale":"2024-09-13T08:40:37.756494+0000","last_undegraded":"2024-09-13T08:40:37.756494+0000","last_fullsized":"2024-09-13T08:40:37.756494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:31.800897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763918+0000","last_change":"2024-09-13T08:40:37.759006+0000","last_active":"2024-09-13T08:40:37.763918+0000","last_peered":"2024-09-13T08:40:37.763918+0000","last_clean":"2024-09-13T08:40:37.763918+0000","last_became_active":"2024-09-13T08:40:37.758700+0000","last_became_peered":"2024-09-13T08:40:37.758700+0000","last_unstale":"2024-09-13T08:40:37.763918+0000","last_undegraded":"2024-09-13T08:40:37.763918+0000","last_fullsized":"2024-09-13T08:40:37.763918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.757199+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.756901+0000","last_became_peered":"2024-09-13T08:40:37.756901+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756708+0000","last_change":"2024-09-13T08:40:37.752015+0000","last_active":"2024-09-13T08:40:37.756708+0000","last_peered":"2024-09-13T08:40:37.756708+0000","last_clean":"2024-09-13T08:40:37.756708+0000","last_became_active":"2024-09-13T08:40:37.751710+0000","last_became_peered":"2024-09-13T08:40:37.751710+0000","last_unstale":"2024-09-13T08:40:37.756708+0000","last_undegraded":"2024-09-13T08:40:37.756708+0000","last_fullsized":"2024-09-13T08:40:37.756708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727096+0000","last_change":"2024-09-13T08:40:36.727096+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727096+0000","last_undegraded":"2024-09-13T08:40:36.727096+0000","last_fullsized":"2024-09-13T08:40:36.727096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764451+0000","last_change":"2024-09-13T08:40:37.751434+0000","last_active":"2024-09-13T08:40:37.764451+0000","last_peered":"2024-09-13T08:40:37.764451+0000","last_clean":"2024-09-13T08:40:37.764451+0000","last_became_active":"2024-09-13T08:40:37.745985+0000","last_became_peered":"2024-09-13T08:40:37.745985+0000","last_unstale":"2024-09-13T08:40:37.764451+0000","last_undegraded":"2024-09-13T08:40:37.764451+0000","last_fullsized":"2024-09-13T08:40:37.764451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:33.990893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727284+0000","last_change":"2024-09-13T08:40:36.727284+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727284+0000","last_undegraded":"2024-09-13T08:40:36.727284+0000","last_fullsized":"2024-09-13T08:40:36.727284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740716+0000","last_change":"2024-09-13T08:40:37.734978+0000","last_active":"2024-09-13T08:40:37.740716+0000","last_peered":"2024-09-13T08:40:37.740716+0000","last_clean":"2024-09-13T08:40:37.740716+0000","last_became_active":"2024-09-13T08:40:37.734070+0000","last_became_peered":"2024-09-13T08:40:37.734070+0000","last_unstale":"2024-09-13T08:40:37.740716+0000","last_undegraded":"2024-09-13T08:40:37.740716+0000","last_fullsized":"2024-09-13T08:40:37.740716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:54.518342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755838+0000","last_change":"2024-09-13T08:40:37.750441+0000","last_active":"2024-09-13T08:40:37.755838+0000","last_peered":"2024-09-13T08:40:37.755838+0000","last_clean":"2024-09-13T08:40:37.755838+0000","last_became_active":"2024-09-13T08:40:37.750132+0000","last_became_peered":"2024-09-13T08:40:37.750132+0000","last_unstale":"2024-09-13T08:40:37.755838+0000","last_undegraded":"2024-09-13T08:40:37.755838+0000","last_fullsized":"2024-09-13T08:40:37.755838+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755293+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.755293+0000","last_peered":"2024-09-13T08:40:37.755293+0000","last_clean":"2024-09-13T08:40:37.755293+0000","last_became_active":"2024-09-13T08:40:37.750728+0000","last_became_peered":"2024-09-13T08:40:37.750728+0000","last_unstale":"2024-09-13T08:40:37.755293+0000","last_undegraded":"2024-09-13T08:40:37.755293+0000","last_fullsized":"2024-09-13T08:40:37.755293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.741441+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.737794+0000","last_became_peered":"2024-09-13T08:40:37.737794+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:41.031994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755203+0000","last_change":"2024-09-13T08:40:37.748613+0000","last_active":"2024-09-13T08:40:37.755203+0000","last_peered":"2024-09-13T08:40:37.755203+0000","last_clean":"2024-09-13T08:40:37.755203+0000","last_became_active":"2024-09-13T08:40:37.745423+0000","last_became_peered":"2024-09-13T08:40:37.745423+0000","last_unstale":"2024-09-13T08:40:37.755203+0000","last_undegraded":"2024-09-13T08:40:37.755203+0000","last_fullsized":"2024-09-13T08:40:37.755203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765146+0000","last_change":"2024-09-13T08:40:37.762367+0000","last_active":"2024-09-13T08:40:37.765146+0000","last_peered":"2024-09-13T08:40:37.765146+0000","last_clean":"2024-09-13T08:40:37.765146+0000","last_became_active":"2024-09-13T08:40:37.762030+0000","last_became_peered":"2024-09-13T08:40:37.762030+0000","last_unstale":"2024-09-13T08:40:37.765146+0000","last_undegraded":"2024-09-13T08:40:37.765146+0000","last_fullsized":"2024-09-13T08:40:37.765146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:20.588520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758045+0000","last_change":"2024-09-13T08:40:37.750469+0000","last_active":"2024-09-13T08:40:37.758045+0000","last_peered":"2024-09-13T08:40:37.758045+0000","last_clean":"2024-09-13T08:40:37.758045+0000","last_became_active":"2024-09-13T08:40:37.748954+0000","last_became_peered":"2024-09-13T08:40:37.748954+0000","last_unstale":"2024-09-13T08:40:37.758045+0000","last_undegraded":"2024-09-13T08:40:37.758045+0000","last_fullsized":"2024-09-13T08:40:37.758045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:04.282317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747600+0000","last_change":"2024-09-13T08:40:37.744539+0000","last_active":"2024-09-13T08:40:37.747600+0000","last_peered":"2024-09-13T08:40:37.747600+0000","last_clean":"2024-09-13T08:40:37.747600+0000","last_became_active":"2024-09-13T08:40:37.744247+0000","last_became_peered":"2024-09-13T08:40:37.744247+0000","last_unstale":"2024-09-13T08:40:37.747600+0000","last_undegraded":"2024-09-13T08:40:37.747600+0000","last_fullsized":"2024-09-13T08:40:37.747600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:14.087231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756561+0000","last_change":"2024-09-13T08:40:37.750709+0000","last_active":"2024-09-13T08:40:37.756561+0000","last_peered":"2024-09-13T08:40:37.756561+0000","last_clean":"2024-09-13T08:40:37.756561+0000","last_became_active":"2024-09-13T08:40:37.749124+0000","last_became_peered":"2024-09-13T08:40:37.749124+0000","last_unstale":"2024-09-13T08:40:37.756561+0000","last_undegraded":"2024-09-13T08:40:37.756561+0000","last_fullsized":"2024-09-13T08:40:37.756561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.728732+0000","last_change":"2024-09-13T08:40:36.728732+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.728732+0000","last_undegraded":"2024-09-13T08:40:36.728732+0000","last_fullsized":"2024-09-13T08:40:36.728732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763275+0000","last_change":"2024-09-13T08:40:37.758027+0000","last_active":"2024-09-13T08:40:37.763275+0000","last_peered":"2024-09-13T08:40:37.763275+0000","last_clean":"2024-09-13T08:40:37.763275+0000","last_became_active":"2024-09-13T08:40:37.757576+0000","last_became_peered":"2024-09-13T08:40:37.757576+0000","last_unstale":"2024-09-13T08:40:37.763275+0000","last_undegraded":"2024-09-13T08:40:37.763275+0000","last_fullsized":"2024-09-13T08:40:37.763275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.744179+0000","last_change":"2024-09-13T08:40:36.740867+0000","last_active":"2024-09-13T08:40:36.744179+0000","last_peered":"2024-09-13T08:40:36.744179+0000","last_clean":"2024-09-13T08:40:36.744179+0000","last_became_active":"2024-09-13T08:40:36.740475+0000","last_became_peered":"2024-09-13T08:40:36.740475+0000","last_unstale":"2024-09-13T08:40:36.744179+0000","last_undegraded":"2024-09-13T08:40:36.744179+0000","last_fullsized":"2024-09-13T08:40:36.744179+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:00.821617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763350+0000","last_change":"2024-09-13T08:40:37.755669+0000","last_active":"2024-09-13T08:40:37.763350+0000","last_peered":"2024-09-13T08:40:37.763350+0000","last_clean":"2024-09-13T08:40:37.763350+0000","last_became_active":"2024-09-13T08:40:37.755390+0000","last_became_peered":"2024-09-13T08:40:37.755390+0000","last_unstale":"2024-09-13T08:40:37.763350+0000","last_undegraded":"2024-09-13T08:40:37.763350+0000","last_fullsized":"2024-09-13T08:40:37.763350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770118+0000","last_change":"2024-09-13T08:40:36.749525+0000","last_active":"2024-09-13T08:40:37.770118+0000","last_peered":"2024-09-13T08:40:37.770118+0000","last_clean":"2024-09-13T08:40:37.770118+0000","last_became_active":"2024-09-13T08:40:36.749236+0000","last_became_peered":"2024-09-13T08:40:36.749236+0000","last_unstale":"2024-09-13T08:40:37.770118+0000","last_undegraded":"2024-09-13T08:40:37.770118+0000","last_fullsized":"2024-09-13T08:40:37.770118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759658+0000","last_change":"2024-09-13T08:40:37.758005+0000","last_active":"2024-09-13T08:40:37.759658+0000","last_peered":"2024-09-13T08:40:37.759658+0000","last_clean":"2024-09-13T08:40:37.759658+0000","last_became_active":"2024-09-13T08:40:37.757312+0000","last_became_peered":"2024-09-13T08:40:37.757312+0000","last_unstale":"2024-09-13T08:40:37.759658+0000","last_undegraded":"2024-09-13T08:40:37.759658+0000","last_fullsized":"2024-09-13T08:40:37.759658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:18:18.952772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769133+0000","last_change":"2024-09-13T08:40:36.751851+0000","last_active":"2024-09-13T08:40:37.769133+0000","last_peered":"2024-09-13T08:40:37.769133+0000","last_clean":"2024-09-13T08:40:37.769133+0000","last_became_active":"2024-09-13T08:40:36.748632+0000","last_became_peered":"2024-09-13T08:40:36.748632+0000","last_unstale":"2024-09-13T08:40:37.769133+0000","last_undegraded":"2024-09-13T08:40:37.769133+0000","last_fullsized":"2024-09-13T08:40:37.769133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:56.150885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757513+0000","last_change":"2024-09-13T08:40:37.751225+0000","last_active":"2024-09-13T08:40:37.757513+0000","last_peered":"2024-09-13T08:40:37.757513+0000","last_clean":"2024-09-13T08:40:37.757513+0000","last_became_active":"2024-09-13T08:40:37.750923+0000","last_became_peered":"2024-09-13T08:40:37.750923+0000","last_unstale":"2024-09-13T08:40:37.757513+0000","last_undegraded":"2024-09-13T08:40:37.757513+0000","last_fullsized":"2024-09-13T08:40:37.757513+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:21.162327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772713+0000","last_change":"2024-09-13T08:40:36.746077+0000","last_active":"2024-09-13T08:40:37.772713+0000","last_peered":"2024-09-13T08:40:37.772713+0000","last_clean":"2024-09-13T08:40:37.772713+0000","last_became_active":"2024-09-13T08:40:36.745814+0000","last_became_peered":"2024-09-13T08:40:36.745814+0000","last_unstale":"2024-09-13T08:40:37.772713+0000","last_undegraded":"2024-09-13T08:40:37.772713+0000","last_fullsized":"2024-09-13T08:40:37.772713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:24.123182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757961+0000","last_change":"2024-09-13T08:40:37.749662+0000","last_active":"2024-09-13T08:40:37.757961+0000","last_peered":"2024-09-13T08:40:37.757961+0000","last_clean":"2024-09-13T08:40:37.757961+0000","last_became_active":"2024-09-13T08:40:37.749218+0000","last_became_peered":"2024-09-13T08:40:37.749218+0000","last_unstale":"2024-09-13T08:40:37.757961+0000","last_undegraded":"2024-09-13T08:40:37.757961+0000","last_fullsized":"2024-09-13T08:40:37.757961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:27.923545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769027+0000","last_change":"2024-09-13T08:40:36.738659+0000","last_active":"2024-09-13T08:40:37.769027+0000","last_peered":"2024-09-13T08:40:37.769027+0000","last_clean":"2024-09-13T08:40:37.769027+0000","last_became_active":"2024-09-13T08:40:36.738315+0000","last_became_peered":"2024-09-13T08:40:36.738315+0000","last_unstale":"2024-09-13T08:40:37.769027+0000","last_undegraded":"2024-09-13T08:40:37.769027+0000","last_fullsized":"2024-09-13T08:40:37.769027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:09.635872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727899+0000","last_change":"2024-09-13T08:40:36.727899+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727899+0000","last_undegraded":"2024-09-13T08:40:36.727899+0000","last_fullsized":"2024-09-13T08:40:36.727899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772875+0000","last_change":"2024-09-13T08:40:36.751002+0000","last_active":"2024-09-13T08:40:37.772875+0000","last_peered":"2024-09-13T08:40:37.772875+0000","last_clean":"2024-09-13T08:40:37.772875+0000","last_became_active":"2024-09-13T08:40:36.750731+0000","last_became_peered":"2024-09-13T08:40:36.750731+0000","last_unstale":"2024-09-13T08:40:37.772875+0000","last_undegraded":"2024-09-13T08:40:37.772875+0000","last_fullsized":"2024-09-13T08:40:37.772875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:24.257806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.742278+0000","last_change":"2024-09-13T08:40:37.736587+0000","last_active":"2024-09-13T08:40:37.742278+0000","last_peered":"2024-09-13T08:40:37.742278+0000","last_clean":"2024-09-13T08:40:37.742278+0000","last_became_active":"2024-09-13T08:40:37.734709+0000","last_became_peered":"2024-09-13T08:40:37.734709+0000","last_unstale":"2024-09-13T08:40:37.742278+0000","last_undegraded":"2024-09-13T08:40:37.742278+0000","last_fullsized":"2024-09-13T08:40:37.742278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730713+0000","last_change":"2024-09-13T08:40:36.740922+0000","last_active":"2024-09-13T08:40:37.730713+0000","last_peered":"2024-09-13T08:40:37.730713+0000","last_clean":"2024-09-13T08:40:37.730713+0000","last_became_active":"2024-09-13T08:40:36.740610+0000","last_became_peered":"2024-09-13T08:40:36.740610+0000","last_unstale":"2024-09-13T08:40:37.730713+0000","last_undegraded":"2024-09-13T08:40:37.730713+0000","last_fullsized":"2024-09-13T08:40:37.730713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758374+0000","last_change":"2024-09-13T08:40:37.753714+0000","last_active":"2024-09-13T08:40:37.758374+0000","last_peered":"2024-09-13T08:40:37.758374+0000","last_clean":"2024-09-13T08:40:37.758374+0000","last_became_active":"2024-09-13T08:40:37.753402+0000","last_became_peered":"2024-09-13T08:40:37.753402+0000","last_unstale":"2024-09-13T08:40:37.758374+0000","last_undegraded":"2024-09-13T08:40:37.758374+0000","last_fullsized":"2024-09-13T08:40:37.758374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:10.642715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731269+0000","last_change":"2024-09-13T08:40:36.737107+0000","last_active":"2024-09-13T08:40:37.731269+0000","last_peered":"2024-09-13T08:40:37.731269+0000","last_clean":"2024-09-13T08:40:37.731269+0000","last_became_active":"2024-09-13T08:40:36.736458+0000","last_became_peered":"2024-09-13T08:40:36.736458+0000","last_unstale":"2024-09-13T08:40:37.731269+0000","last_undegraded":"2024-09-13T08:40:37.731269+0000","last_fullsized":"2024-09-13T08:40:37.731269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:13.644488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.751369+0000","last_change":"2024-09-13T08:40:37.736394+0000","last_active":"2024-09-13T08:40:37.751369+0000","last_peered":"2024-09-13T08:40:37.751369+0000","last_clean":"2024-09-13T08:40:37.751369+0000","last_became_active":"2024-09-13T08:40:37.736142+0000","last_became_peered":"2024-09-13T08:40:37.736142+0000","last_unstale":"2024-09-13T08:40:37.751369+0000","last_undegraded":"2024-09-13T08:40:37.751369+0000","last_fullsized":"2024-09-13T08:40:37.751369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:00.761695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.733495+0000","last_change":"2024-09-13T08:40:36.740047+0000","last_active":"2024-09-13T08:40:37.733495+0000","last_peered":"2024-09-13T08:40:37.733495+0000","last_clean":"2024-09-13T08:40:37.733495+0000","last_became_active":"2024-09-13T08:40:36.739774+0000","last_became_peered":"2024-09-13T08:40:36.739774+0000","last_unstale":"2024-09-13T08:40:37.733495+0000","last_undegraded":"2024-09-13T08:40:37.733495+0000","last_fullsized":"2024-09-13T08:40:37.733495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760164+0000","last_change":"2024-09-13T08:40:37.757097+0000","last_active":"2024-09-13T08:40:37.760164+0000","last_peered":"2024-09-13T08:40:37.760164+0000","last_clean":"2024-09-13T08:40:37.760164+0000","last_became_active":"2024-09-13T08:40:37.756526+0000","last_became_peered":"2024-09-13T08:40:37.756526+0000","last_unstale":"2024-09-13T08:40:37.760164+0000","last_undegraded":"2024-09-13T08:40:37.760164+0000","last_fullsized":"2024-09-13T08:40:37.760164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735925+0000","last_change":"2024-09-13T08:40:36.743002+0000","last_active":"2024-09-13T08:40:37.735925+0000","last_peered":"2024-09-13T08:40:37.735925+0000","last_clean":"2024-09-13T08:40:37.735925+0000","last_became_active":"2024-09-13T08:40:36.740918+0000","last_became_peered":"2024-09-13T08:40:36.740918+0000","last_unstale":"2024-09-13T08:40:37.735925+0000","last_undegraded":"2024-09-13T08:40:37.735925+0000","last_fullsized":"2024-09-13T08:40:37.735925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:05.024141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759635+0000","last_change":"2024-09-13T08:40:37.756377+0000","last_active":"2024-09-13T08:40:37.759635+0000","last_peered":"2024-09-13T08:40:37.759635+0000","last_clean":"2024-09-13T08:40:37.759635+0000","last_became_active":"2024-09-13T08:40:37.756086+0000","last_became_peered":"2024-09-13T08:40:37.756086+0000","last_unstale":"2024-09-13T08:40:37.759635+0000","last_undegraded":"2024-09-13T08:40:37.759635+0000","last_fullsized":"2024-09-13T08:40:37.759635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:41.096839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772900+0000","last_change":"2024-09-13T08:40:36.745979+0000","last_active":"2024-09-13T08:40:37.772900+0000","last_peered":"2024-09-13T08:40:37.772900+0000","last_clean":"2024-09-13T08:40:37.772900+0000","last_became_active":"2024-09-13T08:40:36.745673+0000","last_became_peered":"2024-09-13T08:40:36.745673+0000","last_unstale":"2024-09-13T08:40:37.772900+0000","last_undegraded":"2024-09-13T08:40:37.772900+0000","last_fullsized":"2024-09-13T08:40:37.772900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:11.780765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759430+0000","last_change":"2024-09-13T08:40:37.752368+0000","last_active":"2024-09-13T08:40:37.759430+0000","last_peered":"2024-09-13T08:40:37.759430+0000","last_clean":"2024-09-13T08:40:37.759430+0000","last_became_active":"2024-09-13T08:40:37.752096+0000","last_became_peered":"2024-09-13T08:40:37.752096+0000","last_unstale":"2024-09-13T08:40:37.759430+0000","last_undegraded":"2024-09-13T08:40:37.759430+0000","last_fullsized":"2024-09-13T08:40:37.759430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:33.675957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776405+0000","last_change":"2024-09-13T08:40:36.745140+0000","last_active":"2024-09-13T08:40:37.776405+0000","last_peered":"2024-09-13T08:40:37.776405+0000","last_clean":"2024-09-13T08:40:37.776405+0000","last_became_active":"2024-09-13T08:40:36.743817+0000","last_became_peered":"2024-09-13T08:40:36.743817+0000","last_unstale":"2024-09-13T08:40:37.776405+0000","last_undegraded":"2024-09-13T08:40:37.776405+0000","last_fullsized":"2024-09-13T08:40:37.776405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:15.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745728+0000","last_change":"2024-09-13T08:40:37.740915+0000","last_active":"2024-09-13T08:40:37.745728+0000","last_peered":"2024-09-13T08:40:37.745728+0000","last_clean":"2024-09-13T08:40:37.745728+0000","last_became_active":"2024-09-13T08:40:37.740617+0000","last_became_peered":"2024-09-13T08:40:37.740617+0000","last_unstale":"2024-09-13T08:40:37.745728+0000","last_undegraded":"2024-09-13T08:40:37.745728+0000","last_fullsized":"2024-09-13T08:40:37.745728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770825+0000","last_change":"2024-09-13T08:40:36.740132+0000","last_active":"2024-09-13T08:40:37.770825+0000","last_peered":"2024-09-13T08:40:37.770825+0000","last_clean":"2024-09-13T08:40:37.770825+0000","last_became_active":"2024-09-13T08:40:36.739787+0000","last_became_peered":"2024-09-13T08:40:36.739787+0000","last_unstale":"2024-09-13T08:40:37.770825+0000","last_undegraded":"2024-09-13T08:40:37.770825+0000","last_fullsized":"2024-09-13T08:40:37.770825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.013701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759244+0000","last_change":"2024-09-13T08:40:37.754029+0000","last_active":"2024-09-13T08:40:37.759244+0000","last_peered":"2024-09-13T08:40:37.759244+0000","last_clean":"2024-09-13T08:40:37.759244+0000","last_became_active":"2024-09-13T08:40:37.753773+0000","last_became_peered":"2024-09-13T08:40:37.753773+0000","last_unstale":"2024-09-13T08:40:37.759244+0000","last_undegraded":"2024-09-13T08:40:37.759244+0000","last_fullsized":"2024-09-13T08:40:37.759244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727972+0000","last_change":"2024-09-13T08:40:36.746192+0000","last_active":"2024-09-13T08:40:37.727972+0000","last_peered":"2024-09-13T08:40:37.727972+0000","last_clean":"2024-09-13T08:40:37.727972+0000","last_became_active":"2024-09-13T08:40:36.745703+0000","last_became_peered":"2024-09-13T08:40:36.745703+0000","last_unstale":"2024-09-13T08:40:37.727972+0000","last_undegraded":"2024-09-13T08:40:37.727972+0000","last_fullsized":"2024-09-13T08:40:37.727972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:00.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755812+0000","last_change":"2024-09-13T08:40:37.750252+0000","last_active":"2024-09-13T08:40:37.755812+0000","last_peered":"2024-09-13T08:40:37.755812+0000","last_clean":"2024-09-13T08:40:37.755812+0000","last_became_active":"2024-09-13T08:40:37.749793+0000","last_became_peered":"2024-09-13T08:40:37.749793+0000","last_unstale":"2024-09-13T08:40:37.755812+0000","last_undegraded":"2024-09-13T08:40:37.755812+0000","last_fullsized":"2024-09-13T08:40:37.755812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727573+0000","last_change":"2024-09-13T08:40:36.748499+0000","last_active":"2024-09-13T08:40:37.727573+0000","last_peered":"2024-09-13T08:40:37.727573+0000","last_clean":"2024-09-13T08:40:37.727573+0000","last_became_active":"2024-09-13T08:40:36.745832+0000","last_became_peered":"2024-09-13T08:40:36.745832+0000","last_unstale":"2024-09-13T08:40:37.727573+0000","last_undegraded":"2024-09-13T08:40:37.727573+0000","last_fullsized":"2024-09-13T08:40:37.727573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:42.819908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740690+0000","last_change":"2024-09-13T08:40:37.734550+0000","last_active":"2024-09-13T08:40:37.740690+0000","last_peered":"2024-09-13T08:40:37.740690+0000","last_clean":"2024-09-13T08:40:37.740690+0000","last_became_active":"2024-09-13T08:40:37.734140+0000","last_became_peered":"2024-09-13T08:40:37.734140+0000","last_unstale":"2024-09-13T08:40:37.740690+0000","last_undegraded":"2024-09-13T08:40:37.740690+0000","last_fullsized":"2024-09-13T08:40:37.740690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:11.251715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":21,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776743+0000","last_change":"2024-09-13T08:40:36.746654+0000","last_active":"2024-09-13T08:40:37.776743+0000","last_peered":"2024-09-13T08:40:37.776743+0000","last_clean":"2024-09-13T08:40:37.776743+0000","last_became_active":"2024-09-13T08:40:36.746397+0000","last_became_peered":"2024-09-13T08:40:36.746397+0000","last_unstale":"2024-09-13T08:40:37.776743+0000","last_undegraded":"2024-09-13T08:40:37.776743+0000","last_fullsized":"2024-09-13T08:40:37.776743+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:15.203955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747535+0000","last_change":"2024-09-13T08:40:37.745284+0000","last_active":"2024-09-13T08:40:37.747535+0000","last_peered":"2024-09-13T08:40:37.747535+0000","last_clean":"2024-09-13T08:40:37.747535+0000","last_became_active":"2024-09-13T08:40:37.744957+0000","last_became_peered":"2024-09-13T08:40:37.744957+0000","last_unstale":"2024-09-13T08:40:37.747535+0000","last_undegraded":"2024-09-13T08:40:37.747535+0000","last_fullsized":"2024-09-13T08:40:37.747535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767493+0000","last_change":"2024-09-13T08:40:36.753006+0000","last_active":"2024-09-13T08:40:37.767493+0000","last_peered":"2024-09-13T08:40:37.767493+0000","last_clean":"2024-09-13T08:40:37.767493+0000","last_became_active":"2024-09-13T08:40:36.752498+0000","last_became_peered":"2024-09-13T08:40:36.752498+0000","last_unstale":"2024-09-13T08:40:37.767493+0000","last_undegraded":"2024-09-13T08:40:37.767493+0000","last_fullsized":"2024-09-13T08:40:37.767493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760187+0000","last_change":"2024-09-13T08:40:37.757310+0000","last_active":"2024-09-13T08:40:37.760187+0000","last_peered":"2024-09-13T08:40:37.760187+0000","last_clean":"2024-09-13T08:40:37.760187+0000","last_became_active":"2024-09-13T08:40:37.756852+0000","last_became_peered":"2024-09-13T08:40:37.756852+0000","last_unstale":"2024-09-13T08:40:37.760187+0000","last_undegraded":"2024-09-13T08:40:37.760187+0000","last_fullsized":"2024-09-13T08:40:37.760187+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770763+0000","last_change":"2024-09-13T08:40:36.741233+0000","last_active":"2024-09-13T08:40:37.770763+0000","last_peered":"2024-09-13T08:40:37.770763+0000","last_clean":"2024-09-13T08:40:37.770763+0000","last_became_active":"2024-09-13T08:40:36.740574+0000","last_became_peered":"2024-09-13T08:40:36.740574+0000","last_unstale":"2024-09-13T08:40:37.770763+0000","last_undegraded":"2024-09-13T08:40:37.770763+0000","last_fullsized":"2024-09-13T08:40:37.770763+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:52.377019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762842+0000","last_change":"2024-09-13T08:40:37.761727+0000","last_active":"2024-09-13T08:40:37.762842+0000","last_peered":"2024-09-13T08:40:37.762842+0000","last_clean":"2024-09-13T08:40:37.762842+0000","last_became_active":"2024-09-13T08:40:37.761016+0000","last_became_peered":"2024-09-13T08:40:37.761016+0000","last_unstale":"2024-09-13T08:40:37.762842+0000","last_undegraded":"2024-09-13T08:40:37.762842+0000","last_fullsized":"2024-09-13T08:40:37.762842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:58:06.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769868+0000","last_change":"2024-09-13T08:40:36.740608+0000","last_active":"2024-09-13T08:40:37.769868+0000","last_peered":"2024-09-13T08:40:37.769868+0000","last_clean":"2024-09-13T08:40:37.769868+0000","last_became_active":"2024-09-13T08:40:36.740349+0000","last_became_peered":"2024-09-13T08:40:36.740349+0000","last_unstale":"2024-09-13T08:40:37.769868+0000","last_undegraded":"2024-09-13T08:40:37.769868+0000","last_fullsized":"2024-09-13T08:40:37.769868+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.279565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762124+0000","last_change":"2024-09-13T08:40:37.758748+0000","last_active":"2024-09-13T08:40:37.762124+0000","last_peered":"2024-09-13T08:40:37.762124+0000","last_clean":"2024-09-13T08:40:37.762124+0000","last_became_active":"2024-09-13T08:40:37.756975+0000","last_became_peered":"2024-09-13T08:40:37.756975+0000","last_unstale":"2024-09-13T08:40:37.762124+0000","last_undegraded":"2024-09-13T08:40:37.762124+0000","last_fullsized":"2024-09-13T08:40:37.762124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:43.311241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736532+0000","last_change":"2024-09-13T08:40:36.748868+0000","last_active":"2024-09-13T08:40:37.736532+0000","last_peered":"2024-09-13T08:40:37.736532+0000","last_clean":"2024-09-13T08:40:37.736532+0000","last_became_active":"2024-09-13T08:40:36.748527+0000","last_became_peered":"2024-09-13T08:40:36.748527+0000","last_unstale":"2024-09-13T08:40:37.736532+0000","last_undegraded":"2024-09-13T08:40:37.736532+0000","last_fullsized":"2024-09-13T08:40:37.736532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:05.275938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759771+0000","last_change":"2024-09-13T08:40:37.756050+0000","last_active":"2024-09-13T08:40:37.759771+0000","last_peered":"2024-09-13T08:40:37.759771+0000","last_clean":"2024-09-13T08:40:37.759771+0000","last_became_active":"2024-09-13T08:40:37.755788+0000","last_became_peered":"2024-09-13T08:40:37.755788+0000","last_unstale":"2024-09-13T08:40:37.759771+0000","last_undegraded":"2024-09-13T08:40:37.759771+0000","last_fullsized":"2024-09-13T08:40:37.759771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.741357+0000","last_change":"2024-09-13T08:40:36.737698+0000","last_active":"2024-09-13T08:40:36.741357+0000","last_peered":"2024-09-13T08:40:36.741357+0000","last_clean":"2024-09-13T08:40:36.741357+0000","last_became_active":"2024-09-13T08:40:36.737306+0000","last_became_peered":"2024-09-13T08:40:36.737306+0000","last_unstale":"2024-09-13T08:40:36.741357+0000","last_undegraded":"2024-09-13T08:40:36.741357+0000","last_fullsized":"2024-09-13T08:40:36.741357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:51:28.448035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758021+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.758021+0000","last_peered":"2024-09-13T08:40:37.758021+0000","last_clean":"2024-09-13T08:40:37.758021+0000","last_became_active":"2024-09-13T08:40:37.750682+0000","last_became_peered":"2024-09-13T08:40:37.750682+0000","last_unstale":"2024-09-13T08:40:37.758021+0000","last_undegraded":"2024-09-13T08:40:37.758021+0000","last_fullsized":"2024-09-13T08:40:37.758021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:12.637948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736909+0000","last_change":"2024-09-13T08:40:36.749086+0000","last_active":"2024-09-13T08:40:37.736909+0000","last_peered":"2024-09-13T08:40:37.736909+0000","last_clean":"2024-09-13T08:40:37.736909+0000","last_became_active":"2024-09-13T08:40:36.747729+0000","last_became_peered":"2024-09-13T08:40:36.747729+0000","last_unstale":"2024-09-13T08:40:37.736909+0000","last_undegraded":"2024-09-13T08:40:37.736909+0000","last_fullsized":"2024-09-13T08:40:37.736909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:12:48.295035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767464+0000","last_change":"2024-09-13T08:40:37.765328+0000","last_active":"2024-09-13T08:40:37.767464+0000","last_peered":"2024-09-13T08:40:37.767464+0000","last_clean":"2024-09-13T08:40:37.767464+0000","last_became_active":"2024-09-13T08:40:37.762317+0000","last_became_peered":"2024-09-13T08:40:37.762317+0000","last_unstale":"2024-09-13T08:40:37.767464+0000","last_undegraded":"2024-09-13T08:40:37.767464+0000","last_fullsized":"2024-09-13T08:40:37.767464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:29:55.562604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732662+0000","last_change":"2024-09-13T08:40:36.747739+0000","last_active":"2024-09-13T08:40:37.732662+0000","last_peered":"2024-09-13T08:40:37.732662+0000","last_clean":"2024-09-13T08:40:37.732662+0000","last_became_active":"2024-09-13T08:40:36.747366+0000","last_became_peered":"2024-09-13T08:40:36.747366+0000","last_unstale":"2024-09-13T08:40:37.732662+0000","last_undegraded":"2024-09-13T08:40:37.732662+0000","last_fullsized":"2024-09-13T08:40:37.732662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.748364+0000","last_change":"2024-09-13T08:40:37.746566+0000","last_active":"2024-09-13T08:40:37.748364+0000","last_peered":"2024-09-13T08:40:37.748364+0000","last_clean":"2024-09-13T08:40:37.748364+0000","last_became_active":"2024-09-13T08:40:37.746382+0000","last_became_peered":"2024-09-13T08:40:37.746382+0000","last_unstale":"2024-09-13T08:40:37.748364+0000","last_undegraded":"2024-09-13T08:40:37.748364+0000","last_fullsized":"2024-09-13T08:40:37.748364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.768664+0000","last_change":"2024-09-13T08:40:36.743612+0000","last_active":"2024-09-13T08:40:37.768664+0000","last_peered":"2024-09-13T08:40:37.768664+0000","last_clean":"2024-09-13T08:40:37.768664+0000","last_became_active":"2024-09-13T08:40:36.743337+0000","last_became_peered":"2024-09-13T08:40:36.743337+0000","last_unstale":"2024-09-13T08:40:37.768664+0000","last_undegraded":"2024-09-13T08:40:37.768664+0000","last_fullsized":"2024-09-13T08:40:37.768664+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747771+0000","last_change":"2024-09-13T08:40:37.746100+0000","last_active":"2024-09-13T08:40:37.747771+0000","last_peered":"2024-09-13T08:40:37.747771+0000","last_clean":"2024-09-13T08:40:37.747771+0000","last_became_active":"2024-09-13T08:40:37.745801+0000","last_became_peered":"2024-09-13T08:40:37.745801+0000","last_unstale":"2024-09-13T08:40:37.747771+0000","last_undegraded":"2024-09-13T08:40:37.747771+0000","last_fullsized":"2024-09-13T08:40:37.747771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745934+0000","last_change":"2024-09-13T08:40:36.751068+0000","last_active":"2024-09-13T08:40:37.745934+0000","last_peered":"2024-09-13T08:40:37.745934+0000","last_clean":"2024-09-13T08:40:37.745934+0000","last_became_active":"2024-09-13T08:40:36.748029+0000","last_became_peered":"2024-09-13T08:40:36.748029+0000","last_unstale":"2024-09-13T08:40:37.745934+0000","last_undegraded":"2024-09-13T08:40:37.745934+0000","last_fullsized":"2024-09-13T08:40:37.745934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:25.694691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.749201+0000","last_change":"2024-09-13T08:40:37.743421+0000","last_active":"2024-09-13T08:40:37.749201+0000","last_peered":"2024-09-13T08:40:37.749201+0000","last_clean":"2024-09-13T08:40:37.749201+0000","last_became_active":"2024-09-13T08:40:37.743126+0000","last_became_peered":"2024-09-13T08:40:37.743126+0000","last_unstale":"2024-09-13T08:40:37.749201+0000","last_undegraded":"2024-09-13T08:40:37.749201+0000","last_fullsized":"2024-09-13T08:40:37.749201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731313+0000","last_change":"2024-09-13T08:40:36.742793+0000","last_active":"2024-09-13T08:40:37.731313+0000","last_peered":"2024-09-13T08:40:37.731313+0000","last_clean":"2024-09-13T08:40:37.731313+0000","last_became_active":"2024-09-13T08:40:36.742516+0000","last_became_peered":"2024-09-13T08:40:36.742516+0000","last_unstale":"2024-09-13T08:40:37.731313+0000","last_undegraded":"2024-09-13T08:40:37.731313+0000","last_fullsized":"2024-09-13T08:40:37.731313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767726+0000","last_change":"2024-09-13T08:40:37.763959+0000","last_active":"2024-09-13T08:40:37.767726+0000","last_peered":"2024-09-13T08:40:37.767726+0000","last_clean":"2024-09-13T08:40:37.767726+0000","last_became_active":"2024-09-13T08:40:37.763661+0000","last_became_peered":"2024-09-13T08:40:37.763661+0000","last_unstale":"2024-09-13T08:40:37.767726+0000","last_undegraded":"2024-09-13T08:40:37.767726+0000","last_fullsized":"2024-09-13T08:40:37.767726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:46.895173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735588+0000","last_change":"2024-09-13T08:40:36.750833+0000","last_active":"2024-09-13T08:40:37.735588+0000","last_peered":"2024-09-13T08:40:37.735588+0000","last_clean":"2024-09-13T08:40:37.735588+0000","last_became_active":"2024-09-13T08:40:36.750471+0000","last_became_peered":"2024-09-13T08:40:36.750471+0000","last_unstale":"2024-09-13T08:40:37.735588+0000","last_undegraded":"2024-09-13T08:40:37.735588+0000","last_fullsized":"2024-09-13T08:40:37.735588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:36.394174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762046+0000","last_change":"2024-09-13T08:40:37.756241+0000","last_active":"2024-09-13T08:40:37.762046+0000","last_peered":"2024-09-13T08:40:37.762046+0000","last_clean":"2024-09-13T08:40:37.762046+0000","last_became_active":"2024-09-13T08:40:37.755961+0000","last_became_peered":"2024-09-13T08:40:37.755961+0000","last_unstale":"2024-09-13T08:40:37.762046+0000","last_undegraded":"2024-09-13T08:40:37.762046+0000","last_fullsized":"2024-09-13T08:40:37.762046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:38.286408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730472+0000","last_change":"2024-09-13T08:40:36.750422+0000","last_active":"2024-09-13T08:40:37.730472+0000","last_peered":"2024-09-13T08:40:37.730472+0000","last_clean":"2024-09-13T08:40:37.730472+0000","last_became_active":"2024-09-13T08:40:36.749512+0000","last_became_peered":"2024-09-13T08:40:36.749512+0000","last_unstale":"2024-09-13T08:40:37.730472+0000","last_undegraded":"2024-09-13T08:40:37.730472+0000","last_fullsized":"2024-09-13T08:40:37.730472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764478+0000","last_change":"2024-09-13T08:40:37.756404+0000","last_active":"2024-09-13T08:40:37.764478+0000","last_peered":"2024-09-13T08:40:37.764478+0000","last_clean":"2024-09-13T08:40:37.764478+0000","last_became_active":"2024-09-13T08:40:37.755037+0000","last_became_peered":"2024-09-13T08:40:37.755037+0000","last_unstale":"2024-09-13T08:40:37.764478+0000","last_undegraded":"2024-09-13T08:40:37.764478+0000","last_fullsized":"2024-09-13T08:40:37.764478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:11.905503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732259+0000","last_change":"2024-09-13T08:40:36.740510+0000","last_active":"2024-09-13T08:40:37.732259+0000","last_peered":"2024-09-13T08:40:37.732259+0000","last_clean":"2024-09-13T08:40:37.732259+0000","last_became_active":"2024-09-13T08:40:36.740243+0000","last_became_peered":"2024-09-13T08:40:36.740243+0000","last_unstale":"2024-09-13T08:40:37.732259+0000","last_undegraded":"2024-09-13T08:40:37.732259+0000","last_fullsized":"2024-09-13T08:40:37.732259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:20.155484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758212+0000","last_change":"2024-09-13T08:40:37.751854+0000","last_active":"2024-09-13T08:40:37.758212+0000","last_peered":"2024-09-13T08:40:37.758212+0000","last_clean":"2024-09-13T08:40:37.758212+0000","last_became_active":"2024-09-13T08:40:37.749368+0000","last_became_peered":"2024-09-13T08:40:37.749368+0000","last_unstale":"2024-09-13T08:40:37.758212+0000","last_undegraded":"2024-09-13T08:40:37.758212+0000","last_fullsized":"2024-09-13T08:40:37.758212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:05.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743930+0000","last_change":"2024-09-13T08:40:36.743024+0000","last_active":"2024-09-13T08:40:37.743930+0000","last_peered":"2024-09-13T08:40:37.743930+0000","last_clean":"2024-09-13T08:40:37.743930+0000","last_became_active":"2024-09-13T08:40:36.742739+0000","last_became_peered":"2024-09-13T08:40:36.742739+0000","last_unstale":"2024-09-13T08:40:37.743930+0000","last_undegraded":"2024-09-13T08:40:37.743930+0000","last_fullsized":"2024-09-13T08:40:37.743930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:54.197613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.753493+0000","last_change":"2024-09-13T08:40:37.743948+0000","last_active":"2024-09-13T08:40:37.753493+0000","last_peered":"2024-09-13T08:40:37.753493+0000","last_clean":"2024-09-13T08:40:37.753493+0000","last_became_active":"2024-09-13T08:40:37.743656+0000","last_became_peered":"2024-09-13T08:40:37.743656+0000","last_unstale":"2024-09-13T08:40:37.753493+0000","last_undegraded":"2024-09-13T08:40:37.753493+0000","last_fullsized":"2024-09-13T08:40:37.753493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735241+0000","last_change":"2024-09-13T08:40:36.750854+0000","last_active":"2024-09-13T08:40:37.735241+0000","last_peered":"2024-09-13T08:40:37.735241+0000","last_clean":"2024-09-13T08:40:37.735241+0000","last_became_active":"2024-09-13T08:40:36.750177+0000","last_became_peered":"2024-09-13T08:40:36.750177+0000","last_unstale":"2024-09-13T08:40:37.735241+0000","last_undegraded":"2024-09-13T08:40:37.735241+0000","last_fullsized":"2024-09-13T08:40:37.735241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:20.695241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.754437+0000","last_change":"2024-09-13T08:40:37.746462+0000","last_active":"2024-09-13T08:40:37.754437+0000","last_peered":"2024-09-13T08:40:37.754437+0000","last_clean":"2024-09-13T08:40:37.754437+0000","last_became_active":"2024-09-13T08:40:37.745179+0000","last_became_peered":"2024-09-13T08:40:37.745179+0000","last_unstale":"2024-09-13T08:40:37.754437+0000","last_undegraded":"2024-09-13T08:40:37.754437+0000","last_fullsized":"2024-09-13T08:40:37.754437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:07.391500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.737843+0000","last_change":"2024-09-13T08:40:36.744466+0000","last_active":"2024-09-13T08:40:37.737843+0000","last_peered":"2024-09-13T08:40:37.737843+0000","last_clean":"2024-09-13T08:40:37.737843+0000","last_became_active":"2024-09-13T08:40:36.744206+0000","last_became_peered":"2024-09-13T08:40:36.744206+0000","last_unstale":"2024-09-13T08:40:37.737843+0000","last_undegraded":"2024-09-13T08:40:37.737843+0000","last_fullsized":"2024-09-13T08:40:37.737843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:40.300855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":179,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732740+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:40:37.732740+0000","last_peered":"2024-09-13T08:40:37.732740+0000","last_clean":"2024-09-13T08:40:37.732740+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:40:37.732740+0000","last_undegraded":"2024-09-13T08:40:37.732740+0000","last_fullsized":"2024-09-13T08:40:37.732740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":299641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509487,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1323008,"data_stored":890021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":729088,"data_stored":299709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":745472,"data_stored":301391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1306624,"data_stored":883212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509486,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":737280,"data_stored":301231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36048,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:40:42.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 -- 172.21.15.31:0/2297010426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 msgr2=0x7fa2b80b1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 --2- 172.21.15.31:0/2297010426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80b1d60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0005fd0 tx=0x7fa2b002aaa0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 -- 172.21.15.31:0/2297010426 shutdown_connections 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 --2- 172.21.15.31:0/2297010426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80b1d60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 --2- 172.21.15.31:0/2297010426 >> v2:172.21.15.45:3300/0 conn(0x7fa2b80ca5c0 0x7fa2b80cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 -- 172.21.15.31:0/2297010426 >> 172.21.15.31:0/2297010426 conn(0x7fa2b80cc910 msgr2=0x7fa2b80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 -- 172.21.15.31:0/2297010426 shutdown_connections 2024-09-13T08:40:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.654+0000 7fa2bee00640 1 -- 172.21.15.31:0/2297010426 wait complete. 2024-09-13T08:40:42.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.656+0000 7fa2bee00640 1 Processor -- start 2024-09-13T08:40:42.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.656+0000 7fa2bee00640 1 -- start start 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b815dfc0 0x7fa2b815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b8073b10 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b8073990 con 0x7fa2b815dfc0 2024-09-13T08:40:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b8073c90 con 0x7fa2b80b1910 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45546/0 (socket says 172.21.15.31:45546) 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 -- 172.21.15.31:0/1002933513 learned_addr learned my addr 172.21.15.31:0/1002933513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2b7e00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2bce00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b815dfc0 0x7fa2b815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 -- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 msgr2=0x7fa2b80ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 -- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b815dfc0 msgr2=0x7fa2b815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b815dfc0 0x7fa2b815af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.657+0000 7fa2af400640 1 -- 172.21.15.31:0/1002933513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b80c72b0 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2af400640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000e9f0 tx=0x7fa2a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:42.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a000cd80 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2a0010430 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0010840 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b80c7590 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.658+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b80c6f60 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.659+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b81f8440 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.660+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fa2a00109e0 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.661+0000 7fa2b4c00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 0x7fa29407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:42.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.661+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fa2a009b7f0 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.662+0000 7fa2b7e00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 0x7fa29407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:42.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.662+0000 7fa2b7e00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 0x7fa29407aec0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8001720 tx=0x7fa2a8009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:42.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.665+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa2a0063550 con 0x7fa2b80ca5c0 2024-09-13T08:40:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.859+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2b80b1d60 con 0x7fa294078a00 2024-09-13T08:40:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.864+0000 7fa2b4c00640 1 -- 172.21.15.31:0/1002933513 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251528 (secure 0 0 0) 0x7fa2b80b1d60 con 0x7fa294078a00 2024-09-13T08:40:42.868 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 msgr2=0x7fa29407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 0x7fa29407aec0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8001720 tx=0x7fa2a8009040 comp rx=0 tx=0).stop 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 msgr2=0x7fa2b81f8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000e9f0 tx=0x7fa2a000eeb0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 shutdown_connections 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b815dfc0 0x7fa2b815af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2b80ca5c0 0x7fa2b81f8e50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa294078a00 0x7fa29407aec0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 --2- 172.21.15.31:0/1002933513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2b80b1910 0x7fa2b80ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:42.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.868+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 >> 172.21.15.31:0/1002933513 conn(0x7fa2b80cc910 msgr2=0x7fa2b815d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:42.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.869+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 shutdown_connections 2024-09-13T08:40:42.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:42.869+0000 7fa2bee00640 1 -- 172.21.15.31:0/1002933513 wait complete. 2024-09-13T08:40:42.885 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-13T08:40:41.090232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248052,"kb_used_data":6832,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726668,"statfs":{"total":773094113280,"available":772840108032,"internally_reserved":0,"allocated":6995968,"data_stored":3560893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.374100"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.761134+0000","last_change":"2024-09-13T08:40:37.757129+0000","last_active":"2024-09-13T08:40:37.761134+0000","last_peered":"2024-09-13T08:40:37.761134+0000","last_clean":"2024-09-13T08:40:37.761134+0000","last_became_active":"2024-09-13T08:40:37.756648+0000","last_became_peered":"2024-09-13T08:40:37.756648+0000","last_unstale":"2024-09-13T08:40:37.761134+0000","last_undegraded":"2024-09-13T08:40:37.761134+0000","last_fullsized":"2024-09-13T08:40:37.761134+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757494+0000","last_change":"2024-09-13T08:40:37.756234+0000","last_active":"2024-09-13T08:40:37.757494+0000","last_peered":"2024-09-13T08:40:37.757494+0000","last_clean":"2024-09-13T08:40:37.757494+0000","last_became_active":"2024-09-13T08:40:37.755925+0000","last_became_peered":"2024-09-13T08:40:37.755925+0000","last_unstale":"2024-09-13T08:40:37.757494+0000","last_undegraded":"2024-09-13T08:40:37.757494+0000","last_fullsized":"2024-09-13T08:40:37.757494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:24.421420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760116+0000","last_change":"2024-09-13T08:40:37.756481+0000","last_active":"2024-09-13T08:40:37.760116+0000","last_peered":"2024-09-13T08:40:37.760116+0000","last_clean":"2024-09-13T08:40:37.760116+0000","last_became_active":"2024-09-13T08:40:37.756190+0000","last_became_peered":"2024-09-13T08:40:37.756190+0000","last_unstale":"2024-09-13T08:40:37.760116+0000","last_undegraded":"2024-09-13T08:40:37.760116+0000","last_fullsized":"2024-09-13T08:40:37.760116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:45.588595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759390+0000","last_change":"2024-09-13T08:40:37.752214+0000","last_active":"2024-09-13T08:40:37.759390+0000","last_peered":"2024-09-13T08:40:37.759390+0000","last_clean":"2024-09-13T08:40:37.759390+0000","last_became_active":"2024-09-13T08:40:37.752015+0000","last_became_peered":"2024-09-13T08:40:37.752015+0000","last_unstale":"2024-09-13T08:40:37.759390+0000","last_undegraded":"2024-09-13T08:40:37.759390+0000","last_fullsized":"2024-09-13T08:40:37.759390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:19.281735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755493+0000","last_change":"2024-09-13T08:40:37.749549+0000","last_active":"2024-09-13T08:40:37.755493+0000","last_peered":"2024-09-13T08:40:37.755493+0000","last_clean":"2024-09-13T08:40:37.755493+0000","last_became_active":"2024-09-13T08:40:37.748944+0000","last_became_peered":"2024-09-13T08:40:37.748944+0000","last_unstale":"2024-09-13T08:40:37.755493+0000","last_undegraded":"2024-09-13T08:40:37.755493+0000","last_fullsized":"2024-09-13T08:40:37.755493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:05.311605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762020+0000","last_change":"2024-09-13T08:40:37.756210+0000","last_active":"2024-09-13T08:40:37.762020+0000","last_peered":"2024-09-13T08:40:37.762020+0000","last_clean":"2024-09-13T08:40:37.762020+0000","last_became_active":"2024-09-13T08:40:37.755876+0000","last_became_peered":"2024-09-13T08:40:37.755876+0000","last_unstale":"2024-09-13T08:40:37.762020+0000","last_undegraded":"2024-09-13T08:40:37.762020+0000","last_fullsized":"2024-09-13T08:40:37.762020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:37.401016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743263+0000","last_change":"2024-09-13T08:40:37.738548+0000","last_active":"2024-09-13T08:40:37.743263+0000","last_peered":"2024-09-13T08:40:37.743263+0000","last_clean":"2024-09-13T08:40:37.743263+0000","last_became_active":"2024-09-13T08:40:37.738294+0000","last_became_peered":"2024-09-13T08:40:37.738294+0000","last_unstale":"2024-09-13T08:40:37.743263+0000","last_undegraded":"2024-09-13T08:40:37.743263+0000","last_fullsized":"2024-09-13T08:40:37.743263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:11.567595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758242+0000","last_change":"2024-09-13T08:40:37.752144+0000","last_active":"2024-09-13T08:40:37.758242+0000","last_peered":"2024-09-13T08:40:37.758242+0000","last_clean":"2024-09-13T08:40:37.758242+0000","last_became_active":"2024-09-13T08:40:37.751574+0000","last_became_peered":"2024-09-13T08:40:37.751574+0000","last_unstale":"2024-09-13T08:40:37.758242+0000","last_undegraded":"2024-09-13T08:40:37.758242+0000","last_fullsized":"2024-09-13T08:40:37.758242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.900728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765529+0000","last_change":"2024-09-13T08:40:37.759047+0000","last_active":"2024-09-13T08:40:37.765529+0000","last_peered":"2024-09-13T08:40:37.765529+0000","last_clean":"2024-09-13T08:40:37.765529+0000","last_became_active":"2024-09-13T08:40:37.757616+0000","last_became_peered":"2024-09-13T08:40:37.757616+0000","last_unstale":"2024-09-13T08:40:37.765529+0000","last_undegraded":"2024-09-13T08:40:37.765529+0000","last_fullsized":"2024-09-13T08:40:37.765529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769285+0000","last_change":"2024-09-13T08:40:37.766302+0000","last_active":"2024-09-13T08:40:37.769285+0000","last_peered":"2024-09-13T08:40:37.769285+0000","last_clean":"2024-09-13T08:40:37.769285+0000","last_became_active":"2024-09-13T08:40:37.765985+0000","last_became_peered":"2024-09-13T08:40:37.765985+0000","last_unstale":"2024-09-13T08:40:37.769285+0000","last_undegraded":"2024-09-13T08:40:37.769285+0000","last_fullsized":"2024-09-13T08:40:37.769285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.726360+0000","last_change":"2024-09-13T08:40:36.726360+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.726360+0000","last_undegraded":"2024-09-13T08:40:36.726360+0000","last_fullsized":"2024-09-13T08:40:36.726360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767321+0000","last_change":"2024-09-13T08:40:37.762085+0000","last_active":"2024-09-13T08:40:37.767321+0000","last_peered":"2024-09-13T08:40:37.767321+0000","last_clean":"2024-09-13T08:40:37.767321+0000","last_became_active":"2024-09-13T08:40:37.761803+0000","last_became_peered":"2024-09-13T08:40:37.761803+0000","last_unstale":"2024-09-13T08:40:37.767321+0000","last_undegraded":"2024-09-13T08:40:37.767321+0000","last_fullsized":"2024-09-13T08:40:37.767321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760491+0000","last_change":"2024-09-13T08:40:37.742869+0000","last_active":"2024-09-13T08:40:37.760491+0000","last_peered":"2024-09-13T08:40:37.760491+0000","last_clean":"2024-09-13T08:40:37.760491+0000","last_became_active":"2024-09-13T08:40:37.742566+0000","last_became_peered":"2024-09-13T08:40:37.742566+0000","last_unstale":"2024-09-13T08:40:37.760491+0000","last_undegraded":"2024-09-13T08:40:37.760491+0000","last_fullsized":"2024-09-13T08:40:37.760491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:40.715704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755923+0000","last_change":"2024-09-13T08:40:37.750839+0000","last_active":"2024-09-13T08:40:37.755923+0000","last_peered":"2024-09-13T08:40:37.755923+0000","last_clean":"2024-09-13T08:40:37.755923+0000","last_became_active":"2024-09-13T08:40:37.750572+0000","last_became_peered":"2024-09-13T08:40:37.750572+0000","last_unstale":"2024-09-13T08:40:37.755923+0000","last_undegraded":"2024-09-13T08:40:37.755923+0000","last_fullsized":"2024-09-13T08:40:37.755923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764455+0000","last_change":"2024-09-13T08:40:37.755720+0000","last_active":"2024-09-13T08:40:37.764455+0000","last_peered":"2024-09-13T08:40:37.764455+0000","last_clean":"2024-09-13T08:40:37.764455+0000","last_became_active":"2024-09-13T08:40:37.755419+0000","last_became_peered":"2024-09-13T08:40:37.755419+0000","last_unstale":"2024-09-13T08:40:37.764455+0000","last_undegraded":"2024-09-13T08:40:37.764455+0000","last_fullsized":"2024-09-13T08:40:37.764455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:41.234982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756494+0000","last_change":"2024-09-13T08:40:37.748027+0000","last_active":"2024-09-13T08:40:37.756494+0000","last_peered":"2024-09-13T08:40:37.756494+0000","last_clean":"2024-09-13T08:40:37.756494+0000","last_became_active":"2024-09-13T08:40:37.747736+0000","last_became_peered":"2024-09-13T08:40:37.747736+0000","last_unstale":"2024-09-13T08:40:37.756494+0000","last_undegraded":"2024-09-13T08:40:37.756494+0000","last_fullsized":"2024-09-13T08:40:37.756494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:31.800897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763918+0000","last_change":"2024-09-13T08:40:37.759006+0000","last_active":"2024-09-13T08:40:37.763918+0000","last_peered":"2024-09-13T08:40:37.763918+0000","last_clean":"2024-09-13T08:40:37.763918+0000","last_became_active":"2024-09-13T08:40:37.758700+0000","last_became_peered":"2024-09-13T08:40:37.758700+0000","last_unstale":"2024-09-13T08:40:37.763918+0000","last_undegraded":"2024-09-13T08:40:37.763918+0000","last_fullsized":"2024-09-13T08:40:37.763918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.757199+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.756901+0000","last_became_peered":"2024-09-13T08:40:37.756901+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756708+0000","last_change":"2024-09-13T08:40:37.752015+0000","last_active":"2024-09-13T08:40:37.756708+0000","last_peered":"2024-09-13T08:40:37.756708+0000","last_clean":"2024-09-13T08:40:37.756708+0000","last_became_active":"2024-09-13T08:40:37.751710+0000","last_became_peered":"2024-09-13T08:40:37.751710+0000","last_unstale":"2024-09-13T08:40:37.756708+0000","last_undegraded":"2024-09-13T08:40:37.756708+0000","last_fullsized":"2024-09-13T08:40:37.756708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727096+0000","last_change":"2024-09-13T08:40:36.727096+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727096+0000","last_undegraded":"2024-09-13T08:40:36.727096+0000","last_fullsized":"2024-09-13T08:40:36.727096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764451+0000","last_change":"2024-09-13T08:40:37.751434+0000","last_active":"2024-09-13T08:40:37.764451+0000","last_peered":"2024-09-13T08:40:37.764451+0000","last_clean":"2024-09-13T08:40:37.764451+0000","last_became_active":"2024-09-13T08:40:37.745985+0000","last_became_peered":"2024-09-13T08:40:37.745985+0000","last_unstale":"2024-09-13T08:40:37.764451+0000","last_undegraded":"2024-09-13T08:40:37.764451+0000","last_fullsized":"2024-09-13T08:40:37.764451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:33.990893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727284+0000","last_change":"2024-09-13T08:40:36.727284+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727284+0000","last_undegraded":"2024-09-13T08:40:36.727284+0000","last_fullsized":"2024-09-13T08:40:36.727284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740716+0000","last_change":"2024-09-13T08:40:37.734978+0000","last_active":"2024-09-13T08:40:37.740716+0000","last_peered":"2024-09-13T08:40:37.740716+0000","last_clean":"2024-09-13T08:40:37.740716+0000","last_became_active":"2024-09-13T08:40:37.734070+0000","last_became_peered":"2024-09-13T08:40:37.734070+0000","last_unstale":"2024-09-13T08:40:37.740716+0000","last_undegraded":"2024-09-13T08:40:37.740716+0000","last_fullsized":"2024-09-13T08:40:37.740716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:54.518342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755838+0000","last_change":"2024-09-13T08:40:37.750441+0000","last_active":"2024-09-13T08:40:37.755838+0000","last_peered":"2024-09-13T08:40:37.755838+0000","last_clean":"2024-09-13T08:40:37.755838+0000","last_became_active":"2024-09-13T08:40:37.750132+0000","last_became_peered":"2024-09-13T08:40:37.750132+0000","last_unstale":"2024-09-13T08:40:37.755838+0000","last_undegraded":"2024-09-13T08:40:37.755838+0000","last_fullsized":"2024-09-13T08:40:37.755838+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755293+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.755293+0000","last_peered":"2024-09-13T08:40:37.755293+0000","last_clean":"2024-09-13T08:40:37.755293+0000","last_became_active":"2024-09-13T08:40:37.750728+0000","last_became_peered":"2024-09-13T08:40:37.750728+0000","last_unstale":"2024-09-13T08:40:37.755293+0000","last_undegraded":"2024-09-13T08:40:37.755293+0000","last_fullsized":"2024-09-13T08:40:37.755293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.741441+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.737794+0000","last_became_peered":"2024-09-13T08:40:37.737794+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:41.031994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755203+0000","last_change":"2024-09-13T08:40:37.748613+0000","last_active":"2024-09-13T08:40:37.755203+0000","last_peered":"2024-09-13T08:40:37.755203+0000","last_clean":"2024-09-13T08:40:37.755203+0000","last_became_active":"2024-09-13T08:40:37.745423+0000","last_became_peered":"2024-09-13T08:40:37.745423+0000","last_unstale":"2024-09-13T08:40:37.755203+0000","last_undegraded":"2024-09-13T08:40:37.755203+0000","last_fullsized":"2024-09-13T08:40:37.755203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765146+0000","last_change":"2024-09-13T08:40:37.762367+0000","last_active":"2024-09-13T08:40:37.765146+0000","last_peered":"2024-09-13T08:40:37.765146+0000","last_clean":"2024-09-13T08:40:37.765146+0000","last_became_active":"2024-09-13T08:40:37.762030+0000","last_became_peered":"2024-09-13T08:40:37.762030+0000","last_unstale":"2024-09-13T08:40:37.765146+0000","last_undegraded":"2024-09-13T08:40:37.765146+0000","last_fullsized":"2024-09-13T08:40:37.765146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:20.588520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758045+0000","last_change":"2024-09-13T08:40:37.750469+0000","last_active":"2024-09-13T08:40:37.758045+0000","last_peered":"2024-09-13T08:40:37.758045+0000","last_clean":"2024-09-13T08:40:37.758045+0000","last_became_active":"2024-09-13T08:40:37.748954+0000","last_became_peered":"2024-09-13T08:40:37.748954+0000","last_unstale":"2024-09-13T08:40:37.758045+0000","last_undegraded":"2024-09-13T08:40:37.758045+0000","last_fullsized":"2024-09-13T08:40:37.758045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:04.282317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747600+0000","last_change":"2024-09-13T08:40:37.744539+0000","last_active":"2024-09-13T08:40:37.747600+0000","last_peered":"2024-09-13T08:40:37.747600+0000","last_clean":"2024-09-13T08:40:37.747600+0000","last_became_active":"2024-09-13T08:40:37.744247+0000","last_became_peered":"2024-09-13T08:40:37.744247+0000","last_unstale":"2024-09-13T08:40:37.747600+0000","last_undegraded":"2024-09-13T08:40:37.747600+0000","last_fullsized":"2024-09-13T08:40:37.747600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:14.087231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756561+0000","last_change":"2024-09-13T08:40:37.750709+0000","last_active":"2024-09-13T08:40:37.756561+0000","last_peered":"2024-09-13T08:40:37.756561+0000","last_clean":"2024-09-13T08:40:37.756561+0000","last_became_active":"2024-09-13T08:40:37.749124+0000","last_became_peered":"2024-09-13T08:40:37.749124+0000","last_unstale":"2024-09-13T08:40:37.756561+0000","last_undegraded":"2024-09-13T08:40:37.756561+0000","last_fullsized":"2024-09-13T08:40:37.756561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.728732+0000","last_change":"2024-09-13T08:40:36.728732+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.728732+0000","last_undegraded":"2024-09-13T08:40:36.728732+0000","last_fullsized":"2024-09-13T08:40:36.728732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763275+0000","last_change":"2024-09-13T08:40:37.758027+0000","last_active":"2024-09-13T08:40:37.763275+0000","last_peered":"2024-09-13T08:40:37.763275+0000","last_clean":"2024-09-13T08:40:37.763275+0000","last_became_active":"2024-09-13T08:40:37.757576+0000","last_became_peered":"2024-09-13T08:40:37.757576+0000","last_unstale":"2024-09-13T08:40:37.763275+0000","last_undegraded":"2024-09-13T08:40:37.763275+0000","last_fullsized":"2024-09-13T08:40:37.763275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.744179+0000","last_change":"2024-09-13T08:40:36.740867+0000","last_active":"2024-09-13T08:40:36.744179+0000","last_peered":"2024-09-13T08:40:36.744179+0000","last_clean":"2024-09-13T08:40:36.744179+0000","last_became_active":"2024-09-13T08:40:36.740475+0000","last_became_peered":"2024-09-13T08:40:36.740475+0000","last_unstale":"2024-09-13T08:40:36.744179+0000","last_undegraded":"2024-09-13T08:40:36.744179+0000","last_fullsized":"2024-09-13T08:40:36.744179+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:00.821617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763350+0000","last_change":"2024-09-13T08:40:37.755669+0000","last_active":"2024-09-13T08:40:37.763350+0000","last_peered":"2024-09-13T08:40:37.763350+0000","last_clean":"2024-09-13T08:40:37.763350+0000","last_became_active":"2024-09-13T08:40:37.755390+0000","last_became_peered":"2024-09-13T08:40:37.755390+0000","last_unstale":"2024-09-13T08:40:37.763350+0000","last_undegraded":"2024-09-13T08:40:37.763350+0000","last_fullsized":"2024-09-13T08:40:37.763350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770118+0000","last_change":"2024-09-13T08:40:36.749525+0000","last_active":"2024-09-13T08:40:37.770118+0000","last_peered":"2024-09-13T08:40:37.770118+0000","last_clean":"2024-09-13T08:40:37.770118+0000","last_became_active":"2024-09-13T08:40:36.749236+0000","last_became_peered":"2024-09-13T08:40:36.749236+0000","last_unstale":"2024-09-13T08:40:37.770118+0000","last_undegraded":"2024-09-13T08:40:37.770118+0000","last_fullsized":"2024-09-13T08:40:37.770118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759658+0000","last_change":"2024-09-13T08:40:37.758005+0000","last_active":"2024-09-13T08:40:37.759658+0000","last_peered":"2024-09-13T08:40:37.759658+0000","last_clean":"2024-09-13T08:40:37.759658+0000","last_became_active":"2024-09-13T08:40:37.757312+0000","last_became_peered":"2024-09-13T08:40:37.757312+0000","last_unstale":"2024-09-13T08:40:37.759658+0000","last_undegraded":"2024-09-13T08:40:37.759658+0000","last_fullsized":"2024-09-13T08:40:37.759658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:18:18.952772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769133+0000","last_change":"2024-09-13T08:40:36.751851+0000","last_active":"2024-09-13T08:40:37.769133+0000","last_peered":"2024-09-13T08:40:37.769133+0000","last_clean":"2024-09-13T08:40:37.769133+0000","last_became_active":"2024-09-13T08:40:36.748632+0000","last_became_peered":"2024-09-13T08:40:36.748632+0000","last_unstale":"2024-09-13T08:40:37.769133+0000","last_undegraded":"2024-09-13T08:40:37.769133+0000","last_fullsized":"2024-09-13T08:40:37.769133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:56.150885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757513+0000","last_change":"2024-09-13T08:40:37.751225+0000","last_active":"2024-09-13T08:40:37.757513+0000","last_peered":"2024-09-13T08:40:37.757513+0000","last_clean":"2024-09-13T08:40:37.757513+0000","last_became_active":"2024-09-13T08:40:37.750923+0000","last_became_peered":"2024-09-13T08:40:37.750923+0000","last_unstale":"2024-09-13T08:40:37.757513+0000","last_undegraded":"2024-09-13T08:40:37.757513+0000","last_fullsized":"2024-09-13T08:40:37.757513+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:21.162327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772713+0000","last_change":"2024-09-13T08:40:36.746077+0000","last_active":"2024-09-13T08:40:37.772713+0000","last_peered":"2024-09-13T08:40:37.772713+0000","last_clean":"2024-09-13T08:40:37.772713+0000","last_became_active":"2024-09-13T08:40:36.745814+0000","last_became_peered":"2024-09-13T08:40:36.745814+0000","last_unstale":"2024-09-13T08:40:37.772713+0000","last_undegraded":"2024-09-13T08:40:37.772713+0000","last_fullsized":"2024-09-13T08:40:37.772713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:24.123182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757961+0000","last_change":"2024-09-13T08:40:37.749662+0000","last_active":"2024-09-13T08:40:37.757961+0000","last_peered":"2024-09-13T08:40:37.757961+0000","last_clean":"2024-09-13T08:40:37.757961+0000","last_became_active":"2024-09-13T08:40:37.749218+0000","last_became_peered":"2024-09-13T08:40:37.749218+0000","last_unstale":"2024-09-13T08:40:37.757961+0000","last_undegraded":"2024-09-13T08:40:37.757961+0000","last_fullsized":"2024-09-13T08:40:37.757961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:27.923545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769027+0000","last_change":"2024-09-13T08:40:36.738659+0000","last_active":"2024-09-13T08:40:37.769027+0000","last_peered":"2024-09-13T08:40:37.769027+0000","last_clean":"2024-09-13T08:40:37.769027+0000","last_became_active":"2024-09-13T08:40:36.738315+0000","last_became_peered":"2024-09-13T08:40:36.738315+0000","last_unstale":"2024-09-13T08:40:37.769027+0000","last_undegraded":"2024-09-13T08:40:37.769027+0000","last_fullsized":"2024-09-13T08:40:37.769027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:09.635872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-09-13T08:40:36.727899+0000","last_change":"2024-09-13T08:40:36.727899+0000","last_active":"2024-09-13T08:40:36.715339+0000","last_peered":"2024-09-13T08:40:36.715339+0000","last_clean":"2024-09-13T08:40:36.715339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:40:36.727899+0000","last_undegraded":"2024-09-13T08:40:36.727899+0000","last_fullsized":"2024-09-13T08:40:36.727899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772875+0000","last_change":"2024-09-13T08:40:36.751002+0000","last_active":"2024-09-13T08:40:37.772875+0000","last_peered":"2024-09-13T08:40:37.772875+0000","last_clean":"2024-09-13T08:40:37.772875+0000","last_became_active":"2024-09-13T08:40:36.750731+0000","last_became_peered":"2024-09-13T08:40:36.750731+0000","last_unstale":"2024-09-13T08:40:37.772875+0000","last_undegraded":"2024-09-13T08:40:37.772875+0000","last_fullsized":"2024-09-13T08:40:37.772875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:24.257806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.742278+0000","last_change":"2024-09-13T08:40:37.736587+0000","last_active":"2024-09-13T08:40:37.742278+0000","last_peered":"2024-09-13T08:40:37.742278+0000","last_clean":"2024-09-13T08:40:37.742278+0000","last_became_active":"2024-09-13T08:40:37.734709+0000","last_became_peered":"2024-09-13T08:40:37.734709+0000","last_unstale":"2024-09-13T08:40:37.742278+0000","last_undegraded":"2024-09-13T08:40:37.742278+0000","last_fullsized":"2024-09-13T08:40:37.742278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730713+0000","last_change":"2024-09-13T08:40:36.740922+0000","last_active":"2024-09-13T08:40:37.730713+0000","last_peered":"2024-09-13T08:40:37.730713+0000","last_clean":"2024-09-13T08:40:37.730713+0000","last_became_active":"2024-09-13T08:40:36.740610+0000","last_became_peered":"2024-09-13T08:40:36.740610+0000","last_unstale":"2024-09-13T08:40:37.730713+0000","last_undegraded":"2024-09-13T08:40:37.730713+0000","last_fullsized":"2024-09-13T08:40:37.730713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758374+0000","last_change":"2024-09-13T08:40:37.753714+0000","last_active":"2024-09-13T08:40:37.758374+0000","last_peered":"2024-09-13T08:40:37.758374+0000","last_clean":"2024-09-13T08:40:37.758374+0000","last_became_active":"2024-09-13T08:40:37.753402+0000","last_became_peered":"2024-09-13T08:40:37.753402+0000","last_unstale":"2024-09-13T08:40:37.758374+0000","last_undegraded":"2024-09-13T08:40:37.758374+0000","last_fullsized":"2024-09-13T08:40:37.758374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:10.642715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731269+0000","last_change":"2024-09-13T08:40:36.737107+0000","last_active":"2024-09-13T08:40:37.731269+0000","last_peered":"2024-09-13T08:40:37.731269+0000","last_clean":"2024-09-13T08:40:37.731269+0000","last_became_active":"2024-09-13T08:40:36.736458+0000","last_became_peered":"2024-09-13T08:40:36.736458+0000","last_unstale":"2024-09-13T08:40:37.731269+0000","last_undegraded":"2024-09-13T08:40:37.731269+0000","last_fullsized":"2024-09-13T08:40:37.731269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:13.644488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.751369+0000","last_change":"2024-09-13T08:40:37.736394+0000","last_active":"2024-09-13T08:40:37.751369+0000","last_peered":"2024-09-13T08:40:37.751369+0000","last_clean":"2024-09-13T08:40:37.751369+0000","last_became_active":"2024-09-13T08:40:37.736142+0000","last_became_peered":"2024-09-13T08:40:37.736142+0000","last_unstale":"2024-09-13T08:40:37.751369+0000","last_undegraded":"2024-09-13T08:40:37.751369+0000","last_fullsized":"2024-09-13T08:40:37.751369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:00.761695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.733495+0000","last_change":"2024-09-13T08:40:36.740047+0000","last_active":"2024-09-13T08:40:37.733495+0000","last_peered":"2024-09-13T08:40:37.733495+0000","last_clean":"2024-09-13T08:40:37.733495+0000","last_became_active":"2024-09-13T08:40:36.739774+0000","last_became_peered":"2024-09-13T08:40:36.739774+0000","last_unstale":"2024-09-13T08:40:37.733495+0000","last_undegraded":"2024-09-13T08:40:37.733495+0000","last_fullsized":"2024-09-13T08:40:37.733495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760164+0000","last_change":"2024-09-13T08:40:37.757097+0000","last_active":"2024-09-13T08:40:37.760164+0000","last_peered":"2024-09-13T08:40:37.760164+0000","last_clean":"2024-09-13T08:40:37.760164+0000","last_became_active":"2024-09-13T08:40:37.756526+0000","last_became_peered":"2024-09-13T08:40:37.756526+0000","last_unstale":"2024-09-13T08:40:37.760164+0000","last_undegraded":"2024-09-13T08:40:37.760164+0000","last_fullsized":"2024-09-13T08:40:37.760164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735925+0000","last_change":"2024-09-13T08:40:36.743002+0000","last_active":"2024-09-13T08:40:37.735925+0000","last_peered":"2024-09-13T08:40:37.735925+0000","last_clean":"2024-09-13T08:40:37.735925+0000","last_became_active":"2024-09-13T08:40:36.740918+0000","last_became_peered":"2024-09-13T08:40:36.740918+0000","last_unstale":"2024-09-13T08:40:37.735925+0000","last_undegraded":"2024-09-13T08:40:37.735925+0000","last_fullsized":"2024-09-13T08:40:37.735925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:05.024141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759635+0000","last_change":"2024-09-13T08:40:37.756377+0000","last_active":"2024-09-13T08:40:37.759635+0000","last_peered":"2024-09-13T08:40:37.759635+0000","last_clean":"2024-09-13T08:40:37.759635+0000","last_became_active":"2024-09-13T08:40:37.756086+0000","last_became_peered":"2024-09-13T08:40:37.756086+0000","last_unstale":"2024-09-13T08:40:37.759635+0000","last_undegraded":"2024-09-13T08:40:37.759635+0000","last_fullsized":"2024-09-13T08:40:37.759635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:41.096839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772900+0000","last_change":"2024-09-13T08:40:36.745979+0000","last_active":"2024-09-13T08:40:37.772900+0000","last_peered":"2024-09-13T08:40:37.772900+0000","last_clean":"2024-09-13T08:40:37.772900+0000","last_became_active":"2024-09-13T08:40:36.745673+0000","last_became_peered":"2024-09-13T08:40:36.745673+0000","last_unstale":"2024-09-13T08:40:37.772900+0000","last_undegraded":"2024-09-13T08:40:37.772900+0000","last_fullsized":"2024-09-13T08:40:37.772900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:11.780765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759430+0000","last_change":"2024-09-13T08:40:37.752368+0000","last_active":"2024-09-13T08:40:37.759430+0000","last_peered":"2024-09-13T08:40:37.759430+0000","last_clean":"2024-09-13T08:40:37.759430+0000","last_became_active":"2024-09-13T08:40:37.752096+0000","last_became_peered":"2024-09-13T08:40:37.752096+0000","last_unstale":"2024-09-13T08:40:37.759430+0000","last_undegraded":"2024-09-13T08:40:37.759430+0000","last_fullsized":"2024-09-13T08:40:37.759430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:33.675957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776405+0000","last_change":"2024-09-13T08:40:36.745140+0000","last_active":"2024-09-13T08:40:37.776405+0000","last_peered":"2024-09-13T08:40:37.776405+0000","last_clean":"2024-09-13T08:40:37.776405+0000","last_became_active":"2024-09-13T08:40:36.743817+0000","last_became_peered":"2024-09-13T08:40:36.743817+0000","last_unstale":"2024-09-13T08:40:37.776405+0000","last_undegraded":"2024-09-13T08:40:37.776405+0000","last_fullsized":"2024-09-13T08:40:37.776405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:15.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745728+0000","last_change":"2024-09-13T08:40:37.740915+0000","last_active":"2024-09-13T08:40:37.745728+0000","last_peered":"2024-09-13T08:40:37.745728+0000","last_clean":"2024-09-13T08:40:37.745728+0000","last_became_active":"2024-09-13T08:40:37.740617+0000","last_became_peered":"2024-09-13T08:40:37.740617+0000","last_unstale":"2024-09-13T08:40:37.745728+0000","last_undegraded":"2024-09-13T08:40:37.745728+0000","last_fullsized":"2024-09-13T08:40:37.745728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770825+0000","last_change":"2024-09-13T08:40:36.740132+0000","last_active":"2024-09-13T08:40:37.770825+0000","last_peered":"2024-09-13T08:40:37.770825+0000","last_clean":"2024-09-13T08:40:37.770825+0000","last_became_active":"2024-09-13T08:40:36.739787+0000","last_became_peered":"2024-09-13T08:40:36.739787+0000","last_unstale":"2024-09-13T08:40:37.770825+0000","last_undegraded":"2024-09-13T08:40:37.770825+0000","last_fullsized":"2024-09-13T08:40:37.770825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.013701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759244+0000","last_change":"2024-09-13T08:40:37.754029+0000","last_active":"2024-09-13T08:40:37.759244+0000","last_peered":"2024-09-13T08:40:37.759244+0000","last_clean":"2024-09-13T08:40:37.759244+0000","last_became_active":"2024-09-13T08:40:37.753773+0000","last_became_peered":"2024-09-13T08:40:37.753773+0000","last_unstale":"2024-09-13T08:40:37.759244+0000","last_undegraded":"2024-09-13T08:40:37.759244+0000","last_fullsized":"2024-09-13T08:40:37.759244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727972+0000","last_change":"2024-09-13T08:40:36.746192+0000","last_active":"2024-09-13T08:40:37.727972+0000","last_peered":"2024-09-13T08:40:37.727972+0000","last_clean":"2024-09-13T08:40:37.727972+0000","last_became_active":"2024-09-13T08:40:36.745703+0000","last_became_peered":"2024-09-13T08:40:36.745703+0000","last_unstale":"2024-09-13T08:40:37.727972+0000","last_undegraded":"2024-09-13T08:40:37.727972+0000","last_fullsized":"2024-09-13T08:40:37.727972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:00.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755812+0000","last_change":"2024-09-13T08:40:37.750252+0000","last_active":"2024-09-13T08:40:37.755812+0000","last_peered":"2024-09-13T08:40:37.755812+0000","last_clean":"2024-09-13T08:40:37.755812+0000","last_became_active":"2024-09-13T08:40:37.749793+0000","last_became_peered":"2024-09-13T08:40:37.749793+0000","last_unstale":"2024-09-13T08:40:37.755812+0000","last_undegraded":"2024-09-13T08:40:37.755812+0000","last_fullsized":"2024-09-13T08:40:37.755812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727573+0000","last_change":"2024-09-13T08:40:36.748499+0000","last_active":"2024-09-13T08:40:37.727573+0000","last_peered":"2024-09-13T08:40:37.727573+0000","last_clean":"2024-09-13T08:40:37.727573+0000","last_became_active":"2024-09-13T08:40:36.745832+0000","last_became_peered":"2024-09-13T08:40:36.745832+0000","last_unstale":"2024-09-13T08:40:37.727573+0000","last_undegraded":"2024-09-13T08:40:37.727573+0000","last_fullsized":"2024-09-13T08:40:37.727573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:42.819908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740690+0000","last_change":"2024-09-13T08:40:37.734550+0000","last_active":"2024-09-13T08:40:37.740690+0000","last_peered":"2024-09-13T08:40:37.740690+0000","last_clean":"2024-09-13T08:40:37.740690+0000","last_became_active":"2024-09-13T08:40:37.734140+0000","last_became_peered":"2024-09-13T08:40:37.734140+0000","last_unstale":"2024-09-13T08:40:37.740690+0000","last_undegraded":"2024-09-13T08:40:37.740690+0000","last_fullsized":"2024-09-13T08:40:37.740690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:11.251715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":21,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776743+0000","last_change":"2024-09-13T08:40:36.746654+0000","last_active":"2024-09-13T08:40:37.776743+0000","last_peered":"2024-09-13T08:40:37.776743+0000","last_clean":"2024-09-13T08:40:37.776743+0000","last_became_active":"2024-09-13T08:40:36.746397+0000","last_became_peered":"2024-09-13T08:40:36.746397+0000","last_unstale":"2024-09-13T08:40:37.776743+0000","last_undegraded":"2024-09-13T08:40:37.776743+0000","last_fullsized":"2024-09-13T08:40:37.776743+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:15.203955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747535+0000","last_change":"2024-09-13T08:40:37.745284+0000","last_active":"2024-09-13T08:40:37.747535+0000","last_peered":"2024-09-13T08:40:37.747535+0000","last_clean":"2024-09-13T08:40:37.747535+0000","last_became_active":"2024-09-13T08:40:37.744957+0000","last_became_peered":"2024-09-13T08:40:37.744957+0000","last_unstale":"2024-09-13T08:40:37.747535+0000","last_undegraded":"2024-09-13T08:40:37.747535+0000","last_fullsized":"2024-09-13T08:40:37.747535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767493+0000","last_change":"2024-09-13T08:40:36.753006+0000","last_active":"2024-09-13T08:40:37.767493+0000","last_peered":"2024-09-13T08:40:37.767493+0000","last_clean":"2024-09-13T08:40:37.767493+0000","last_became_active":"2024-09-13T08:40:36.752498+0000","last_became_peered":"2024-09-13T08:40:36.752498+0000","last_unstale":"2024-09-13T08:40:37.767493+0000","last_undegraded":"2024-09-13T08:40:37.767493+0000","last_fullsized":"2024-09-13T08:40:37.767493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760187+0000","last_change":"2024-09-13T08:40:37.757310+0000","last_active":"2024-09-13T08:40:37.760187+0000","last_peered":"2024-09-13T08:40:37.760187+0000","last_clean":"2024-09-13T08:40:37.760187+0000","last_became_active":"2024-09-13T08:40:37.756852+0000","last_became_peered":"2024-09-13T08:40:37.756852+0000","last_unstale":"2024-09-13T08:40:37.760187+0000","last_undegraded":"2024-09-13T08:40:37.760187+0000","last_fullsized":"2024-09-13T08:40:37.760187+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770763+0000","last_change":"2024-09-13T08:40:36.741233+0000","last_active":"2024-09-13T08:40:37.770763+0000","last_peered":"2024-09-13T08:40:37.770763+0000","last_clean":"2024-09-13T08:40:37.770763+0000","last_became_active":"2024-09-13T08:40:36.740574+0000","last_became_peered":"2024-09-13T08:40:36.740574+0000","last_unstale":"2024-09-13T08:40:37.770763+0000","last_undegraded":"2024-09-13T08:40:37.770763+0000","last_fullsized":"2024-09-13T08:40:37.770763+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:52.377019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762842+0000","last_change":"2024-09-13T08:40:37.761727+0000","last_active":"2024-09-13T08:40:37.762842+0000","last_peered":"2024-09-13T08:40:37.762842+0000","last_clean":"2024-09-13T08:40:37.762842+0000","last_became_active":"2024-09-13T08:40:37.761016+0000","last_became_peered":"2024-09-13T08:40:37.761016+0000","last_unstale":"2024-09-13T08:40:37.762842+0000","last_undegraded":"2024-09-13T08:40:37.762842+0000","last_fullsized":"2024-09-13T08:40:37.762842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:58:06.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769868+0000","last_change":"2024-09-13T08:40:36.740608+0000","last_active":"2024-09-13T08:40:37.769868+0000","last_peered":"2024-09-13T08:40:37.769868+0000","last_clean":"2024-09-13T08:40:37.769868+0000","last_became_active":"2024-09-13T08:40:36.740349+0000","last_became_peered":"2024-09-13T08:40:36.740349+0000","last_unstale":"2024-09-13T08:40:37.769868+0000","last_undegraded":"2024-09-13T08:40:37.769868+0000","last_fullsized":"2024-09-13T08:40:37.769868+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.279565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762124+0000","last_change":"2024-09-13T08:40:37.758748+0000","last_active":"2024-09-13T08:40:37.762124+0000","last_peered":"2024-09-13T08:40:37.762124+0000","last_clean":"2024-09-13T08:40:37.762124+0000","last_became_active":"2024-09-13T08:40:37.756975+0000","last_became_peered":"2024-09-13T08:40:37.756975+0000","last_unstale":"2024-09-13T08:40:37.762124+0000","last_undegraded":"2024-09-13T08:40:37.762124+0000","last_fullsized":"2024-09-13T08:40:37.762124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:43.311241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736532+0000","last_change":"2024-09-13T08:40:36.748868+0000","last_active":"2024-09-13T08:40:37.736532+0000","last_peered":"2024-09-13T08:40:37.736532+0000","last_clean":"2024-09-13T08:40:37.736532+0000","last_became_active":"2024-09-13T08:40:36.748527+0000","last_became_peered":"2024-09-13T08:40:36.748527+0000","last_unstale":"2024-09-13T08:40:37.736532+0000","last_undegraded":"2024-09-13T08:40:37.736532+0000","last_fullsized":"2024-09-13T08:40:37.736532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:05.275938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759771+0000","last_change":"2024-09-13T08:40:37.756050+0000","last_active":"2024-09-13T08:40:37.759771+0000","last_peered":"2024-09-13T08:40:37.759771+0000","last_clean":"2024-09-13T08:40:37.759771+0000","last_became_active":"2024-09-13T08:40:37.755788+0000","last_became_peered":"2024-09-13T08:40:37.755788+0000","last_unstale":"2024-09-13T08:40:37.759771+0000","last_undegraded":"2024-09-13T08:40:37.759771+0000","last_fullsized":"2024-09-13T08:40:37.759771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T08:40:36.741357+0000","last_change":"2024-09-13T08:40:36.737698+0000","last_active":"2024-09-13T08:40:36.741357+0000","last_peered":"2024-09-13T08:40:36.741357+0000","last_clean":"2024-09-13T08:40:36.741357+0000","last_became_active":"2024-09-13T08:40:36.737306+0000","last_became_peered":"2024-09-13T08:40:36.737306+0000","last_unstale":"2024-09-13T08:40:36.741357+0000","last_undegraded":"2024-09-13T08:40:36.741357+0000","last_fullsized":"2024-09-13T08:40:36.741357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:51:28.448035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758021+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.758021+0000","last_peered":"2024-09-13T08:40:37.758021+0000","last_clean":"2024-09-13T08:40:37.758021+0000","last_became_active":"2024-09-13T08:40:37.750682+0000","last_became_peered":"2024-09-13T08:40:37.750682+0000","last_unstale":"2024-09-13T08:40:37.758021+0000","last_undegraded":"2024-09-13T08:40:37.758021+0000","last_fullsized":"2024-09-13T08:40:37.758021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:12.637948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736909+0000","last_change":"2024-09-13T08:40:36.749086+0000","last_active":"2024-09-13T08:40:37.736909+0000","last_peered":"2024-09-13T08:40:37.736909+0000","last_clean":"2024-09-13T08:40:37.736909+0000","last_became_active":"2024-09-13T08:40:36.747729+0000","last_became_peered":"2024-09-13T08:40:36.747729+0000","last_unstale":"2024-09-13T08:40:37.736909+0000","last_undegraded":"2024-09-13T08:40:37.736909+0000","last_fullsized":"2024-09-13T08:40:37.736909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:12:48.295035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767464+0000","last_change":"2024-09-13T08:40:37.765328+0000","last_active":"2024-09-13T08:40:37.767464+0000","last_peered":"2024-09-13T08:40:37.767464+0000","last_clean":"2024-09-13T08:40:37.767464+0000","last_became_active":"2024-09-13T08:40:37.762317+0000","last_became_peered":"2024-09-13T08:40:37.762317+0000","last_unstale":"2024-09-13T08:40:37.767464+0000","last_undegraded":"2024-09-13T08:40:37.767464+0000","last_fullsized":"2024-09-13T08:40:37.767464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:29:55.562604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732662+0000","last_change":"2024-09-13T08:40:36.747739+0000","last_active":"2024-09-13T08:40:37.732662+0000","last_peered":"2024-09-13T08:40:37.732662+0000","last_clean":"2024-09-13T08:40:37.732662+0000","last_became_active":"2024-09-13T08:40:36.747366+0000","last_became_peered":"2024-09-13T08:40:36.747366+0000","last_unstale":"2024-09-13T08:40:37.732662+0000","last_undegraded":"2024-09-13T08:40:37.732662+0000","last_fullsized":"2024-09-13T08:40:37.732662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.748364+0000","last_change":"2024-09-13T08:40:37.746566+0000","last_active":"2024-09-13T08:40:37.748364+0000","last_peered":"2024-09-13T08:40:37.748364+0000","last_clean":"2024-09-13T08:40:37.748364+0000","last_became_active":"2024-09-13T08:40:37.746382+0000","last_became_peered":"2024-09-13T08:40:37.746382+0000","last_unstale":"2024-09-13T08:40:37.748364+0000","last_undegraded":"2024-09-13T08:40:37.748364+0000","last_fullsized":"2024-09-13T08:40:37.748364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.768664+0000","last_change":"2024-09-13T08:40:36.743612+0000","last_active":"2024-09-13T08:40:37.768664+0000","last_peered":"2024-09-13T08:40:37.768664+0000","last_clean":"2024-09-13T08:40:37.768664+0000","last_became_active":"2024-09-13T08:40:36.743337+0000","last_became_peered":"2024-09-13T08:40:36.743337+0000","last_unstale":"2024-09-13T08:40:37.768664+0000","last_undegraded":"2024-09-13T08:40:37.768664+0000","last_fullsized":"2024-09-13T08:40:37.768664+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747771+0000","last_change":"2024-09-13T08:40:37.746100+0000","last_active":"2024-09-13T08:40:37.747771+0000","last_peered":"2024-09-13T08:40:37.747771+0000","last_clean":"2024-09-13T08:40:37.747771+0000","last_became_active":"2024-09-13T08:40:37.745801+0000","last_became_peered":"2024-09-13T08:40:37.745801+0000","last_unstale":"2024-09-13T08:40:37.747771+0000","last_undegraded":"2024-09-13T08:40:37.747771+0000","last_fullsized":"2024-09-13T08:40:37.747771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745934+0000","last_change":"2024-09-13T08:40:36.751068+0000","last_active":"2024-09-13T08:40:37.745934+0000","last_peered":"2024-09-13T08:40:37.745934+0000","last_clean":"2024-09-13T08:40:37.745934+0000","last_became_active":"2024-09-13T08:40:36.748029+0000","last_became_peered":"2024-09-13T08:40:36.748029+0000","last_unstale":"2024-09-13T08:40:37.745934+0000","last_undegraded":"2024-09-13T08:40:37.745934+0000","last_fullsized":"2024-09-13T08:40:37.745934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:25.694691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.749201+0000","last_change":"2024-09-13T08:40:37.743421+0000","last_active":"2024-09-13T08:40:37.749201+0000","last_peered":"2024-09-13T08:40:37.749201+0000","last_clean":"2024-09-13T08:40:37.749201+0000","last_became_active":"2024-09-13T08:40:37.743126+0000","last_became_peered":"2024-09-13T08:40:37.743126+0000","last_unstale":"2024-09-13T08:40:37.749201+0000","last_undegraded":"2024-09-13T08:40:37.749201+0000","last_fullsized":"2024-09-13T08:40:37.749201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731313+0000","last_change":"2024-09-13T08:40:36.742793+0000","last_active":"2024-09-13T08:40:37.731313+0000","last_peered":"2024-09-13T08:40:37.731313+0000","last_clean":"2024-09-13T08:40:37.731313+0000","last_became_active":"2024-09-13T08:40:36.742516+0000","last_became_peered":"2024-09-13T08:40:36.742516+0000","last_unstale":"2024-09-13T08:40:37.731313+0000","last_undegraded":"2024-09-13T08:40:37.731313+0000","last_fullsized":"2024-09-13T08:40:37.731313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767726+0000","last_change":"2024-09-13T08:40:37.763959+0000","last_active":"2024-09-13T08:40:37.767726+0000","last_peered":"2024-09-13T08:40:37.767726+0000","last_clean":"2024-09-13T08:40:37.767726+0000","last_became_active":"2024-09-13T08:40:37.763661+0000","last_became_peered":"2024-09-13T08:40:37.763661+0000","last_unstale":"2024-09-13T08:40:37.767726+0000","last_undegraded":"2024-09-13T08:40:37.767726+0000","last_fullsized":"2024-09-13T08:40:37.767726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:46.895173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735588+0000","last_change":"2024-09-13T08:40:36.750833+0000","last_active":"2024-09-13T08:40:37.735588+0000","last_peered":"2024-09-13T08:40:37.735588+0000","last_clean":"2024-09-13T08:40:37.735588+0000","last_became_active":"2024-09-13T08:40:36.750471+0000","last_became_peered":"2024-09-13T08:40:36.750471+0000","last_unstale":"2024-09-13T08:40:37.735588+0000","last_undegraded":"2024-09-13T08:40:37.735588+0000","last_fullsized":"2024-09-13T08:40:37.735588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:36.394174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762046+0000","last_change":"2024-09-13T08:40:37.756241+0000","last_active":"2024-09-13T08:40:37.762046+0000","last_peered":"2024-09-13T08:40:37.762046+0000","last_clean":"2024-09-13T08:40:37.762046+0000","last_became_active":"2024-09-13T08:40:37.755961+0000","last_became_peered":"2024-09-13T08:40:37.755961+0000","last_unstale":"2024-09-13T08:40:37.762046+0000","last_undegraded":"2024-09-13T08:40:37.762046+0000","last_fullsized":"2024-09-13T08:40:37.762046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:38.286408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730472+0000","last_change":"2024-09-13T08:40:36.750422+0000","last_active":"2024-09-13T08:40:37.730472+0000","last_peered":"2024-09-13T08:40:37.730472+0000","last_clean":"2024-09-13T08:40:37.730472+0000","last_became_active":"2024-09-13T08:40:36.749512+0000","last_became_peered":"2024-09-13T08:40:36.749512+0000","last_unstale":"2024-09-13T08:40:37.730472+0000","last_undegraded":"2024-09-13T08:40:37.730472+0000","last_fullsized":"2024-09-13T08:40:37.730472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764478+0000","last_change":"2024-09-13T08:40:37.756404+0000","last_active":"2024-09-13T08:40:37.764478+0000","last_peered":"2024-09-13T08:40:37.764478+0000","last_clean":"2024-09-13T08:40:37.764478+0000","last_became_active":"2024-09-13T08:40:37.755037+0000","last_became_peered":"2024-09-13T08:40:37.755037+0000","last_unstale":"2024-09-13T08:40:37.764478+0000","last_undegraded":"2024-09-13T08:40:37.764478+0000","last_fullsized":"2024-09-13T08:40:37.764478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:11.905503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732259+0000","last_change":"2024-09-13T08:40:36.740510+0000","last_active":"2024-09-13T08:40:37.732259+0000","last_peered":"2024-09-13T08:40:37.732259+0000","last_clean":"2024-09-13T08:40:37.732259+0000","last_became_active":"2024-09-13T08:40:36.740243+0000","last_became_peered":"2024-09-13T08:40:36.740243+0000","last_unstale":"2024-09-13T08:40:37.732259+0000","last_undegraded":"2024-09-13T08:40:37.732259+0000","last_fullsized":"2024-09-13T08:40:37.732259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:20.155484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758212+0000","last_change":"2024-09-13T08:40:37.751854+0000","last_active":"2024-09-13T08:40:37.758212+0000","last_peered":"2024-09-13T08:40:37.758212+0000","last_clean":"2024-09-13T08:40:37.758212+0000","last_became_active":"2024-09-13T08:40:37.749368+0000","last_became_peered":"2024-09-13T08:40:37.749368+0000","last_unstale":"2024-09-13T08:40:37.758212+0000","last_undegraded":"2024-09-13T08:40:37.758212+0000","last_fullsized":"2024-09-13T08:40:37.758212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:05.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743930+0000","last_change":"2024-09-13T08:40:36.743024+0000","last_active":"2024-09-13T08:40:37.743930+0000","last_peered":"2024-09-13T08:40:37.743930+0000","last_clean":"2024-09-13T08:40:37.743930+0000","last_became_active":"2024-09-13T08:40:36.742739+0000","last_became_peered":"2024-09-13T08:40:36.742739+0000","last_unstale":"2024-09-13T08:40:37.743930+0000","last_undegraded":"2024-09-13T08:40:37.743930+0000","last_fullsized":"2024-09-13T08:40:37.743930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:54.197613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.753493+0000","last_change":"2024-09-13T08:40:37.743948+0000","last_active":"2024-09-13T08:40:37.753493+0000","last_peered":"2024-09-13T08:40:37.753493+0000","last_clean":"2024-09-13T08:40:37.753493+0000","last_became_active":"2024-09-13T08:40:37.743656+0000","last_became_peered":"2024-09-13T08:40:37.743656+0000","last_unstale":"2024-09-13T08:40:37.753493+0000","last_undegraded":"2024-09-13T08:40:37.753493+0000","last_fullsized":"2024-09-13T08:40:37.753493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735241+0000","last_change":"2024-09-13T08:40:36.750854+0000","last_active":"2024-09-13T08:40:37.735241+0000","last_peered":"2024-09-13T08:40:37.735241+0000","last_clean":"2024-09-13T08:40:37.735241+0000","last_became_active":"2024-09-13T08:40:36.750177+0000","last_became_peered":"2024-09-13T08:40:36.750177+0000","last_unstale":"2024-09-13T08:40:37.735241+0000","last_undegraded":"2024-09-13T08:40:37.735241+0000","last_fullsized":"2024-09-13T08:40:37.735241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:20.695241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.754437+0000","last_change":"2024-09-13T08:40:37.746462+0000","last_active":"2024-09-13T08:40:37.754437+0000","last_peered":"2024-09-13T08:40:37.754437+0000","last_clean":"2024-09-13T08:40:37.754437+0000","last_became_active":"2024-09-13T08:40:37.745179+0000","last_became_peered":"2024-09-13T08:40:37.745179+0000","last_unstale":"2024-09-13T08:40:37.754437+0000","last_undegraded":"2024-09-13T08:40:37.754437+0000","last_fullsized":"2024-09-13T08:40:37.754437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:07.391500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.737843+0000","last_change":"2024-09-13T08:40:36.744466+0000","last_active":"2024-09-13T08:40:37.737843+0000","last_peered":"2024-09-13T08:40:37.737843+0000","last_clean":"2024-09-13T08:40:37.737843+0000","last_became_active":"2024-09-13T08:40:36.744206+0000","last_became_peered":"2024-09-13T08:40:36.744206+0000","last_unstale":"2024-09-13T08:40:37.737843+0000","last_undegraded":"2024-09-13T08:40:37.737843+0000","last_fullsized":"2024-09-13T08:40:37.737843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:40.300855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":179,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732740+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:40:37.732740+0000","last_peered":"2024-09-13T08:40:37.732740+0000","last_clean":"2024-09-13T08:40:37.732740+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:40:37.732740+0000","last_undegraded":"2024-09-13T08:40:37.732740+0000","last_fullsized":"2024-09-13T08:40:37.732740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":299641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509487,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1323008,"data_stored":890021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":729088,"data_stored":299709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":745472,"data_stored":301391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1306624,"data_stored":883212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509486,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":737280,"data_stored":301231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36048,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:40:42.889 INFO:tasks.ceph.ceph_manager.ceph:PG 9.35 is not active+clean 2024-09-13T08:40:42.889 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-09-13T08:40:36.726360+0000', 'last_change': '2024-09-13T08:40:36.726360+0000', 'last_active': '2024-09-13T08:40:36.715339+0000', 'last_peered': '2024-09-13T08:40:36.715339+0000', 'last_clean': '2024-09-13T08:40:36.715339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T08:40:36.726360+0000', 'last_undegraded': '2024-09-13T08:40:36.726360+0000', 'last_fullsized': '2024-09-13T08:40:36.726360+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_clean_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T08:40:42.889 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2c is not active+clean 2024-09-13T08:40:42.889 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-09-13T08:40:36.727096+0000', 'last_change': '2024-09-13T08:40:36.727096+0000', 'last_active': '2024-09-13T08:40:36.715339+0000', 'last_peered': '2024-09-13T08:40:36.715339+0000', 'last_clean': '2024-09-13T08:40:36.715339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T08:40:36.727096+0000', 'last_undegraded': '2024-09-13T08:40:36.727096+0000', 'last_fullsized': '2024-09-13T08:40:36.727096+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_clean_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2a is not active+clean 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-09-13T08:40:36.727284+0000', 'last_change': '2024-09-13T08:40:36.727284+0000', 'last_active': '2024-09-13T08:40:36.715339+0000', 'last_peered': '2024-09-13T08:40:36.715339+0000', 'last_clean': '2024-09-13T08:40:36.715339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T08:40:36.727284+0000', 'last_undegraded': '2024-09-13T08:40:36.727284+0000', 'last_fullsized': '2024-09-13T08:40:36.727284+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_clean_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:PG 9.20 is not active+clean 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-09-13T08:40:36.728732+0000', 'last_change': '2024-09-13T08:40:36.728732+0000', 'last_active': '2024-09-13T08:40:36.715339+0000', 'last_peered': '2024-09-13T08:40:36.715339+0000', 'last_clean': '2024-09-13T08:40:36.715339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T08:40:36.728732+0000', 'last_undegraded': '2024-09-13T08:40:36.728732+0000', 'last_fullsized': '2024-09-13T08:40:36.728732+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_clean_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1b is not active+clean 2024-09-13T08:40:42.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-09-13T08:40:36.727899+0000', 'last_change': '2024-09-13T08:40:36.727899+0000', 'last_active': '2024-09-13T08:40:36.715339+0000', 'last_peered': '2024-09-13T08:40:36.715339+0000', 'last_clean': '2024-09-13T08:40:36.715339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T08:40:36.727899+0000', 'last_undegraded': '2024-09-13T08:40:36.727899+0000', 'last_fullsized': '2024-09-13T08:40:36.727899+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'last_clean_scrub_stamp': '2024-09-13T08:40:36.715339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T08:40:42.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T08:40:43.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 -- 172.21.15.31:0/321891674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 msgr2=0x7fb71c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 --2- 172.21.15.31:0/321891674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb71c004c80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb720009b50 tx=0x7fb71c005830 comp rx=0 tx=0).stop 2024-09-13T08:40:43.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73a400640 1 -- 172.21.15.31:0/321891674 reap_dead start 2024-09-13T08:40:43.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 -- 172.21.15.31:0/321891674 shutdown_connections 2024-09-13T08:40:43.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 -- 172.21.15.31:0/321891674 >> 172.21.15.31:0/321891674 conn(0x7fb7340cc910 msgr2=0x7fb7340cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:43.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 -- 172.21.15.31:0/321891674 shutdown_connections 2024-09-13T08:40:43.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.085+0000 7fb73c400640 1 -- 172.21.15.31:0/321891674 wait complete. 2024-09-13T08:40:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.087+0000 7fb73c400640 1 Processor -- start 2024-09-13T08:40:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.087+0000 7fb73c400640 1 -- start start 2024-09-13T08:40:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 0x7fb734157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7340c88e0 0x7fb7340c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7340cc480 con 0x7fb71c004800 2024-09-13T08:40:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7340cc300 con 0x7fb7340c9930 2024-09-13T08:40:43.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73c400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb7340cc600 con 0x7fb7340c88e0 2024-09-13T08:40:43.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45564/0 (socket says 172.21.15.31:45564) 2024-09-13T08:40:43.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 -- 172.21.15.31:0/4245541871 learned_addr learned my addr 172.21.15.31:0/4245541871 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:43.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739000640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 0x7fb734157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb73a400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7340c88e0 0x7fb7340c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 -- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7340c88e0 msgr2=0x7fb7340c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7340c88e0 0x7fb7340c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 -- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 msgr2=0x7fb734157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 0x7fb734157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739a00640 1 -- 172.21.15.31:0/4245541871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7341f58d0 con 0x7fb71c004800 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.088+0000 7fb739000640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 0x7fb734157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb739a00640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e5e0 tx=0x7fb72800eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728015070 con 0x7fb71c004800 2024-09-13T08:40:43.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb72800ca60 con 0x7fb71c004800 2024-09-13T08:40:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72800ce50 con 0x7fb71c004800 2024-09-13T08:40:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7341f5bb0 con 0x7fb71c004800 2024-09-13T08:40:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.089+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7340c80e0 con 0x7fb71c004800 2024-09-13T08:40:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.090+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7340b1da0 con 0x7fb71c004800 2024-09-13T08:40:43.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.091+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fb728018070 con 0x7fb71c004800 2024-09-13T08:40:43.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.092+0000 7fb731600640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 0x7fb71407b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.092+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fb72809b280 con 0x7fb71c004800 2024-09-13T08:40:43.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.093+0000 7fb739000640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 0x7fb71407b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.094+0000 7fb739000640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 0x7fb71407b060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb720035000 tx=0x7fb72002d040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.096+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb7340cbe40 con 0x7fb71c004800 2024-09-13T08:40:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.365+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb7340adc10 con 0x7fb71c004800 2024-09-13T08:40:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.366+0000 7fb731600640 1 -- 172.21.15.31:0/4245541871 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1141 (secure 0 0 0) 0x7fb7280634c0 con 0x7fb71c004800 2024-09-13T08:40:43.367 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:43.367 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"0251b593-a02a-47fa-b938-c065b7284822","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":239,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":8,"num_up_osds":8,"osd_up_since":1726216617,"num_in_osds":8,"osd_in_since":1726216607,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":92},{"state_name":"creating+peering","count":5}],"num_pgs":97,"num_pools":3,"num_objects":20,"data_bytes":592196,"bytes_used":254005248,"bytes_avail":772840108032,"bytes_total":773094113280,"inactive_pgs_ratio":0.051546391099691391,"write_bytes_sec":2477,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":37,"btime":"2024-09-13T08:40:40:762682+0000","id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":5748}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-13T08:39:43.074239+0000","services":{}},"progress_events":{}} 2024-09-13T08:40:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 msgr2=0x7fb71407b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 0x7fb71407b060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb720035000 tx=0x7fb72002d040 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 msgr2=0x7fb7341f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e5e0 tx=0x7fb72800eaa0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 shutdown_connections 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.368+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb714078ba0 0x7fb71407b060 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7340c88e0 0x7fb7340c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7340c9930 0x7fb734157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 --2- 172.21.15.31:0/4245541871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb71c004800 0x7fb7341f5030 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 >> 172.21.15.31:0/4245541871 conn(0x7fb7340cc910 msgr2=0x7fb7340b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 shutdown_connections 2024-09-13T08:40:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.369+0000 7fb73c400640 1 -- 172.21.15.31:0/4245541871 wait complete. 2024-09-13T08:40:43.385 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T08:40:43.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.546+0000 7fa7b3200640 1 -- 172.21.15.31:0/3126688215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 msgr2=0x7fa7b4152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.546+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3126688215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b4152680 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40cec50 tx=0x7fa7ac02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 -- 172.21.15.31:0/3126688215 shutdown_connections 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3126688215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b4152680 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3126688215 >> v2:172.21.15.45:3300/0 conn(0x7fa7b40b52d0 0x7fa7b406f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 -- 172.21.15.31:0/3126688215 >> 172.21.15.31:0/3126688215 conn(0x7fa7b406cb00 msgr2=0x7fa7b406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 -- 172.21.15.31:0/3126688215 shutdown_connections 2024-09-13T08:40:43.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.547+0000 7fa7b3200640 1 -- 172.21.15.31:0/3126688215 wait complete. 2024-09-13T08:40:43.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.548+0000 7fa7b3200640 1 Processor -- start 2024-09-13T08:40:43.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.548+0000 7fa7b3200640 1 -- start start 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7b40b52d0 0x7fa7b40c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7b40cc720 0x7fa7b40c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4152bc0 con 0x7fa7b40ccb00 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b40703b0 con 0x7fa7b40b52d0 2024-09-13T08:40:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b4152d40 con 0x7fa7b40cc720 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.549+0000 7fa7b2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45582/0 (socket says 172.21.15.31:45582) 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 -- 172.21.15.31:0/3789038387 learned_addr learned my addr 172.21.15.31:0/3789038387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b1e00640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7b40b52d0 0x7fa7b40c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7a9400640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7b40cc720 0x7fa7b40c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 -- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7b40cc720 msgr2=0x7fa7b40c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7b40cc720 0x7fa7b40c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 -- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7b40b52d0 msgr2=0x7fa7b40c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7b40b52d0 0x7fa7b40c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 -- 172.21.15.31:0/3789038387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40cad70 con 0x7fa7b40ccb00 2024-09-13T08:40:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.550+0000 7fa7b2800640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009ca0 tx=0x7fa7ac003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.551+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac031990 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.551+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7ac031c20 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.551+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b40cb050 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.551+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0330d0 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.551+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b40cb610 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.552+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b406f9b0 con 0x7fa7b40ccb00 2024-09-13T08:40:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.553+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7fa7ac033290 con 0x7fa7b40ccb00 2024-09-13T08:40:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.554+0000 7fa7aaa00640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 0x7fa79007aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.554+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7fa7ac0bc120 con 0x7fa7b40ccb00 2024-09-13T08:40:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.554+0000 7fa7b1e00640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 0x7fa79007aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.555+0000 7fa7b1e00640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 0x7fa79007aef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4007fd0 tx=0x7fa7a4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.558+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa7ac03f030 con 0x7fa7b40ccb00 2024-09-13T08:40:43.714 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-13T08:40:43.713+0000 7fd07a400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:40:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.751+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7b40cb980 con 0x7fa790078a30 2024-09-13T08:40:43.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.756+0000 7fa7aaa00640 1 -- 172.21.15.31:0/3789038387 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7fa7b40cb980 con 0x7fa790078a30 2024-09-13T08:40:43.761 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:43.763 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:40:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 msgr2=0x7fa79007aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 0x7fa79007aef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4007fd0 tx=0x7fa7a4009040 comp rx=0 tx=0).stop 2024-09-13T08:40:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 msgr2=0x7fa7b40ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009ca0 tx=0x7fa7ac003990 comp rx=0 tx=0).stop 2024-09-13T08:40:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 shutdown_connections 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b40ccb00 0x7fa7b40ca0f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa790078a30 0x7fa79007aef0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7b40cc720 0x7fa7b40c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 --2- 172.21.15.31:0/3789038387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7b40b52d0 0x7fa7b40c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 >> 172.21.15.31:0/3789038387 conn(0x7fa7b406cb00 msgr2=0x7fa7b41552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 shutdown_connections 2024-09-13T08:40:43.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:43.760+0000 7fa7b3200640 1 -- 172.21.15.31:0/3789038387 wait complete. 2024-09-13T08:40:43.777 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-13T08:40:43.090691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248080,"kb_used_data":6864,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726640,"statfs":{"total":773094113280,"available":772840079360,"internally_reserved":0,"allocated":7028736,"data_stored":3574584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.367216"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.761134+0000","last_change":"2024-09-13T08:40:37.757129+0000","last_active":"2024-09-13T08:40:37.761134+0000","last_peered":"2024-09-13T08:40:37.761134+0000","last_clean":"2024-09-13T08:40:37.761134+0000","last_became_active":"2024-09-13T08:40:37.756648+0000","last_became_peered":"2024-09-13T08:40:37.756648+0000","last_unstale":"2024-09-13T08:40:37.761134+0000","last_undegraded":"2024-09-13T08:40:37.761134+0000","last_fullsized":"2024-09-13T08:40:37.761134+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757494+0000","last_change":"2024-09-13T08:40:37.756234+0000","last_active":"2024-09-13T08:40:37.757494+0000","last_peered":"2024-09-13T08:40:37.757494+0000","last_clean":"2024-09-13T08:40:37.757494+0000","last_became_active":"2024-09-13T08:40:37.755925+0000","last_became_peered":"2024-09-13T08:40:37.755925+0000","last_unstale":"2024-09-13T08:40:37.757494+0000","last_undegraded":"2024-09-13T08:40:37.757494+0000","last_fullsized":"2024-09-13T08:40:37.757494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:24.421420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760116+0000","last_change":"2024-09-13T08:40:37.756481+0000","last_active":"2024-09-13T08:40:37.760116+0000","last_peered":"2024-09-13T08:40:37.760116+0000","last_clean":"2024-09-13T08:40:37.760116+0000","last_became_active":"2024-09-13T08:40:37.756190+0000","last_became_peered":"2024-09-13T08:40:37.756190+0000","last_unstale":"2024-09-13T08:40:37.760116+0000","last_undegraded":"2024-09-13T08:40:37.760116+0000","last_fullsized":"2024-09-13T08:40:37.760116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:45.588595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759390+0000","last_change":"2024-09-13T08:40:37.752214+0000","last_active":"2024-09-13T08:40:37.759390+0000","last_peered":"2024-09-13T08:40:37.759390+0000","last_clean":"2024-09-13T08:40:37.759390+0000","last_became_active":"2024-09-13T08:40:37.752015+0000","last_became_peered":"2024-09-13T08:40:37.752015+0000","last_unstale":"2024-09-13T08:40:37.759390+0000","last_undegraded":"2024-09-13T08:40:37.759390+0000","last_fullsized":"2024-09-13T08:40:37.759390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:19.281735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755493+0000","last_change":"2024-09-13T08:40:37.749549+0000","last_active":"2024-09-13T08:40:37.755493+0000","last_peered":"2024-09-13T08:40:37.755493+0000","last_clean":"2024-09-13T08:40:37.755493+0000","last_became_active":"2024-09-13T08:40:37.748944+0000","last_became_peered":"2024-09-13T08:40:37.748944+0000","last_unstale":"2024-09-13T08:40:37.755493+0000","last_undegraded":"2024-09-13T08:40:37.755493+0000","last_fullsized":"2024-09-13T08:40:37.755493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:05.311605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762020+0000","last_change":"2024-09-13T08:40:37.756210+0000","last_active":"2024-09-13T08:40:37.762020+0000","last_peered":"2024-09-13T08:40:37.762020+0000","last_clean":"2024-09-13T08:40:37.762020+0000","last_became_active":"2024-09-13T08:40:37.755876+0000","last_became_peered":"2024-09-13T08:40:37.755876+0000","last_unstale":"2024-09-13T08:40:37.762020+0000","last_undegraded":"2024-09-13T08:40:37.762020+0000","last_fullsized":"2024-09-13T08:40:37.762020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:37.401016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743263+0000","last_change":"2024-09-13T08:40:37.738548+0000","last_active":"2024-09-13T08:40:37.743263+0000","last_peered":"2024-09-13T08:40:37.743263+0000","last_clean":"2024-09-13T08:40:37.743263+0000","last_became_active":"2024-09-13T08:40:37.738294+0000","last_became_peered":"2024-09-13T08:40:37.738294+0000","last_unstale":"2024-09-13T08:40:37.743263+0000","last_undegraded":"2024-09-13T08:40:37.743263+0000","last_fullsized":"2024-09-13T08:40:37.743263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:11.567595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758242+0000","last_change":"2024-09-13T08:40:37.752144+0000","last_active":"2024-09-13T08:40:37.758242+0000","last_peered":"2024-09-13T08:40:37.758242+0000","last_clean":"2024-09-13T08:40:37.758242+0000","last_became_active":"2024-09-13T08:40:37.751574+0000","last_became_peered":"2024-09-13T08:40:37.751574+0000","last_unstale":"2024-09-13T08:40:37.758242+0000","last_undegraded":"2024-09-13T08:40:37.758242+0000","last_fullsized":"2024-09-13T08:40:37.758242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.900728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765529+0000","last_change":"2024-09-13T08:40:37.759047+0000","last_active":"2024-09-13T08:40:37.765529+0000","last_peered":"2024-09-13T08:40:37.765529+0000","last_clean":"2024-09-13T08:40:37.765529+0000","last_became_active":"2024-09-13T08:40:37.757616+0000","last_became_peered":"2024-09-13T08:40:37.757616+0000","last_unstale":"2024-09-13T08:40:37.765529+0000","last_undegraded":"2024-09-13T08:40:37.765529+0000","last_fullsized":"2024-09-13T08:40:37.765529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769285+0000","last_change":"2024-09-13T08:40:37.766302+0000","last_active":"2024-09-13T08:40:37.769285+0000","last_peered":"2024-09-13T08:40:37.769285+0000","last_clean":"2024-09-13T08:40:37.769285+0000","last_became_active":"2024-09-13T08:40:37.765985+0000","last_became_peered":"2024-09-13T08:40:37.765985+0000","last_unstale":"2024-09-13T08:40:37.769285+0000","last_undegraded":"2024-09-13T08:40:37.769285+0000","last_fullsized":"2024-09-13T08:40:37.769285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750260+0000","last_change":"2024-09-13T08:40:37.750961+0000","last_active":"2024-09-13T08:40:38.750260+0000","last_peered":"2024-09-13T08:40:38.750260+0000","last_clean":"2024-09-13T08:40:38.750260+0000","last_became_active":"2024-09-13T08:40:37.750706+0000","last_became_peered":"2024-09-13T08:40:37.750706+0000","last_unstale":"2024-09-13T08:40:38.750260+0000","last_undegraded":"2024-09-13T08:40:38.750260+0000","last_fullsized":"2024-09-13T08:40:38.750260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:57:35.246408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767321+0000","last_change":"2024-09-13T08:40:37.762085+0000","last_active":"2024-09-13T08:40:37.767321+0000","last_peered":"2024-09-13T08:40:37.767321+0000","last_clean":"2024-09-13T08:40:37.767321+0000","last_became_active":"2024-09-13T08:40:37.761803+0000","last_became_peered":"2024-09-13T08:40:37.761803+0000","last_unstale":"2024-09-13T08:40:37.767321+0000","last_undegraded":"2024-09-13T08:40:37.767321+0000","last_fullsized":"2024-09-13T08:40:37.767321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760491+0000","last_change":"2024-09-13T08:40:37.742869+0000","last_active":"2024-09-13T08:40:37.760491+0000","last_peered":"2024-09-13T08:40:37.760491+0000","last_clean":"2024-09-13T08:40:37.760491+0000","last_became_active":"2024-09-13T08:40:37.742566+0000","last_became_peered":"2024-09-13T08:40:37.742566+0000","last_unstale":"2024-09-13T08:40:37.760491+0000","last_undegraded":"2024-09-13T08:40:37.760491+0000","last_fullsized":"2024-09-13T08:40:37.760491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:40.715704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755923+0000","last_change":"2024-09-13T08:40:37.750839+0000","last_active":"2024-09-13T08:40:37.755923+0000","last_peered":"2024-09-13T08:40:37.755923+0000","last_clean":"2024-09-13T08:40:37.755923+0000","last_became_active":"2024-09-13T08:40:37.750572+0000","last_became_peered":"2024-09-13T08:40:37.750572+0000","last_unstale":"2024-09-13T08:40:37.755923+0000","last_undegraded":"2024-09-13T08:40:37.755923+0000","last_fullsized":"2024-09-13T08:40:37.755923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:09:45.194596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764455+0000","last_change":"2024-09-13T08:40:37.755720+0000","last_active":"2024-09-13T08:40:37.764455+0000","last_peered":"2024-09-13T08:40:37.764455+0000","last_clean":"2024-09-13T08:40:37.764455+0000","last_became_active":"2024-09-13T08:40:37.755419+0000","last_became_peered":"2024-09-13T08:40:37.755419+0000","last_unstale":"2024-09-13T08:40:37.764455+0000","last_undegraded":"2024-09-13T08:40:37.764455+0000","last_fullsized":"2024-09-13T08:40:37.764455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:41.234982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756494+0000","last_change":"2024-09-13T08:40:37.748027+0000","last_active":"2024-09-13T08:40:37.756494+0000","last_peered":"2024-09-13T08:40:37.756494+0000","last_clean":"2024-09-13T08:40:37.756494+0000","last_became_active":"2024-09-13T08:40:37.747736+0000","last_became_peered":"2024-09-13T08:40:37.747736+0000","last_unstale":"2024-09-13T08:40:37.756494+0000","last_undegraded":"2024-09-13T08:40:37.756494+0000","last_fullsized":"2024-09-13T08:40:37.756494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:31.800897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763918+0000","last_change":"2024-09-13T08:40:37.759006+0000","last_active":"2024-09-13T08:40:37.763918+0000","last_peered":"2024-09-13T08:40:37.763918+0000","last_clean":"2024-09-13T08:40:37.763918+0000","last_became_active":"2024-09-13T08:40:37.758700+0000","last_became_peered":"2024-09-13T08:40:37.758700+0000","last_unstale":"2024-09-13T08:40:37.763918+0000","last_undegraded":"2024-09-13T08:40:37.763918+0000","last_fullsized":"2024-09-13T08:40:37.763918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.757199+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.756901+0000","last_became_peered":"2024-09-13T08:40:37.756901+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756708+0000","last_change":"2024-09-13T08:40:37.752015+0000","last_active":"2024-09-13T08:40:37.756708+0000","last_peered":"2024-09-13T08:40:37.756708+0000","last_clean":"2024-09-13T08:40:37.756708+0000","last_became_active":"2024-09-13T08:40:37.751710+0000","last_became_peered":"2024-09-13T08:40:37.751710+0000","last_unstale":"2024-09-13T08:40:37.756708+0000","last_undegraded":"2024-09-13T08:40:37.756708+0000","last_fullsized":"2024-09-13T08:40:37.756708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:14.240052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749566+0000","last_change":"2024-09-13T08:40:37.752392+0000","last_active":"2024-09-13T08:40:38.749566+0000","last_peered":"2024-09-13T08:40:38.749566+0000","last_clean":"2024-09-13T08:40:38.749566+0000","last_became_active":"2024-09-13T08:40:37.752052+0000","last_became_peered":"2024-09-13T08:40:37.752052+0000","last_unstale":"2024-09-13T08:40:38.749566+0000","last_undegraded":"2024-09-13T08:40:38.749566+0000","last_fullsized":"2024-09-13T08:40:38.749566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:24.659958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764451+0000","last_change":"2024-09-13T08:40:37.751434+0000","last_active":"2024-09-13T08:40:37.764451+0000","last_peered":"2024-09-13T08:40:37.764451+0000","last_clean":"2024-09-13T08:40:37.764451+0000","last_became_active":"2024-09-13T08:40:37.745985+0000","last_became_peered":"2024-09-13T08:40:37.745985+0000","last_unstale":"2024-09-13T08:40:37.764451+0000","last_undegraded":"2024-09-13T08:40:37.764451+0000","last_fullsized":"2024-09-13T08:40:37.764451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:33.990893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749808+0000","last_change":"2024-09-13T08:40:37.741617+0000","last_active":"2024-09-13T08:40:38.749808+0000","last_peered":"2024-09-13T08:40:38.749808+0000","last_clean":"2024-09-13T08:40:38.749808+0000","last_became_active":"2024-09-13T08:40:37.741317+0000","last_became_peered":"2024-09-13T08:40:37.741317+0000","last_unstale":"2024-09-13T08:40:38.749808+0000","last_undegraded":"2024-09-13T08:40:38.749808+0000","last_fullsized":"2024-09-13T08:40:38.749808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:51.165235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740716+0000","last_change":"2024-09-13T08:40:37.734978+0000","last_active":"2024-09-13T08:40:37.740716+0000","last_peered":"2024-09-13T08:40:37.740716+0000","last_clean":"2024-09-13T08:40:37.740716+0000","last_became_active":"2024-09-13T08:40:37.734070+0000","last_became_peered":"2024-09-13T08:40:37.734070+0000","last_unstale":"2024-09-13T08:40:37.740716+0000","last_undegraded":"2024-09-13T08:40:37.740716+0000","last_fullsized":"2024-09-13T08:40:37.740716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:54.518342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755838+0000","last_change":"2024-09-13T08:40:37.750441+0000","last_active":"2024-09-13T08:40:37.755838+0000","last_peered":"2024-09-13T08:40:37.755838+0000","last_clean":"2024-09-13T08:40:37.755838+0000","last_became_active":"2024-09-13T08:40:37.750132+0000","last_became_peered":"2024-09-13T08:40:37.750132+0000","last_unstale":"2024-09-13T08:40:37.755838+0000","last_undegraded":"2024-09-13T08:40:37.755838+0000","last_fullsized":"2024-09-13T08:40:37.755838+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755293+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.755293+0000","last_peered":"2024-09-13T08:40:37.755293+0000","last_clean":"2024-09-13T08:40:37.755293+0000","last_became_active":"2024-09-13T08:40:37.750728+0000","last_became_peered":"2024-09-13T08:40:37.750728+0000","last_unstale":"2024-09-13T08:40:37.755293+0000","last_undegraded":"2024-09-13T08:40:37.755293+0000","last_fullsized":"2024-09-13T08:40:37.755293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:59.465026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760327+0000","last_change":"2024-09-13T08:40:37.741441+0000","last_active":"2024-09-13T08:40:37.760327+0000","last_peered":"2024-09-13T08:40:37.760327+0000","last_clean":"2024-09-13T08:40:37.760327+0000","last_became_active":"2024-09-13T08:40:37.737794+0000","last_became_peered":"2024-09-13T08:40:37.737794+0000","last_unstale":"2024-09-13T08:40:37.760327+0000","last_undegraded":"2024-09-13T08:40:37.760327+0000","last_fullsized":"2024-09-13T08:40:37.760327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:41.031994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755203+0000","last_change":"2024-09-13T08:40:37.748613+0000","last_active":"2024-09-13T08:40:37.755203+0000","last_peered":"2024-09-13T08:40:37.755203+0000","last_clean":"2024-09-13T08:40:37.755203+0000","last_became_active":"2024-09-13T08:40:37.745423+0000","last_became_peered":"2024-09-13T08:40:37.745423+0000","last_unstale":"2024-09-13T08:40:37.755203+0000","last_undegraded":"2024-09-13T08:40:37.755203+0000","last_fullsized":"2024-09-13T08:40:37.755203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:24.871551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.765146+0000","last_change":"2024-09-13T08:40:37.762367+0000","last_active":"2024-09-13T08:40:37.765146+0000","last_peered":"2024-09-13T08:40:37.765146+0000","last_clean":"2024-09-13T08:40:37.765146+0000","last_became_active":"2024-09-13T08:40:37.762030+0000","last_became_peered":"2024-09-13T08:40:37.762030+0000","last_unstale":"2024-09-13T08:40:37.765146+0000","last_undegraded":"2024-09-13T08:40:37.765146+0000","last_fullsized":"2024-09-13T08:40:37.765146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:20.588520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758045+0000","last_change":"2024-09-13T08:40:37.750469+0000","last_active":"2024-09-13T08:40:37.758045+0000","last_peered":"2024-09-13T08:40:37.758045+0000","last_clean":"2024-09-13T08:40:37.758045+0000","last_became_active":"2024-09-13T08:40:37.748954+0000","last_became_peered":"2024-09-13T08:40:37.748954+0000","last_unstale":"2024-09-13T08:40:37.758045+0000","last_undegraded":"2024-09-13T08:40:37.758045+0000","last_fullsized":"2024-09-13T08:40:37.758045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:04.282317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747600+0000","last_change":"2024-09-13T08:40:37.744539+0000","last_active":"2024-09-13T08:40:37.747600+0000","last_peered":"2024-09-13T08:40:37.747600+0000","last_clean":"2024-09-13T08:40:37.747600+0000","last_became_active":"2024-09-13T08:40:37.744247+0000","last_became_peered":"2024-09-13T08:40:37.744247+0000","last_unstale":"2024-09-13T08:40:37.747600+0000","last_undegraded":"2024-09-13T08:40:37.747600+0000","last_fullsized":"2024-09-13T08:40:37.747600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:14.087231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.756561+0000","last_change":"2024-09-13T08:40:37.750709+0000","last_active":"2024-09-13T08:40:37.756561+0000","last_peered":"2024-09-13T08:40:37.756561+0000","last_clean":"2024-09-13T08:40:37.756561+0000","last_became_active":"2024-09-13T08:40:37.749124+0000","last_became_peered":"2024-09-13T08:40:37.749124+0000","last_unstale":"2024-09-13T08:40:37.756561+0000","last_undegraded":"2024-09-13T08:40:37.756561+0000","last_fullsized":"2024-09-13T08:40:37.756561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749189+0000","last_change":"2024-09-13T08:40:37.741816+0000","last_active":"2024-09-13T08:40:38.749189+0000","last_peered":"2024-09-13T08:40:38.749189+0000","last_clean":"2024-09-13T08:40:38.749189+0000","last_became_active":"2024-09-13T08:40:37.741441+0000","last_became_peered":"2024-09-13T08:40:37.741441+0000","last_unstale":"2024-09-13T08:40:38.749189+0000","last_undegraded":"2024-09-13T08:40:38.749189+0000","last_fullsized":"2024-09-13T08:40:38.749189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:05.154898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763275+0000","last_change":"2024-09-13T08:40:37.758027+0000","last_active":"2024-09-13T08:40:37.763275+0000","last_peered":"2024-09-13T08:40:37.763275+0000","last_clean":"2024-09-13T08:40:37.763275+0000","last_became_active":"2024-09-13T08:40:37.757576+0000","last_became_peered":"2024-09-13T08:40:37.757576+0000","last_unstale":"2024-09-13T08:40:37.763275+0000","last_undegraded":"2024-09-13T08:40:37.763275+0000","last_fullsized":"2024-09-13T08:40:37.763275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"61'4","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749923+0000","last_change":"2024-09-13T08:40:36.740867+0000","last_active":"2024-09-13T08:40:38.749923+0000","last_peered":"2024-09-13T08:40:38.749923+0000","last_clean":"2024-09-13T08:40:38.749923+0000","last_became_active":"2024-09-13T08:40:36.740475+0000","last_became_peered":"2024-09-13T08:40:36.740475+0000","last_unstale":"2024-09-13T08:40:38.749923+0000","last_undegraded":"2024-09-13T08:40:38.749923+0000","last_fullsized":"2024-09-13T08:40:38.749923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:29.262920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.763350+0000","last_change":"2024-09-13T08:40:37.755669+0000","last_active":"2024-09-13T08:40:37.763350+0000","last_peered":"2024-09-13T08:40:37.763350+0000","last_clean":"2024-09-13T08:40:37.763350+0000","last_became_active":"2024-09-13T08:40:37.755390+0000","last_became_peered":"2024-09-13T08:40:37.755390+0000","last_unstale":"2024-09-13T08:40:37.763350+0000","last_undegraded":"2024-09-13T08:40:37.763350+0000","last_fullsized":"2024-09-13T08:40:37.763350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770118+0000","last_change":"2024-09-13T08:40:36.749525+0000","last_active":"2024-09-13T08:40:37.770118+0000","last_peered":"2024-09-13T08:40:37.770118+0000","last_clean":"2024-09-13T08:40:37.770118+0000","last_became_active":"2024-09-13T08:40:36.749236+0000","last_became_peered":"2024-09-13T08:40:36.749236+0000","last_unstale":"2024-09-13T08:40:37.770118+0000","last_undegraded":"2024-09-13T08:40:37.770118+0000","last_fullsized":"2024-09-13T08:40:37.770118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759658+0000","last_change":"2024-09-13T08:40:37.758005+0000","last_active":"2024-09-13T08:40:37.759658+0000","last_peered":"2024-09-13T08:40:37.759658+0000","last_clean":"2024-09-13T08:40:37.759658+0000","last_became_active":"2024-09-13T08:40:37.757312+0000","last_became_peered":"2024-09-13T08:40:37.757312+0000","last_unstale":"2024-09-13T08:40:37.759658+0000","last_undegraded":"2024-09-13T08:40:37.759658+0000","last_fullsized":"2024-09-13T08:40:37.759658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:18:18.952772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769133+0000","last_change":"2024-09-13T08:40:36.751851+0000","last_active":"2024-09-13T08:40:37.769133+0000","last_peered":"2024-09-13T08:40:37.769133+0000","last_clean":"2024-09-13T08:40:37.769133+0000","last_became_active":"2024-09-13T08:40:36.748632+0000","last_became_peered":"2024-09-13T08:40:36.748632+0000","last_unstale":"2024-09-13T08:40:37.769133+0000","last_undegraded":"2024-09-13T08:40:37.769133+0000","last_fullsized":"2024-09-13T08:40:37.769133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:56.150885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757513+0000","last_change":"2024-09-13T08:40:37.751225+0000","last_active":"2024-09-13T08:40:37.757513+0000","last_peered":"2024-09-13T08:40:37.757513+0000","last_clean":"2024-09-13T08:40:37.757513+0000","last_became_active":"2024-09-13T08:40:37.750923+0000","last_became_peered":"2024-09-13T08:40:37.750923+0000","last_unstale":"2024-09-13T08:40:37.757513+0000","last_undegraded":"2024-09-13T08:40:37.757513+0000","last_fullsized":"2024-09-13T08:40:37.757513+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:21.162327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772713+0000","last_change":"2024-09-13T08:40:36.746077+0000","last_active":"2024-09-13T08:40:37.772713+0000","last_peered":"2024-09-13T08:40:37.772713+0000","last_clean":"2024-09-13T08:40:37.772713+0000","last_became_active":"2024-09-13T08:40:36.745814+0000","last_became_peered":"2024-09-13T08:40:36.745814+0000","last_unstale":"2024-09-13T08:40:37.772713+0000","last_undegraded":"2024-09-13T08:40:37.772713+0000","last_fullsized":"2024-09-13T08:40:37.772713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:24.123182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.757961+0000","last_change":"2024-09-13T08:40:37.749662+0000","last_active":"2024-09-13T08:40:37.757961+0000","last_peered":"2024-09-13T08:40:37.757961+0000","last_clean":"2024-09-13T08:40:37.757961+0000","last_became_active":"2024-09-13T08:40:37.749218+0000","last_became_peered":"2024-09-13T08:40:37.749218+0000","last_unstale":"2024-09-13T08:40:37.757961+0000","last_undegraded":"2024-09-13T08:40:37.757961+0000","last_fullsized":"2024-09-13T08:40:37.757961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:27.923545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769027+0000","last_change":"2024-09-13T08:40:36.738659+0000","last_active":"2024-09-13T08:40:37.769027+0000","last_peered":"2024-09-13T08:40:37.769027+0000","last_clean":"2024-09-13T08:40:37.769027+0000","last_became_active":"2024-09-13T08:40:36.738315+0000","last_became_peered":"2024-09-13T08:40:36.738315+0000","last_unstale":"2024-09-13T08:40:37.769027+0000","last_undegraded":"2024-09-13T08:40:37.769027+0000","last_fullsized":"2024-09-13T08:40:37.769027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:09.635872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749651+0000","last_change":"2024-09-13T08:40:37.741822+0000","last_active":"2024-09-13T08:40:38.749651+0000","last_peered":"2024-09-13T08:40:38.749651+0000","last_clean":"2024-09-13T08:40:38.749651+0000","last_became_active":"2024-09-13T08:40:37.741264+0000","last_became_peered":"2024-09-13T08:40:37.741264+0000","last_unstale":"2024-09-13T08:40:38.749651+0000","last_undegraded":"2024-09-13T08:40:38.749651+0000","last_fullsized":"2024-09-13T08:40:38.749651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:25:36.907735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772875+0000","last_change":"2024-09-13T08:40:36.751002+0000","last_active":"2024-09-13T08:40:37.772875+0000","last_peered":"2024-09-13T08:40:37.772875+0000","last_clean":"2024-09-13T08:40:37.772875+0000","last_became_active":"2024-09-13T08:40:36.750731+0000","last_became_peered":"2024-09-13T08:40:36.750731+0000","last_unstale":"2024-09-13T08:40:37.772875+0000","last_undegraded":"2024-09-13T08:40:37.772875+0000","last_fullsized":"2024-09-13T08:40:37.772875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:24.257806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.742278+0000","last_change":"2024-09-13T08:40:37.736587+0000","last_active":"2024-09-13T08:40:37.742278+0000","last_peered":"2024-09-13T08:40:37.742278+0000","last_clean":"2024-09-13T08:40:37.742278+0000","last_became_active":"2024-09-13T08:40:37.734709+0000","last_became_peered":"2024-09-13T08:40:37.734709+0000","last_unstale":"2024-09-13T08:40:37.742278+0000","last_undegraded":"2024-09-13T08:40:37.742278+0000","last_fullsized":"2024-09-13T08:40:37.742278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:29.013799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730713+0000","last_change":"2024-09-13T08:40:36.740922+0000","last_active":"2024-09-13T08:40:37.730713+0000","last_peered":"2024-09-13T08:40:37.730713+0000","last_clean":"2024-09-13T08:40:37.730713+0000","last_became_active":"2024-09-13T08:40:36.740610+0000","last_became_peered":"2024-09-13T08:40:36.740610+0000","last_unstale":"2024-09-13T08:40:37.730713+0000","last_undegraded":"2024-09-13T08:40:37.730713+0000","last_fullsized":"2024-09-13T08:40:37.730713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758374+0000","last_change":"2024-09-13T08:40:37.753714+0000","last_active":"2024-09-13T08:40:37.758374+0000","last_peered":"2024-09-13T08:40:37.758374+0000","last_clean":"2024-09-13T08:40:37.758374+0000","last_became_active":"2024-09-13T08:40:37.753402+0000","last_became_peered":"2024-09-13T08:40:37.753402+0000","last_unstale":"2024-09-13T08:40:37.758374+0000","last_undegraded":"2024-09-13T08:40:37.758374+0000","last_fullsized":"2024-09-13T08:40:37.758374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:10.642715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731269+0000","last_change":"2024-09-13T08:40:36.737107+0000","last_active":"2024-09-13T08:40:37.731269+0000","last_peered":"2024-09-13T08:40:37.731269+0000","last_clean":"2024-09-13T08:40:37.731269+0000","last_became_active":"2024-09-13T08:40:36.736458+0000","last_became_peered":"2024-09-13T08:40:36.736458+0000","last_unstale":"2024-09-13T08:40:37.731269+0000","last_undegraded":"2024-09-13T08:40:37.731269+0000","last_fullsized":"2024-09-13T08:40:37.731269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:13.644488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.751369+0000","last_change":"2024-09-13T08:40:37.736394+0000","last_active":"2024-09-13T08:40:37.751369+0000","last_peered":"2024-09-13T08:40:37.751369+0000","last_clean":"2024-09-13T08:40:37.751369+0000","last_became_active":"2024-09-13T08:40:37.736142+0000","last_became_peered":"2024-09-13T08:40:37.736142+0000","last_unstale":"2024-09-13T08:40:37.751369+0000","last_undegraded":"2024-09-13T08:40:37.751369+0000","last_fullsized":"2024-09-13T08:40:37.751369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:00.761695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.733495+0000","last_change":"2024-09-13T08:40:36.740047+0000","last_active":"2024-09-13T08:40:37.733495+0000","last_peered":"2024-09-13T08:40:37.733495+0000","last_clean":"2024-09-13T08:40:37.733495+0000","last_became_active":"2024-09-13T08:40:36.739774+0000","last_became_peered":"2024-09-13T08:40:36.739774+0000","last_unstale":"2024-09-13T08:40:37.733495+0000","last_undegraded":"2024-09-13T08:40:37.733495+0000","last_fullsized":"2024-09-13T08:40:37.733495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760164+0000","last_change":"2024-09-13T08:40:37.757097+0000","last_active":"2024-09-13T08:40:37.760164+0000","last_peered":"2024-09-13T08:40:37.760164+0000","last_clean":"2024-09-13T08:40:37.760164+0000","last_became_active":"2024-09-13T08:40:37.756526+0000","last_became_peered":"2024-09-13T08:40:37.756526+0000","last_unstale":"2024-09-13T08:40:37.760164+0000","last_undegraded":"2024-09-13T08:40:37.760164+0000","last_fullsized":"2024-09-13T08:40:37.760164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735925+0000","last_change":"2024-09-13T08:40:36.743002+0000","last_active":"2024-09-13T08:40:37.735925+0000","last_peered":"2024-09-13T08:40:37.735925+0000","last_clean":"2024-09-13T08:40:37.735925+0000","last_became_active":"2024-09-13T08:40:36.740918+0000","last_became_peered":"2024-09-13T08:40:36.740918+0000","last_unstale":"2024-09-13T08:40:37.735925+0000","last_undegraded":"2024-09-13T08:40:37.735925+0000","last_fullsized":"2024-09-13T08:40:37.735925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:05.024141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759635+0000","last_change":"2024-09-13T08:40:37.756377+0000","last_active":"2024-09-13T08:40:37.759635+0000","last_peered":"2024-09-13T08:40:37.759635+0000","last_clean":"2024-09-13T08:40:37.759635+0000","last_became_active":"2024-09-13T08:40:37.756086+0000","last_became_peered":"2024-09-13T08:40:37.756086+0000","last_unstale":"2024-09-13T08:40:37.759635+0000","last_undegraded":"2024-09-13T08:40:37.759635+0000","last_fullsized":"2024-09-13T08:40:37.759635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:41.096839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.772900+0000","last_change":"2024-09-13T08:40:36.745979+0000","last_active":"2024-09-13T08:40:37.772900+0000","last_peered":"2024-09-13T08:40:37.772900+0000","last_clean":"2024-09-13T08:40:37.772900+0000","last_became_active":"2024-09-13T08:40:36.745673+0000","last_became_peered":"2024-09-13T08:40:36.745673+0000","last_unstale":"2024-09-13T08:40:37.772900+0000","last_undegraded":"2024-09-13T08:40:37.772900+0000","last_fullsized":"2024-09-13T08:40:37.772900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:11.780765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759430+0000","last_change":"2024-09-13T08:40:37.752368+0000","last_active":"2024-09-13T08:40:37.759430+0000","last_peered":"2024-09-13T08:40:37.759430+0000","last_clean":"2024-09-13T08:40:37.759430+0000","last_became_active":"2024-09-13T08:40:37.752096+0000","last_became_peered":"2024-09-13T08:40:37.752096+0000","last_unstale":"2024-09-13T08:40:37.759430+0000","last_undegraded":"2024-09-13T08:40:37.759430+0000","last_fullsized":"2024-09-13T08:40:37.759430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:33.675957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776405+0000","last_change":"2024-09-13T08:40:36.745140+0000","last_active":"2024-09-13T08:40:37.776405+0000","last_peered":"2024-09-13T08:40:37.776405+0000","last_clean":"2024-09-13T08:40:37.776405+0000","last_became_active":"2024-09-13T08:40:36.743817+0000","last_became_peered":"2024-09-13T08:40:36.743817+0000","last_unstale":"2024-09-13T08:40:37.776405+0000","last_undegraded":"2024-09-13T08:40:37.776405+0000","last_fullsized":"2024-09-13T08:40:37.776405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:15.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745728+0000","last_change":"2024-09-13T08:40:37.740915+0000","last_active":"2024-09-13T08:40:37.745728+0000","last_peered":"2024-09-13T08:40:37.745728+0000","last_clean":"2024-09-13T08:40:37.745728+0000","last_became_active":"2024-09-13T08:40:37.740617+0000","last_became_peered":"2024-09-13T08:40:37.740617+0000","last_unstale":"2024-09-13T08:40:37.745728+0000","last_undegraded":"2024-09-13T08:40:37.745728+0000","last_fullsized":"2024-09-13T08:40:37.745728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770825+0000","last_change":"2024-09-13T08:40:36.740132+0000","last_active":"2024-09-13T08:40:37.770825+0000","last_peered":"2024-09-13T08:40:37.770825+0000","last_clean":"2024-09-13T08:40:37.770825+0000","last_became_active":"2024-09-13T08:40:36.739787+0000","last_became_peered":"2024-09-13T08:40:36.739787+0000","last_unstale":"2024-09-13T08:40:37.770825+0000","last_undegraded":"2024-09-13T08:40:37.770825+0000","last_fullsized":"2024-09-13T08:40:37.770825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.013701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759244+0000","last_change":"2024-09-13T08:40:37.754029+0000","last_active":"2024-09-13T08:40:37.759244+0000","last_peered":"2024-09-13T08:40:37.759244+0000","last_clean":"2024-09-13T08:40:37.759244+0000","last_became_active":"2024-09-13T08:40:37.753773+0000","last_became_peered":"2024-09-13T08:40:37.753773+0000","last_unstale":"2024-09-13T08:40:37.759244+0000","last_undegraded":"2024-09-13T08:40:37.759244+0000","last_fullsized":"2024-09-13T08:40:37.759244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727972+0000","last_change":"2024-09-13T08:40:36.746192+0000","last_active":"2024-09-13T08:40:37.727972+0000","last_peered":"2024-09-13T08:40:37.727972+0000","last_clean":"2024-09-13T08:40:37.727972+0000","last_became_active":"2024-09-13T08:40:36.745703+0000","last_became_peered":"2024-09-13T08:40:36.745703+0000","last_unstale":"2024-09-13T08:40:37.727972+0000","last_undegraded":"2024-09-13T08:40:37.727972+0000","last_fullsized":"2024-09-13T08:40:37.727972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:00.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.755812+0000","last_change":"2024-09-13T08:40:37.750252+0000","last_active":"2024-09-13T08:40:37.755812+0000","last_peered":"2024-09-13T08:40:37.755812+0000","last_clean":"2024-09-13T08:40:37.755812+0000","last_became_active":"2024-09-13T08:40:37.749793+0000","last_became_peered":"2024-09-13T08:40:37.749793+0000","last_unstale":"2024-09-13T08:40:37.755812+0000","last_undegraded":"2024-09-13T08:40:37.755812+0000","last_fullsized":"2024-09-13T08:40:37.755812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.727573+0000","last_change":"2024-09-13T08:40:36.748499+0000","last_active":"2024-09-13T08:40:37.727573+0000","last_peered":"2024-09-13T08:40:37.727573+0000","last_clean":"2024-09-13T08:40:37.727573+0000","last_became_active":"2024-09-13T08:40:36.745832+0000","last_became_peered":"2024-09-13T08:40:36.745832+0000","last_unstale":"2024-09-13T08:40:37.727573+0000","last_undegraded":"2024-09-13T08:40:37.727573+0000","last_fullsized":"2024-09-13T08:40:37.727573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:42.819908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.740690+0000","last_change":"2024-09-13T08:40:37.734550+0000","last_active":"2024-09-13T08:40:37.740690+0000","last_peered":"2024-09-13T08:40:37.740690+0000","last_clean":"2024-09-13T08:40:37.740690+0000","last_became_active":"2024-09-13T08:40:37.734140+0000","last_became_peered":"2024-09-13T08:40:37.734140+0000","last_unstale":"2024-09-13T08:40:37.740690+0000","last_undegraded":"2024-09-13T08:40:37.740690+0000","last_fullsized":"2024-09-13T08:40:37.740690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:11.251715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":21,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.776743+0000","last_change":"2024-09-13T08:40:36.746654+0000","last_active":"2024-09-13T08:40:37.776743+0000","last_peered":"2024-09-13T08:40:37.776743+0000","last_clean":"2024-09-13T08:40:37.776743+0000","last_became_active":"2024-09-13T08:40:36.746397+0000","last_became_peered":"2024-09-13T08:40:36.746397+0000","last_unstale":"2024-09-13T08:40:37.776743+0000","last_undegraded":"2024-09-13T08:40:37.776743+0000","last_fullsized":"2024-09-13T08:40:37.776743+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:15.203955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747535+0000","last_change":"2024-09-13T08:40:37.745284+0000","last_active":"2024-09-13T08:40:37.747535+0000","last_peered":"2024-09-13T08:40:37.747535+0000","last_clean":"2024-09-13T08:40:37.747535+0000","last_became_active":"2024-09-13T08:40:37.744957+0000","last_became_peered":"2024-09-13T08:40:37.744957+0000","last_unstale":"2024-09-13T08:40:37.747535+0000","last_undegraded":"2024-09-13T08:40:37.747535+0000","last_fullsized":"2024-09-13T08:40:37.747535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:10.429012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767493+0000","last_change":"2024-09-13T08:40:36.753006+0000","last_active":"2024-09-13T08:40:37.767493+0000","last_peered":"2024-09-13T08:40:37.767493+0000","last_clean":"2024-09-13T08:40:37.767493+0000","last_became_active":"2024-09-13T08:40:36.752498+0000","last_became_peered":"2024-09-13T08:40:36.752498+0000","last_unstale":"2024-09-13T08:40:37.767493+0000","last_undegraded":"2024-09-13T08:40:37.767493+0000","last_fullsized":"2024-09-13T08:40:37.767493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.760187+0000","last_change":"2024-09-13T08:40:37.757310+0000","last_active":"2024-09-13T08:40:37.760187+0000","last_peered":"2024-09-13T08:40:37.760187+0000","last_clean":"2024-09-13T08:40:37.760187+0000","last_became_active":"2024-09-13T08:40:37.756852+0000","last_became_peered":"2024-09-13T08:40:37.756852+0000","last_unstale":"2024-09-13T08:40:37.760187+0000","last_undegraded":"2024-09-13T08:40:37.760187+0000","last_fullsized":"2024-09-13T08:40:37.760187+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.770763+0000","last_change":"2024-09-13T08:40:36.741233+0000","last_active":"2024-09-13T08:40:37.770763+0000","last_peered":"2024-09-13T08:40:37.770763+0000","last_clean":"2024-09-13T08:40:37.770763+0000","last_became_active":"2024-09-13T08:40:36.740574+0000","last_became_peered":"2024-09-13T08:40:36.740574+0000","last_unstale":"2024-09-13T08:40:37.770763+0000","last_undegraded":"2024-09-13T08:40:37.770763+0000","last_fullsized":"2024-09-13T08:40:37.770763+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:52.377019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762842+0000","last_change":"2024-09-13T08:40:37.761727+0000","last_active":"2024-09-13T08:40:37.762842+0000","last_peered":"2024-09-13T08:40:37.762842+0000","last_clean":"2024-09-13T08:40:37.762842+0000","last_became_active":"2024-09-13T08:40:37.761016+0000","last_became_peered":"2024-09-13T08:40:37.761016+0000","last_unstale":"2024-09-13T08:40:37.762842+0000","last_undegraded":"2024-09-13T08:40:37.762842+0000","last_fullsized":"2024-09-13T08:40:37.762842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:58:06.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.769868+0000","last_change":"2024-09-13T08:40:36.740608+0000","last_active":"2024-09-13T08:40:37.769868+0000","last_peered":"2024-09-13T08:40:37.769868+0000","last_clean":"2024-09-13T08:40:37.769868+0000","last_became_active":"2024-09-13T08:40:36.740349+0000","last_became_peered":"2024-09-13T08:40:36.740349+0000","last_unstale":"2024-09-13T08:40:37.769868+0000","last_undegraded":"2024-09-13T08:40:37.769868+0000","last_fullsized":"2024-09-13T08:40:37.769868+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.279565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762124+0000","last_change":"2024-09-13T08:40:37.758748+0000","last_active":"2024-09-13T08:40:37.762124+0000","last_peered":"2024-09-13T08:40:37.762124+0000","last_clean":"2024-09-13T08:40:37.762124+0000","last_became_active":"2024-09-13T08:40:37.756975+0000","last_became_peered":"2024-09-13T08:40:37.756975+0000","last_unstale":"2024-09-13T08:40:37.762124+0000","last_undegraded":"2024-09-13T08:40:37.762124+0000","last_fullsized":"2024-09-13T08:40:37.762124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:43.311241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736532+0000","last_change":"2024-09-13T08:40:36.748868+0000","last_active":"2024-09-13T08:40:37.736532+0000","last_peered":"2024-09-13T08:40:37.736532+0000","last_clean":"2024-09-13T08:40:37.736532+0000","last_became_active":"2024-09-13T08:40:36.748527+0000","last_became_peered":"2024-09-13T08:40:36.748527+0000","last_unstale":"2024-09-13T08:40:37.736532+0000","last_undegraded":"2024-09-13T08:40:37.736532+0000","last_fullsized":"2024-09-13T08:40:37.736532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:05.275938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.759771+0000","last_change":"2024-09-13T08:40:37.756050+0000","last_active":"2024-09-13T08:40:37.759771+0000","last_peered":"2024-09-13T08:40:37.759771+0000","last_clean":"2024-09-13T08:40:37.759771+0000","last_became_active":"2024-09-13T08:40:37.755788+0000","last_became_peered":"2024-09-13T08:40:37.755788+0000","last_unstale":"2024-09-13T08:40:37.759771+0000","last_undegraded":"2024-09-13T08:40:37.759771+0000","last_fullsized":"2024-09-13T08:40:37.759771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749727+0000","last_change":"2024-09-13T08:40:36.737698+0000","last_active":"2024-09-13T08:40:38.749727+0000","last_peered":"2024-09-13T08:40:38.749727+0000","last_clean":"2024-09-13T08:40:38.749727+0000","last_became_active":"2024-09-13T08:40:36.737306+0000","last_became_peered":"2024-09-13T08:40:36.737306+0000","last_unstale":"2024-09-13T08:40:38.749727+0000","last_undegraded":"2024-09-13T08:40:38.749727+0000","last_fullsized":"2024-09-13T08:40:38.749727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:52.538374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758021+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:37.758021+0000","last_peered":"2024-09-13T08:40:37.758021+0000","last_clean":"2024-09-13T08:40:37.758021+0000","last_became_active":"2024-09-13T08:40:37.750682+0000","last_became_peered":"2024-09-13T08:40:37.750682+0000","last_unstale":"2024-09-13T08:40:37.758021+0000","last_undegraded":"2024-09-13T08:40:37.758021+0000","last_fullsized":"2024-09-13T08:40:37.758021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:12.637948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.736909+0000","last_change":"2024-09-13T08:40:36.749086+0000","last_active":"2024-09-13T08:40:37.736909+0000","last_peered":"2024-09-13T08:40:37.736909+0000","last_clean":"2024-09-13T08:40:37.736909+0000","last_became_active":"2024-09-13T08:40:36.747729+0000","last_became_peered":"2024-09-13T08:40:36.747729+0000","last_unstale":"2024-09-13T08:40:37.736909+0000","last_undegraded":"2024-09-13T08:40:37.736909+0000","last_fullsized":"2024-09-13T08:40:37.736909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:12:48.295035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767464+0000","last_change":"2024-09-13T08:40:37.765328+0000","last_active":"2024-09-13T08:40:37.767464+0000","last_peered":"2024-09-13T08:40:37.767464+0000","last_clean":"2024-09-13T08:40:37.767464+0000","last_became_active":"2024-09-13T08:40:37.762317+0000","last_became_peered":"2024-09-13T08:40:37.762317+0000","last_unstale":"2024-09-13T08:40:37.767464+0000","last_undegraded":"2024-09-13T08:40:37.767464+0000","last_fullsized":"2024-09-13T08:40:37.767464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:29:55.562604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732662+0000","last_change":"2024-09-13T08:40:36.747739+0000","last_active":"2024-09-13T08:40:37.732662+0000","last_peered":"2024-09-13T08:40:37.732662+0000","last_clean":"2024-09-13T08:40:37.732662+0000","last_became_active":"2024-09-13T08:40:36.747366+0000","last_became_peered":"2024-09-13T08:40:36.747366+0000","last_unstale":"2024-09-13T08:40:37.732662+0000","last_undegraded":"2024-09-13T08:40:37.732662+0000","last_fullsized":"2024-09-13T08:40:37.732662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.748364+0000","last_change":"2024-09-13T08:40:37.746566+0000","last_active":"2024-09-13T08:40:37.748364+0000","last_peered":"2024-09-13T08:40:37.748364+0000","last_clean":"2024-09-13T08:40:37.748364+0000","last_became_active":"2024-09-13T08:40:37.746382+0000","last_became_peered":"2024-09-13T08:40:37.746382+0000","last_unstale":"2024-09-13T08:40:37.748364+0000","last_undegraded":"2024-09-13T08:40:37.748364+0000","last_fullsized":"2024-09-13T08:40:37.748364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:38:16.254288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.768664+0000","last_change":"2024-09-13T08:40:36.743612+0000","last_active":"2024-09-13T08:40:37.768664+0000","last_peered":"2024-09-13T08:40:37.768664+0000","last_clean":"2024-09-13T08:40:37.768664+0000","last_became_active":"2024-09-13T08:40:36.743337+0000","last_became_peered":"2024-09-13T08:40:36.743337+0000","last_unstale":"2024-09-13T08:40:37.768664+0000","last_undegraded":"2024-09-13T08:40:37.768664+0000","last_fullsized":"2024-09-13T08:40:37.768664+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.747771+0000","last_change":"2024-09-13T08:40:37.746100+0000","last_active":"2024-09-13T08:40:37.747771+0000","last_peered":"2024-09-13T08:40:37.747771+0000","last_clean":"2024-09-13T08:40:37.747771+0000","last_became_active":"2024-09-13T08:40:37.745801+0000","last_became_peered":"2024-09-13T08:40:37.745801+0000","last_unstale":"2024-09-13T08:40:37.747771+0000","last_undegraded":"2024-09-13T08:40:37.747771+0000","last_fullsized":"2024-09-13T08:40:37.747771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:38.977755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.745934+0000","last_change":"2024-09-13T08:40:36.751068+0000","last_active":"2024-09-13T08:40:37.745934+0000","last_peered":"2024-09-13T08:40:37.745934+0000","last_clean":"2024-09-13T08:40:37.745934+0000","last_became_active":"2024-09-13T08:40:36.748029+0000","last_became_peered":"2024-09-13T08:40:36.748029+0000","last_unstale":"2024-09-13T08:40:37.745934+0000","last_undegraded":"2024-09-13T08:40:37.745934+0000","last_fullsized":"2024-09-13T08:40:37.745934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:25.694691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.749201+0000","last_change":"2024-09-13T08:40:37.743421+0000","last_active":"2024-09-13T08:40:37.749201+0000","last_peered":"2024-09-13T08:40:37.749201+0000","last_clean":"2024-09-13T08:40:37.749201+0000","last_became_active":"2024-09-13T08:40:37.743126+0000","last_became_peered":"2024-09-13T08:40:37.743126+0000","last_unstale":"2024-09-13T08:40:37.749201+0000","last_undegraded":"2024-09-13T08:40:37.749201+0000","last_fullsized":"2024-09-13T08:40:37.749201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:55.204657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.731313+0000","last_change":"2024-09-13T08:40:36.742793+0000","last_active":"2024-09-13T08:40:37.731313+0000","last_peered":"2024-09-13T08:40:37.731313+0000","last_clean":"2024-09-13T08:40:37.731313+0000","last_became_active":"2024-09-13T08:40:36.742516+0000","last_became_peered":"2024-09-13T08:40:36.742516+0000","last_unstale":"2024-09-13T08:40:37.731313+0000","last_undegraded":"2024-09-13T08:40:37.731313+0000","last_fullsized":"2024-09-13T08:40:37.731313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.767726+0000","last_change":"2024-09-13T08:40:37.763959+0000","last_active":"2024-09-13T08:40:37.767726+0000","last_peered":"2024-09-13T08:40:37.767726+0000","last_clean":"2024-09-13T08:40:37.767726+0000","last_became_active":"2024-09-13T08:40:37.763661+0000","last_became_peered":"2024-09-13T08:40:37.763661+0000","last_unstale":"2024-09-13T08:40:37.767726+0000","last_undegraded":"2024-09-13T08:40:37.767726+0000","last_fullsized":"2024-09-13T08:40:37.767726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:46.895173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735588+0000","last_change":"2024-09-13T08:40:36.750833+0000","last_active":"2024-09-13T08:40:37.735588+0000","last_peered":"2024-09-13T08:40:37.735588+0000","last_clean":"2024-09-13T08:40:37.735588+0000","last_became_active":"2024-09-13T08:40:36.750471+0000","last_became_peered":"2024-09-13T08:40:36.750471+0000","last_unstale":"2024-09-13T08:40:37.735588+0000","last_undegraded":"2024-09-13T08:40:37.735588+0000","last_fullsized":"2024-09-13T08:40:37.735588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:36.394174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.762046+0000","last_change":"2024-09-13T08:40:37.756241+0000","last_active":"2024-09-13T08:40:37.762046+0000","last_peered":"2024-09-13T08:40:37.762046+0000","last_clean":"2024-09-13T08:40:37.762046+0000","last_became_active":"2024-09-13T08:40:37.755961+0000","last_became_peered":"2024-09-13T08:40:37.755961+0000","last_unstale":"2024-09-13T08:40:37.762046+0000","last_undegraded":"2024-09-13T08:40:37.762046+0000","last_fullsized":"2024-09-13T08:40:37.762046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:38.286408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.730472+0000","last_change":"2024-09-13T08:40:36.750422+0000","last_active":"2024-09-13T08:40:37.730472+0000","last_peered":"2024-09-13T08:40:37.730472+0000","last_clean":"2024-09-13T08:40:37.730472+0000","last_became_active":"2024-09-13T08:40:36.749512+0000","last_became_peered":"2024-09-13T08:40:36.749512+0000","last_unstale":"2024-09-13T08:40:37.730472+0000","last_undegraded":"2024-09-13T08:40:37.730472+0000","last_fullsized":"2024-09-13T08:40:37.730472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.764478+0000","last_change":"2024-09-13T08:40:37.756404+0000","last_active":"2024-09-13T08:40:37.764478+0000","last_peered":"2024-09-13T08:40:37.764478+0000","last_clean":"2024-09-13T08:40:37.764478+0000","last_became_active":"2024-09-13T08:40:37.755037+0000","last_became_peered":"2024-09-13T08:40:37.755037+0000","last_unstale":"2024-09-13T08:40:37.764478+0000","last_undegraded":"2024-09-13T08:40:37.764478+0000","last_fullsized":"2024-09-13T08:40:37.764478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:11.905503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732259+0000","last_change":"2024-09-13T08:40:36.740510+0000","last_active":"2024-09-13T08:40:37.732259+0000","last_peered":"2024-09-13T08:40:37.732259+0000","last_clean":"2024-09-13T08:40:37.732259+0000","last_became_active":"2024-09-13T08:40:36.740243+0000","last_became_peered":"2024-09-13T08:40:36.740243+0000","last_unstale":"2024-09-13T08:40:37.732259+0000","last_undegraded":"2024-09-13T08:40:37.732259+0000","last_fullsized":"2024-09-13T08:40:37.732259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:20.155484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.758212+0000","last_change":"2024-09-13T08:40:37.751854+0000","last_active":"2024-09-13T08:40:37.758212+0000","last_peered":"2024-09-13T08:40:37.758212+0000","last_clean":"2024-09-13T08:40:37.758212+0000","last_became_active":"2024-09-13T08:40:37.749368+0000","last_became_peered":"2024-09-13T08:40:37.749368+0000","last_unstale":"2024-09-13T08:40:37.758212+0000","last_undegraded":"2024-09-13T08:40:37.758212+0000","last_fullsized":"2024-09-13T08:40:37.758212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:05.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.743930+0000","last_change":"2024-09-13T08:40:36.743024+0000","last_active":"2024-09-13T08:40:37.743930+0000","last_peered":"2024-09-13T08:40:37.743930+0000","last_clean":"2024-09-13T08:40:37.743930+0000","last_became_active":"2024-09-13T08:40:36.742739+0000","last_became_peered":"2024-09-13T08:40:36.742739+0000","last_unstale":"2024-09-13T08:40:37.743930+0000","last_undegraded":"2024-09-13T08:40:37.743930+0000","last_fullsized":"2024-09-13T08:40:37.743930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:54.197613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.753493+0000","last_change":"2024-09-13T08:40:37.743948+0000","last_active":"2024-09-13T08:40:37.753493+0000","last_peered":"2024-09-13T08:40:37.753493+0000","last_clean":"2024-09-13T08:40:37.753493+0000","last_became_active":"2024-09-13T08:40:37.743656+0000","last_became_peered":"2024-09-13T08:40:37.743656+0000","last_unstale":"2024-09-13T08:40:37.753493+0000","last_undegraded":"2024-09-13T08:40:37.753493+0000","last_fullsized":"2024-09-13T08:40:37.753493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:50.636997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.735241+0000","last_change":"2024-09-13T08:40:36.750854+0000","last_active":"2024-09-13T08:40:37.735241+0000","last_peered":"2024-09-13T08:40:37.735241+0000","last_clean":"2024-09-13T08:40:37.735241+0000","last_became_active":"2024-09-13T08:40:36.750177+0000","last_became_peered":"2024-09-13T08:40:36.750177+0000","last_unstale":"2024-09-13T08:40:37.735241+0000","last_undegraded":"2024-09-13T08:40:37.735241+0000","last_fullsized":"2024-09-13T08:40:37.735241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:20.695241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.754437+0000","last_change":"2024-09-13T08:40:37.746462+0000","last_active":"2024-09-13T08:40:37.754437+0000","last_peered":"2024-09-13T08:40:37.754437+0000","last_clean":"2024-09-13T08:40:37.754437+0000","last_became_active":"2024-09-13T08:40:37.745179+0000","last_became_peered":"2024-09-13T08:40:37.745179+0000","last_unstale":"2024-09-13T08:40:37.754437+0000","last_undegraded":"2024-09-13T08:40:37.754437+0000","last_fullsized":"2024-09-13T08:40:37.754437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:07.391500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.737843+0000","last_change":"2024-09-13T08:40:36.744466+0000","last_active":"2024-09-13T08:40:37.737843+0000","last_peered":"2024-09-13T08:40:37.737843+0000","last_clean":"2024-09-13T08:40:37.737843+0000","last_became_active":"2024-09-13T08:40:36.744206+0000","last_became_peered":"2024-09-13T08:40:36.744206+0000","last_unstale":"2024-09-13T08:40:37.737843+0000","last_undegraded":"2024-09-13T08:40:37.737843+0000","last_fullsized":"2024-09-13T08:40:37.737843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:40.300855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":179,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T08:40:37.732740+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:40:37.732740+0000","last_peered":"2024-09-13T08:40:37.732740+0000","last_clean":"2024-09-13T08:40:37.732740+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:40:37.732740+0000","last_undegraded":"2024-09-13T08:40:37.732740+0000","last_fullsized":"2024-09-13T08:40:37.732740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":292844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":299641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509487,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1323008,"data_stored":890021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":729088,"data_stored":299709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":745472,"data_stored":301391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1306624,"data_stored":883212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509486,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":737280,"data_stored":301231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36076,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335764,"statfs":{"total":96636764160,"available":96599822336,"internally_reserved":0,"allocated":745472,"data_stored":306535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:40:46.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:40:46.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 -- 172.21.15.31:0/4251275854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780d0240 msgr2=0x7f47780b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:46.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 --2- 172.21.15.31:0/4251275854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780d0240 0x7f47780b34d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4760005fd0 tx=0x7f476002a670 comp rx=0 tx=0).stop 2024-09-13T08:40:46.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 -- 172.21.15.31:0/4251275854 shutdown_connections 2024-09-13T08:40:46.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 --2- 172.21.15.31:0/4251275854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780d0240 0x7f47780b34d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:46.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 -- 172.21.15.31:0/4251275854 >> 172.21.15.31:0/4251275854 conn(0x7f47780d3e90 msgr2=0x7f47780d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:46.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 -- 172.21.15.31:0/4251275854 shutdown_connections 2024-09-13T08:40:46.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.960+0000 7f4777e00640 1 -- 172.21.15.31:0/4251275854 wait complete. 2024-09-13T08:40:46.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.961+0000 7f4777e00640 1 Processor -- start 2024-09-13T08:40:46.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.962+0000 7f4777e00640 1 -- start start 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.962+0000 7f4777e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.962+0000 7f4777e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47780d08a0 0x7f47781f8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.962+0000 7f4777e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47781f86d0 0x7f47780c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4777e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47780b4120 con 0x7f47780b2f80 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4777e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47780d15a0 con 0x7f47780d08a0 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4777e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f47780b42a0 con 0x7f47781f86d0 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:46.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43508/0 (socket says 172.21.15.31:43508) 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 -- 172.21.15.31:0/289521764 learned_addr learned my addr 172.21.15.31:0/289521764 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4777400640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47781f86d0 0x7f47780c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f476fe00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47780d08a0 0x7f47781f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 -- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47781f86d0 msgr2=0x7f47780c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47781f86d0 0x7f47780c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 -- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47780d08a0 msgr2=0x7f47781f8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47780d08a0 0x7f47781f8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.963+0000 7f4776a00640 1 -- 172.21.15.31:0/289521764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47781f5410 con 0x7f47780b2f80 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f4776a00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f476800c3d0 tx=0x7f476800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768017070 con 0x7f47780b2f80 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47680053c0 con 0x7f47780b2f80 2024-09-13T08:40:46.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47780c9bb0 con 0x7f47780b2f80 2024-09-13T08:40:46.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768003850 con 0x7f47780b2f80 2024-09-13T08:40:46.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.964+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47780ca170 con 0x7f47780b2f80 2024-09-13T08:40:46.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.965+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47780d0c80 con 0x7f47780b2f80 2024-09-13T08:40:46.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.966+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f476801a070 con 0x7f47780b2f80 2024-09-13T08:40:46.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.967+0000 7f476ea00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 0x7f475407aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:46.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.967+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f476809a0c0 con 0x7f47780b2f80 2024-09-13T08:40:46.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.968+0000 7f476fe00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 0x7f475407aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:46.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.969+0000 7f476fe00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 0x7f475407aef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4760033000 tx=0x7f4760004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:46.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:46.971+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4768062300 con 0x7f47780b2f80 2024-09-13T08:40:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.164+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4778076160 con 0x7f4754078a30 2024-09-13T08:40:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.170+0000 7f476ea00640 1 -- 172.21.15.31:0/289521764 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251920 (secure 0 0 0) 0x7f4778076160 con 0x7f4754078a30 2024-09-13T08:40:47.174 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 msgr2=0x7f475407aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 0x7f475407aef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4760033000 tx=0x7f4760004730 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 msgr2=0x7f47780ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f476800c3d0 tx=0x7f476800c890 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 shutdown_connections 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4754078a30 0x7f475407aef0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47781f86d0 0x7f47780c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47780d08a0 0x7f47781f8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 --2- 172.21.15.31:0/289521764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47780b2f80 0x7f47780ca690 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 >> 172.21.15.31:0/289521764 conn(0x7f47780d3e90 msgr2=0x7f47780b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 shutdown_connections 2024-09-13T08:40:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.174+0000 7f4777e00640 1 -- 172.21.15.31:0/289521764 wait complete. 2024-09-13T08:40:47.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-13T08:40:47.091938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248200,"kb_used_data":6984,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726520,"statfs":{"total":773094113280,"available":772839956480,"internally_reserved":0,"allocated":7151616,"data_stored":3632908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.365553"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761092+0000","last_change":"2024-09-13T08:40:37.757129+0000","last_active":"2024-09-13T08:40:38.761092+0000","last_peered":"2024-09-13T08:40:38.761092+0000","last_clean":"2024-09-13T08:40:38.761092+0000","last_became_active":"2024-09-13T08:40:37.756648+0000","last_became_peered":"2024-09-13T08:40:37.756648+0000","last_unstale":"2024-09-13T08:40:38.761092+0000","last_undegraded":"2024-09-13T08:40:38.761092+0000","last_fullsized":"2024-09-13T08:40:38.761092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:47.020544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750663+0000","last_change":"2024-09-13T08:40:37.756234+0000","last_active":"2024-09-13T08:40:38.750663+0000","last_peered":"2024-09-13T08:40:38.750663+0000","last_clean":"2024-09-13T08:40:38.750663+0000","last_became_active":"2024-09-13T08:40:37.755925+0000","last_became_peered":"2024-09-13T08:40:37.755925+0000","last_unstale":"2024-09-13T08:40:38.750663+0000","last_undegraded":"2024-09-13T08:40:38.750663+0000","last_fullsized":"2024-09-13T08:40:38.750663+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:34:51.389968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750766+0000","last_change":"2024-09-13T08:40:37.756481+0000","last_active":"2024-09-13T08:40:38.750766+0000","last_peered":"2024-09-13T08:40:38.750766+0000","last_clean":"2024-09-13T08:40:38.750766+0000","last_became_active":"2024-09-13T08:40:37.756190+0000","last_became_peered":"2024-09-13T08:40:37.756190+0000","last_unstale":"2024-09-13T08:40:38.750766+0000","last_undegraded":"2024-09-13T08:40:38.750766+0000","last_fullsized":"2024-09-13T08:40:38.750766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:34:11.066900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329572+0000","last_change":"2024-09-13T08:40:37.752214+0000","last_active":"2024-09-13T08:40:39.329572+0000","last_peered":"2024-09-13T08:40:39.329572+0000","last_clean":"2024-09-13T08:40:39.329572+0000","last_became_active":"2024-09-13T08:40:37.752015+0000","last_became_peered":"2024-09-13T08:40:37.752015+0000","last_unstale":"2024-09-13T08:40:39.329572+0000","last_undegraded":"2024-09-13T08:40:39.329572+0000","last_fullsized":"2024-09-13T08:40:39.329572+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:57.778782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760704+0000","last_change":"2024-09-13T08:40:37.749549+0000","last_active":"2024-09-13T08:40:38.760704+0000","last_peered":"2024-09-13T08:40:38.760704+0000","last_clean":"2024-09-13T08:40:38.760704+0000","last_became_active":"2024-09-13T08:40:37.748944+0000","last_became_peered":"2024-09-13T08:40:37.748944+0000","last_unstale":"2024-09-13T08:40:38.760704+0000","last_undegraded":"2024-09-13T08:40:38.760704+0000","last_fullsized":"2024-09-13T08:40:38.760704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760635+0000","last_change":"2024-09-13T08:40:37.756210+0000","last_active":"2024-09-13T08:40:38.760635+0000","last_peered":"2024-09-13T08:40:38.760635+0000","last_clean":"2024-09-13T08:40:38.760635+0000","last_became_active":"2024-09-13T08:40:37.755876+0000","last_became_peered":"2024-09-13T08:40:37.755876+0000","last_unstale":"2024-09-13T08:40:38.760635+0000","last_undegraded":"2024-09-13T08:40:38.760635+0000","last_fullsized":"2024-09-13T08:40:38.760635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328574+0000","last_change":"2024-09-13T08:40:37.738548+0000","last_active":"2024-09-13T08:40:39.328574+0000","last_peered":"2024-09-13T08:40:39.328574+0000","last_clean":"2024-09-13T08:40:39.328574+0000","last_became_active":"2024-09-13T08:40:37.738294+0000","last_became_peered":"2024-09-13T08:40:37.738294+0000","last_unstale":"2024-09-13T08:40:39.328574+0000","last_undegraded":"2024-09-13T08:40:39.328574+0000","last_fullsized":"2024-09-13T08:40:39.328574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:35.277363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328543+0000","last_change":"2024-09-13T08:40:37.752144+0000","last_active":"2024-09-13T08:40:39.328543+0000","last_peered":"2024-09-13T08:40:39.328543+0000","last_clean":"2024-09-13T08:40:39.328543+0000","last_became_active":"2024-09-13T08:40:37.751574+0000","last_became_peered":"2024-09-13T08:40:37.751574+0000","last_unstale":"2024-09-13T08:40:39.328543+0000","last_undegraded":"2024-09-13T08:40:39.328543+0000","last_fullsized":"2024-09-13T08:40:39.328543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:25.130025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329126+0000","last_change":"2024-09-13T08:40:37.759047+0000","last_active":"2024-09-13T08:40:39.329126+0000","last_peered":"2024-09-13T08:40:39.329126+0000","last_clean":"2024-09-13T08:40:39.329126+0000","last_became_active":"2024-09-13T08:40:37.757616+0000","last_became_peered":"2024-09-13T08:40:37.757616+0000","last_unstale":"2024-09-13T08:40:39.329126+0000","last_undegraded":"2024-09-13T08:40:39.329126+0000","last_fullsized":"2024-09-13T08:40:39.329126+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:18.978754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760963+0000","last_change":"2024-09-13T08:40:37.766302+0000","last_active":"2024-09-13T08:40:38.760963+0000","last_peered":"2024-09-13T08:40:38.760963+0000","last_clean":"2024-09-13T08:40:38.760963+0000","last_became_active":"2024-09-13T08:40:37.765985+0000","last_became_peered":"2024-09-13T08:40:37.765985+0000","last_unstale":"2024-09-13T08:40:38.760963+0000","last_undegraded":"2024-09-13T08:40:38.760963+0000","last_fullsized":"2024-09-13T08:40:38.760963+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:02.573536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750260+0000","last_change":"2024-09-13T08:40:37.750961+0000","last_active":"2024-09-13T08:40:38.750260+0000","last_peered":"2024-09-13T08:40:38.750260+0000","last_clean":"2024-09-13T08:40:38.750260+0000","last_became_active":"2024-09-13T08:40:37.750706+0000","last_became_peered":"2024-09-13T08:40:37.750706+0000","last_unstale":"2024-09-13T08:40:38.750260+0000","last_undegraded":"2024-09-13T08:40:38.750260+0000","last_fullsized":"2024-09-13T08:40:38.750260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:57:35.246408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328905+0000","last_change":"2024-09-13T08:40:37.762085+0000","last_active":"2024-09-13T08:40:39.328905+0000","last_peered":"2024-09-13T08:40:39.328905+0000","last_clean":"2024-09-13T08:40:39.328905+0000","last_became_active":"2024-09-13T08:40:37.761803+0000","last_became_peered":"2024-09-13T08:40:37.761803+0000","last_unstale":"2024-09-13T08:40:39.328905+0000","last_undegraded":"2024-09-13T08:40:39.328905+0000","last_fullsized":"2024-09-13T08:40:39.328905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:34.977391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329315+0000","last_change":"2024-09-13T08:40:37.742869+0000","last_active":"2024-09-13T08:40:39.329315+0000","last_peered":"2024-09-13T08:40:39.329315+0000","last_clean":"2024-09-13T08:40:39.329315+0000","last_became_active":"2024-09-13T08:40:37.742566+0000","last_became_peered":"2024-09-13T08:40:37.742566+0000","last_unstale":"2024-09-13T08:40:39.329315+0000","last_undegraded":"2024-09-13T08:40:39.329315+0000","last_fullsized":"2024-09-13T08:40:39.329315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:16.296586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750304+0000","last_change":"2024-09-13T08:40:37.750839+0000","last_active":"2024-09-13T08:40:38.750304+0000","last_peered":"2024-09-13T08:40:38.750304+0000","last_clean":"2024-09-13T08:40:38.750304+0000","last_became_active":"2024-09-13T08:40:37.750572+0000","last_became_peered":"2024-09-13T08:40:37.750572+0000","last_unstale":"2024-09-13T08:40:38.750304+0000","last_undegraded":"2024-09-13T08:40:38.750304+0000","last_fullsized":"2024-09-13T08:40:38.750304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:02.573536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329467+0000","last_change":"2024-09-13T08:40:37.755720+0000","last_active":"2024-09-13T08:40:39.329467+0000","last_peered":"2024-09-13T08:40:39.329467+0000","last_clean":"2024-09-13T08:40:39.329467+0000","last_became_active":"2024-09-13T08:40:37.755419+0000","last_became_peered":"2024-09-13T08:40:37.755419+0000","last_unstale":"2024-09-13T08:40:39.329467+0000","last_undegraded":"2024-09-13T08:40:39.329467+0000","last_fullsized":"2024-09-13T08:40:39.329467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:35:58.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760566+0000","last_change":"2024-09-13T08:40:37.748027+0000","last_active":"2024-09-13T08:40:38.760566+0000","last_peered":"2024-09-13T08:40:38.760566+0000","last_clean":"2024-09-13T08:40:38.760566+0000","last_became_active":"2024-09-13T08:40:37.747736+0000","last_became_peered":"2024-09-13T08:40:37.747736+0000","last_unstale":"2024-09-13T08:40:38.760566+0000","last_undegraded":"2024-09-13T08:40:38.760566+0000","last_fullsized":"2024-09-13T08:40:38.760566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761124+0000","last_change":"2024-09-13T08:40:37.759006+0000","last_active":"2024-09-13T08:40:38.761124+0000","last_peered":"2024-09-13T08:40:38.761124+0000","last_clean":"2024-09-13T08:40:38.761124+0000","last_became_active":"2024-09-13T08:40:37.758700+0000","last_became_peered":"2024-09-13T08:40:37.758700+0000","last_unstale":"2024-09-13T08:40:38.761124+0000","last_undegraded":"2024-09-13T08:40:38.761124+0000","last_fullsized":"2024-09-13T08:40:38.761124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:07.391500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750815+0000","last_change":"2024-09-13T08:40:37.757199+0000","last_active":"2024-09-13T08:40:38.750815+0000","last_peered":"2024-09-13T08:40:38.750815+0000","last_clean":"2024-09-13T08:40:38.750815+0000","last_became_active":"2024-09-13T08:40:37.756901+0000","last_became_peered":"2024-09-13T08:40:37.756901+0000","last_unstale":"2024-09-13T08:40:38.750815+0000","last_undegraded":"2024-09-13T08:40:38.750815+0000","last_fullsized":"2024-09-13T08:40:38.750815+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:44:11.983878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750334+0000","last_change":"2024-09-13T08:40:37.752015+0000","last_active":"2024-09-13T08:40:38.750334+0000","last_peered":"2024-09-13T08:40:38.750334+0000","last_clean":"2024-09-13T08:40:38.750334+0000","last_became_active":"2024-09-13T08:40:37.751710+0000","last_became_peered":"2024-09-13T08:40:37.751710+0000","last_unstale":"2024-09-13T08:40:38.750334+0000","last_undegraded":"2024-09-13T08:40:38.750334+0000","last_fullsized":"2024-09-13T08:40:38.750334+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:48.663537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749566+0000","last_change":"2024-09-13T08:40:37.752392+0000","last_active":"2024-09-13T08:40:38.749566+0000","last_peered":"2024-09-13T08:40:38.749566+0000","last_clean":"2024-09-13T08:40:38.749566+0000","last_became_active":"2024-09-13T08:40:37.752052+0000","last_became_peered":"2024-09-13T08:40:37.752052+0000","last_unstale":"2024-09-13T08:40:38.749566+0000","last_undegraded":"2024-09-13T08:40:38.749566+0000","last_fullsized":"2024-09-13T08:40:38.749566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:24.659958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329519+0000","last_change":"2024-09-13T08:40:37.751434+0000","last_active":"2024-09-13T08:40:39.329519+0000","last_peered":"2024-09-13T08:40:39.329519+0000","last_clean":"2024-09-13T08:40:39.329519+0000","last_became_active":"2024-09-13T08:40:37.745985+0000","last_became_peered":"2024-09-13T08:40:37.745985+0000","last_unstale":"2024-09-13T08:40:39.329519+0000","last_undegraded":"2024-09-13T08:40:39.329519+0000","last_fullsized":"2024-09-13T08:40:39.329519+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749808+0000","last_change":"2024-09-13T08:40:37.741617+0000","last_active":"2024-09-13T08:40:38.749808+0000","last_peered":"2024-09-13T08:40:38.749808+0000","last_clean":"2024-09-13T08:40:38.749808+0000","last_became_active":"2024-09-13T08:40:37.741317+0000","last_became_peered":"2024-09-13T08:40:37.741317+0000","last_unstale":"2024-09-13T08:40:38.749808+0000","last_undegraded":"2024-09-13T08:40:38.749808+0000","last_fullsized":"2024-09-13T08:40:38.749808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:51.165235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328648+0000","last_change":"2024-09-13T08:40:37.734978+0000","last_active":"2024-09-13T08:40:39.328648+0000","last_peered":"2024-09-13T08:40:39.328648+0000","last_clean":"2024-09-13T08:40:39.328648+0000","last_became_active":"2024-09-13T08:40:37.734070+0000","last_became_peered":"2024-09-13T08:40:37.734070+0000","last_unstale":"2024-09-13T08:40:39.328648+0000","last_undegraded":"2024-09-13T08:40:39.328648+0000","last_fullsized":"2024-09-13T08:40:39.328648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:34.924669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329904+0000","last_change":"2024-09-13T08:40:37.750441+0000","last_active":"2024-09-13T08:40:39.329904+0000","last_peered":"2024-09-13T08:40:39.329904+0000","last_clean":"2024-09-13T08:40:39.329904+0000","last_became_active":"2024-09-13T08:40:37.750132+0000","last_became_peered":"2024-09-13T08:40:37.750132+0000","last_unstale":"2024-09-13T08:40:39.329904+0000","last_undegraded":"2024-09-13T08:40:39.329904+0000","last_fullsized":"2024-09-13T08:40:39.329904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:34:28.022804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750559+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:38.750559+0000","last_peered":"2024-09-13T08:40:38.750559+0000","last_clean":"2024-09-13T08:40:38.750559+0000","last_became_active":"2024-09-13T08:40:37.750728+0000","last_became_peered":"2024-09-13T08:40:37.750728+0000","last_unstale":"2024-09-13T08:40:38.750559+0000","last_undegraded":"2024-09-13T08:40:38.750559+0000","last_fullsized":"2024-09-13T08:40:38.750559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:19.268020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328968+0000","last_change":"2024-09-13T08:40:37.741441+0000","last_active":"2024-09-13T08:40:39.328968+0000","last_peered":"2024-09-13T08:40:39.328968+0000","last_clean":"2024-09-13T08:40:39.328968+0000","last_became_active":"2024-09-13T08:40:37.737794+0000","last_became_peered":"2024-09-13T08:40:37.737794+0000","last_unstale":"2024-09-13T08:40:39.328968+0000","last_undegraded":"2024-09-13T08:40:39.328968+0000","last_fullsized":"2024-09-13T08:40:39.328968+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:03.212987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750668+0000","last_change":"2024-09-13T08:40:37.748613+0000","last_active":"2024-09-13T08:40:38.750668+0000","last_peered":"2024-09-13T08:40:38.750668+0000","last_clean":"2024-09-13T08:40:38.750668+0000","last_became_active":"2024-09-13T08:40:37.745423+0000","last_became_peered":"2024-09-13T08:40:37.745423+0000","last_unstale":"2024-09-13T08:40:38.750668+0000","last_undegraded":"2024-09-13T08:40:38.750668+0000","last_fullsized":"2024-09-13T08:40:38.750668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:12:13.311559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328777+0000","last_change":"2024-09-13T08:40:37.762367+0000","last_active":"2024-09-13T08:40:39.328777+0000","last_peered":"2024-09-13T08:40:39.328777+0000","last_clean":"2024-09-13T08:40:39.328777+0000","last_became_active":"2024-09-13T08:40:37.762030+0000","last_became_peered":"2024-09-13T08:40:37.762030+0000","last_unstale":"2024-09-13T08:40:39.328777+0000","last_undegraded":"2024-09-13T08:40:39.328777+0000","last_fullsized":"2024-09-13T08:40:39.328777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:11.251715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328832+0000","last_change":"2024-09-13T08:40:37.750469+0000","last_active":"2024-09-13T08:40:39.328832+0000","last_peered":"2024-09-13T08:40:39.328832+0000","last_clean":"2024-09-13T08:40:39.328832+0000","last_became_active":"2024-09-13T08:40:37.748954+0000","last_became_peered":"2024-09-13T08:40:37.748954+0000","last_unstale":"2024-09-13T08:40:39.328832+0000","last_undegraded":"2024-09-13T08:40:39.328832+0000","last_fullsized":"2024-09-13T08:40:39.328832+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:54.518342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329029+0000","last_change":"2024-09-13T08:40:37.744539+0000","last_active":"2024-09-13T08:40:39.329029+0000","last_peered":"2024-09-13T08:40:39.329029+0000","last_clean":"2024-09-13T08:40:39.329029+0000","last_became_active":"2024-09-13T08:40:37.744247+0000","last_became_peered":"2024-09-13T08:40:37.744247+0000","last_unstale":"2024-09-13T08:40:39.329029+0000","last_undegraded":"2024-09-13T08:40:39.329029+0000","last_fullsized":"2024-09-13T08:40:39.329029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:33.390626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749624+0000","last_change":"2024-09-13T08:40:37.750709+0000","last_active":"2024-09-13T08:40:38.749624+0000","last_peered":"2024-09-13T08:40:38.749624+0000","last_clean":"2024-09-13T08:40:38.749624+0000","last_became_active":"2024-09-13T08:40:37.749124+0000","last_became_peered":"2024-09-13T08:40:37.749124+0000","last_unstale":"2024-09-13T08:40:38.749624+0000","last_undegraded":"2024-09-13T08:40:38.749624+0000","last_fullsized":"2024-09-13T08:40:38.749624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:34.977391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749189+0000","last_change":"2024-09-13T08:40:37.741816+0000","last_active":"2024-09-13T08:40:38.749189+0000","last_peered":"2024-09-13T08:40:38.749189+0000","last_clean":"2024-09-13T08:40:38.749189+0000","last_became_active":"2024-09-13T08:40:37.741441+0000","last_became_peered":"2024-09-13T08:40:37.741441+0000","last_unstale":"2024-09-13T08:40:38.749189+0000","last_undegraded":"2024-09-13T08:40:38.749189+0000","last_fullsized":"2024-09-13T08:40:38.749189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:05.154898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761143+0000","last_change":"2024-09-13T08:40:37.758027+0000","last_active":"2024-09-13T08:40:38.761143+0000","last_peered":"2024-09-13T08:40:38.761143+0000","last_clean":"2024-09-13T08:40:38.761143+0000","last_became_active":"2024-09-13T08:40:37.757576+0000","last_became_peered":"2024-09-13T08:40:37.757576+0000","last_unstale":"2024-09-13T08:40:38.761143+0000","last_undegraded":"2024-09-13T08:40:38.761143+0000","last_fullsized":"2024-09-13T08:40:38.761143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:44:11.983878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"61'4","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749923+0000","last_change":"2024-09-13T08:40:36.740867+0000","last_active":"2024-09-13T08:40:38.749923+0000","last_peered":"2024-09-13T08:40:38.749923+0000","last_clean":"2024-09-13T08:40:38.749923+0000","last_became_active":"2024-09-13T08:40:36.740475+0000","last_became_peered":"2024-09-13T08:40:36.740475+0000","last_unstale":"2024-09-13T08:40:38.749923+0000","last_undegraded":"2024-09-13T08:40:38.749923+0000","last_fullsized":"2024-09-13T08:40:38.749923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:29.262920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761364+0000","last_change":"2024-09-13T08:40:37.755669+0000","last_active":"2024-09-13T08:40:38.761364+0000","last_peered":"2024-09-13T08:40:38.761364+0000","last_clean":"2024-09-13T08:40:38.761364+0000","last_became_active":"2024-09-13T08:40:37.755390+0000","last_became_peered":"2024-09-13T08:40:37.755390+0000","last_unstale":"2024-09-13T08:40:38.761364+0000","last_undegraded":"2024-09-13T08:40:38.761364+0000","last_fullsized":"2024-09-13T08:40:38.761364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:32.240200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750611+0000","last_change":"2024-09-13T08:40:36.749525+0000","last_active":"2024-09-13T08:40:38.750611+0000","last_peered":"2024-09-13T08:40:38.750611+0000","last_clean":"2024-09-13T08:40:38.750611+0000","last_became_active":"2024-09-13T08:40:36.749236+0000","last_became_peered":"2024-09-13T08:40:36.749236+0000","last_unstale":"2024-09-13T08:40:38.750611+0000","last_undegraded":"2024-09-13T08:40:38.750611+0000","last_fullsized":"2024-09-13T08:40:38.750611+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328814+0000","last_change":"2024-09-13T08:40:37.758005+0000","last_active":"2024-09-13T08:40:39.328814+0000","last_peered":"2024-09-13T08:40:39.328814+0000","last_clean":"2024-09-13T08:40:39.328814+0000","last_became_active":"2024-09-13T08:40:37.757312+0000","last_became_peered":"2024-09-13T08:40:37.757312+0000","last_unstale":"2024-09-13T08:40:39.328814+0000","last_undegraded":"2024-09-13T08:40:39.328814+0000","last_fullsized":"2024-09-13T08:40:39.328814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:35.645007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761311+0000","last_change":"2024-09-13T08:40:36.751851+0000","last_active":"2024-09-13T08:40:38.761311+0000","last_peered":"2024-09-13T08:40:38.761311+0000","last_clean":"2024-09-13T08:40:38.761311+0000","last_became_active":"2024-09-13T08:40:36.748632+0000","last_became_peered":"2024-09-13T08:40:36.748632+0000","last_unstale":"2024-09-13T08:40:38.761311+0000","last_undegraded":"2024-09-13T08:40:38.761311+0000","last_fullsized":"2024-09-13T08:40:38.761311+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:56.150885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761043+0000","last_change":"2024-09-13T08:40:37.751225+0000","last_active":"2024-09-13T08:40:38.761043+0000","last_peered":"2024-09-13T08:40:38.761043+0000","last_clean":"2024-09-13T08:40:38.761043+0000","last_became_active":"2024-09-13T08:40:37.750923+0000","last_became_peered":"2024-09-13T08:40:37.750923+0000","last_unstale":"2024-09-13T08:40:38.761043+0000","last_undegraded":"2024-09-13T08:40:38.761043+0000","last_fullsized":"2024-09-13T08:40:38.761043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:33.464144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329794+0000","last_change":"2024-09-13T08:40:36.746077+0000","last_active":"2024-09-13T08:40:39.329794+0000","last_peered":"2024-09-13T08:40:39.329794+0000","last_clean":"2024-09-13T08:40:39.329794+0000","last_became_active":"2024-09-13T08:40:36.745814+0000","last_became_peered":"2024-09-13T08:40:36.745814+0000","last_unstale":"2024-09-13T08:40:39.329794+0000","last_undegraded":"2024-09-13T08:40:39.329794+0000","last_fullsized":"2024-09-13T08:40:39.329794+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:24.123182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328584+0000","last_change":"2024-09-13T08:40:37.749662+0000","last_active":"2024-09-13T08:40:39.328584+0000","last_peered":"2024-09-13T08:40:39.328584+0000","last_clean":"2024-09-13T08:40:39.328584+0000","last_became_active":"2024-09-13T08:40:37.749218+0000","last_became_peered":"2024-09-13T08:40:37.749218+0000","last_unstale":"2024-09-13T08:40:39.328584+0000","last_undegraded":"2024-09-13T08:40:39.328584+0000","last_fullsized":"2024-09-13T08:40:39.328584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:43.826750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750476+0000","last_change":"2024-09-13T08:40:36.738659+0000","last_active":"2024-09-13T08:40:38.750476+0000","last_peered":"2024-09-13T08:40:38.750476+0000","last_clean":"2024-09-13T08:40:38.750476+0000","last_became_active":"2024-09-13T08:40:36.738315+0000","last_became_peered":"2024-09-13T08:40:36.738315+0000","last_unstale":"2024-09-13T08:40:38.750476+0000","last_undegraded":"2024-09-13T08:40:38.750476+0000","last_fullsized":"2024-09-13T08:40:38.750476+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:09.635872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749651+0000","last_change":"2024-09-13T08:40:37.741822+0000","last_active":"2024-09-13T08:40:38.749651+0000","last_peered":"2024-09-13T08:40:38.749651+0000","last_clean":"2024-09-13T08:40:38.749651+0000","last_became_active":"2024-09-13T08:40:37.741264+0000","last_became_peered":"2024-09-13T08:40:37.741264+0000","last_unstale":"2024-09-13T08:40:38.749651+0000","last_undegraded":"2024-09-13T08:40:38.749651+0000","last_fullsized":"2024-09-13T08:40:38.749651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:25:36.907735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760840+0000","last_change":"2024-09-13T08:40:36.751002+0000","last_active":"2024-09-13T08:40:38.760840+0000","last_peered":"2024-09-13T08:40:38.760840+0000","last_clean":"2024-09-13T08:40:38.760840+0000","last_became_active":"2024-09-13T08:40:36.750731+0000","last_became_peered":"2024-09-13T08:40:36.750731+0000","last_unstale":"2024-09-13T08:40:38.760840+0000","last_undegraded":"2024-09-13T08:40:38.760840+0000","last_fullsized":"2024-09-13T08:40:38.760840+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:15:24.257806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750280+0000","last_change":"2024-09-13T08:40:37.736587+0000","last_active":"2024-09-13T08:40:38.750280+0000","last_peered":"2024-09-13T08:40:38.750280+0000","last_clean":"2024-09-13T08:40:38.750280+0000","last_became_active":"2024-09-13T08:40:37.734709+0000","last_became_peered":"2024-09-13T08:40:37.734709+0000","last_unstale":"2024-09-13T08:40:38.750280+0000","last_undegraded":"2024-09-13T08:40:38.750280+0000","last_fullsized":"2024-09-13T08:40:38.750280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:33.464144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328731+0000","last_change":"2024-09-13T08:40:36.740922+0000","last_active":"2024-09-13T08:40:39.328731+0000","last_peered":"2024-09-13T08:40:39.328731+0000","last_clean":"2024-09-13T08:40:39.328731+0000","last_became_active":"2024-09-13T08:40:36.740610+0000","last_became_peered":"2024-09-13T08:40:36.740610+0000","last_unstale":"2024-09-13T08:40:39.328731+0000","last_undegraded":"2024-09-13T08:40:39.328731+0000","last_fullsized":"2024-09-13T08:40:39.328731+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760586+0000","last_change":"2024-09-13T08:40:37.753714+0000","last_active":"2024-09-13T08:40:38.760586+0000","last_peered":"2024-09-13T08:40:38.760586+0000","last_clean":"2024-09-13T08:40:38.760586+0000","last_became_active":"2024-09-13T08:40:37.753402+0000","last_became_peered":"2024-09-13T08:40:37.753402+0000","last_unstale":"2024-09-13T08:40:38.760586+0000","last_undegraded":"2024-09-13T08:40:38.760586+0000","last_fullsized":"2024-09-13T08:40:38.760586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:33.743925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329286+0000","last_change":"2024-09-13T08:40:36.737107+0000","last_active":"2024-09-13T08:40:39.329286+0000","last_peered":"2024-09-13T08:40:39.329286+0000","last_clean":"2024-09-13T08:40:39.329286+0000","last_became_active":"2024-09-13T08:40:36.736458+0000","last_became_peered":"2024-09-13T08:40:36.736458+0000","last_unstale":"2024-09-13T08:40:39.329286+0000","last_undegraded":"2024-09-13T08:40:39.329286+0000","last_fullsized":"2024-09-13T08:40:39.329286+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:13.644488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329008+0000","last_change":"2024-09-13T08:40:37.736394+0000","last_active":"2024-09-13T08:40:39.329008+0000","last_peered":"2024-09-13T08:40:39.329008+0000","last_clean":"2024-09-13T08:40:39.329008+0000","last_became_active":"2024-09-13T08:40:37.736142+0000","last_became_peered":"2024-09-13T08:40:37.736142+0000","last_unstale":"2024-09-13T08:40:39.329008+0000","last_undegraded":"2024-09-13T08:40:39.329008+0000","last_fullsized":"2024-09-13T08:40:39.329008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:45.588595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329330+0000","last_change":"2024-09-13T08:40:36.740047+0000","last_active":"2024-09-13T08:40:39.329330+0000","last_peered":"2024-09-13T08:40:39.329330+0000","last_clean":"2024-09-13T08:40:39.329330+0000","last_became_active":"2024-09-13T08:40:36.739774+0000","last_became_peered":"2024-09-13T08:40:36.739774+0000","last_unstale":"2024-09-13T08:40:39.329330+0000","last_undegraded":"2024-09-13T08:40:39.329330+0000","last_fullsized":"2024-09-13T08:40:39.329330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.751071+0000","last_change":"2024-09-13T08:40:37.757097+0000","last_active":"2024-09-13T08:40:38.751071+0000","last_peered":"2024-09-13T08:40:38.751071+0000","last_clean":"2024-09-13T08:40:38.751071+0000","last_became_active":"2024-09-13T08:40:37.756526+0000","last_became_peered":"2024-09-13T08:40:37.756526+0000","last_unstale":"2024-09-13T08:40:38.751071+0000","last_undegraded":"2024-09-13T08:40:38.751071+0000","last_fullsized":"2024-09-13T08:40:38.751071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:01.557871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328753+0000","last_change":"2024-09-13T08:40:36.743002+0000","last_active":"2024-09-13T08:40:39.328753+0000","last_peered":"2024-09-13T08:40:39.328753+0000","last_clean":"2024-09-13T08:40:39.328753+0000","last_became_active":"2024-09-13T08:40:36.740918+0000","last_became_peered":"2024-09-13T08:40:36.740918+0000","last_unstale":"2024-09-13T08:40:39.328753+0000","last_undegraded":"2024-09-13T08:40:39.328753+0000","last_fullsized":"2024-09-13T08:40:39.328753+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:05.024141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329262+0000","last_change":"2024-09-13T08:40:37.756377+0000","last_active":"2024-09-13T08:40:39.329262+0000","last_peered":"2024-09-13T08:40:39.329262+0000","last_clean":"2024-09-13T08:40:39.329262+0000","last_became_active":"2024-09-13T08:40:37.756086+0000","last_became_peered":"2024-09-13T08:40:37.756086+0000","last_unstale":"2024-09-13T08:40:39.329262+0000","last_undegraded":"2024-09-13T08:40:39.329262+0000","last_fullsized":"2024-09-13T08:40:39.329262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:33.675957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329755+0000","last_change":"2024-09-13T08:40:36.745979+0000","last_active":"2024-09-13T08:40:39.329755+0000","last_peered":"2024-09-13T08:40:39.329755+0000","last_clean":"2024-09-13T08:40:39.329755+0000","last_became_active":"2024-09-13T08:40:36.745673+0000","last_became_peered":"2024-09-13T08:40:36.745673+0000","last_unstale":"2024-09-13T08:40:39.329755+0000","last_undegraded":"2024-09-13T08:40:39.329755+0000","last_fullsized":"2024-09-13T08:40:39.329755+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:11.780765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329572+0000","last_change":"2024-09-13T08:40:37.752368+0000","last_active":"2024-09-13T08:40:39.329572+0000","last_peered":"2024-09-13T08:40:39.329572+0000","last_clean":"2024-09-13T08:40:39.329572+0000","last_became_active":"2024-09-13T08:40:37.752096+0000","last_became_peered":"2024-09-13T08:40:37.752096+0000","last_unstale":"2024-09-13T08:40:39.329572+0000","last_undegraded":"2024-09-13T08:40:39.329572+0000","last_fullsized":"2024-09-13T08:40:39.329572+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:54.227482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":21,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.330001+0000","last_change":"2024-09-13T08:40:36.745140+0000","last_active":"2024-09-13T08:40:39.330001+0000","last_peered":"2024-09-13T08:40:39.330001+0000","last_clean":"2024-09-13T08:40:39.330001+0000","last_became_active":"2024-09-13T08:40:36.743817+0000","last_became_peered":"2024-09-13T08:40:36.743817+0000","last_unstale":"2024-09-13T08:40:39.330001+0000","last_undegraded":"2024-09-13T08:40:39.330001+0000","last_fullsized":"2024-09-13T08:40:39.330001+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:15.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328362+0000","last_change":"2024-09-13T08:40:37.740915+0000","last_active":"2024-09-13T08:40:39.328362+0000","last_peered":"2024-09-13T08:40:39.328362+0000","last_clean":"2024-09-13T08:40:39.328362+0000","last_became_active":"2024-09-13T08:40:37.740617+0000","last_became_peered":"2024-09-13T08:40:37.740617+0000","last_unstale":"2024-09-13T08:40:39.328362+0000","last_undegraded":"2024-09-13T08:40:39.328362+0000","last_fullsized":"2024-09-13T08:40:39.328362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:24.421420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329067+0000","last_change":"2024-09-13T08:40:36.740132+0000","last_active":"2024-09-13T08:40:39.329067+0000","last_peered":"2024-09-13T08:40:39.329067+0000","last_clean":"2024-09-13T08:40:39.329067+0000","last_became_active":"2024-09-13T08:40:36.739787+0000","last_became_peered":"2024-09-13T08:40:36.739787+0000","last_unstale":"2024-09-13T08:40:39.329067+0000","last_undegraded":"2024-09-13T08:40:39.329067+0000","last_fullsized":"2024-09-13T08:40:39.329067+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.013701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750486+0000","last_change":"2024-09-13T08:40:37.754029+0000","last_active":"2024-09-13T08:40:38.750486+0000","last_peered":"2024-09-13T08:40:38.750486+0000","last_clean":"2024-09-13T08:40:38.750486+0000","last_became_active":"2024-09-13T08:40:37.753773+0000","last_became_peered":"2024-09-13T08:40:37.753773+0000","last_unstale":"2024-09-13T08:40:38.750486+0000","last_undegraded":"2024-09-13T08:40:38.750486+0000","last_fullsized":"2024-09-13T08:40:38.750486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:27.237888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329177+0000","last_change":"2024-09-13T08:40:36.746192+0000","last_active":"2024-09-13T08:40:39.329177+0000","last_peered":"2024-09-13T08:40:39.329177+0000","last_clean":"2024-09-13T08:40:39.329177+0000","last_became_active":"2024-09-13T08:40:36.745703+0000","last_became_peered":"2024-09-13T08:40:36.745703+0000","last_unstale":"2024-09-13T08:40:39.329177+0000","last_undegraded":"2024-09-13T08:40:39.329177+0000","last_fullsized":"2024-09-13T08:40:39.329177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:00.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329485+0000","last_change":"2024-09-13T08:40:37.750252+0000","last_active":"2024-09-13T08:40:39.329485+0000","last_peered":"2024-09-13T08:40:39.329485+0000","last_clean":"2024-09-13T08:40:39.329485+0000","last_became_active":"2024-09-13T08:40:37.749793+0000","last_became_peered":"2024-09-13T08:40:37.749793+0000","last_unstale":"2024-09-13T08:40:39.329485+0000","last_undegraded":"2024-09-13T08:40:39.329485+0000","last_fullsized":"2024-09-13T08:40:39.329485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:40.674973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329012+0000","last_change":"2024-09-13T08:40:36.748499+0000","last_active":"2024-09-13T08:40:39.329012+0000","last_peered":"2024-09-13T08:40:39.329012+0000","last_clean":"2024-09-13T08:40:39.329012+0000","last_became_active":"2024-09-13T08:40:36.745832+0000","last_became_peered":"2024-09-13T08:40:36.745832+0000","last_unstale":"2024-09-13T08:40:39.329012+0000","last_undegraded":"2024-09-13T08:40:39.329012+0000","last_fullsized":"2024-09-13T08:40:39.329012+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:42.819908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328859+0000","last_change":"2024-09-13T08:40:37.734550+0000","last_active":"2024-09-13T08:40:39.328859+0000","last_peered":"2024-09-13T08:40:39.328859+0000","last_clean":"2024-09-13T08:40:39.328859+0000","last_became_active":"2024-09-13T08:40:37.734140+0000","last_became_peered":"2024-09-13T08:40:37.734140+0000","last_unstale":"2024-09-13T08:40:39.328859+0000","last_undegraded":"2024-09-13T08:40:39.328859+0000","last_fullsized":"2024-09-13T08:40:39.328859+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:54.797593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":23,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328384+0000","last_change":"2024-09-13T08:40:36.746654+0000","last_active":"2024-09-13T08:40:39.328384+0000","last_peered":"2024-09-13T08:40:39.328384+0000","last_clean":"2024-09-13T08:40:39.328384+0000","last_became_active":"2024-09-13T08:40:36.746397+0000","last_became_peered":"2024-09-13T08:40:36.746397+0000","last_unstale":"2024-09-13T08:40:39.328384+0000","last_undegraded":"2024-09-13T08:40:39.328384+0000","last_fullsized":"2024-09-13T08:40:39.328384+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:15.203955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750994+0000","last_change":"2024-09-13T08:40:37.745284+0000","last_active":"2024-09-13T08:40:38.750994+0000","last_peered":"2024-09-13T08:40:38.750994+0000","last_clean":"2024-09-13T08:40:38.750994+0000","last_became_active":"2024-09-13T08:40:37.744957+0000","last_became_peered":"2024-09-13T08:40:37.744957+0000","last_unstale":"2024-09-13T08:40:38.750994+0000","last_undegraded":"2024-09-13T08:40:38.750994+0000","last_fullsized":"2024-09-13T08:40:38.750994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:32.240200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750560+0000","last_change":"2024-09-13T08:40:36.753006+0000","last_active":"2024-09-13T08:40:38.750560+0000","last_peered":"2024-09-13T08:40:38.750560+0000","last_clean":"2024-09-13T08:40:38.750560+0000","last_became_active":"2024-09-13T08:40:36.752498+0000","last_became_peered":"2024-09-13T08:40:36.752498+0000","last_unstale":"2024-09-13T08:40:38.750560+0000","last_undegraded":"2024-09-13T08:40:38.750560+0000","last_fullsized":"2024-09-13T08:40:38.750560+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.751248+0000","last_change":"2024-09-13T08:40:37.757310+0000","last_active":"2024-09-13T08:40:38.751248+0000","last_peered":"2024-09-13T08:40:38.751248+0000","last_clean":"2024-09-13T08:40:38.751248+0000","last_became_active":"2024-09-13T08:40:37.756852+0000","last_became_peered":"2024-09-13T08:40:37.756852+0000","last_unstale":"2024-09-13T08:40:38.751248+0000","last_undegraded":"2024-09-13T08:40:38.751248+0000","last_fullsized":"2024-09-13T08:40:38.751248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:10.473515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328981+0000","last_change":"2024-09-13T08:40:36.741233+0000","last_active":"2024-09-13T08:40:39.328981+0000","last_peered":"2024-09-13T08:40:39.328981+0000","last_clean":"2024-09-13T08:40:39.328981+0000","last_became_active":"2024-09-13T08:40:36.740574+0000","last_became_peered":"2024-09-13T08:40:36.740574+0000","last_unstale":"2024-09-13T08:40:39.328981+0000","last_undegraded":"2024-09-13T08:40:39.328981+0000","last_fullsized":"2024-09-13T08:40:39.328981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:52.377019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750236+0000","last_change":"2024-09-13T08:40:37.761727+0000","last_active":"2024-09-13T08:40:38.750236+0000","last_peered":"2024-09-13T08:40:38.750236+0000","last_clean":"2024-09-13T08:40:38.750236+0000","last_became_active":"2024-09-13T08:40:37.761016+0000","last_became_peered":"2024-09-13T08:40:37.761016+0000","last_unstale":"2024-09-13T08:40:38.750236+0000","last_undegraded":"2024-09-13T08:40:38.750236+0000","last_fullsized":"2024-09-13T08:40:38.750236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:34:28.022804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750795+0000","last_change":"2024-09-13T08:40:36.740608+0000","last_active":"2024-09-13T08:40:38.750795+0000","last_peered":"2024-09-13T08:40:38.750795+0000","last_clean":"2024-09-13T08:40:38.750795+0000","last_became_active":"2024-09-13T08:40:36.740349+0000","last_became_peered":"2024-09-13T08:40:36.740349+0000","last_unstale":"2024-09-13T08:40:38.750795+0000","last_undegraded":"2024-09-13T08:40:38.750795+0000","last_fullsized":"2024-09-13T08:40:38.750795+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.279565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329689+0000","last_change":"2024-09-13T08:40:37.758748+0000","last_active":"2024-09-13T08:40:39.329689+0000","last_peered":"2024-09-13T08:40:39.329689+0000","last_clean":"2024-09-13T08:40:39.329689+0000","last_became_active":"2024-09-13T08:40:37.756975+0000","last_became_peered":"2024-09-13T08:40:37.756975+0000","last_unstale":"2024-09-13T08:40:39.329689+0000","last_undegraded":"2024-09-13T08:40:39.329689+0000","last_fullsized":"2024-09-13T08:40:39.329689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:36.078214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329197+0000","last_change":"2024-09-13T08:40:36.748868+0000","last_active":"2024-09-13T08:40:39.329197+0000","last_peered":"2024-09-13T08:40:39.329197+0000","last_clean":"2024-09-13T08:40:39.329197+0000","last_became_active":"2024-09-13T08:40:36.748527+0000","last_became_peered":"2024-09-13T08:40:36.748527+0000","last_unstale":"2024-09-13T08:40:39.329197+0000","last_undegraded":"2024-09-13T08:40:39.329197+0000","last_fullsized":"2024-09-13T08:40:39.329197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:05.275938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328591+0000","last_change":"2024-09-13T08:40:37.756050+0000","last_active":"2024-09-13T08:40:39.328591+0000","last_peered":"2024-09-13T08:40:39.328591+0000","last_clean":"2024-09-13T08:40:39.328591+0000","last_became_active":"2024-09-13T08:40:37.755788+0000","last_became_peered":"2024-09-13T08:40:37.755788+0000","last_unstale":"2024-09-13T08:40:39.328591+0000","last_undegraded":"2024-09-13T08:40:39.328591+0000","last_fullsized":"2024-09-13T08:40:39.328591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:58:06.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749727+0000","last_change":"2024-09-13T08:40:36.737698+0000","last_active":"2024-09-13T08:40:38.749727+0000","last_peered":"2024-09-13T08:40:38.749727+0000","last_clean":"2024-09-13T08:40:38.749727+0000","last_became_active":"2024-09-13T08:40:36.737306+0000","last_became_peered":"2024-09-13T08:40:36.737306+0000","last_unstale":"2024-09-13T08:40:38.749727+0000","last_undegraded":"2024-09-13T08:40:38.749727+0000","last_fullsized":"2024-09-13T08:40:38.749727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:52.538374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328987+0000","last_change":"2024-09-13T08:40:37.751010+0000","last_active":"2024-09-13T08:40:39.328987+0000","last_peered":"2024-09-13T08:40:39.328987+0000","last_clean":"2024-09-13T08:40:39.328987+0000","last_became_active":"2024-09-13T08:40:37.750682+0000","last_became_peered":"2024-09-13T08:40:37.750682+0000","last_unstale":"2024-09-13T08:40:39.328987+0000","last_undegraded":"2024-09-13T08:40:39.328987+0000","last_fullsized":"2024-09-13T08:40:39.328987+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:14.087231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328604+0000","last_change":"2024-09-13T08:40:36.749086+0000","last_active":"2024-09-13T08:40:39.328604+0000","last_peered":"2024-09-13T08:40:39.328604+0000","last_clean":"2024-09-13T08:40:39.328604+0000","last_became_active":"2024-09-13T08:40:36.747729+0000","last_became_peered":"2024-09-13T08:40:36.747729+0000","last_unstale":"2024-09-13T08:40:39.328604+0000","last_undegraded":"2024-09-13T08:40:39.328604+0000","last_fullsized":"2024-09-13T08:40:39.328604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:12:48.295035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328599+0000","last_change":"2024-09-13T08:40:37.765328+0000","last_active":"2024-09-13T08:40:39.328599+0000","last_peered":"2024-09-13T08:40:39.328599+0000","last_clean":"2024-09-13T08:40:39.328599+0000","last_became_active":"2024-09-13T08:40:37.762317+0000","last_became_peered":"2024-09-13T08:40:37.762317+0000","last_unstale":"2024-09-13T08:40:39.328599+0000","last_undegraded":"2024-09-13T08:40:39.328599+0000","last_fullsized":"2024-09-13T08:40:39.328599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:01.293446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329635+0000","last_change":"2024-09-13T08:40:36.747739+0000","last_active":"2024-09-13T08:40:39.329635+0000","last_peered":"2024-09-13T08:40:39.329635+0000","last_clean":"2024-09-13T08:40:39.329635+0000","last_became_active":"2024-09-13T08:40:36.747366+0000","last_became_peered":"2024-09-13T08:40:36.747366+0000","last_unstale":"2024-09-13T08:40:39.329635+0000","last_undegraded":"2024-09-13T08:40:39.329635+0000","last_fullsized":"2024-09-13T08:40:39.329635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:44.202688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.751033+0000","last_change":"2024-09-13T08:40:37.746566+0000","last_active":"2024-09-13T08:40:38.751033+0000","last_peered":"2024-09-13T08:40:38.751033+0000","last_clean":"2024-09-13T08:40:38.751033+0000","last_became_active":"2024-09-13T08:40:37.746382+0000","last_became_peered":"2024-09-13T08:40:37.746382+0000","last_unstale":"2024-09-13T08:40:38.751033+0000","last_undegraded":"2024-09-13T08:40:38.751033+0000","last_fullsized":"2024-09-13T08:40:38.751033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:16.638312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750698+0000","last_change":"2024-09-13T08:40:36.743612+0000","last_active":"2024-09-13T08:40:38.750698+0000","last_peered":"2024-09-13T08:40:38.750698+0000","last_clean":"2024-09-13T08:40:38.750698+0000","last_became_active":"2024-09-13T08:40:36.743337+0000","last_became_peered":"2024-09-13T08:40:36.743337+0000","last_unstale":"2024-09-13T08:40:38.750698+0000","last_undegraded":"2024-09-13T08:40:38.750698+0000","last_fullsized":"2024-09-13T08:40:38.750698+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:43.142011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.751284+0000","last_change":"2024-09-13T08:40:37.746100+0000","last_active":"2024-09-13T08:40:38.751284+0000","last_peered":"2024-09-13T08:40:38.751284+0000","last_clean":"2024-09-13T08:40:38.751284+0000","last_became_active":"2024-09-13T08:40:37.745801+0000","last_became_peered":"2024-09-13T08:40:37.745801+0000","last_unstale":"2024-09-13T08:40:38.751284+0000","last_undegraded":"2024-09-13T08:40:38.751284+0000","last_fullsized":"2024-09-13T08:40:38.751284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:54.090892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"62'1","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.761228+0000","last_change":"2024-09-13T08:40:36.751068+0000","last_active":"2024-09-13T08:40:38.761228+0000","last_peered":"2024-09-13T08:40:38.761228+0000","last_clean":"2024-09-13T08:40:38.761228+0000","last_became_active":"2024-09-13T08:40:36.748029+0000","last_became_peered":"2024-09-13T08:40:36.748029+0000","last_unstale":"2024-09-13T08:40:38.761228+0000","last_undegraded":"2024-09-13T08:40:38.761228+0000","last_fullsized":"2024-09-13T08:40:38.761228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:25.694691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750209+0000","last_change":"2024-09-13T08:40:37.743421+0000","last_active":"2024-09-13T08:40:38.750209+0000","last_peered":"2024-09-13T08:40:38.750209+0000","last_clean":"2024-09-13T08:40:38.750209+0000","last_became_active":"2024-09-13T08:40:37.743126+0000","last_became_peered":"2024-09-13T08:40:37.743126+0000","last_unstale":"2024-09-13T08:40:38.750209+0000","last_undegraded":"2024-09-13T08:40:38.750209+0000","last_fullsized":"2024-09-13T08:40:38.750209+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:40.674973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328445+0000","last_change":"2024-09-13T08:40:36.742793+0000","last_active":"2024-09-13T08:40:39.328445+0000","last_peered":"2024-09-13T08:40:39.328445+0000","last_clean":"2024-09-13T08:40:39.328445+0000","last_became_active":"2024-09-13T08:40:36.742516+0000","last_became_peered":"2024-09-13T08:40:36.742516+0000","last_unstale":"2024-09-13T08:40:39.328445+0000","last_undegraded":"2024-09-13T08:40:39.328445+0000","last_fullsized":"2024-09-13T08:40:39.328445+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:08:02.890149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329248+0000","last_change":"2024-09-13T08:40:37.763959+0000","last_active":"2024-09-13T08:40:39.329248+0000","last_peered":"2024-09-13T08:40:39.329248+0000","last_clean":"2024-09-13T08:40:39.329248+0000","last_became_active":"2024-09-13T08:40:37.763661+0000","last_became_peered":"2024-09-13T08:40:37.763661+0000","last_unstale":"2024-09-13T08:40:39.329248+0000","last_undegraded":"2024-09-13T08:40:39.329248+0000","last_fullsized":"2024-09-13T08:40:39.329248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:43.311241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750568+0000","last_change":"2024-09-13T08:40:36.750833+0000","last_active":"2024-09-13T08:40:38.750568+0000","last_peered":"2024-09-13T08:40:38.750568+0000","last_clean":"2024-09-13T08:40:38.750568+0000","last_became_active":"2024-09-13T08:40:36.750471+0000","last_became_peered":"2024-09-13T08:40:36.750471+0000","last_unstale":"2024-09-13T08:40:38.750568+0000","last_undegraded":"2024-09-13T08:40:38.750568+0000","last_fullsized":"2024-09-13T08:40:38.750568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:36.394174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760873+0000","last_change":"2024-09-13T08:40:37.756241+0000","last_active":"2024-09-13T08:40:38.760873+0000","last_peered":"2024-09-13T08:40:38.760873+0000","last_clean":"2024-09-13T08:40:38.760873+0000","last_became_active":"2024-09-13T08:40:37.755961+0000","last_became_peered":"2024-09-13T08:40:37.755961+0000","last_unstale":"2024-09-13T08:40:38.760873+0000","last_undegraded":"2024-09-13T08:40:38.760873+0000","last_fullsized":"2024-09-13T08:40:38.760873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:20.705612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749969+0000","last_change":"2024-09-13T08:40:36.750422+0000","last_active":"2024-09-13T08:40:38.749969+0000","last_peered":"2024-09-13T08:40:38.749969+0000","last_clean":"2024-09-13T08:40:38.749969+0000","last_became_active":"2024-09-13T08:40:36.749512+0000","last_became_peered":"2024-09-13T08:40:36.749512+0000","last_unstale":"2024-09-13T08:40:38.749969+0000","last_undegraded":"2024-09-13T08:40:38.749969+0000","last_fullsized":"2024-09-13T08:40:38.749969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:20.298192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329724+0000","last_change":"2024-09-13T08:40:37.756404+0000","last_active":"2024-09-13T08:40:39.329724+0000","last_peered":"2024-09-13T08:40:39.329724+0000","last_clean":"2024-09-13T08:40:39.329724+0000","last_became_active":"2024-09-13T08:40:37.755037+0000","last_became_peered":"2024-09-13T08:40:37.755037+0000","last_unstale":"2024-09-13T08:40:39.329724+0000","last_undegraded":"2024-09-13T08:40:39.329724+0000","last_fullsized":"2024-09-13T08:40:39.329724+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:02.573536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.750310+0000","last_change":"2024-09-13T08:40:36.740510+0000","last_active":"2024-09-13T08:40:38.750310+0000","last_peered":"2024-09-13T08:40:38.750310+0000","last_clean":"2024-09-13T08:40:38.750310+0000","last_became_active":"2024-09-13T08:40:36.740243+0000","last_became_peered":"2024-09-13T08:40:36.740243+0000","last_unstale":"2024-09-13T08:40:38.750310+0000","last_undegraded":"2024-09-13T08:40:38.750310+0000","last_fullsized":"2024-09-13T08:40:38.750310+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:20.155484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328685+0000","last_change":"2024-09-13T08:40:37.751854+0000","last_active":"2024-09-13T08:40:39.328685+0000","last_peered":"2024-09-13T08:40:39.328685+0000","last_clean":"2024-09-13T08:40:39.328685+0000","last_became_active":"2024-09-13T08:40:37.749368+0000","last_became_peered":"2024-09-13T08:40:37.749368+0000","last_unstale":"2024-09-13T08:40:39.328685+0000","last_undegraded":"2024-09-13T08:40:39.328685+0000","last_fullsized":"2024-09-13T08:40:39.328685+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:12.787608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.760584+0000","last_change":"2024-09-13T08:40:36.743024+0000","last_active":"2024-09-13T08:40:38.760584+0000","last_peered":"2024-09-13T08:40:38.760584+0000","last_clean":"2024-09-13T08:40:38.760584+0000","last_became_active":"2024-09-13T08:40:36.742739+0000","last_became_peered":"2024-09-13T08:40:36.742739+0000","last_unstale":"2024-09-13T08:40:38.760584+0000","last_undegraded":"2024-09-13T08:40:38.760584+0000","last_fullsized":"2024-09-13T08:40:38.760584+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:54.197613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:38.749842+0000","last_change":"2024-09-13T08:40:37.743948+0000","last_active":"2024-09-13T08:40:38.749842+0000","last_peered":"2024-09-13T08:40:38.749842+0000","last_clean":"2024-09-13T08:40:38.749842+0000","last_became_active":"2024-09-13T08:40:37.743656+0000","last_became_peered":"2024-09-13T08:40:37.743656+0000","last_unstale":"2024-09-13T08:40:38.749842+0000","last_undegraded":"2024-09-13T08:40:38.749842+0000","last_fullsized":"2024-09-13T08:40:38.749842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:18.978754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328883+0000","last_change":"2024-09-13T08:40:36.750854+0000","last_active":"2024-09-13T08:40:39.328883+0000","last_peered":"2024-09-13T08:40:39.328883+0000","last_clean":"2024-09-13T08:40:39.328883+0000","last_became_active":"2024-09-13T08:40:36.750177+0000","last_became_peered":"2024-09-13T08:40:36.750177+0000","last_unstale":"2024-09-13T08:40:39.328883+0000","last_undegraded":"2024-09-13T08:40:39.328883+0000","last_fullsized":"2024-09-13T08:40:39.328883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:20.695241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.328964+0000","last_change":"2024-09-13T08:40:37.746462+0000","last_active":"2024-09-13T08:40:39.328964+0000","last_peered":"2024-09-13T08:40:39.328964+0000","last_clean":"2024-09-13T08:40:39.328964+0000","last_became_active":"2024-09-13T08:40:37.745179+0000","last_became_peered":"2024-09-13T08:40:37.745179+0000","last_unstale":"2024-09-13T08:40:39.328964+0000","last_undegraded":"2024-09-13T08:40:39.328964+0000","last_fullsized":"2024-09-13T08:40:39.328964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:36.715339+0000","last_clean_scrub_stamp":"2024-09-13T08:40:36.715339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:19.281735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329342+0000","last_change":"2024-09-13T08:40:36.744466+0000","last_active":"2024-09-13T08:40:39.329342+0000","last_peered":"2024-09-13T08:40:39.329342+0000","last_clean":"2024-09-13T08:40:39.329342+0000","last_became_active":"2024-09-13T08:40:36.744206+0000","last_became_peered":"2024-09-13T08:40:36.744206+0000","last_unstale":"2024-09-13T08:40:39.329342+0000","last_undegraded":"2024-09-13T08:40:39.329342+0000","last_fullsized":"2024-09-13T08:40:39.329342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:40:35.708497+0000","last_clean_scrub_stamp":"2024-09-13T08:40:35.708497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:40.300855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T08:40:39.329729+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:40:39.329729+0000","last_peered":"2024-09-13T08:40:39.329729+0000","last_clean":"2024-09-13T08:40:39.329729+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:40:39.329729+0000","last_undegraded":"2024-09-13T08:40:39.329729+0000","last_fullsized":"2024-09-13T08:40:39.329729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509488,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":745472,"data_stored":306011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":741376,"data_stored":305941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1335296,"data_stored":896321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509488,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":741376,"data_stored":306009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":757760,"data_stored":307691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1335296,"data_stored":896869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509487,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":749568,"data_stored":307531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36076,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335764,"statfs":{"total":96636764160,"available":96599822336,"internally_reserved":0,"allocated":745472,"data_stored":306535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:40:47.197 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:40:47.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 -- 172.21.15.31:0/48827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18640745f0 msgr2=0x7f18640cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 --2- 172.21.15.31:0/48827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18640745f0 0x7f18640cdcc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f18640d0cb0 tx=0x7f185c02aa30 comp rx=0 tx=0).stop 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 -- 172.21.15.31:0/48827 shutdown_connections 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 --2- 172.21.15.31:0/48827 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 0x7f18640cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 --2- 172.21.15.31:0/48827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18640745f0 0x7f18640cdcc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 --2- 172.21.15.31:0/48827 >> v2:172.21.15.45:3300/0 conn(0x7f186415acb0 0x7f18640cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 -- 172.21.15.31:0/48827 >> 172.21.15.31:0/48827 conn(0x7f18640cfbb0 msgr2=0x7f18640cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 -- 172.21.15.31:0/48827 shutdown_connections 2024-09-13T08:40:47.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.399+0000 7f186c400640 1 -- 172.21.15.31:0/48827 wait complete. 2024-09-13T08:40:47.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.401+0000 7f186c400640 1 Processor -- start 2024-09-13T08:40:47.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.401+0000 7f186c400640 1 -- start start 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 0x7f18641f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18640745f0 0x7f186414d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f18640cad70 con 0x7f186415acb0 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18640cabf0 con 0x7f18640745f0 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186c400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f18640caef0 con 0x7f18640741b0 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43526/0 (socket says 172.21.15.31:43526) 2024-09-13T08:40:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f186a400640 1 -- 172.21.15.31:0/1636522005 learned_addr learned my addr 172.21.15.31:0/1636522005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f1869000640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18640745f0 0x7f186414d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.402+0000 7f1869a00640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 0x7f18641f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 -- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 msgr2=0x7f18641f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 0x7f18641f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 -- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18640745f0 msgr2=0x7f186414d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18640745f0 0x7f186414d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 -- 172.21.15.31:0/1636522005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641f8dc0 con 0x7f186415acb0 2024-09-13T08:40:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186a400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f185c009d60 tx=0x7f185c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c047070 con 0x7f186415acb0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f185c02f690 con 0x7f186415acb0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c033050 con 0x7f186415acb0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186415b560 con 0x7f186415acb0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.403+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186415bb20 con 0x7f186415acb0 2024-09-13T08:40:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.405+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186406ac90 con 0x7f186415acb0 2024-09-13T08:40:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.406+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f185c04a070 con 0x7f186415acb0 2024-09-13T08:40:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.407+0000 7f1861600640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 0x7f184407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.407+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f185c0bbd30 con 0x7f186415acb0 2024-09-13T08:40:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.407+0000 7f1869a00640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 0x7f184407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.408+0000 7f1869a00640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 0x7f184407aec0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1858009b30 tx=0x7f1858005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.411+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f186406ac90 con 0x7f186415acb0 2024-09-13T08:40:47.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.682+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f18640cf4d0 con 0x7f186415acb0 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.685+0000 7f1861600640 1 -- 172.21.15.31:0/1636522005 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f18641f8dc0 con 0x7f186415acb0 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:40:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:40:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 msgr2=0x7f184407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 0x7f184407aec0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1858009b30 tx=0x7f1858005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 msgr2=0x7f186415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f185c009d60 tx=0x7f185c003f10 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 shutdown_connections 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f186415acb0 0x7f186415c600 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18640745f0 0x7f186414d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1844078a00 0x7f184407aec0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 --2- 172.21.15.31:0/1636522005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18640741b0 0x7f18641f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 >> 172.21.15.31:0/1636522005 conn(0x7f18640cfbb0 msgr2=0x7f18640b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 shutdown_connections 2024-09-13T08:40:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.688+0000 7f186c400640 1 -- 172.21.15.31:0/1636522005 wait complete. 2024-09-13T08:40:47.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:40:47.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 -- 172.21.15.31:0/1253491756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 msgr2=0x7f54000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 --2- 172.21.15.31:0/1253491756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f54000051c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5410001e90 tx=0x7f5400005830 comp rx=0 tx=0).stop 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541de00640 1 -- 172.21.15.31:0/1253491756 reap_dead start 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 -- 172.21.15.31:0/1253491756 shutdown_connections 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 -- 172.21.15.31:0/1253491756 >> 172.21.15.31:0/1253491756 conn(0x7f54180cc910 msgr2=0x7f54180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 -- 172.21.15.31:0/1253491756 shutdown_connections 2024-09-13T08:40:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.866+0000 7f541fe00640 1 -- 172.21.15.31:0/1253491756 wait complete. 2024-09-13T08:40:47.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.868+0000 7f541fe00640 1 Processor -- start 2024-09-13T08:40:47.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.868+0000 7f541fe00640 1 -- start start 2024-09-13T08:40:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f54180b58f0 0x7f541809c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 0x7f54180b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54181600f0 con 0x7f5400004d40 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f54180cb850 con 0x7f541806f430 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f541806f9a0 con 0x7f54180b58f0 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43554/0 (socket says 172.21.15.31:43554) 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541d400640 1 -- 172.21.15.31:0/2907304617 learned_addr learned my addr 172.21.15.31:0/2907304617 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.869+0000 7f541ca00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f54180b58f0 0x7f541809c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541de00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 0x7f54180b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 -- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f54180b58f0 msgr2=0x7f541809c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f54180b58f0 0x7f541809c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 -- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 msgr2=0x7f54180b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 0x7f54180b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 -- 172.21.15.31:0/2907304617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541809b4c0 con 0x7f5400004d40 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541de00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 0x7f54180b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541d400640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f540c000fd0 tx=0x7f540c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c015070 con 0x7f5400004d40 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f540c011840 con 0x7f5400004d40 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c011c90 con 0x7f5400004d40 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541809b7a0 con 0x7f5400004d40 2024-09-13T08:40:47.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.870+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541809bd60 con 0x7f5400004d40 2024-09-13T08:40:47.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.872+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541809bf60 con 0x7f5400004d40 2024-09-13T08:40:47.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.873+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f540c018070 con 0x7f5400004d40 2024-09-13T08:40:47.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.874+0000 7f5415600640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 0x7f53f807aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:47.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.874+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f540c09ba30 con 0x7f5400004d40 2024-09-13T08:40:47.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.875+0000 7f541ca00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 0x7f53f807aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:47.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.875+0000 7f541ca00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 0x7f53f807aef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f541806e940 tx=0x7f5404005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:47.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:47.879+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f541809bf60 con 0x7f5400004d40 2024-09-13T08:40:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.111+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54180cc3f0 con 0x7f5400004d40 2024-09-13T08:40:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.111+0000 7f5415600640 1 -- 172.21.15.31:0/2907304617 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5217 (secure 0 0 0) 0x7f540c063c80 con 0x7f5400004d40 2024-09-13T08:40:48.112 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:40:48.112 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":37,"btime":"2024-09-13T08:40:40:762682+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:40:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 37 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 msgr2=0x7f53f807aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 0x7f53f807aef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f541806e940 tx=0x7f5404005f90 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 msgr2=0x7f541809c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f540c000fd0 tx=0x7f540c0039c0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 shutdown_connections 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541806f430 0x7f54180b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f53f8078a30 0x7f53f807aef0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f54180b58f0 0x7f541809c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 --2- 172.21.15.31:0/2907304617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5400004d40 0x7f541809c280 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 >> 172.21.15.31:0/2907304617 conn(0x7f54180cc910 msgr2=0x7f54180ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 shutdown_connections 2024-09-13T08:40:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.114+0000 7f541fe00640 1 -- 172.21.15.31:0/2907304617 wait complete. 2024-09-13T08:40:48.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:40:37.170179+0000', 'modified': '2024-09-13T08:40:40.200201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5748}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5748': {'gid': 5748, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6835/707614758', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6833', 'nonce': 707614758}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 707614758}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5748, 'qdb_cluster': [5748]} 2024-09-13T08:40:48.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:40:48.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 -- 172.21.15.31:0/3659205211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 msgr2=0x7f080c06e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 --2- 172.21.15.31:0/3659205211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c06e660 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f07fc005fd0 tx=0x7f07fc02a530 comp rx=0 tx=0).stop 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 -- 172.21.15.31:0/3659205211 shutdown_connections 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 --2- 172.21.15.31:0/3659205211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06eba0 0x7f080c06f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 --2- 172.21.15.31:0/3659205211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c06e660 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 -- 172.21.15.31:0/3659205211 >> 172.21.15.31:0/3659205211 conn(0x7f080c0b18f0 msgr2=0x7f080c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 -- 172.21.15.31:0/3659205211 shutdown_connections 2024-09-13T08:40:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.696+0000 7f0810a00640 1 -- 172.21.15.31:0/3659205211 wait complete. 2024-09-13T08:40:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.698+0000 7f0810a00640 1 Processor -- start 2024-09-13T08:40:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.698+0000 7f0810a00640 1 -- start start 2024-09-13T08:40:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f080c06eba0 0x7f080c09d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06f8c0 0x7f080c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f080c0b4b40 con 0x7f080c158700 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f080c0b49c0 con 0x7f080c06eba0 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0810a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f080c0ccd70 con 0x7f080c06f8c0 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f080a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0809e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f080c06eba0 0x7f080c09d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f080a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43578/0 (socket says 172.21.15.31:43578) 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f080a800640 1 -- 172.21.15.31:0/1888381858 learned_addr learned my addr 172.21.15.31:0/1888381858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.699+0000 7f0801400640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06f8c0 0x7f080c14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 -- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06f8c0 msgr2=0x7f080c14df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06f8c0 0x7f080c14df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 -- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f080c06eba0 msgr2=0x7f080c09d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f080c06eba0 0x7f080c09d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 -- 172.21.15.31:0/1888381858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c073050 con 0x7f080c158700 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f080a800640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f080400c7b0 tx=0x7f080400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f0802a00640 1 -- 172.21.15.31:0/1888381858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804004960 con 0x7f080c158700 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f0802a00640 1 -- 172.21.15.31:0/1888381858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0804004bf0 con 0x7f080c158700 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080c09ddf0 con 0x7f080c158700 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f0802a00640 1 -- 172.21.15.31:0/1888381858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080400a850 con 0x7f080c158700 2024-09-13T08:40:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.700+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080c073310 con 0x7f080c158700 2024-09-13T08:40:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.703+0000 7f0802a00640 1 -- 172.21.15.31:0/1888381858 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f080400a9f0 con 0x7f080c158700 2024-09-13T08:40:48.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.704+0000 7f0802a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 0x7f07e807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.704+0000 7f0802a00640 1 -- 172.21.15.31:0/1888381858 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f080409b1a0 con 0x7f080c158700 2024-09-13T08:40:48.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.704+0000 7f081384f740 1 Processor -- start 2024-09-13T08:40:48.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.704+0000 7f0809e00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 0x7f07e807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.704+0000 7f081384f740 1 -- start start 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f0809e00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 0x7f07e807aec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f07fc033000 tx=0x7f07fc004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x55606248fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55606244afa0 0x55606248e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5560623cd430 con 0x55606244aa20 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5560623bd5d0 con 0x55606244afa0 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x5560623aca00 con 0x55606248f8b0 2024-09-13T08:40:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.705+0000 7f0801400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43582/0 (socket says 172.21.15.31:43582) 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 -- 172.21.15.31:0/463037472 learned_addr learned my addr 172.21.15.31:0/463037472 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0809e00640 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x55606248fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f080a800640 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55606244afa0 0x55606248e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 -- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 msgr2=0x55606248fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x55606248fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 -- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55606244afa0 msgr2=0x55606248e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55606244afa0 0x55606248e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 -- 172.21.15.31:0/463037472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556062375520 con 0x55606244aa20 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f0801400640 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009a00 tx=0x7f07f4009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb76caf47a60b13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f07fa600640 1 -- 172.21.15.31:0/463037472 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4017070 con 0x55606244aa20 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f07fa600640 1 -- 172.21.15.31:0/463037472 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07f400eb70 con 0x55606244aa20 2024-09-13T08:40:48.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.706+0000 7f07fa600640 1 -- 172.21.15.31:0/463037472 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4013470 con 0x55606244aa20 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 -- 172.21.15.31:0/463037472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 msgr2=0x55606248e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009a00 tx=0x7f07f4009e00 comp rx=0 tx=0).stop 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 -- 172.21.15.31:0/463037472 shutdown_connections 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55606244afa0 0x55606248e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244aa20 0x55606248e390 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 --2- 172.21.15.31:0/463037472 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x55606248fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 -- 172.21.15.31:0/463037472 >> 172.21.15.31:0/463037472 conn(0x556062445ff0 msgr2=0x556062449a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 -- 172.21.15.31:0/463037472 shutdown_connections 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.707+0000 7f081384f740 1 -- 172.21.15.31:0/463037472 wait complete. 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.708+0000 7f081384f740 1 -- start start 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.708+0000 7f081384f740 1 Processor -- start 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x556062491d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606248e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556062353330 con 0x55606244afa0 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5560623b0810 con 0x556062492240 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55606211d620 con 0x55606248f8b0 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f081384f740 10 client.0 _ll_register_callbacks cb 0x556062358420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:40:48.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f0809e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f0809e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43584/0 (socket says 172.21.15.31:43584) 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f0809e00640 1 -- 172.21.15.31:0/994990077 learned_addr learned my addr 172.21.15.31:0/994990077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f0801400640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x556062491d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.709+0000 7f080a800640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606248e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 -- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 msgr2=0x556062491d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x556062491d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 -- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 msgr2=0x55606248e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606248e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 -- 172.21.15.31:0/994990077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55606210e850 con 0x55606244afa0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f0809e00640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f07fc009230 tx=0x7f07fc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3ad7979955dd12 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f07eea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:40:48.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc04d070 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f07fc03cdd0 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.710+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07f400eb70 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07fc058050 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 10 client.5895 fetch_fsmap learned FSMap version 37 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x5560621856b0 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07f400eb70 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7f07fc05b020 con 0x55606244afa0 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 10 client.5895 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 10 client.5895 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 20 client.5895 populate_metadata read hostname 'smithi031' 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x5560624722b0 0x556062472690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 4 client.5895 mds_command: new command op to 5748 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:40:48.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 --> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560623bf520 con 0x5560624722b0 2024-09-13T08:40:48.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.711+0000 7f0801400640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x5560624722b0 0x556062472690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.712+0000 7f0801400640 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x5560624722b0 0x556062472690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc58ec72b2b6a4c1 server_cookie=d86447b45db18621 in_seq=0 out_seq=0 2024-09-13T08:40:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.712+0000 7f07eea00640 10 client.5895 ms_handle_connect on v2:172.21.15.31:6833/707614758 2024-09-13T08:40:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f07eea00640 1 -- 172.21.15.31:0/994990077 <== mds.0 v2:172.21.15.31:6833/707614758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5560623bf520 con 0x5560624722b0 2024-09-13T08:40:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f07eea00640 10 client.5895 handle_command_reply: tid=0 2024-09-13T08:40:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 client.5895 shutdown 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 msgr2=0x5560624a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f07fc009230 tx=0x7f07fc009260 comp rx=0 tx=0).stop 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 shutdown_connections 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x5560624722b0 0x556062472690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606248e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x556062491d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f07eea00640 0 client.5895 ms_handle_reset on v2:172.21.15.31:6833/707614758 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 --2- 172.21.15.31:0/994990077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a2700 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 >> 172.21.15.31:0/994990077 conn(0x55606244aa20 msgr2=0x556062443660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 shutdown_connections 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 1 -- 172.21.15.31:0/994990077 wait complete. 2024-09-13T08:40:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.713+0000 7f081384f740 20 client.5895 trim_cache size 0 max 16384 2024-09-13T08:40:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.733+0000 7f081384f740 1 Processor -- start 2024-09-13T08:40:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.733+0000 7f081384f740 1 -- start start 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.733+0000 7f081384f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.733+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x5560624b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.733+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x5560624b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f081384f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1f9220 con 0x55606244afa0 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f080c0aebf0 con 0x556062492240 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f080c1f8bd0 con 0x55606248f8b0 2024-09-13T08:40:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43586/0 (socket says 172.21.15.31:43586) 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 -- 172.21.15.31:0/1910351488 learned_addr learned my addr 172.21.15.31:0/1910351488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f080a800640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x5560624b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0801400640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x5560624b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 -- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 msgr2=0x5560624b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x5560624b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 -- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 msgr2=0x5560624b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x5560624b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 -- 172.21.15.31:0/1910351488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556062470b00 con 0x55606244afa0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f080a800640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x5560624b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0801400640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x5560624b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.734+0000 7f0809e00640 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f07fc052460 tx=0x7f07fc003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3faeee2cedff9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f07f9c00640 1 -- 172.21.15.31:0/1910351488 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc04d040 con 0x55606244afa0 2024-09-13T08:40:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f07f9c00640 1 -- 172.21.15.31:0/1910351488 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07fc043030 con 0x55606244afa0 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f07f9c00640 1 -- 172.21.15.31:0/1910351488 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc035040 con 0x55606244afa0 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 -- 172.21.15.31:0/1910351488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 msgr2=0x5560624a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f07fc052460 tx=0x7f07fc003990 comp rx=0 tx=0).stop 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 -- 172.21.15.31:0/1910351488 shutdown_connections 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x5560624b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606248f8b0 0x5560624b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 --2- 172.21.15.31:0/1910351488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606244afa0 0x5560624a23a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 -- 172.21.15.31:0/1910351488 >> 172.21.15.31:0/1910351488 conn(0x55606244aa20 msgr2=0x556062491b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 -- 172.21.15.31:0/1910351488 shutdown_connections 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.735+0000 7f081384f740 1 -- 172.21.15.31:0/1910351488 wait complete. 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.736+0000 7f081384f740 1 -- start start 2024-09-13T08:40:48.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.736+0000 7f081384f740 1 Processor -- start 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 0x556062447e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606246b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556062473a00 con 0x55606248f8b0 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556062472cb0 con 0x556062492240 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x556062487180 con 0x55606244afa0 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f081384f740 10 client.0 _ll_register_callbacks cb 0x7f07fc003a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:40:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f0801400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f0801400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43588/0 (socket says 172.21.15.31:43588) 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.737+0000 7f0801400640 1 -- 172.21.15.31:0/1756586953 learned_addr learned my addr 172.21.15.31:0/1756586953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f080a800640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606246b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0809e00640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 0x556062447e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 -- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 msgr2=0x556062447e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 0x556062447e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 -- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 msgr2=0x55606246b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606246b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 -- 172.21.15.31:0/1756586953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55606248e4d0 con 0x55606248f8b0 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0809e00640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 0x556062447e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f0801400640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f07f4016000 tx=0x7f07f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e62e420636ea5c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f07eea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4017040 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f07f4013860 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.738+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07fc043030 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07f4033050 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 10 client.5901 fetch_fsmap learned FSMap version 37 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x5560625113c0 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07fc043030 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7f07f4036020 con 0x55606248f8b0 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 10 client.5901 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 10 client.5901 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 20 client.5901 populate_metadata read hostname 'smithi031' 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x556062510e80 0x556062484a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 4 client.5901 mds_command: new command op to 5748 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 --> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] -- command(tid 0: {"prefix": "status"}) -- 0x5560623bafd0 con 0x556062510e80 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.739+0000 7f0809e00640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x556062510e80 0x556062484a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f0809e00640 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x556062510e80 0x556062484a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a882244fce35e784 server_cookie=464da79b8f385429 in_seq=0 out_seq=0 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f07eea00640 10 client.5901 ms_handle_connect on v2:172.21.15.31:6833/707614758 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f07eea00640 1 -- 172.21.15.31:0/1756586953 <== mds.0 v2:172.21.15.31:6833/707614758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x5560623bafd0 con 0x556062510e80 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f07eea00640 10 client.5901 handle_command_reply: tid=0 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 client.5901 shutdown 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 msgr2=0x55606250c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f07f4016000 tx=0x7f07f40046c0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 shutdown_connections 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:6833/707614758,v1:172.21.15.31:6835/707614758] conn(0x556062510e80 0x556062484a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556062492240 0x55606246b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55606248f8b0 0x55606250c710 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 --2- 172.21.15.31:0/1756586953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55606244afa0 0x556062447e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 >> 172.21.15.31:0/1756586953 conn(0x55606244aa20 msgr2=0x5560624497d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f07eea00640 0 client.5901 ms_handle_reset on v2:172.21.15.31:6833/707614758 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.740+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 shutdown_connections 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f081384f740 1 -- 172.21.15.31:0/1756586953 wait complete. 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f081384f740 20 client.5901 trim_cache size 0 max 16384 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:40:48.747 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 5748, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 11.00582854, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 37, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 62, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 62, 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 16.274297033 2024-09-13T08:40:48.748 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 msgr2=0x7f07e807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 0x7f07e807aec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f07fc033000 tx=0x7f07fc004730 comp rx=0 tx=0).stop 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 msgr2=0x7f080c0b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f080400c7b0 tx=0x7f080400cbb0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 shutdown_connections 2024-09-13T08:40:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f080c158700 0x7f080c0b08c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f07e8078a00 0x7f07e807aec0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f080c06f8c0 0x7f080c14df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 --2- 172.21.15.31:0/1888381858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f080c06eba0 0x7f080c09d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 >> 172.21.15.31:0/1888381858 conn(0x7f080c0b18f0 msgr2=0x7f080c160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 shutdown_connections 2024-09-13T08:40:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.741+0000 7f0810a00640 1 -- 172.21.15.31:0/1888381858 wait complete. 2024-09-13T08:40:48.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 -- 172.21.15.31:0/1481671619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000b52d0 msgr2=0x7f5d0006f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 --2- 172.21.15.31:0/1481671619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000b52d0 0x7f5d0006f9b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009830 tx=0x7f5ce802a510 comp rx=0 tx=0).stop 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 -- 172.21.15.31:0/1481671619 shutdown_connections 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 --2- 172.21.15.31:0/1481671619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000b52d0 0x7f5d0006f9b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 -- 172.21.15.31:0/1481671619 >> 172.21.15.31:0/1481671619 conn(0x7f5d0006cb00 msgr2=0x7f5d0006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 -- 172.21.15.31:0/1481671619 shutdown_connections 2024-09-13T08:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.919+0000 7f5d05200640 1 -- 172.21.15.31:0/1481671619 wait complete. 2024-09-13T08:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.921+0000 7f5d05200640 1 Processor -- start 2024-09-13T08:40:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.921+0000 7f5d05200640 1 -- start start 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d000b52d0 0x7f5d00153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d000ccb00 0x7f5d000cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000ce050 con 0x7f5d000cc720 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000703b0 con 0x7f5d000ccb00 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5d05200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5d00152d90 con 0x7f5d000b52d0 2024-09-13T08:40:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cf6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cf6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43606/0 (socket says 172.21.15.31:43606) 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cf6a00640 1 -- 172.21.15.31:0/2660436600 learned_addr learned my addr 172.21.15.31:0/2660436600 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cff400640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d000b52d0 0x7f5d00153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cffe00640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d000ccb00 0x7f5d000cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cf6a00640 1 -- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d000b52d0 msgr2=0x7f5d00153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.922+0000 7f5cf6a00640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d000b52d0 0x7f5d00153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf6a00640 1 -- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d000ccb00 msgr2=0x7f5d000cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf6a00640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d000ccb00 0x7f5d000cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf6a00640 1 -- 172.21.15.31:0/2660436600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d001f81e0 con 0x7f5d000cc720 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf6a00640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8000c00 tx=0x7f5ce8032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8040070 con 0x7f5d000cc720 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ce8003af0 con 0x7f5d000cc720 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8045460 con 0x7f5d000cc720 2024-09-13T08:40:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001f84c0 con 0x7f5d000cc720 2024-09-13T08:40:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.923+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d001f8a80 con 0x7f5d000cc720 2024-09-13T08:40:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.925+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00153260 con 0x7f5d000cc720 2024-09-13T08:40:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.926+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f5ce8039040 con 0x7f5d000cc720 2024-09-13T08:40:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.926+0000 7f5cf7e00640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 0x7f5cdc07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.927+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f5ce80bc3c0 con 0x7f5d000cc720 2024-09-13T08:40:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.927+0000 7f5cff400640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 0x7f5cdc07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.928+0000 7f5cff400640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 0x7f5cdc07aef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0004ea0 tx=0x7f5cf0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:48.931+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5ce8084690 con 0x7f5d000cc720 2024-09-13T08:40:49.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.165+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5d0006cf80 con 0x7f5d000cc720 2024-09-13T08:40:49.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.165+0000 7f5cf7e00640 1 -- 172.21.15.31:0/2660436600 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f5ce8083f90 con 0x7f5d000cc720 2024-09-13T08:40:49.167 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 msgr2=0x7f5cdc07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 0x7f5cdc07aef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0004ea0 tx=0x7f5cf0009040 comp rx=0 tx=0).stop 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 msgr2=0x7f5d00154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8000c00 tx=0x7f5ce8032d50 comp rx=0 tx=0).stop 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 shutdown_connections 2024-09-13T08:40:49.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d000ccb00 0x7f5d000cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d000cc720 0x7f5d00154070 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5cdc078a30 0x7f5cdc07aef0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 --2- 172.21.15.31:0/2660436600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5d000b52d0 0x7f5d00153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 >> 172.21.15.31:0/2660436600 conn(0x7f5d0006cb00 msgr2=0x7f5d0015f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 shutdown_connections 2024-09-13T08:40:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49.168+0000 7f5d05200640 1 -- 172.21.15.31:0/2660436600 wait complete. 2024-09-13T08:40:49.185 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:40:49.185 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:40:49.185 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:40:49.185 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:40:49.185 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:40:49.186 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:40:49.186 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:40:49.186 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:40:49.186 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:40:49.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.186 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:40:49.222 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:40:49.222 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:40:49.223 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42483sec preferred_lft 42483sec 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.224 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:40:49.225 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:49.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.226 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:40:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:40:49.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.319 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:40:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:40:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:40:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:40:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:40:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:40:49.356 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:40:49.356 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:40:49.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:40:49.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:40:49.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:40:49.357 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:40:49.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:40:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:40:49.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.555 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:40:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:40:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:40:49.591 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:40:49.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:40:49.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:40:49.687 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:40:49.722 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:40:49.722 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:40:49.723 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:40:49.798 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:40:49.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:49.926 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:49.927 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:40:49.953 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:49.954 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:40:49.954 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:40:49.954 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:40:49.954 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:40:49.956 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:40:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:40:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:40:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:40:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:40:49.959 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:40:49.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:49.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:40:50.018 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:40:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:40:50.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:40:50.115 INFO:teuthology.orchestra.run.smithi031.stdout:client5327 2024-09-13T08:40:50.115 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:40:50.115 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:50.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:40:50.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 -- 172.21.15.31:0/2461529058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8003a00 msgr2=0x7f2dd8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 --2- 172.21.15.31:0/2461529058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8003a00 0x7f2dd8003e20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2de400c290 tx=0x7f2dd8005830 comp rx=0 tx=0).stop 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2df9200640 1 -- 172.21.15.31:0/2461529058 reap_dead start 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 -- 172.21.15.31:0/2461529058 shutdown_connections 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 -- 172.21.15.31:0/2461529058 >> 172.21.15.31:0/2461529058 conn(0x7f2df40cc910 msgr2=0x7f2df40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 -- 172.21.15.31:0/2461529058 shutdown_connections 2024-09-13T08:40:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.351+0000 7f2dfb200640 1 -- 172.21.15.31:0/2461529058 wait complete. 2024-09-13T08:40:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.353+0000 7f2dfb200640 1 Processor -- start 2024-09-13T08:40:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.353+0000 7f2dfb200640 1 -- start start 2024-09-13T08:40:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2dd8003a00 0x7f2df409d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2df4156520 0x7f2df40adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2df41600f0 con 0x7f2df409bf80 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2df40cb850 con 0x7f2df4156520 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2dfb200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2df40b3720 con 0x7f2dd8003a00 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2debe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2debe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43632/0 (socket says 172.21.15.31:43632) 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2debe00640 1 -- 172.21.15.31:0/3641510939 learned_addr learned my addr 172.21.15.31:0/3641510939 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2df3e00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2dd8003a00 0x7f2df409d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.354+0000 7f2df9200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2df4156520 0x7f2df40adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 -- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2dd8003a00 msgr2=0x7f2df409d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2dd8003a00 0x7f2df409d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 -- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2df4156520 msgr2=0x7f2df40adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2df4156520 0x7f2df40adba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 -- 172.21.15.31:0/3641510939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df409c6f0 con 0x7f2df409bf80 2024-09-13T08:40:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2debe00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc011ab0 tx=0x7f2ddc011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc01e070 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ddc00fbb0 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df40c3ee0 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc019410 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.355+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df40b30d0 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.356+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df40ce3c0 con 0x7f2df409bf80 2024-09-13T08:40:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.358+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f2ddc021070 con 0x7f2df409bf80 2024-09-13T08:40:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.358+0000 7f2df0c00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 0x7f2dd007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.359+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5709+0+0 (secure 0 0 0) 0x7f2ddc09b3f0 con 0x7f2df409bf80 2024-09-13T08:40:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.359+0000 7f2df3e00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 0x7f2dd007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.360+0000 7f2df3e00640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 0x7f2dd007aec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2de4033000 tx=0x7f2de4038040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:50.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.363+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2df409c6f0 con 0x7f2df409bf80 2024-09-13T08:40:50.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.595+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2df40aef60 con 0x7f2df409bf80 2024-09-13T08:40:50.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.595+0000 7f2df0c00640 1 -- 172.21.15.31:0/3641510939 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f2ddc063120 con 0x7f2df409bf80 2024-09-13T08:40:50.596 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 msgr2=0x7f2dd007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 0x7f2dd007aec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2de4033000 tx=0x7f2de4038040 comp rx=0 tx=0).stop 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 msgr2=0x7f2df40c39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc011ab0 tx=0x7f2ddc011eb0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 shutdown_connections 2024-09-13T08:40:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2df4156520 0x7f2df40adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df409bf80 0x7f2df40c39a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2dd0078a00 0x7f2dd007aec0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 --2- 172.21.15.31:0/3641510939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2dd8003a00 0x7f2df409d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 >> 172.21.15.31:0/3641510939 conn(0x7f2df40cc910 msgr2=0x7f2df415e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 shutdown_connections 2024-09-13T08:40:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:50.598+0000 7f2dfb200640 1 -- 172.21.15.31:0/3641510939 wait complete. 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:40:50.615 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:40:50.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:50.616 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:40:50.656 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.657 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:40:50.658 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42486sec preferred_lft 42486sec 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:40:50.659 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.660 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:40:50.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:50.661 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:40:50.755 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:40:50.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:50.759 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:40:50.796 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:40:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:40:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:40:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:40:50.797 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:40:50.797 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:40:50.797 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:40:50.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:40:50.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:40:50.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:40:50.798 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:40:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:40:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:40:50.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:40:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:40:50.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:50.996 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:40:51.031 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:40:51.031 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:40:51.032 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:40:51.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:51.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:40:51.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:40:51.128 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:40:51.163 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:40:51.164 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:40:51.164 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:40:51.238 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:40:51.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:51.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:51.368 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:51.368 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:40:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:40:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:40:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:40:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:40:51.396 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:40:51.398 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:40:51.399 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:40:51.399 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:40:51.399 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:40:51.399 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:40:51.402 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:40:51.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:51.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:40:51.458 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:40:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:40:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:40:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:40:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:40:51.550 INFO:teuthology.orchestra.run.smithi045.stdout:client5336 2024-09-13T08:40:51.550 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:40:51.550 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:40:51.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:40:51.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 -- 172.21.15.31:0/2950410592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc8152a30 msgr2=0x7f7bc80aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:51.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/2950410592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc8152a30 0x7f7bc80aea00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80d0600 tx=0x7f7bb802a560 comp rx=0 tx=0).stop 2024-09-13T08:40:51.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 -- 172.21.15.31:0/2950410592 shutdown_connections 2024-09-13T08:40:51.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/2950410592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8072450 0x7f7bc8073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/2950410592 >> v2:172.21.15.45:3300/0 conn(0x7f7bc80740e0 0x7f7bc80b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/2950410592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc8152a30 0x7f7bc80aea00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 -- 172.21.15.31:0/2950410592 >> 172.21.15.31:0/2950410592 conn(0x7f7bc80cf9b0 msgr2=0x7f7bc80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 -- 172.21.15.31:0/2950410592 shutdown_connections 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.759+0000 7f7bc7e00640 1 -- 172.21.15.31:0/2950410592 wait complete. 2024-09-13T08:40:51.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.761+0000 7f7bc7e00640 1 Processor -- start 2024-09-13T08:40:51.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.761+0000 7f7bc7e00640 1 -- start start 2024-09-13T08:40:51.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 0x7f7bc80b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:51.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:51.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8152a30 0x7f7bc81e7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc80b4f00 con 0x7f7bc80740e0 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc80b4d80 con 0x7f7bc8072450 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7bc80b5080 con 0x7f7bc8152a30 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bbe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bbe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43642/0 (socket says 172.21.15.31:43642) 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bbe000640 1 -- 172.21.15.31:0/901396609 learned_addr learned my addr 172.21.15.31:0/901396609 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc7400640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8152a30 0x7f7bc81e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.762+0000 7f7bc6a00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 0x7f7bc80b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 -- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8152a30 msgr2=0x7f7bc81e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8152a30 0x7f7bc81e7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:51.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 -- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 msgr2=0x7f7bc80b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 0x7f7bc80b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 -- 172.21.15.31:0/901396609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc80c34e0 con 0x7f7bc80740e0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bc6a00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 0x7f7bc80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbe000640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d980 tx=0x7f7bb000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0017070 con 0x7f7bc80740e0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bb0009d20 con 0x7f7bc80740e0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc80c37c0 con 0x7f7bc80740e0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb00106c0 con 0x7f7bc80740e0 2024-09-13T08:40:51.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.763+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc80c3d80 con 0x7f7bc80740e0 2024-09-13T08:40:51.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.765+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc80b4840 con 0x7f7bc80740e0 2024-09-13T08:40:51.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.766+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f7bb0010860 con 0x7f7bc80740e0 2024-09-13T08:40:51.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.767+0000 7f7bbf400640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 0x7f7ba407aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:51.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.767+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5767+0+0 (secure 0 0 0) 0x7f7bb009b1d0 con 0x7f7bc80740e0 2024-09-13T08:40:51.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.767+0000 7f7bc6a00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 0x7f7ba407aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:51.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.768+0000 7f7bc6a00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 0x7f7ba407aec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8033000 tx=0x7f7bb8004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:51.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:51.771+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7bb00633d0 con 0x7f7bc80740e0 2024-09-13T08:40:52.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.014+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f7bc80b1300 con 0x7f7bc80740e0 2024-09-13T08:40:52.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.820+0000 7f7bbf400640 1 -- 172.21.15.31:0/901396609 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v38) ==== 136+0+0 (secure 0 0 0) 0x7f7bb0062cd0 con 0x7f7bc80740e0 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 msgr2=0x7f7ba407aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 0x7f7ba407aec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8033000 tx=0x7f7bb8004730 comp rx=0 tx=0).stop 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 msgr2=0x7f7bc81e6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d980 tx=0x7f7bb000de40 comp rx=0 tx=0).stop 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 shutdown_connections 2024-09-13T08:40:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7ba4078a00 0x7f7ba407aec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bc8152a30 0x7f7bc81e7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bc80740e0 0x7f7bc81e6b30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 --2- 172.21.15.31:0/901396609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bc8072450 0x7f7bc80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 >> 172.21.15.31:0/901396609 conn(0x7f7bc80cf9b0 msgr2=0x7f7bc80cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 shutdown_connections 2024-09-13T08:40:52.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:52.824+0000 7f7bc7e00640 1 -- 172.21.15.31:0/901396609 wait complete. 2024-09-13T08:40:52.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.008+0000 7f464cc00640 1 -- 172.21.15.31:0/3833051013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480d0cd0 msgr2=0x7f46480d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.008+0000 7f464cc00640 1 --2- 172.21.15.31:0/3833051013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480d0cd0 0x7f46480d1110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4634009860 tx=0x7f463402a540 comp rx=0 tx=0).stop 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 -- 172.21.15.31:0/3833051013 shutdown_connections 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 --2- 172.21.15.31:0/3833051013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480d0cd0 0x7f46480d1110 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 --2- 172.21.15.31:0/3833051013 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 0x7f46480d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 -- 172.21.15.31:0/3833051013 >> 172.21.15.31:0/3833051013 conn(0x7f46480d3f50 msgr2=0x7f46480d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 -- 172.21.15.31:0/3833051013 shutdown_connections 2024-09-13T08:40:53.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.009+0000 7f464cc00640 1 -- 172.21.15.31:0/3833051013 wait complete. 2024-09-13T08:40:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.010+0000 7f464cc00640 1 Processor -- start 2024-09-13T08:40:53.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.010+0000 7f464cc00640 1 -- start start 2024-09-13T08:40:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46480d0cd0 0x7f46480b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 0x7f46480b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46480cfe00 con 0x7f46480c8340 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46480cfc80 con 0x7f46480d0cd0 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f464cc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f46480c8f00 con 0x7f464815e160 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.011+0000 7f4647400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43662/0 (socket says 172.21.15.31:43662) 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 -- 172.21.15.31:0/2864278611 learned_addr learned my addr 172.21.15.31:0/2864278611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4646a00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46480d0cd0 0x7f46480b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647e00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 0x7f46480b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 -- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 msgr2=0x7f46480b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 0x7f46480b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 -- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46480d0cd0 msgr2=0x7f46480b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46480d0cd0 0x7f46480b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 -- 172.21.15.31:0/2864278611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4648153270 con 0x7f46480c8340 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.012+0000 7f4647400640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f463800d740 tx=0x7f463800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.013+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638017070 con 0x7f46480c8340 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.013+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4638009d20 con 0x7f46480c8340 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.013+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4648153550 con 0x7f46480c8340 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.013+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46380106c0 con 0x7f46480c8340 2024-09-13T08:40:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.013+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4648153b10 con 0x7f46480c8340 2024-09-13T08:40:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.014+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46480c93b0 con 0x7f46480c8340 2024-09-13T08:40:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.015+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f463801a070 con 0x7f46480c8340 2024-09-13T08:40:53.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.016+0000 7f463f400640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 0x7f462807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.016+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f463809af70 con 0x7f46480c8340 2024-09-13T08:40:53.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.016+0000 7f4646a00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 0x7f462807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.017+0000 7f4646a00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 0x7f462807aec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4634034000 tx=0x7f4634004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:53.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.020+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4638062aa0 con 0x7f46480c8340 2024-09-13T08:40:53.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.239+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f46480d1ff0 con 0x7f4628078a00 2024-09-13T08:40:53.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.242+0000 7f463f400640 1 -- 172.21.15.31:0/2864278611 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f46480d1ff0 con 0x7f4628078a00 2024-09-13T08:40:53.243 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:40:53.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:40:53.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:40:53.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:40:53.244 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.243+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 msgr2=0x7f462807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.243+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 0x7f462807aec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4634034000 tx=0x7f4634004730 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.243+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 msgr2=0x7f46480b8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.243+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f463800d740 tx=0x7f463800dc00 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 shutdown_connections 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4628078a00 0x7f462807aec0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f464815e160 0x7f46480b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46480d0cd0 0x7f46480b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 --2- 172.21.15.31:0/2864278611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46480c8340 0x7f46480b8800 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 >> 172.21.15.31:0/2864278611 conn(0x7f46480d3f50 msgr2=0x7f46480b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 shutdown_connections 2024-09-13T08:40:53.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.244+0000 7f464cc00640 1 -- 172.21.15.31:0/2864278611 wait complete. 2024-09-13T08:40:53.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 -- 172.21.15.31:0/1929426773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 msgr2=0x7f62a40ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 --2- 172.21.15.31:0/1929426773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40ca020 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6294005fd0 tx=0x7f629402a530 comp rx=0 tx=0).stop 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 -- 172.21.15.31:0/1929426773 shutdown_connections 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 --2- 172.21.15.31:0/1929426773 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 0x7f62a40ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 --2- 172.21.15.31:0/1929426773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40ca020 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 -- 172.21.15.31:0/1929426773 >> 172.21.15.31:0/1929426773 conn(0x7f62a40b1520 msgr2=0x7f62a40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 -- 172.21.15.31:0/1929426773 shutdown_connections 2024-09-13T08:40:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.424+0000 7f62aa400640 1 -- 172.21.15.31:0/1929426773 wait complete. 2024-09-13T08:40:53.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.426+0000 7f62aa400640 1 Processor -- start 2024-09-13T08:40:53.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.426+0000 7f62aa400640 1 -- start start 2024-09-13T08:40:53.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a406f530 0x7f62a4158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 0x7f62a4155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62a415e560 con 0x7f62a415abf0 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62a40ca5f0 con 0x7f62a406f530 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62aa400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f62a415e6e0 con 0x7f62a406f970 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62a9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62a9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43680/0 (socket says 172.21.15.31:43680) 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62a9a00640 1 -- 172.21.15.31:0/279835174 learned_addr learned my addr 172.21.15.31:0/279835174 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f62a9000640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a406f530 0x7f62a4158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.427+0000 7f629be00640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 0x7f62a4155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 -- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 msgr2=0x7f62a4155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 0x7f62a4155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 -- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a406f530 msgr2=0x7f62a4158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a406f530 0x7f62a4158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 -- 172.21.15.31:0/279835174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a41f5f30 con 0x7f62a415abf0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a9a00640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b970 tx=0x7f629c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c019070 con 0x7f62a415abf0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f629c00cc50 con 0x7f62a415abf0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a41f6210 con 0x7f62a415abf0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c0146a0 con 0x7f62a415abf0 2024-09-13T08:40:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.428+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a41585c0 con 0x7f62a415abf0 2024-09-13T08:40:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.430+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a41f6600 con 0x7f62a415abf0 2024-09-13T08:40:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.431+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f629c01c070 con 0x7f62a415abf0 2024-09-13T08:40:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.431+0000 7f62a1600640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 0x7f628007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.432+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f629c09a9d0 con 0x7f62a415abf0 2024-09-13T08:40:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.432+0000 7f62a9000640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 0x7f628007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.433+0000 7f62a9000640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 0x7f628007aec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6294033000 tx=0x7f6294004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:53.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.435+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f629c062630 con 0x7f62a415abf0 2024-09-13T08:40:53.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.630+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f62a41f63e0 con 0x7f62a415abf0 2024-09-13T08:40:53.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.631+0000 7f62a1600640 1 -- 172.21.15.31:0/279835174 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f629c067060 con 0x7f62a415abf0 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 msgr2=0x7f628007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 0x7f628007aec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6294033000 tx=0x7f6294004730 comp rx=0 tx=0).stop 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 msgr2=0x7f62a40afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b970 tx=0x7f629c00be30 comp rx=0 tx=0).stop 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 shutdown_connections 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62a415abf0 0x7f62a40afa40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6280078a00 0x7f628007aec0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62a406f970 0x7f62a4155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 --2- 172.21.15.31:0/279835174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62a406f530 0x7f62a4158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:53.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 >> 172.21.15.31:0/279835174 conn(0x7f62a40b1520 msgr2=0x7f62a40cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:53.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 shutdown_connections 2024-09-13T08:40:53.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:53.633+0000 7f62aa400640 1 -- 172.21.15.31:0/279835174 wait complete. 2024-09-13T08:40:53.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:53.651 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol42010) 2024-09-13T08:40:53.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:53.714 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:40:53.751 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:40:53.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:53.752 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/_nogroup/subvol42010) 2024-09-13T08:40:53.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol93641 2024-09-13T08:40:54.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.017+0000 7f3d42200640 1 -- 172.21.15.31:0/298713961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cc190 msgr2=0x7f3d3c0cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:54.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.017+0000 7f3d42200640 1 --2- 172.21.15.31:0/298713961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cc190 0x7f3d3c0cdcc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3d24005fd0 tx=0x7f3d2402a540 comp rx=0 tx=0).stop 2024-09-13T08:40:54.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 -- 172.21.15.31:0/298713961 shutdown_connections 2024-09-13T08:40:54.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 --2- 172.21.15.31:0/298713961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cc190 0x7f3d3c0cdcc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 --2- 172.21.15.31:0/298713961 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c161480 0x7f3d3c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 -- 172.21.15.31:0/298713961 >> 172.21.15.31:0/298713961 conn(0x7f3d3c0cfbb0 msgr2=0x7f3d3c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:54.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 -- 172.21.15.31:0/298713961 shutdown_connections 2024-09-13T08:40:54.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.018+0000 7f3d42200640 1 -- 172.21.15.31:0/298713961 wait complete. 2024-09-13T08:40:54.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.019+0000 7f3d42200640 1 Processor -- start 2024-09-13T08:40:54.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.019+0000 7f3d42200640 1 -- start start 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c0cc190 0x7f3d3c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 0x7f3d3c0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c06ffd0 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0cb590 con 0x7f3d3c161480 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.020+0000 7f3d42200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c1622c0 con 0x7f3d3c0cc190 2024-09-13T08:40:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43702/0 (socket says 172.21.15.31:43702) 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 -- 172.21.15.31:0/4225244900 learned_addr learned my addr 172.21.15.31:0/4225244900 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d32a00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c0cc190 0x7f3d3c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3be00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 0x7f3d3c0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 -- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c0cc190 msgr2=0x7f3d3c0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c0cc190 0x7f3d3c0b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 -- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 msgr2=0x7f3d3c0c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 0x7f3d3c0c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 -- 172.21.15.31:0/4225244900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0b3da0 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3be00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 0x7f3d3c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.021+0000 7f3d3b400640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00ea20 tx=0x7f3d2c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:54.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.022+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00cd80 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.022+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d2c011560 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.022+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c0b2aa0 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.022+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c011990 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.022+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c0b3060 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.023+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c162770 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.024+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 101440+0+0 (secure 0 0 0) 0x7f3d2c011b30 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.025+0000 7f3d33e00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 0x7f3d1807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.025+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f3d2c09b940 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.025+0000 7f3d32a00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 0x7f3d1807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:54.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.026+0000 7f3d32a00640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 0x7f3d1807aec0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d24033000 tx=0x7f3d24004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:54.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.029+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3d2c016030 con 0x7f3d3c0cbc40 2024-09-13T08:40:54.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.250+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol93641", "target": ["mon-mgr", ""]}) -- 0x7f3d3c15eed0 con 0x7f3d18078a00 2024-09-13T08:40:54.259 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:40:54.258+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:40:54.259 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:40:54.258+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:40:54.259 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:40:54.258+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:40:54.259 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:40:54.258+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:40:54.259 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:40:54.258+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:40:54.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.300+0000 7f3d33e00640 1 -- 172.21.15.31:0/4225244900 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3d3c15eed0 con 0x7f3d18078a00 2024-09-13T08:40:54.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 msgr2=0x7f3d1807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:54.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 0x7f3d1807aec0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d24033000 tx=0x7f3d24004730 comp rx=0 tx=0).stop 2024-09-13T08:40:54.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 msgr2=0x7f3d3c0b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00ea20 tx=0x7f3d2c00eee0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 shutdown_connections 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d3c161480 0x7f3d3c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3d18078a00 0x7f3d1807aec0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3d3c0cc190 0x7f3d3c0b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 --2- 172.21.15.31:0/4225244900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d3c0cbc40 0x7f3d3c0b3860 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.302+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 >> 172.21.15.31:0/4225244900 conn(0x7f3d3c0cfbb0 msgr2=0x7f3d3c15fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.303+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 shutdown_connections 2024-09-13T08:40:54.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:54.303+0000 7f3d42200640 1 -- 172.21.15.31:0/4225244900 wait complete. 2024-09-13T08:40:54.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:54.320 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cp ./volumes/_nogroup/subvol93641/.meta ./volumes/_nogroup/subvol42010) 2024-09-13T08:40:57.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol42010 2024-09-13T08:40:57.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 -- 172.21.15.31:0/298678929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc928003a50 msgr2=0x7fc928003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 --2- 172.21.15.31:0/298678929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc928003a50 0x7fc928003e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc93400bbb0 tx=0x7fc928005830 comp rx=0 tx=0).stop 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949200640 1 -- 172.21.15.31:0/298678929 reap_dead start 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 -- 172.21.15.31:0/298678929 shutdown_connections 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 -- 172.21.15.31:0/298678929 >> 172.21.15.31:0/298678929 conn(0x7fc9440b1520 msgr2=0x7fc9440b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 -- 172.21.15.31:0/298678929 shutdown_connections 2024-09-13T08:40:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.659+0000 7fc949c00640 1 -- 172.21.15.31:0/298678929 wait complete. 2024-09-13T08:40:57.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.661+0000 7fc949c00640 1 Processor -- start 2024-09-13T08:40:57.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.661+0000 7fc949c00640 1 -- start start 2024-09-13T08:40:57.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc928003a50 0x7fc944158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:57.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9440c6130 0x7fc9440c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc94406fc10 con 0x7fc94415e6a0 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440cccd0 con 0x7fc9440c6130 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc94406fd90 con 0x7fc928003a50 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57860/0 (socket says 172.21.15.31:57860) 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc949200640 1 -- 172.21.15.31:0/3044732619 learned_addr learned my addr 172.21.15.31:0/3044732619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc943e00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc928003a50 0x7fc944158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.662+0000 7fc943400640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9440c6130 0x7fc9440c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 -- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc928003a50 msgr2=0x7fc944158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc928003a50 0x7fc944158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 -- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9440c6130 msgr2=0x7fc9440c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9440c6130 0x7fc9440c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 -- 172.21.15.31:0/3044732619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9440c3750 con 0x7fc94415e6a0 2024-09-13T08:40:57.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949200640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc944070910 tx=0x7fc93c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c015070 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc93c00ccb0 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00cea0 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9440c3a30 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.663+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9440c3ff0 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.664+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc944073ac0 con 0x7fc94415e6a0 2024-09-13T08:40:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.666+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7fc93c018070 con 0x7fc94415e6a0 2024-09-13T08:40:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.667+0000 7fc93be00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 0x7fc92407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.667+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7fc93c09b220 con 0x7fc94415e6a0 2024-09-13T08:40:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.667+0000 7fc943e00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 0x7fc92407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.668+0000 7fc943e00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 0x7fc92407af50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc934036000 tx=0x7fc9340046c0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.671+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc93c001a30 con 0x7fc94415e6a0 2024-09-13T08:40:57.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.899+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol42010", "target": ["mon-mgr", ""]}) -- 0x7fc9440c4510 con 0x7fc924078a90 2024-09-13T08:40:57.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.938+0000 7fc93be00640 1 -- 172.21.15.31:0/3044732619 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7fc9340315c0 con 0x7fc924078a90 2024-09-13T08:40:57.940 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol42010 2024-09-13T08:40:57.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 msgr2=0x7fc92407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:57.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 0x7fc92407af50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc934036000 tx=0x7fc9340046c0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 msgr2=0x7fc94415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc944070910 tx=0x7fc93c00ed10 comp rx=0 tx=0).stop 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 shutdown_connections 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc94415e6a0 0x7fc94415eb50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9440c6130 0x7fc9440c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc924078a90 0x7fc92407af50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 --2- 172.21.15.31:0/3044732619 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc928003a50 0x7fc944158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 >> 172.21.15.31:0/3044732619 conn(0x7fc9440b1520 msgr2=0x7fc9440b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 shutdown_connections 2024-09-13T08:40:57.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:57.941+0000 7fc949c00640 1 -- 172.21.15.31:0/3044732619 wait complete. 2024-09-13T08:40:57.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol42010 2024-09-13T08:40:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 -- 172.21.15.31:0/3629263333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fec0 msgr2=0x7f04300b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 --2- 172.21.15.31:0/3629263333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fec0 0x7f04300b29e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0428005fd0 tx=0x7f042802aaa0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 -- 172.21.15.31:0/3629263333 shutdown_connections 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 --2- 172.21.15.31:0/3629263333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fec0 0x7f04300b29e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 --2- 172.21.15.31:0/3629263333 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f043015bfc0 0x7f0430072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 -- 172.21.15.31:0/3629263333 >> 172.21.15.31:0/3629263333 conn(0x7f04300cfbb0 msgr2=0x7f04300cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 -- 172.21.15.31:0/3629263333 shutdown_connections 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.121+0000 7f0434e00640 1 -- 172.21.15.31:0/3629263333 wait complete. 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.123+0000 7f0434e00640 1 Processor -- start 2024-09-13T08:40:58.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.123+0000 7f0434e00640 1 -- start start 2024-09-13T08:40:58.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:58.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 0x7f04301f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04300b2030 0x7f04301f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04300cbfd0 con 0x7f043006fa60 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04300cbe50 con 0x7f043015bfc0 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f0434e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f04300b2f20 con 0x7f04300b2030 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f042de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f042de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57880/0 (socket says 172.21.15.31:57880) 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f042de00640 1 -- 172.21.15.31:0/3710863211 learned_addr learned my addr 172.21.15.31:0/3710863211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f042e800640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04300b2030 0x7f04301f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:58.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.124+0000 7f042d400640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 0x7f04301f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 -- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04300b2030 msgr2=0x7f04301f4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04300b2030 0x7f04301f4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 -- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 msgr2=0x7f04301f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 0x7f04301f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 -- 172.21.15.31:0/3710863211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04301f4670 con 0x7f043006fa60 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042d400640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 0x7f04301f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:40:58.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f042de00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f042000c970 tx=0x7f042000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420007bb0 con 0x7f043006fa60 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0420007e40 con 0x7f043006fa60 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042000f690 con 0x7f043006fa60 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04301f83b0 con 0x7f043006fa60 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.125+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04301f8970 con 0x7f043006fa60 2024-09-13T08:40:58.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.126+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04300b33d0 con 0x7f043006fa60 2024-09-13T08:40:58.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.127+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7f042000f830 con 0x7f043006fa60 2024-09-13T08:40:58.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.128+0000 7f0426000640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 0x7f041007af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:40:58.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.128+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f042009b620 con 0x7f043006fa60 2024-09-13T08:40:58.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.129+0000 7f042d400640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 0x7f041007af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:40:58.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.129+0000 7f042d400640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 0x7f041007af50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0430156f50 tx=0x7f041c009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:40:58.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.133+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f04301f4670 con 0x7f043006fa60 2024-09-13T08:40:58.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.356+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol42010", "target": ["mon-mgr", ""]}) -- 0x7f04301f8b40 con 0x7f0410078a90 2024-09-13T08:40:58.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.362+0000 7f0426000640 1 -- 172.21.15.31:0/3710863211 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f041c001410 con 0x7f0410078a90 2024-09-13T08:40:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol42010 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 msgr2=0x7f041007af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 0x7f041007af50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0430156f50 tx=0x7f041c009040 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 msgr2=0x7f0430154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f042000c970 tx=0x7f042000ce30 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 shutdown_connections 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0410078a90 0x7f041007af50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f04300b2030 0x7f04301f4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f043015bfc0 0x7f04301f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 --2- 172.21.15.31:0/3710863211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f043006fa60 0x7f0430154830 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 >> 172.21.15.31:0/3710863211 conn(0x7f04300cfbb0 msgr2=0x7f04300cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:40:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 shutdown_connections 2024-09-13T08:40:58.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:40:58.364+0000 7f0434e00640 1 -- 172.21.15.31:0/3710863211 wait complete. 2024-09-13T08:40:58.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:40:58.383 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/19065d408c2bfe2d238d947e95a4fcf7.meta) 2024-09-13T08:41:02.478 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:41:02.478 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:41:02.478 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:41:02.479 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol42010 2024-09-13T08:41:02.479 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:41:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:02.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume authorize cephfs subvol42010 alice 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.655+0000 7f7c84e00640 1 -- 172.21.15.31:0/3684038495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 msgr2=0x7f7c6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.655+0000 7f7c84e00640 1 --2- 172.21.15.31:0/3684038495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c6400b9c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7c78006fd0 tx=0x7f7c64005830 comp rx=0 tx=0).stop 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.655+0000 7f7c84e00640 1 -- 172.21.15.31:0/3684038495 shutdown_connections 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.655+0000 7f7c84e00640 1 --2- 172.21.15.31:0/3684038495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c6400b9c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.655+0000 7f7c84e00640 1 -- 172.21.15.31:0/3684038495 >> 172.21.15.31:0/3684038495 conn(0x7f7c800cfbb0 msgr2=0x7f7c800cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.656+0000 7f7c84e00640 1 -- 172.21.15.31:0/3684038495 shutdown_connections 2024-09-13T08:41:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.656+0000 7f7c84e00640 1 -- 172.21.15.31:0/3684038495 wait complete. 2024-09-13T08:41:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.657+0000 7f7c84e00640 1 Processor -- start 2024-09-13T08:41:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.657+0000 7f7c84e00640 1 -- start start 2024-09-13T08:41:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c80155150 0x7f7c800c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c80153c60 0x7f7c80154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c800cbdc0 con 0x7f7c640052e0 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c800cbc40 con 0x7f7c80153c60 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c84e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7c800cbf40 con 0x7f7c80155150 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c7de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c7de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57910/0 (socket says 172.21.15.31:57910) 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c7de00640 1 -- 172.21.15.31:0/4245488830 learned_addr learned my addr 172.21.15.31:0/4245488830 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.658+0000 7f7c7e800640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c80153c60 0x7f7c80154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c75400640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c80155150 0x7f7c800c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 -- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c80155150 msgr2=0x7f7c800c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c80155150 0x7f7c800c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 -- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c80153c60 msgr2=0x7f7c80154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c80153c60 0x7f7c80154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 -- 172.21.15.31:0/4245488830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c800c81c0 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c7de00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7c70011ae0 tx=0x7f7c7000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7001e070 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c7000fda0 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c800c9a20 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70019670 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.659+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c800c8720 con 0x7f7c640052e0 2024-09-13T08:41:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.660+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c800cd8f0 con 0x7f7c640052e0 2024-09-13T08:41:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.662+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7f7c70021070 con 0x7f7c640052e0 2024-09-13T08:41:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.663+0000 7f7c76a00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 0x7f7c5c07af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.663+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f7c7009b8b0 con 0x7f7c640052e0 2024-09-13T08:41:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.664+0000 7f7c75400640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 0x7f7c5c07af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.664+0000 7f7c75400640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 0x7f7c5c07af80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7c800b5ad0 tx=0x7f7c68006b70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.668+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7c800c81c0 con 0x7f7c640052e0 2024-09-13T08:41:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.890+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvol42010", "auth_id": "alice", "target": ["mon-mgr", ""]}) -- 0x7f7c800ae530 con 0x7f7c5c078ac0 2024-09-13T08:41:02.904 INFO:tasks.ceph.mgr.x.smithi045.stderr:Creating meta for ID alice with tenant None 2024-09-13T08:41:02.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.947+0000 7f7c76a00640 1 -- 172.21.15.31:0/4245488830 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7c800ae530 con 0x7f7c5c078ac0 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 msgr2=0x7f7c5c07af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 0x7f7c5c07af80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7c800b5ad0 tx=0x7f7c68006b70 comp rx=0 tx=0).stop 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 msgr2=0x7f7c800c65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7c70011ae0 tx=0x7f7c7000f3c0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 shutdown_connections 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c80153c60 0x7f7c80154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7c5c078ac0 0x7f7c5c07af80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c80155150 0x7f7c800c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 --2- 172.21.15.31:0/4245488830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c640052e0 0x7f7c800c65c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 >> 172.21.15.31:0/4245488830 conn(0x7f7c800cfbb0 msgr2=0x7f7c8015c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 shutdown_connections 2024-09-13T08:41:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:02.949+0000 7f7c84e00640 1 -- 172.21.15.31:0/4245488830 wait complete. 2024-09-13T08:41:02.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.alice --format=json-pretty 2024-09-13T08:41:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 -- 172.21.15.31:0/4285726059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde70005210 msgr2=0x7fde7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 --2- 172.21.15.31:0/4285726059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde70005210 0x7fde7000b9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fde84001d90 tx=0x7fde70005830 comp rx=0 tx=0).stop 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 -- 172.21.15.31:0/4285726059 shutdown_connections 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 --2- 172.21.15.31:0/4285726059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde70005210 0x7fde7000b9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 -- 172.21.15.31:0/4285726059 >> 172.21.15.31:0/4285726059 conn(0x7fde8c0b18f0 msgr2=0x7fde8c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 -- 172.21.15.31:0/4285726059 shutdown_connections 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.128+0000 7fde92200640 1 -- 172.21.15.31:0/4285726059 wait complete. 2024-09-13T08:41:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.130+0000 7fde92200640 1 Processor -- start 2024-09-13T08:41:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.130+0000 7fde92200640 1 -- start start 2024-09-13T08:41:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde70005210 0x7fde8c14bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde8c155570 0x7fde8c1f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c073ec0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c073400 con 0x7fde70005210 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde92200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c074040 con 0x7fde8c155570 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57918/0 (socket says 172.21.15.31:57918) 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde8be00640 1 -- 172.21.15.31:0/3351491464 learned_addr learned my addr 172.21.15.31:0/3351491464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde8b400640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde70005210 0x7fde8c14bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.131+0000 7fde82a00640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde8c155570 0x7fde8c1f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 -- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde8c155570 msgr2=0x7fde8c1f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde8c155570 0x7fde8c1f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 -- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde70005210 msgr2=0x7fde8c14bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde70005210 0x7fde8c14bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 -- 172.21.15.31:0/3351491464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c09ab20 con 0x7fde8c1f8f30 2024-09-13T08:41:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde8be00640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fde84001d90 tx=0x7fde84006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84003d80 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde84005ba0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84005ff0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c09ae00 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.132+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1f81f0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.133+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c0745a0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.135+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7fde84005030 con 0x7fde8c1f8f30 2024-09-13T08:41:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.136+0000 7fde83e00640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 0x7fde6807af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.136+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7fde840ba480 con 0x7fde8c1f8f30 2024-09-13T08:41:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.136+0000 7fde8b400640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 0x7fde6807af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.137+0000 7fde8b400640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 0x7fde6807af50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fde7c006bb0 tx=0x7fde7c006720 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.140+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fde840820e0 con 0x7fde8c1f8f30 2024-09-13T08:41:03.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.406+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"} v 0) -- 0x7fde8c156970 con 0x7fde8c1f8f30 2024-09-13T08:41:03.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.407+0000 7fde83e00640 1 -- 172.21.15.31:0/3351491464 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"}]=0 v12) ==== 107+0+280 (secure 0 0 0) 0x7fde84086b10 con 0x7fde8c1f8f30 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.alice", 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCe+uNmre3GNhAAhD2EPuS8cMbQjF+Ssa06fg==", 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol42010", 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:41:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 msgr2=0x7fde6807af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 0x7fde6807af50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fde7c006bb0 tx=0x7fde7c006720 comp rx=0 tx=0).stop 2024-09-13T08:41:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 msgr2=0x7fde8c09d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fde84001d90 tx=0x7fde84006d00 comp rx=0 tx=0).stop 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 shutdown_connections 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8c1f8f30 0x7fde8c09d5c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fde68078a90 0x7fde6807af50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde8c155570 0x7fde8c1f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 --2- 172.21.15.31:0/3351491464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde70005210 0x7fde8c14bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 >> 172.21.15.31:0/3351491464 conn(0x7fde8c0b18f0 msgr2=0x7fde8c0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 shutdown_connections 2024-09-13T08:41:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.409+0000 7fde92200640 1 -- 172.21.15.31:0/3351491464 wait complete. 2024-09-13T08:41:03.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol42010 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 -- 172.21.15.31:0/1941245888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c1613f0 msgr2=0x7fe92c0d49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 --2- 172.21.15.31:0/1941245888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c1613f0 0x7fe92c0d49c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe920005fd0 tx=0x7fe92002a720 comp rx=0 tx=0).stop 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 -- 172.21.15.31:0/1941245888 shutdown_connections 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 --2- 172.21.15.31:0/1941245888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c1613f0 0x7fe92c0d49c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 -- 172.21.15.31:0/1941245888 >> 172.21.15.31:0/1941245888 conn(0x7fe92c0d3df0 msgr2=0x7fe92c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 -- 172.21.15.31:0/1941245888 shutdown_connections 2024-09-13T08:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.590+0000 7fe932600640 1 -- 172.21.15.31:0/1941245888 wait complete. 2024-09-13T08:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.591+0000 7fe932600640 1 Processor -- start 2024-09-13T08:41:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.592+0000 7fe932600640 1 -- start start 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.592+0000 7fe932600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.592+0000 7fe932600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 0x7fe92c0a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe932600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 0x7fe92c0cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe932600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c0b7ca0 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe932600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c0b7b20 con 0x7fe92c0cf3d0 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe932600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe92c0b7e20 con 0x7fe92c1613f0 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57932/0 (socket says 172.21.15.31:57932) 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 -- 172.21.15.31:0/179419849 learned_addr learned my addr 172.21.15.31:0/179419849 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe92be00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 0x7fe92c0a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 -- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 msgr2=0x7fe92c0cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931c00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 0x7fe92c0cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 0x7fe92c0cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 -- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 msgr2=0x7fe92c0a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 0x7fe92c0a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 -- 172.21.15.31:0/179419849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c0b1b20 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe92be00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 0x7fe92c0a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931c00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 0x7fe92c0cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.593+0000 7fe931200640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe920034040 tx=0x7fe9200039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.594+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92003e030 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.594+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9200047a0 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.594+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92c1f5880 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.594+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe920035040 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.594+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92c0b3270 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.595+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe92c0b8340 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.596+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7fe920041030 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.597+0000 7fe928c00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 0x7fe90c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:03.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.597+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7fe9200bb510 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.597+0000 7fe92be00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 0x7fe90c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.598+0000 7fe92be00640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 0x7fe90c07af50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe92c1f92f0 tx=0x7fe918009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.601+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe920083790 con 0x7fe92c0ce8c0 2024-09-13T08:41:03.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.827+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol42010", "target": ["mon-mgr", ""]}) -- 0x7fe92c0b38a0 con 0x7fe90c078a90 2024-09-13T08:41:03.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.855+0000 7fe928c00640 1 -- 172.21.15.31:0/179419849 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe92c0b38a0 con 0x7fe90c078a90 2024-09-13T08:41:03.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 msgr2=0x7fe90c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 0x7fe90c07af50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe92c1f92f0 tx=0x7fe918009040 comp rx=0 tx=0).stop 2024-09-13T08:41:03.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 msgr2=0x7fe92c0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:03.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe920034040 tx=0x7fe9200039e0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 shutdown_connections 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe90c078a90 0x7fe90c07af50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe92c1613f0 0x7fe92c0cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c0cf3d0 0x7fe92c0a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 --2- 172.21.15.31:0/179419849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe92c0ce8c0 0x7fe92c0b15e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 >> 172.21.15.31:0/179419849 conn(0x7fe92c0d3df0 msgr2=0x7fe92c0b8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 shutdown_connections 2024-09-13T08:41:03.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:03.858+0000 7fe932600640 1 -- 172.21.15.31:0/179419849 wait complete. 2024-09-13T08:41:03.863 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.862+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.863 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.862+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.863 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.862+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.863 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.862+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.864 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.862+0000 7fe269a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol93641 2024-09-13T08:41:03.884 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.883+0000 7fe26ae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.884 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.883+0000 7fe26ae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.884 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.883+0000 7fe26ae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.884 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.883+0000 7fe26ae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:03.884 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:03.883+0000 7fe26ae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:04.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 -- 172.21.15.31:0/2985071212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef8005210 msgr2=0x7f1ef800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:04.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/2985071212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef8005210 0x7f1ef800b9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1f140b5a90 tx=0x7f1ef8005830 comp rx=0 tx=0).stop 2024-09-13T08:41:04.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 -- 172.21.15.31:0/2985071212 shutdown_connections 2024-09-13T08:41:04.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/2985071212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef8005210 0x7f1ef800b9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:04.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 -- 172.21.15.31:0/2985071212 >> 172.21.15.31:0/2985071212 conn(0x7f1f140cfbb0 msgr2=0x7f1f140cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:04.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 -- 172.21.15.31:0/2985071212 shutdown_connections 2024-09-13T08:41:04.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.043+0000 7f1f1aa00640 1 -- 172.21.15.31:0/2985071212 wait complete. 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.045+0000 7f1f1aa00640 1 Processor -- start 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.045+0000 7f1f1aa00640 1 -- start start 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8005210 0x7f1f14152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f140c36e0 0x7f1f140c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140726e0 con 0x7f1f140c3b60 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f14072560 con 0x7f1ef8005210 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f1aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1f14072860 con 0x7f1f140c36e0 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f18a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f18a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57960/0 (socket says 172.21.15.31:57960) 2024-09-13T08:41:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f18a00640 1 -- 172.21.15.31:0/3927088712 learned_addr learned my addr 172.21.15.31:0/3927088712 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f13e00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8005210 0x7f1f14152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.046+0000 7f1f0be00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f140c36e0 0x7f1f140c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 -- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f140c36e0 msgr2=0x7f1f140c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f140c36e0 0x7f1f140c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 -- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8005210 msgr2=0x7f1f14152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8005210 0x7f1f14152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 -- 172.21.15.31:0/3927088712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140c40c0 con 0x7f1f140c3b60 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f18a00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1f14072510 tx=0x7f1f0c006d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c003d80 con 0x7f1f140c3b60 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f0c032470 con 0x7f1f140c3b60 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0328c0 con 0x7f1f140c3b60 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f140c43a0 con 0x7f1f140c3b60 2024-09-13T08:41:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.047+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f14152640 con 0x7f1f140c3b60 2024-09-13T08:41:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.049+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f140cba30 con 0x7f1f140c3b60 2024-09-13T08:41:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.050+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 10) ==== 101491+0+0 (secure 0 0 0) 0x7f1f0c032a60 con 0x7f1f140c3b60 2024-09-13T08:41:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.051+0000 7f1f10c00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 0x7f1ef007af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.051+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f1f0c0ba690 con 0x7f1f140c3b60 2024-09-13T08:41:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.051+0000 7f1f13e00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 0x7f1ef007af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.052+0000 7f1f13e00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 0x7f1ef007af80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009bd0 tx=0x7f1f04005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.055+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1f0c044030 con 0x7f1f140c3b60 2024-09-13T08:41:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:04.280+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol93641", "target": ["mon-mgr", ""]}) -- 0x7f1f14159770 con 0x7f1ef0078ac0 2024-09-13T08:41:05.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:05.938+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f1f0c07e540 con 0x7f1f140c3b60 2024-09-13T08:41:06.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:41:06.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:41:12.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.487+0000 7f1f10c00640 1 -- 172.21.15.31:0/3927088712 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1f14159770 con 0x7f1ef0078ac0 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 msgr2=0x7f1ef007af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 0x7f1ef007af80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009bd0 tx=0x7f1f04005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 msgr2=0x7f1f140c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1f14072510 tx=0x7f1f0c006d40 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 shutdown_connections 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f140c3b60 0x7f1f140c56a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f1ef0078ac0 0x7f1ef007af80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f140c36e0 0x7f1f140c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/3927088712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8005210 0x7f1f14152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:12.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.490+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 >> 172.21.15.31:0/3927088712 conn(0x7f1f140cfbb0 msgr2=0x7f1f1415ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.491+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 shutdown_connections 2024-09-13T08:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12.491+0000 7f1f1aa00640 1 -- 172.21.15.31:0/3927088712 wait complete. 2024-09-13T08:41:12.513 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:41:12.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:41:12.604 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:41:12.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:41:12.613 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:41:12.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:12.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:12.798 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:41:12.798 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:12.799 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:12.799 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:12.836 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:12.836 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:12.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:41:13.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 -- 172.21.15.31:0/331051188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba415e930 msgr2=0x7f8ba415ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 --2- 172.21.15.31:0/331051188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba415e930 0x7f8ba415ed70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005fd0 tx=0x7f8b8c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 -- 172.21.15.31:0/331051188 shutdown_connections 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 --2- 172.21.15.31:0/331051188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba415e930 0x7f8ba415ed70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 --2- 172.21.15.31:0/331051188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba41446e0 0x7f8ba415bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 -- 172.21.15.31:0/331051188 >> 172.21.15.31:0/331051188 conn(0x7f8ba40b18f0 msgr2=0x7f8ba40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 -- 172.21.15.31:0/331051188 shutdown_connections 2024-09-13T08:41:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.112+0000 7f8baba00640 1 -- 172.21.15.31:0/331051188 wait complete. 2024-09-13T08:41:13.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.114+0000 7f8baba00640 1 Processor -- start 2024-09-13T08:41:13.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.114+0000 7f8baba00640 1 -- start start 2024-09-13T08:41:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba40cec90 0x7f8ba41f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba415e930 0x7f8ba415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba40cf2c0 con 0x7f8ba41446e0 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba40cf140 con 0x7f8ba415e930 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8baba00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba40cf440 con 0x7f8ba40cec90 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37702/0 (socket says 172.21.15.31:37702) 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8b9be00640 1 -- 172.21.15.31:0/2229424303 learned_addr learned my addr 172.21.15.31:0/2229424303 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8ba9000640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba40cec90 0x7f8ba41f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.115+0000 7f8ba9a00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba415e930 0x7f8ba415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 -- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba40cec90 msgr2=0x7f8ba41f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba40cec90 0x7f8ba41f7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 -- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba415e930 msgr2=0x7f8ba415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba415e930 0x7f8ba415a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 -- 172.21.15.31:0/2229424303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba4141310 con 0x7f8ba41446e0 2024-09-13T08:41:13.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8b9be00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034040 tx=0x7f8b8c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c03e030 con 0x7f8ba41446e0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b8c003ca0 con 0x7f8ba41446e0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c035040 con 0x7f8ba41446e0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba41415f0 con 0x7f8ba41446e0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.116+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba41420d0 con 0x7f8ba41446e0 2024-09-13T08:41:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.117+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba415bdc0 con 0x7f8ba41446e0 2024-09-13T08:41:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.119+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f8b8c041030 con 0x7f8ba41446e0 2024-09-13T08:41:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.120+0000 7f8ba1600640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 0x7f8b8007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.120+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f8b8c0bb310 con 0x7f8ba41446e0 2024-09-13T08:41:13.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.120+0000 7f8ba9000640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 0x7f8b8007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.121+0000 7f8ba9000640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 0x7f8b8007b070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400bbe0 tx=0x7f8b9400b390 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:13.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.124+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8b8c082e20 con 0x7f8ba41446e0 2024-09-13T08:41:13.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.331+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8ba40cd240 con 0x7f8ba41446e0 2024-09-13T08:41:13.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.332+0000 7f8ba1600640 1 -- 172.21.15.31:0/2229424303 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) ==== 80+0+0 (secure 0 0 0) 0x7f8b8c087850 con 0x7f8ba41446e0 2024-09-13T08:41:13.334 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:41:13.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 msgr2=0x7f8b8007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 0x7f8b8007b070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400bbe0 tx=0x7f8b9400b390 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 msgr2=0x7f8ba40ad680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034040 tx=0x7f8b8c0038a0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 shutdown_connections 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba415e930 0x7f8ba415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ba41446e0 0x7f8ba40ad680 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8b80078bb0 0x7f8b8007b070 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 --2- 172.21.15.31:0/2229424303 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8ba40cec90 0x7f8ba41f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 >> 172.21.15.31:0/2229424303 conn(0x7f8ba40b18f0 msgr2=0x7f8ba415c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 shutdown_connections 2024-09-13T08:41:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.334+0000 7f8baba00640 1 -- 172.21.15.31:0/2229424303 wait complete. 2024-09-13T08:41:13.353 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:41:13.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:13.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:13.439 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:41:13.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:13.440 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:41:13.488 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:13.488 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:13.489 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:13.526 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:13.527 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:13.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.765+0000 7f504d800640 1 -- 172.21.15.31:0/2863669959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50480cbbd0 msgr2=0x7f504806cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.765+0000 7f504d800640 1 --2- 172.21.15.31:0/2863669959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50480cbbd0 0x7f504806cb00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5038009830 tx=0x7f503802a480 comp rx=0 tx=0).stop 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 -- 172.21.15.31:0/2863669959 shutdown_connections 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 --2- 172.21.15.31:0/2863669959 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 --2- 172.21.15.31:0/2863669959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50480cbbd0 0x7f504806cb00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 -- 172.21.15.31:0/2863669959 >> 172.21.15.31:0/2863669959 conn(0x7f50480d0e30 msgr2=0x7f50480d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 -- 172.21.15.31:0/2863669959 shutdown_connections 2024-09-13T08:41:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.766+0000 7f504d800640 1 -- 172.21.15.31:0/2863669959 wait complete. 2024-09-13T08:41:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.767+0000 7f504d800640 1 Processor -- start 2024-09-13T08:41:13.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.767+0000 7f504d800640 1 -- start start 2024-09-13T08:41:13.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 0x7f50481f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f504815ed60 con 0x7f504806d040 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f504815ebe0 con 0x7f50480cbbd0 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.768+0000 7f504d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f504815eee0 con 0x7f50480d0840 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37716/0 (socket says 172.21.15.31:37716) 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 -- 172.21.15.31:0/2831131180 learned_addr learned my addr 172.21.15.31:0/2831131180 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f503f400640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 0x7f50481f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f504ce00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 -- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 msgr2=0x7f504815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 -- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 msgr2=0x7f50481f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 0x7f50481f2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 -- 172.21.15.31:0/2831131180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50481f4ea0 con 0x7f504806d040 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f503f400640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 0x7f50481f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f504ce00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.769+0000 7f5047e00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5038009950 tx=0x7f5038032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.770+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038040070 con 0x7f504806d040 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.770+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5038005310 con 0x7f504806d040 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.770+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50380047a0 con 0x7f504806d040 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.770+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50481f5180 con 0x7f504806d040 2024-09-13T08:41:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.770+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50481f5740 con 0x7f504806d040 2024-09-13T08:41:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.771+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f504806a190 con 0x7f504806d040 2024-09-13T08:41:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.772+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f5038039040 con 0x7f504806d040 2024-09-13T08:41:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.773+0000 7f5044c00640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 0x7f502407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:13.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.773+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f50380bc3d0 con 0x7f504806d040 2024-09-13T08:41:13.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.773+0000 7f503f400640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 0x7f502407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:13.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.774+0000 7f503f400640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 0x7f502407b070 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f50480b6f20 tx=0x7f5030009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:13.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.777+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5038084530 con 0x7f504806d040 2024-09-13T08:41:13.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.984+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5048160590 con 0x7f504806d040 2024-09-13T08:41:13.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.985+0000 7f5044c00640 1 -- 172.21.15.31:0/2831131180 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) ==== 80+0+0 (secure 0 0 0) 0x7f5038083e30 con 0x7f504806d040 2024-09-13T08:41:13.986 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:41:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 msgr2=0x7f502407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 0x7f502407b070 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f50480b6f20 tx=0x7f5030009040 comp rx=0 tx=0).stop 2024-09-13T08:41:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 msgr2=0x7f50480b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5038009950 tx=0x7f5038032c10 comp rx=0 tx=0).stop 2024-09-13T08:41:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 shutdown_connections 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5024078bb0 0x7f502407b070 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50480d0840 0x7f504815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50480cbbd0 0x7f50481f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 --2- 172.21.15.31:0/2831131180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f504806d040 0x7f50480b4570 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 >> 172.21.15.31:0/2831131180 conn(0x7f50480d0e30 msgr2=0x7f50480add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 shutdown_connections 2024-09-13T08:41:13.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:13.987+0000 7f504d800640 1 -- 172.21.15.31:0/2831131180 wait complete. 2024-09-13T08:41:14.004 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:41:14.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:14.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:14.071 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:41:14.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:14.071 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:41:14.108 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:14.109 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:14.109 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:14.145 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:41:14.145 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:14.145 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:14.145 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:14.185 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:41:14.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:14.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 -- 172.21.15.31:0/601405197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8152a70 msgr2=0x7f47d8152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 --2- 172.21.15.31:0/601405197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8152a70 0x7f47d8152e50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f47d80b2a00 tx=0x7f47c802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 -- 172.21.15.31:0/601405197 shutdown_connections 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 --2- 172.21.15.31:0/601405197 >> v2:172.21.15.45:3300/0 conn(0x7f47d80cf980 0x7f47d80cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 --2- 172.21.15.31:0/601405197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8152a70 0x7f47d8152e50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 -- 172.21.15.31:0/601405197 >> 172.21.15.31:0/601405197 conn(0x7f47d80cc910 msgr2=0x7f47d80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 -- 172.21.15.31:0/601405197 shutdown_connections 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.343+0000 7f47dec00640 1 -- 172.21.15.31:0/601405197 wait complete. 2024-09-13T08:41:14.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.345+0000 7f47dec00640 1 Processor -- start 2024-09-13T08:41:14.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.345+0000 7f47dec00640 1 -- start start 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d80cf980 0x7f47d80b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d8152a70 0x7f47d81f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47d80b1ca0 con 0x7f47d8073fc0 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47d80b1b20 con 0x7f47d8152a70 2024-09-13T08:41:14.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dec00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f47d80b1e20 con 0x7f47d80cf980 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37742/0 (socket says 172.21.15.31:37742) 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47d7e00640 1 -- 172.21.15.31:0/247648895 learned_addr learned my addr 172.21.15.31:0/247648895 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47dcc00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d8152a70 0x7f47d81f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.346+0000 7f47cf400640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d80cf980 0x7f47d80b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 -- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d80cf980 msgr2=0x7f47d80b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d80cf980 0x7f47d80b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 -- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d8152a70 msgr2=0x7f47d81f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d8152a70 0x7f47d81f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 -- 172.21.15.31:0/247648895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d80b3f60 con 0x7f47d8073fc0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d7e00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009850 tx=0x7f47c8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c803f070 con 0x7f47d8073fc0 2024-09-13T08:41:14.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47c80047a0 con 0x7f47d8073fc0 2024-09-13T08:41:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8035030 con 0x7f47d8073fc0 2024-09-13T08:41:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d80b4240 con 0x7f47d8073fc0 2024-09-13T08:41:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.347+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d80b4800 con 0x7f47d8073fc0 2024-09-13T08:41:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.348+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d80b22d0 con 0x7f47d8073fc0 2024-09-13T08:41:14.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.349+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f47c8042070 con 0x7f47d8073fc0 2024-09-13T08:41:14.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.350+0000 7f47d4c00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 0x7f47b407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.351+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f47c80bbd40 con 0x7f47d8073fc0 2024-09-13T08:41:14.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.351+0000 7f47cf400640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 0x7f47b407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.352+0000 7f47cf400640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 0x7f47b407b070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f47d815c2b0 tx=0x7f47c0005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:14.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.355+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f47c80837c0 con 0x7f47d8073fc0 2024-09-13T08:41:14.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.584+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47d80adc00 con 0x7f47d8073fc0 2024-09-13T08:41:14.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.585+0000 7f47d4c00640 1 -- 172.21.15.31:0/247648895 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5217 (secure 0 0 0) 0x7f47c80881f0 con 0x7f47d8073fc0 2024-09-13T08:41:14.587 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:14.588 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":38,"btime":"2024-09-13T08:40:52:813222+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:41:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 38 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 msgr2=0x7f47b407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 0x7f47b407b070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f47d815c2b0 tx=0x7f47c0005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 msgr2=0x7f47d80b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009850 tx=0x7f47c8003820 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 shutdown_connections 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d8152a70 0x7f47d81f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f47b4078bb0 0x7f47b407b070 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d80cf980 0x7f47d80b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 --2- 172.21.15.31:0/247648895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d8073fc0 0x7f47d80b0b50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.587+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 >> 172.21.15.31:0/247648895 conn(0x7f47d80cc910 msgr2=0x7f47d80adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.588+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 shutdown_connections 2024-09-13T08:41:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.588+0000 7f47dec00640 1 -- 172.21.15.31:0/247648895 wait complete. 2024-09-13T08:41:14.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 -- 172.21.15.31:0/2370452582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 msgr2=0x7fa9980d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 --2- 172.21.15.31:0/2370452582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9980d2500 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa98c005fd0 tx=0x7fa98c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 -- 172.21.15.31:0/2370452582 shutdown_connections 2024-09-13T08:41:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 --2- 172.21.15.31:0/2370452582 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980b47f0 0x7fa9980d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 --2- 172.21.15.31:0/2370452582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9980d2500 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 -- 172.21.15.31:0/2370452582 >> 172.21.15.31:0/2370452582 conn(0x7fa99808d8b0 msgr2=0x7fa99809d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 -- 172.21.15.31:0/2370452582 shutdown_connections 2024-09-13T08:41:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.765+0000 7fa99e400640 1 -- 172.21.15.31:0/2370452582 wait complete. 2024-09-13T08:41:14.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.767+0000 7fa99e400640 1 Processor -- start 2024-09-13T08:41:14.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.767+0000 7fa99e400640 1 -- start start 2024-09-13T08:41:14.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980b47f0 0x7fa998153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980d4110 0x7fa9980d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981620b0 con 0x7fa99815dfa0 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9980b21a0 con 0x7fa9980b47f0 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99e400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa998162230 con 0x7fa9980d4110 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37756/0 (socket says 172.21.15.31:37756) 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99da00640 1 -- 172.21.15.31:0/985582852 learned_addr learned my addr 172.21.15.31:0/985582852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa997e00640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980d4110 0x7fa9980d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.768+0000 7fa99d000640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980b47f0 0x7fa998153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 -- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980d4110 msgr2=0x7fa9980d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980d4110 0x7fa9980d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 -- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980b47f0 msgr2=0x7fa998153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980b47f0 0x7fa998153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:14.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 -- 172.21.15.31:0/985582852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9980d12f0 con 0x7fa99815dfa0 2024-09-13T08:41:14.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99da00640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c7d0 tx=0x7fa99000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:14.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa990003b70 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa990003d30 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99000aab0 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9980d0a50 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.769+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9980d1010 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.771+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9981626e0 con 0x7fa99815dfa0 2024-09-13T08:41:14.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.772+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fa99000ac50 con 0x7fa99815dfa0 2024-09-13T08:41:14.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.773+0000 7fa994c00640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 0x7fa97807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:14.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.773+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7fa99009adb0 con 0x7fa99815dfa0 2024-09-13T08:41:14.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.773+0000 7fa99d000640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 0x7fa97807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:14.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.774+0000 7fa99d000640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 0x7fa97807b070 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa98c033000 tx=0x7fa98c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:14.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:14.777+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa99009f070 con 0x7fa99815dfa0 2024-09-13T08:41:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.011+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9980d4cf0 con 0x7fa99815dfa0 2024-09-13T08:41:15.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.011+0000 7fa994c00640 1 -- 172.21.15.31:0/985582852 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5217 (secure 0 0 0) 0x7fa990062730 con 0x7fa99815dfa0 2024-09-13T08:41:15.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:15.014 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":38,"btime":"2024-09-13T08:40:52:813222+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:41:15.014 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 38 2024-09-13T08:41:15.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 msgr2=0x7fa97807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 0x7fa97807b070 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa98c033000 tx=0x7fa98c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 msgr2=0x7fa9981f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c7d0 tx=0x7fa99000cc90 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 shutdown_connections 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa99815dfa0 0x7fa9981f7fa0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa978078bb0 0x7fa97807b070 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa9980d4110 0x7fa9980d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 --2- 172.21.15.31:0/985582852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9980b47f0 0x7fa998153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.013+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 >> 172.21.15.31:0/985582852 conn(0x7fa99808d8b0 msgr2=0x7fa9980b8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.014+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 shutdown_connections 2024-09-13T08:41:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.014+0000 7fa99e400640 1 -- 172.21.15.31:0/985582852 wait complete. 2024-09-13T08:41:15.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 -- 172.21.15.31:0/3232002484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc0cf850 msgr2=0x7f94bc0ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 --2- 172.21.15.31:0/3232002484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc0cf850 0x7f94bc0ce3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009830 tx=0x7f94a802a510 comp rx=0 tx=0).stop 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 -- 172.21.15.31:0/3232002484 shutdown_connections 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 --2- 172.21.15.31:0/3232002484 >> v2:172.21.15.45:3300/0 conn(0x7f94bc0ce900 0x7f94bc0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 --2- 172.21.15.31:0/3232002484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc0cf850 0x7f94bc0ce3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 -- 172.21.15.31:0/3232002484 >> 172.21.15.31:0/3232002484 conn(0x7f94bc0cc910 msgr2=0x7f94bc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 -- 172.21.15.31:0/3232002484 shutdown_connections 2024-09-13T08:41:15.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.193+0000 7f94c4600640 1 -- 172.21.15.31:0/3232002484 wait complete. 2024-09-13T08:41:15.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 Processor -- start 2024-09-13T08:41:15.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 -- start start 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 0x7f94bc09bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc15bd70 0x7f94bc155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1600f0 con 0x7f94bc0cf850 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc0cb850 con 0x7f94bc0ce900 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.194+0000 7f94c4600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc09d460 con 0x7f94bc15bd70 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39994/0 (socket says 172.21.15.31:39994) 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 -- 172.21.15.31:0/4227816495 learned_addr learned my addr 172.21.15.31:0/4227816495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:15.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c2600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc15bd70 0x7f94bc155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1c00640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 0x7f94bc09bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 -- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc15bd70 msgr2=0x7f94bc155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc15bd70 0x7f94bc155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 -- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 msgr2=0x7f94bc09bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 0x7f94bc09bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1200640 1 -- 172.21.15.31:0/4227816495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc0b5cb0 con 0x7f94bc0cf850 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.195+0000 7f94c1c00640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 0x7f94bc09bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94c1200640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f94a8000c00 tx=0x7f94a8032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8040070 con 0x7f94bc0cf850 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94a80038e0 con 0x7f94bc0cf850 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8003d30 con 0x7f94bc0cf850 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc09c3c0 con 0x7f94bc0cf850 2024-09-13T08:41:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.196+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc09cc60 con 0x7f94bc0cf850 2024-09-13T08:41:15.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.197+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc0d0dc0 con 0x7f94bc0cf850 2024-09-13T08:41:15.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.198+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f94a8039040 con 0x7f94bc0cf850 2024-09-13T08:41:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.199+0000 7f94b9600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 0x7f949c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.199+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f94a80bc5d0 con 0x7f94bc0cf850 2024-09-13T08:41:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.200+0000 7f94c1c00640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 0x7f949c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.200+0000 7f94c1c00640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 0x7f949c07b070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f94b0007fd0 tx=0x7f94b0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.203+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f94bc0d0dc0 con 0x7f94bc0cf850 2024-09-13T08:41:15.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.400+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94bc0ce3c0 con 0x7f94bc0cf850 2024-09-13T08:41:15.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.401+0000 7f94b9600640 1 -- 172.21.15.31:0/4227816495 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16773 (secure 0 0 0) 0x7f94a8084730 con 0x7f94bc0cf850 2024-09-13T08:41:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:15.403 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:51.806287+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:40:35.056540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T08:40:36.137508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.31","mappings":[{"from":6,"to":1}]},{"pgid":"9.37","mappings":[{"from":6,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 msgr2=0x7f949c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 0x7f949c07b070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f94b0007fd0 tx=0x7f94b0009040 comp rx=0 tx=0).stop 2024-09-13T08:41:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 msgr2=0x7f94bc06f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f94a8000c00 tx=0x7f94a8032c10 comp rx=0 tx=0).stop 2024-09-13T08:41:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 shutdown_connections 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f949c078bb0 0x7f949c07b070 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94bc15bd70 0x7f94bc155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94bc0cf850 0x7f94bc06f7b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 --2- 172.21.15.31:0/4227816495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc0ce900 0x7f94bc09bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 >> 172.21.15.31:0/4227816495 conn(0x7f94bc0cc910 msgr2=0x7f94bc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 shutdown_connections 2024-09-13T08:41:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.403+0000 7f94c4600640 1 -- 172.21.15.31:0/4227816495 wait complete. 2024-09-13T08:41:15.421 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:41:15.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:15.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 -- 172.21.15.31:0/3280303138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c0cf980 msgr2=0x7fe22c0cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 --2- 172.21.15.31:0/3280303138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c0cf980 0x7fe22c0cbf60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe22c06daa0 tx=0x7fe21402a530 comp rx=0 tx=0).stop 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 -- 172.21.15.31:0/3280303138 shutdown_connections 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 --2- 172.21.15.31:0/3280303138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c0cf980 0x7fe22c0cbf60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 --2- 172.21.15.31:0/3280303138 >> v2:172.21.15.45:3300/0 conn(0x7fe22c152a70 0x7fe22c152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 -- 172.21.15.31:0/3280303138 >> 172.21.15.31:0/3280303138 conn(0x7fe22c0cc910 msgr2=0x7fe22c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.579+0000 7fe230e00640 1 -- 172.21.15.31:0/3280303138 shutdown_connections 2024-09-13T08:41:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.580+0000 7fe230e00640 1 -- 172.21.15.31:0/3280303138 wait complete. 2024-09-13T08:41:15.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.581+0000 7fe230e00640 1 Processor -- start 2024-09-13T08:41:15.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.581+0000 7fe230e00640 1 -- start start 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe22c0cf980 0x7fe22c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 0x7fe22c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c0b1ca0 con 0x7fe22c073fc0 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c0b1b20 con 0x7fe22c152a70 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe230e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe22c0b1e20 con 0x7fe22c0cf980 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe229e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.582+0000 7fe229e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40010/0 (socket says 172.21.15.31:40010) 2024-09-13T08:41:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 -- 172.21.15.31:0/422262489 learned_addr learned my addr 172.21.15.31:0/422262489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe221400640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe22c0cf980 0x7fe22c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe22a800640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 0x7fe22c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 -- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe22c0cf980 msgr2=0x7fe22c0c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe22c0cf980 0x7fe22c0c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 -- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 msgr2=0x7fe22c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 0x7fe22c0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 -- 172.21.15.31:0/422262489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22c1f8800 con 0x7fe22c073fc0 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe22a800640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 0x7fe22c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.583+0000 7fe229e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00d950 tx=0x7fe21c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.584+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c017070 con 0x7fe22c073fc0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.584+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe21c009d20 con 0x7fe22c073fc0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.584+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c004280 con 0x7fe22c073fc0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.584+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22c1f7950 con 0x7fe22c073fc0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.584+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22c1f81b0 con 0x7fe22c073fc0 2024-09-13T08:41:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.586+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe22c0b22d0 con 0x7fe22c073fc0 2024-09-13T08:41:15.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.586+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fe21c0035f0 con 0x7fe22c073fc0 2024-09-13T08:41:15.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.587+0000 7fe222a00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 0x7fe20807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:15.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.587+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7fe21c09b1a0 con 0x7fe22c073fc0 2024-09-13T08:41:15.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.587+0000 7fe221400640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 0x7fe20807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:15.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.588+0000 7fe221400640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 0x7fe20807b070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe214034000 tx=0x7fe214004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:15.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.591+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe21c062bb0 con 0x7fe22c073fc0 2024-09-13T08:41:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.825+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe22c069a50 con 0x7fe22c073fc0 2024-09-13T08:41:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.826+0000 7fe222a00640 1 -- 172.21.15.31:0/422262489 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5217 (secure 0 0 0) 0x7fe21c0675e0 con 0x7fe22c073fc0 2024-09-13T08:41:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:15.827 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":38,"btime":"2024-09-13T08:40:52:813222+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:41:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 38 2024-09-13T08:41:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 msgr2=0x7fe20807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 0x7fe20807b070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe214034000 tx=0x7fe214004730 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 msgr2=0x7fe22c159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00d950 tx=0x7fe21c00de10 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 shutdown_connections 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22c152a70 0x7fe22c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe208078bb0 0x7fe20807b070 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe22c0cf980 0x7fe22c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 --2- 172.21.15.31:0/422262489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe22c073fc0 0x7fe22c159020 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.828+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 >> 172.21.15.31:0/422262489 conn(0x7fe22c0cc910 msgr2=0x7fe22c0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.829+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 shutdown_connections 2024-09-13T08:41:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:15.829+0000 7fe230e00640 1 -- 172.21.15.31:0/422262489 wait complete. 2024-09-13T08:41:15.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 -- 172.21.15.31:0/291597847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 msgr2=0x7f706c0d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 --2- 172.21.15.31:0/291597847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c0d0350 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f706c0cc370 tx=0x7f705402a560 comp rx=0 tx=0).stop 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 -- 172.21.15.31:0/291597847 shutdown_connections 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 --2- 172.21.15.31:0/291597847 >> v2:172.21.15.45:3300/0 conn(0x7f706c0c70b0 0x7f706c0c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 --2- 172.21.15.31:0/291597847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c0d0350 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 --2- 172.21.15.31:0/291597847 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c15bf80 0x7f706c0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 -- 172.21.15.31:0/291597847 >> 172.21.15.31:0/291597847 conn(0x7f706c0d1e20 msgr2=0x7f706c0d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 -- 172.21.15.31:0/291597847 shutdown_connections 2024-09-13T08:41:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.005+0000 7f7073000640 1 -- 172.21.15.31:0/291597847 wait complete. 2024-09-13T08:41:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.007+0000 7f7073000640 1 Processor -- start 2024-09-13T08:41:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.007+0000 7f7073000640 1 -- start start 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c0c70b0 0x7f706c0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f706c15bf80 0x7f706c15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f706c0c7e10 con 0x7f706c0d2940 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f706c0c7c90 con 0x7f706c15bf80 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7073000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f706c0c7f90 con 0x7f706c0c70b0 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7063400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7063400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40032/0 (socket says 172.21.15.31:40032) 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7063400640 1 -- 172.21.15.31:0/2863379766 learned_addr learned my addr 172.21.15.31:0/2863379766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:16.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f706be00640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c0c70b0 0x7f706c0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.008+0000 7f7071000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f706c15bf80 0x7f706c15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 -- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c0c70b0 msgr2=0x7f706c0cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c0c70b0 0x7f706c0cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 -- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f706c15bf80 msgr2=0x7f706c15e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f706c15bf80 0x7f706c15e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 -- 172.21.15.31:0/2863379766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c0cbf90 con 0x7f706c0d2940 2024-09-13T08:41:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7063400640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7054009850 tx=0x7f7054003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.009+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705403f070 con 0x7f706c0d2940 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.010+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70540047a0 con 0x7f706c0d2940 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.010+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054035030 con 0x7f706c0d2940 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.010+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c1f2ea0 con 0x7f706c0d2940 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.010+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c0c8f90 con 0x7f706c0d2940 2024-09-13T08:41:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.011+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c0c7540 con 0x7f706c0d2940 2024-09-13T08:41:16.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.012+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f7054042070 con 0x7f706c0d2940 2024-09-13T08:41:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.013+0000 7f7068c00640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 0x7f704807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.013+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f70540bbdd0 con 0x7f706c0d2940 2024-09-13T08:41:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.013+0000 7f706be00640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 0x7f704807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.014+0000 7f706be00640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 0x7f704807b070 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f705c00bab0 tx=0x7f705c00b2e0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.017+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f706c0cbf90 con 0x7f706c0d2940 2024-09-13T08:41:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.248+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f706c0d2450 con 0x7f706c0d2940 2024-09-13T08:41:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.248+0000 7f7068c00640 1 -- 172.21.15.31:0/2863379766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5217 (secure 0 0 0) 0x7f70540838e0 con 0x7f706c0d2940 2024-09-13T08:41:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:16.250 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":38,"btime":"2024-09-13T08:40:52:813222+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:40:37.170179+0000","modified":"2024-09-13T08:40:40.200201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/707614758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":707614758},{"type":"v1","addr":"172.21.15.31:6835","nonce":707614758}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-13T08:41:16.250 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 38 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 msgr2=0x7f704807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 0x7f704807b070 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f705c00bab0 tx=0x7f705c00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 msgr2=0x7f706c1f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7054009850 tx=0x7f7054003910 comp rx=0 tx=0).stop 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 shutdown_connections 2024-09-13T08:41:16.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f706c15bf80 0x7f706c15e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f706c0d2940 0x7f706c1f2520 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f7048078bb0 0x7f704807b070 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 --2- 172.21.15.31:0/2863379766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f706c0c70b0 0x7f706c0cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 >> 172.21.15.31:0/2863379766 conn(0x7f706c0d1e20 msgr2=0x7f706c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 shutdown_connections 2024-09-13T08:41:16.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.251+0000 7f7073000640 1 -- 172.21.15.31:0/2863379766 wait complete. 2024-09-13T08:41:16.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.429+0000 7f1415a00640 1 -- 172.21.15.31:0/157267475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f4003a00 msgr2=0x7f13f4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.429+0000 7f1415a00640 1 --2- 172.21.15.31:0/157267475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f4003a00 0x7f13f4003e20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f14100c78e0 tx=0x7f13f4005830 comp rx=0 tx=0).stop 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.429+0000 7f1415000640 1 -- 172.21.15.31:0/157267475 reap_dead start 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.430+0000 7f1415a00640 1 -- 172.21.15.31:0/157267475 shutdown_connections 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.430+0000 7f1415a00640 1 -- 172.21.15.31:0/157267475 >> 172.21.15.31:0/157267475 conn(0x7f141009d190 msgr2=0x7f14100d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.430+0000 7f1415a00640 1 -- 172.21.15.31:0/157267475 shutdown_connections 2024-09-13T08:41:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.430+0000 7f1415a00640 1 -- 172.21.15.31:0/157267475 wait complete. 2024-09-13T08:41:16.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.431+0000 7f1415a00640 1 Processor -- start 2024-09-13T08:41:16.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.431+0000 7f1415a00640 1 -- start start 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13f4003a00 0x7f141015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14100b7d70 0x7f1410160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14100d2f20 con 0x7f14101612f0 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14100d2da0 con 0x7f14100b7d70 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.432+0000 7f1415a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1410162a90 con 0x7f13f4003a00 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40050/0 (socket says 172.21.15.31:40050) 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 -- 172.21.15.31:0/3450928247 learned_addr learned my addr 172.21.15.31:0/3450928247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f140fe00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13f4003a00 0x7f141015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1407e00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14100b7d70 0x7f1410160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 -- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13f4003a00 msgr2=0x7f141015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13f4003a00 0x7f141015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 -- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14100b7d70 msgr2=0x7f1410160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14100b7d70 0x7f1410160db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 -- 172.21.15.31:0/3450928247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14100c7a90 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.433+0000 7f1415000640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f14100b33d0 tx=0x7f140800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.434+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408003d80 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.434+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1408004b20 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.434+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408004030 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.434+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14100b8200 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.434+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14100b87c0 con 0x7f14101612f0 2024-09-13T08:41:16.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.435+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1410162550 con 0x7f14101612f0 2024-09-13T08:41:16.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.436+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f14080041d0 con 0x7f14101612f0 2024-09-13T08:41:16.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.437+0000 7f140cc00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 0x7f13ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.437+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f140809bc20 con 0x7f14101612f0 2024-09-13T08:41:16.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.437+0000 7f140fe00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 0x7f13ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.438+0000 7f140fe00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 0x7f13ec07b0a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1400035000 tx=0x7f140002d040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.441+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f140800de20 con 0x7f14101612f0 2024-09-13T08:41:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.637+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14100d0340 con 0x7f14101612f0 2024-09-13T08:41:16.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.638+0000 7f140cc00640 1 -- 172.21.15.31:0/3450928247 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16773 (secure 0 0 0) 0x7f1408063d80 con 0x7f14101612f0 2024-09-13T08:41:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:16.640 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:40:51.806287+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:40:35.056540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T08:40:36.137508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.31","mappings":[{"from":6,"to":1}]},{"pgid":"9.37","mappings":[{"from":6,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 msgr2=0x7f13ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 0x7f13ec07b0a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1400035000 tx=0x7f140002d040 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 msgr2=0x7f141006de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f14100b33d0 tx=0x7f140800f790 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 shutdown_connections 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101612f0 0x7f141006de10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14100b7d70 0x7f1410160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f13ec078be0 0x7f13ec07b0a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.640+0000 7f1415a00640 1 --2- 172.21.15.31:0/3450928247 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13f4003a00 0x7f141015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.641+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 >> 172.21.15.31:0/3450928247 conn(0x7f141009d190 msgr2=0x7f14100b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.641+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 shutdown_connections 2024-09-13T08:41:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.641+0000 7f1415a00640 1 -- 172.21.15.31:0/3450928247 wait complete. 2024-09-13T08:41:16.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:41:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 -- 172.21.15.31:0/1438659219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d00ced20 msgr2=0x7f52d00ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 --2- 172.21.15.31:0/1438659219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d00ced20 0x7f52d00ce290 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009830 tx=0x7f52b802a510 comp rx=0 tx=0).stop 2024-09-13T08:41:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 -- 172.21.15.31:0/1438659219 shutdown_connections 2024-09-13T08:41:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 --2- 172.21.15.31:0/1438659219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d00ced20 0x7f52d00ce290 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 -- 172.21.15.31:0/1438659219 >> 172.21.15.31:0/1438659219 conn(0x7f52d00b18f0 msgr2=0x7f52d00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 -- 172.21.15.31:0/1438659219 shutdown_connections 2024-09-13T08:41:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.821+0000 7f52d7800640 1 -- 172.21.15.31:0/1438659219 wait complete. 2024-09-13T08:41:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.823+0000 7f52d7800640 1 Processor -- start 2024-09-13T08:41:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.823+0000 7f52d7800640 1 -- start start 2024-09-13T08:41:16.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 0x7f52d009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52d00ced20 0x7f52d00cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52d00733e0 con 0x7f52d015be30 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52d00ce9e0 con 0x7f52d00ced20 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d7800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f52d00743c0 con 0x7f52d0073860 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40078/0 (socket says 172.21.15.31:40078) 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d5800640 1 -- 172.21.15.31:0/1886522900 learned_addr learned my addr 172.21.15.31:0/1886522900 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52c7e00640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52d00ced20 0x7f52d00cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.824+0000 7f52d4e00640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 0x7f52d009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 -- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 msgr2=0x7f52d009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 0x7f52d009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 -- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52d00ced20 msgr2=0x7f52d00cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52d00ced20 0x7f52d00cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 -- 172.21.15.31:0/1886522900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d009aff0 con 0x7f52d015be30 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d4e00640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 0x7f52d009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d5800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c780 tx=0x7f52c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8004b60 con 0x7f52d015be30 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52c8004df0 con 0x7f52d015be30 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c800aa70 con 0x7f52d015be30 2024-09-13T08:41:16.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.825+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d009b2d0 con 0x7f52d015be30 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.826+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d009b890 con 0x7f52d015be30 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.827+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d0074840 con 0x7f52d015be30 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.828+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f52c800ac10 con 0x7f52d015be30 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.829+0000 7f52cd600640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 0x7f52ac07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.829+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5767+0+0 (secure 0 0 0) 0x7f52c809b350 con 0x7f52d015be30 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.829+0000 7f52d4e00640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 0x7f52ac07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.830+0000 7f52d4e00640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 0x7f52ac07b070 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:16.833+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f52c8020070 con 0x7f52d015be30 2024-09-13T08:41:17.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.067+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f52d006eac0 con 0x7f52d015be30 2024-09-13T08:41:17.068 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:41:17.067+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:41:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.069+0000 7f52cd600640 1 -- 172.21.15.31:0/1886522900 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) ==== 161+0+0 (secure 0 0 0) 0x7f52c8063320 con 0x7f52d015be30 2024-09-13T08:41:17.070 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:41:17.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 msgr2=0x7f52ac07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 0x7f52ac07b070 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 msgr2=0x7f52d0156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c780 tx=0x7f52c800cc40 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 shutdown_connections 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52d015be30 0x7f52d0156870 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52d00ced20 0x7f52d00cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f52ac078bb0 0x7f52ac07b070 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 --2- 172.21.15.31:0/1886522900 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52d0073860 0x7f52d009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.072+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 >> 172.21.15.31:0/1886522900 conn(0x7f52d00b18f0 msgr2=0x7f52d00aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:17.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.073+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 shutdown_connections 2024-09-13T08:41:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.073+0000 7f52d7800640 1 -- 172.21.15.31:0/1886522900 wait complete. 2024-09-13T08:41:17.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:41:17.139 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:41:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 -- 172.21.15.31:0/904286769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 msgr2=0x7faab00d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:17.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 --2- 172.21.15.31:0/904286769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab00d0690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7faab00cb310 tx=0x7faaa402a560 comp rx=0 tx=0).stop 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 -- 172.21.15.31:0/904286769 shutdown_connections 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 --2- 172.21.15.31:0/904286769 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab0072560 0x7faab00b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 --2- 172.21.15.31:0/904286769 >> v2:172.21.15.45:3300/0 conn(0x7faab00cb570 0x7faab00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 --2- 172.21.15.31:0/904286769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab00d0690 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 -- 172.21.15.31:0/904286769 >> 172.21.15.31:0/904286769 conn(0x7faab00cfb70 msgr2=0x7faab00cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 -- 172.21.15.31:0/904286769 shutdown_connections 2024-09-13T08:41:17.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.254+0000 7faab8800640 1 -- 172.21.15.31:0/904286769 wait complete. 2024-09-13T08:41:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.256+0000 7faab8800640 1 Processor -- start 2024-09-13T08:41:17.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.256+0000 7faab8800640 1 -- start start 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 0x7faab00cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab00cb570 0x7faab00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faab006f810 con 0x7faab015c040 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faab006f690 con 0x7faab0072560 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab8800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7faab006f990 con 0x7faab00cb570 2024-09-13T08:41:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40102/0 (socket says 172.21.15.31:40102) 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 -- 172.21.15.31:0/4108820531 learned_addr learned my addr 172.21.15.31:0/4108820531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab5e00640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 0x7faab00cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab5400640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab00cb570 0x7faab00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 -- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab00cb570 msgr2=0x7faab00b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab00cb570 0x7faab00b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 -- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 msgr2=0x7faab00cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 0x7faab00cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab6800640 1 -- 172.21.15.31:0/4108820531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab00ce1f0 con 0x7faab015c040 2024-09-13T08:41:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.257+0000 7faab5e00640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 0x7faab00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faab6800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faaac00cce0 tx=0x7faaac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac00f450 con 0x7faab015c040 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaac00f700 con 0x7faab015c040 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac00fb50 con 0x7faab015c040 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab00ce4d0 con 0x7faab015c040 2024-09-13T08:41:17.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.258+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab00af6e0 con 0x7faab015c040 2024-09-13T08:41:17.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.259+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab006a180 con 0x7faab015c040 2024-09-13T08:41:17.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.260+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7faaac00fcf0 con 0x7faab015c040 2024-09-13T08:41:17.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.261+0000 7faaaa000640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 0x7faa9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:17.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.262+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5853+0+0 (secure 0 0 0) 0x7faaac09bca0 con 0x7faab015c040 2024-09-13T08:41:17.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.262+0000 7faab5e00640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 0x7faa9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:17.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.263+0000 7faab5e00640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 0x7faa9407b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faaa4033000 tx=0x7faaa4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:17.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.266+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7faab006a180 con 0x7faab015c040 2024-09-13T08:41:17.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:17.500+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7faab006fef0 con 0x7faab015c040 2024-09-13T08:41:18.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.074+0000 7faaaa000640 1 -- 172.21.15.31:0/4108820531 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) ==== 104+0+0 (secure 0 0 0) 0x7faaac014030 con 0x7faab015c040 2024-09-13T08:41:18.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 msgr2=0x7faa9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:18.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 0x7faa9407b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faaa4033000 tx=0x7faaa4004730 comp rx=0 tx=0).stop 2024-09-13T08:41:18.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 msgr2=0x7faab0151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faaac00cce0 tx=0x7faaac00a620 comp rx=0 tx=0).stop 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 shutdown_connections 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab015c040 0x7faab0151fc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7faa94078be0 0x7faa9407b0a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faab00cb570 0x7faab00b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 --2- 172.21.15.31:0/4108820531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faab0072560 0x7faab00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.077+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 >> 172.21.15.31:0/4108820531 conn(0x7faab00cfb70 msgr2=0x7faab0159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.078+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 shutdown_connections 2024-09-13T08:41:18.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.078+0000 7faab8800640 1 -- 172.21.15.31:0/4108820531 wait complete. 2024-09-13T08:41:18.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:41:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 -- 172.21.15.31:0/64805888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4003aa0 msgr2=0x7f63f4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 --2- 172.21.15.31:0/64805888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4003aa0 0x7f63f4003ec0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f640000c310 tx=0x7f63f4005830 comp rx=0 tx=0).stop 2024-09-13T08:41:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 -- 172.21.15.31:0/64805888 shutdown_connections 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 --2- 172.21.15.31:0/64805888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4003aa0 0x7f63f4003ec0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 -- 172.21.15.31:0/64805888 >> 172.21.15.31:0/64805888 conn(0x7f64100ccd00 msgr2=0x7f64100cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 -- 172.21.15.31:0/64805888 shutdown_connections 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.264+0000 7f640fe00640 1 -- 172.21.15.31:0/64805888 wait complete. 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.266+0000 7f640fe00640 1 Processor -- start 2024-09-13T08:41:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.266+0000 7f640fe00640 1 -- start start 2024-09-13T08:41:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 0x7f64101f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64100c9850 0x7f64101e6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64100688b0 con 0x7f64100c9cd0 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64100b2560 con 0x7f64100c9850 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6410068a30 con 0x7f63f4003aa0 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40106/0 (socket says 172.21.15.31:40106) 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640f400640 1 -- 172.21.15.31:0/797069895 learned_addr learned my addr 172.21.15.31:0/797069895 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f6407e00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64100c9850 0x7f64101e6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.267+0000 7f640ea00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 0x7f64101f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 -- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 msgr2=0x7f64101f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 0x7f64101f8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 -- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64100c9850 msgr2=0x7f64101e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64100c9850 0x7f64101e6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 -- 172.21.15.31:0/797069895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64101f57e0 con 0x7f64100c9cd0 2024-09-13T08:41:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640ea00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 0x7f64101f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640f400640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6408005fd0 tx=0x7f640800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408003d80 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6408004b20 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64101f5ac0 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408004030 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.268+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64101f8530 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.270+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64100cea50 con 0x7f64100c9cd0 2024-09-13T08:41:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.271+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f64080041d0 con 0x7f64100c9cd0 2024-09-13T08:41:18.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.272+0000 7f6406a00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 0x7f63ec07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:18.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.272+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5853+0+0 (secure 0 0 0) 0x7f640809be40 con 0x7f64100c9cd0 2024-09-13T08:41:18.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.273+0000 7f640ea00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 0x7f63ec07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:18.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.273+0000 7f640ea00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 0x7f63ec07b070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6400033000 tx=0x7f64000019f0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:18.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.277+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6408063e90 con 0x7f64100c9cd0 2024-09-13T08:41:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:18.477+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f64100b2ae0 con 0x7f64100c9cd0 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.076+0000 7f6406a00640 1 -- 172.21.15.31:0/797069895 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) ==== 181+0+0 (secure 0 0 0) 0x7f6408063790 con 0x7f64100c9cd0 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 msgr2=0x7f63ec07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 0x7f63ec07b070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6400033000 tx=0x7f64000019f0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 msgr2=0x7f64101e73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6408005fd0 tx=0x7f640800f790 comp rx=0 tx=0).stop 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 shutdown_connections 2024-09-13T08:41:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64100c9cd0 0x7f64101e73c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64100c9850 0x7f64101e6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f63ec078bb0 0x7f63ec07b070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 --2- 172.21.15.31:0/797069895 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f63f4003aa0 0x7f64101f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.081+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 >> 172.21.15.31:0/797069895 conn(0x7f64100ccd00 msgr2=0x7f64100b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.082+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 shutdown_connections 2024-09-13T08:41:19.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.082+0000 7f640fe00640 1 -- 172.21.15.31:0/797069895 wait complete. 2024-09-13T08:41:19.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:41:19.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 -- 172.21.15.31:0/800264572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3040cd780 msgr2=0x7fa3040ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 --2- 172.21.15.31:0/800264572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3040cd780 0x7fa3040ce660 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009830 tx=0x7fa2f002a510 comp rx=0 tx=0).stop 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 -- 172.21.15.31:0/800264572 shutdown_connections 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 --2- 172.21.15.31:0/800264572 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040ca200 0x7fa3040ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 --2- 172.21.15.31:0/800264572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3040cd780 0x7fa3040ce660 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 -- 172.21.15.31:0/800264572 >> 172.21.15.31:0/800264572 conn(0x7fa3040cfbb0 msgr2=0x7fa3040cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 -- 172.21.15.31:0/800264572 shutdown_connections 2024-09-13T08:41:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.273+0000 7fa30a400640 1 -- 172.21.15.31:0/800264572 wait complete. 2024-09-13T08:41:19.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.275+0000 7fa30a400640 1 Processor -- start 2024-09-13T08:41:19.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.275+0000 7fa30a400640 1 -- start start 2024-09-13T08:41:19.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3040ca200 0x7fa3040c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:19.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040cd780 0x7fa3041f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa30406f500 con 0x7fa30415d5e0 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa304070860 con 0x7fa3040ca200 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa30a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa30406f680 con 0x7fa3040cd780 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa303e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.276+0000 7fa303e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40122/0 (socket says 172.21.15.31:40122) 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 -- 172.21.15.31:0/2553048510 learned_addr learned my addr 172.21.15.31:0/2553048510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa302a00640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040cd780 0x7fa3041f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3040ca200 0x7fa3040c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 -- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040cd780 msgr2=0x7fa3041f7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040cd780 0x7fa3041f7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 -- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3040ca200 msgr2=0x7fa3040c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:19.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3040ca200 0x7fa3040c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 -- 172.21.15.31:0/2553048510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3040c5510 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.277+0000 7fa303e00640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb10 tx=0x7fa2fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.278+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc004b20 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.278+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2fc004ce0 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.278+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc005280 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.278+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3040c57f0 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.278+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30414eac0 con 0x7fa30415d5e0 2024-09-13T08:41:19.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.279+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3040ca690 con 0x7fa30415d5e0 2024-09-13T08:41:19.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.280+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fa2fc005420 con 0x7fa30415d5e0 2024-09-13T08:41:19.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.281+0000 7fa2fb400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 0x7fa2e407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:19.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.281+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5415+0+0 (secure 0 0 0) 0x7fa2fc09bb60 con 0x7fa30415d5e0 2024-09-13T08:41:19.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.281+0000 7fa303400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 0x7fa2e407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:19.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.282+0000 7fa303400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 0x7fa2e407b070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000f60 tx=0x7fa2f4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:19.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.285+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa2fc063640 con 0x7fa30415d5e0 2024-09-13T08:41:19.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:19.484+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa304158ae0 con 0x7fa30415d5e0 2024-09-13T08:41:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.082+0000 7fa2fb400640 1 -- 172.21.15.31:0/2553048510 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) ==== 169+0+0 (secure 0 0 0) 0x7fa2fc068070 con 0x7fa30415d5e0 2024-09-13T08:41:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:41:20.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.086+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 msgr2=0x7fa2e407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.086+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 0x7fa2e407b070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000f60 tx=0x7fa2f4009040 comp rx=0 tx=0).stop 2024-09-13T08:41:20.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.086+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 msgr2=0x7fa3041f8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.086+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb10 tx=0x7fa2fc00c2d0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 shutdown_connections 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa30415d5e0 0x7fa3041f8430 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fa2e4078bb0 0x7fa2e407b070 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa3040cd780 0x7fa3041f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 --2- 172.21.15.31:0/2553048510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3040ca200 0x7fa3040c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 >> 172.21.15.31:0/2553048510 conn(0x7fa3040cfbb0 msgr2=0x7fa3040cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 shutdown_connections 2024-09-13T08:41:20.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.087+0000 7fa30a400640 1 -- 172.21.15.31:0/2553048510 wait complete. 2024-09-13T08:41:20.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 -- 172.21.15.31:0/435251814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc004800 msgr2=0x7f26fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 --2- 172.21.15.31:0/435251814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2700006f60 tx=0x7f26fc005830 comp rx=0 tx=0).stop 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 -- 172.21.15.31:0/435251814 shutdown_connections 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 --2- 172.21.15.31:0/435251814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc004800 0x7f26fc004c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 -- 172.21.15.31:0/435251814 >> 172.21.15.31:0/435251814 conn(0x7f27140b18f0 msgr2=0x7f27140b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.274+0000 7f271c000640 1 -- 172.21.15.31:0/435251814 shutdown_connections 2024-09-13T08:41:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.275+0000 7f271c000640 1 -- 172.21.15.31:0/435251814 wait complete. 2024-09-13T08:41:20.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.276+0000 7f271c000640 1 Processor -- start 2024-09-13T08:41:20.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.276+0000 7f271c000640 1 -- start start 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27140ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f27141f8260 0x7f2714152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f27140cdd00 con 0x7f271409b240 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f27140cdb80 con 0x7f26fc004800 2024-09-13T08:41:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271c000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f27140cde80 con 0x7f27141f8260 2024-09-13T08:41:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40140/0 (socket says 172.21.15.31:40140) 2024-09-13T08:41:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f271a000640 1 -- 172.21.15.31:0/3711947362 learned_addr learned my addr 172.21.15.31:0/3711947362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f2719600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27140ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.277+0000 7f2718c00640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f27141f8260 0x7f2714152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 -- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f27141f8260 msgr2=0x7f2714152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f27141f8260 0x7f2714152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 -- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 msgr2=0x7f27140ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27140ae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 -- 172.21.15.31:0/3711947362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27140aefd0 con 0x7f271409b240 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f271a000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f27140cf770 tx=0x7f270c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.278+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c003d80 con 0x7f271409b240 2024-09-13T08:41:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.279+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f270c004b20 con 0x7f271409b240 2024-09-13T08:41:20.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.279+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c004050 con 0x7f271409b240 2024-09-13T08:41:20.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.279+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27141f8720 con 0x7f271409b240 2024-09-13T08:41:20.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.279+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27141f8ce0 con 0x7f271409b240 2024-09-13T08:41:20.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.280+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27140ce3e0 con 0x7f271409b240 2024-09-13T08:41:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.281+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f270c0041f0 con 0x7f271409b240 2024-09-13T08:41:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.282+0000 7f2711600640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 0x7f26f407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.282+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7f270c09ba10 con 0x7f271409b240 2024-09-13T08:41:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.282+0000 7f2719600640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 0x7f26f407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.283+0000 7f2719600640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 0x7f26f407b070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f27140ae2d0 tx=0x7f2708006c70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:20.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.286+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f270c063650 con 0x7f271409b240 2024-09-13T08:41:20.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.491+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f27140b25f0 con 0x7f271409b240 2024-09-13T08:41:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.495+0000 7f2711600640 1 -- 172.21.15.31:0/3711947362 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v33) ==== 71+0+0 (secure 0 0 0) 0x7f270c068080 con 0x7f271409b240 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.497+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 msgr2=0x7f26f407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.497+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 0x7f26f407b070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f27140ae2d0 tx=0x7f2708006c70 comp rx=0 tx=0).stop 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.497+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 msgr2=0x7f2714156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.497+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f27140cf770 tx=0x7f270c00f790 comp rx=0 tx=0).stop 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 shutdown_connections 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f271409b240 0x7f2714156560 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f26f4078bb0 0x7f26f407b070 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f27141f8260 0x7f2714152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 --2- 172.21.15.31:0/3711947362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26fc004800 0x7f27140ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 >> 172.21.15.31:0/3711947362 conn(0x7f27140b18f0 msgr2=0x7f2714164610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 shutdown_connections 2024-09-13T08:41:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.498+0000 7f271c000640 1 -- 172.21.15.31:0/3711947362 wait complete. 2024-09-13T08:41:20.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:41:20.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3942507762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 msgr2=0x7ff9e00b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3942507762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e00b5100 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00b26c0 tx=0x7ff9d802aa30 comp rx=0 tx=0).stop 2024-09-13T08:41:20.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3942507762 shutdown_connections 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3942507762 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3942507762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e00b5100 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3942507762 >> v2:172.21.15.45:3300/0 conn(0x7ff9e00cc720 0x7ff9e006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3942507762 >> 172.21.15.31:0/3942507762 conn(0x7ff9e006cb00 msgr2=0x7ff9e006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3942507762 shutdown_connections 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.674+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3942507762 wait complete. 2024-09-13T08:41:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.676+0000 7ff9e4e00640 1 Processor -- start 2024-09-13T08:41:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.676+0000 7ff9e4e00640 1 -- start start 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 0x7ff9e01f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0152a10 con 0x7ff9e00701a0 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0152890 con 0x7ff9e00cc720 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9e4e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e0152b90 con 0x7ff9e00ccb00 2024-09-13T08:41:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40158/0 (socket says 172.21.15.31:40158) 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9df400640 1 -- 172.21.15.31:0/3115506648 learned_addr learned my addr 172.21.15.31:0/3115506648 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9dfe00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.677+0000 7ff9dea00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 0x7ff9e01f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 -- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 msgr2=0x7ff9e00cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 -- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 msgr2=0x7ff9e01f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 0x7ff9e01f5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 -- 172.21.15.31:0/3115506648 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e01f5b50 con 0x7ff9e00701a0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9dfe00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9dea00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 0x7ff9e01f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9df400640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000e9f0 tx=0x7ff9d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000ccc0 con 0x7ff9e00701a0 2024-09-13T08:41:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9d0010430 con 0x7ff9e00701a0 2024-09-13T08:41:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0010840 con 0x7ff9e00701a0 2024-09-13T08:41:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e00c6cf0 con 0x7ff9e00701a0 2024-09-13T08:41:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.678+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e00c7290 con 0x7ff9e00701a0 2024-09-13T08:41:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.680+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e0153040 con 0x7ff9e00701a0 2024-09-13T08:41:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.681+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7ff9d00109e0 con 0x7ff9e00701a0 2024-09-13T08:41:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.681+0000 7ff9d7400640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 0x7ff9c007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.682+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7ff9d009b7a0 con 0x7ff9e00701a0 2024-09-13T08:41:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.682+0000 7ff9dea00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 0x7ff9c007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.683+0000 7ff9dea00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 0x7ff9c007b070 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff9e015ba60 tx=0x7ff9cc005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:20.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.686+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff9d0063bf0 con 0x7ff9e00701a0 2024-09-13T08:41:20.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:20.955+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7ff9e006f1b0 con 0x7ff9e00701a0 2024-09-13T08:41:21.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.084+0000 7ff9d7400640 1 -- 172.21.15.31:0/3115506648 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v229) ==== 161+0+0 (secure 0 0 0) 0x7ff9d00634f0 con 0x7ff9e00701a0 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 msgr2=0x7ff9c007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 0x7ff9c007b070 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff9e015ba60 tx=0x7ff9cc005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 msgr2=0x7ff9e01f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000e9f0 tx=0x7ff9d000eeb0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 shutdown_connections 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff9c0078bb0 0x7ff9c007b070 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff9e00ccb00 0x7ff9e00cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9e00cc720 0x7ff9e01f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 --2- 172.21.15.31:0/3115506648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e00701a0 0x7ff9e01f4fd0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 >> 172.21.15.31:0/3115506648 conn(0x7ff9e006cb00 msgr2=0x7ff9e006e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 shutdown_connections 2024-09-13T08:41:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.087+0000 7ff9e4e00640 1 -- 172.21.15.31:0/3115506648 wait complete. 2024-09-13T08:41:21.104 INFO:tasks.cephfs_test_runner:Validate handcrafted .meta file on legacy subvol root doesn't break the system ... ok 2024-09-13T08:41:21.104 INFO:tasks.cephfs_test_runner:Starting test: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:41:21.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 -- 172.21.15.31:0/4270813265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415f9e0 msgr2=0x7fe6d415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/4270813265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415f9e0 0x7fe6d415fe20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40cc8c0 tx=0x7fe6c002a560 comp rx=0 tx=0).stop 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 -- 172.21.15.31:0/4270813265 shutdown_connections 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/4270813265 >> v2:172.21.15.45:3300/0 conn(0x7fe6d40ce3c0 0x7fe6d40ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/4270813265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415f9e0 0x7fe6d415fe20 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/4270813265 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d415bd70 0x7fe6d40cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 -- 172.21.15.31:0/4270813265 >> 172.21.15.31:0/4270813265 conn(0x7fe6d40cc910 msgr2=0x7fe6d40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 -- 172.21.15.31:0/4270813265 shutdown_connections 2024-09-13T08:41:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.265+0000 7fe6dbe00640 1 -- 172.21.15.31:0/4270813265 wait complete. 2024-09-13T08:41:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.266+0000 7fe6dbe00640 1 Processor -- start 2024-09-13T08:41:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.266+0000 7fe6dbe00640 1 -- start start 2024-09-13T08:41:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d40ce3c0 0x7fe6d409d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6d415f9e0 0x7fe6d40b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d40cb9d0 con 0x7fe6d415bd70 2024-09-13T08:41:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d40cb850 con 0x7fe6d415f9e0 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.267+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d40cbc50 con 0x7fe6d40ce3c0 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40184/0 (socket says 172.21.15.31:40184) 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 -- 172.21.15.31:0/193926898 learned_addr learned my addr 172.21.15.31:0/193926898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d9400640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d40ce3c0 0x7fe6d409d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d9e00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6d415f9e0 0x7fe6d40b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 -- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d40ce3c0 msgr2=0x7fe6d409d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d40ce3c0 0x7fe6d409d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 -- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6d415f9e0 msgr2=0x7fe6d40b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6d415f9e0 0x7fe6d40b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 -- 172.21.15.31:0/193926898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d406e670 con 0x7fe6d415bd70 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.268+0000 7fe6d8a00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0034040 tx=0x7fe6c0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.269+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c003e030 con 0x7fe6d415bd70 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.269+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6c0003e00 con 0x7fe6d415bd70 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.269+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0035040 con 0x7fe6d415bd70 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.269+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d406e950 con 0x7fe6d415bd70 2024-09-13T08:41:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.269+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d40c2a90 con 0x7fe6d415bd70 2024-09-13T08:41:21.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.270+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d409d8d0 con 0x7fe6d415bd70 2024-09-13T08:41:21.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.271+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fe6c0041030 con 0x7fe6d415bd70 2024-09-13T08:41:21.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.272+0000 7fe6d1600640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 0x7fe6b407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.273+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7fe6c00bb240 con 0x7fe6d415bd70 2024-09-13T08:41:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.273+0000 7fe6d9400640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 0x7fe6b407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.274+0000 7fe6d9400640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 0x7fe6b407b070 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8005fd0 tx=0x7fe6c8009680 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:21.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.277+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe6c0083000 con 0x7fe6d415bd70 2024-09-13T08:41:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.477+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6d4072710 con 0x7fe6d415bd70 2024-09-13T08:41:21.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.478+0000 7fe6d1600640 1 -- 172.21.15.31:0/193926898 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+12982 (secure 0 0 0) 0x7fe6c0087a30 con 0x7fe6d415bd70 2024-09-13T08:41:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:21.479 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:20.077882+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6835/707614758":"2024-09-14T08:41:17.068216+0000","172.21.15.31:6833/707614758":"2024-09-14T08:41:17.068216+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.480+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 msgr2=0x7fe6b407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.480+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 0x7fe6b407b070 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8005fd0 tx=0x7fe6c8009680 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.480+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 msgr2=0x7fe6d409b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.480+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0034040 tx=0x7fe6c0003910 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 shutdown_connections 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6d415f9e0 0x7fe6d40b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6d415bd70 0x7fe6d409b100 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fe6b4078bb0 0x7fe6b407b070 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 --2- 172.21.15.31:0/193926898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe6d40ce3c0 0x7fe6d409d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 >> 172.21.15.31:0/193926898 conn(0x7fe6d40cc910 msgr2=0x7fe6d40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 shutdown_connections 2024-09-13T08:41:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.481+0000 7fe6dbe00640 1 -- 172.21.15.31:0/193926898 wait complete. 2024-09-13T08:41:21.498 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:41:21.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 -- 172.21.15.31:0/3258852777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c152a30 msgr2=0x7ff79c0aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 --2- 172.21.15.31:0/3258852777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c152a30 0x7ff79c0aea00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0738f0 tx=0x7ff78c02a560 comp rx=0 tx=0).stop 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 -- 172.21.15.31:0/3258852777 shutdown_connections 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 --2- 172.21.15.31:0/3258852777 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c0740e0 0x7ff79c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 --2- 172.21.15.31:0/3258852777 >> v2:172.21.15.45:3300/0 conn(0x7ff79c072450 0x7ff79c073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 --2- 172.21.15.31:0/3258852777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c152a30 0x7ff79c0aea00 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 -- 172.21.15.31:0/3258852777 >> 172.21.15.31:0/3258852777 conn(0x7ff79c0cf9b0 msgr2=0x7ff79c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 -- 172.21.15.31:0/3258852777 shutdown_connections 2024-09-13T08:41:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.661+0000 7ff7a1400640 1 -- 172.21.15.31:0/3258852777 wait complete. 2024-09-13T08:41:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.663+0000 7ff7a1400640 1 Processor -- start 2024-09-13T08:41:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.663+0000 7ff7a1400640 1 -- start start 2024-09-13T08:41:21.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c072450 0x7ff79c0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff79c152a30 0x7ff79c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c0b4f00 con 0x7ff79c0740e0 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c0b4d80 con 0x7ff79c152a30 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a1400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff79c0b5080 con 0x7ff79c072450 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40206/0 (socket says 172.21.15.31:40206) 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 -- 172.21.15.31:0/149747221 learned_addr learned my addr 172.21.15.31:0/149747221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff79be00640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c072450 0x7ff79c0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff7a0a00640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff79c152a30 0x7ff79c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 -- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c072450 msgr2=0x7ff79c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c072450 0x7ff79c0cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 -- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff79c152a30 msgr2=0x7ff79c1554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff79c152a30 0x7ff79c1554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.664+0000 7ff793400640 1 -- 172.21.15.31:0/149747221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79c155a00 con 0x7ff79c0740e0 2024-09-13T08:41:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff793400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff78400d980 tx=0x7ff78400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784017030 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff784009b20 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784004020 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79c155c80 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.665+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79c1f44a0 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.666+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79c0b4840 con 0x7ff79c0740e0 2024-09-13T08:41:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.667+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7ff7840041c0 con 0x7ff79c0740e0 2024-09-13T08:41:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.668+0000 7ff798c00640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 0x7ff77807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.669+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7ff78409acb0 con 0x7ff79c0740e0 2024-09-13T08:41:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.669+0000 7ff79be00640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 0x7ff77807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.670+0000 7ff79be00640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 0x7ff77807b070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff78c033000 tx=0x7ff78c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.672+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff79c155a00 con 0x7ff79c0740e0 2024-09-13T08:41:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.867+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff79c0008d0 con 0x7ff79c0740e0 2024-09-13T08:41:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.867+0000 7ff798c00640 1 -- 172.21.15.31:0/149747221 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7ff784062a70 con 0x7ff79c0740e0 2024-09-13T08:41:21.868 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 msgr2=0x7ff77807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 0x7ff77807b070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff78c033000 tx=0x7ff78c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 msgr2=0x7ff79c1f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff78400d980 tx=0x7ff78400de40 comp rx=0 tx=0).stop 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 shutdown_connections 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff79c152a30 0x7ff79c1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff79c0740e0 0x7ff79c1f4ca0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff778078bb0 0x7ff77807b070 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 --2- 172.21.15.31:0/149747221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff79c072450 0x7ff79c0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:21.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 >> 172.21.15.31:0/149747221 conn(0x7ff79c0cf9b0 msgr2=0x7ff79c0cde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:21.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 shutdown_connections 2024-09-13T08:41:21.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:21.870+0000 7ff7a1400640 1 -- 172.21.15.31:0/149747221 wait complete. 2024-09-13T08:41:21.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-13T08:41:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 -- 172.21.15.31:0/429039854 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 msgr2=0x7f39380cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 --2- 172.21.15.31:0/429039854 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 0x7f39380cbc50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3930005fd0 tx=0x7f393002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:41:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 -- 172.21.15.31:0/429039854 shutdown_connections 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 --2- 172.21.15.31:0/429039854 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 0x7f39380cbc50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 --2- 172.21.15.31:0/429039854 >> v2:172.21.15.45:3300/0 conn(0x7f393815bd70 0x7f39380cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 -- 172.21.15.31:0/429039854 >> 172.21.15.31:0/429039854 conn(0x7f39380cc910 msgr2=0x7f39380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 -- 172.21.15.31:0/429039854 shutdown_connections 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.047+0000 7f393da00640 1 -- 172.21.15.31:0/429039854 wait complete. 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.049+0000 7f393da00640 1 Processor -- start 2024-09-13T08:41:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.049+0000 7f393da00640 1 -- start start 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 0x7f393806fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 0x7f393806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39380cec50 con 0x7f393815bd70 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39380cead0 con 0x7f393815f9e0 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f393da00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39380cb850 con 0x7f393815fe20 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40210/0 (socket says 172.21.15.31:40210) 2024-09-13T08:41:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 -- 172.21.15.31:0/678029569 learned_addr learned my addr 172.21.15.31:0/678029569 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 -- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 msgr2=0x7f393806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 0x7f393806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f392e000640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 0x7f393806fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 -- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 msgr2=0x7f393806fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 0x7f393806fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f3936a00640 1 -- 172.21.15.31:0/678029569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39380cb4f0 con 0x7f393815bd70 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.050+0000 7f392e000640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 0x7f393806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f3936a00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f392800e9f0 tx=0x7f392800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392800cd80 con 0x7f393815bd70 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3928010430 con 0x7f393815bd70 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39380b4a00 con 0x7f393815bd70 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3928010840 con 0x7f393815bd70 2024-09-13T08:41:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.051+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39380cb200 con 0x7f393815bd70 2024-09-13T08:41:22.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.052+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39380cf850 con 0x7f393815bd70 2024-09-13T08:41:22.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.053+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f39280109e0 con 0x7f393815bd70 2024-09-13T08:41:22.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.054+0000 7f392f400640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 0x7f391407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:22.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.054+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7f392809b7b0 con 0x7f393815bd70 2024-09-13T08:41:22.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.055+0000 7f392e000640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 0x7f391407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:22.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.056+0000 7f392e000640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 0x7f391407b070 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3920001220 tx=0x7f3920009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.058+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3928063d10 con 0x7f393815bd70 2024-09-13T08:41:22.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:22.329+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f39380728c0 con 0x7f393815bd70 2024-09-13T08:41:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.092+0000 7f392f400640 1 -- 172.21.15.31:0/678029569 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v231) ==== 134+0+0 (secure 0 0 0) 0x7f3928063610 con 0x7f393815bd70 2024-09-13T08:41:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 msgr2=0x7f391407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 0x7f391407b070 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3920001220 tx=0x7f3920009040 comp rx=0 tx=0).stop 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 msgr2=0x7f393815b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f392800e9f0 tx=0x7f392800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 shutdown_connections 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3914078bb0 0x7f391407b070 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393815fe20 0x7f393806ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393815f9e0 0x7f393806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 --2- 172.21.15.31:0/678029569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393815bd70 0x7f393815b370 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 >> 172.21.15.31:0/678029569 conn(0x7f39380cc910 msgr2=0x7f39380ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 shutdown_connections 2024-09-13T08:41:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.096+0000 7f393da00640 1 -- 172.21.15.31:0/678029569 wait complete. 2024-09-13T08:41:23.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:41:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 -- 172.21.15.31:0/1630256669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540ca5c0 msgr2=0x7f64540cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 --2- 172.21.15.31:0/1630256669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540ca5c0 0x7f64540cc300 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f643c009830 tx=0x7f643c02a510 comp rx=0 tx=0).stop 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 -- 172.21.15.31:0/1630256669 shutdown_connections 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 --2- 172.21.15.31:0/1630256669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f64540b1910 0x7f64540b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 --2- 172.21.15.31:0/1630256669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540ca5c0 0x7f64540cc300 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 -- 172.21.15.31:0/1630256669 >> 172.21.15.31:0/1630256669 conn(0x7f64540cc910 msgr2=0x7f64540cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 -- 172.21.15.31:0/1630256669 shutdown_connections 2024-09-13T08:41:23.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.282+0000 7f645a000640 1 -- 172.21.15.31:0/1630256669 wait complete. 2024-09-13T08:41:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.284+0000 7f645a000640 1 Processor -- start 2024-09-13T08:41:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.284+0000 7f645a000640 1 -- start start 2024-09-13T08:41:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64540ca5c0 0x7f645415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f645415dfc0 0x7f645415b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6454073b10 con 0x7f64540b1910 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6454073990 con 0x7f64540ca5c0 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f645a000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6454073c90 con 0x7f645415dfc0 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.285+0000 7f6453400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40222/0 (socket says 172.21.15.31:40222) 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 -- 172.21.15.31:0/132869394 learned_addr learned my addr 172.21.15.31:0/132869394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f644aa00640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64540ca5c0 0x7f645415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453e00640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f645415dfc0 0x7f645415b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 -- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f645415dfc0 msgr2=0x7f645415b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f645415dfc0 0x7f645415b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 -- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64540ca5c0 msgr2=0x7f645415aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64540ca5c0 0x7f645415aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 -- 172.21.15.31:0/132869394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64541e8740 con 0x7f64540b1910 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.286+0000 7f6453400640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f644400c3d0 tx=0x7f644400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.287+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444011790 con 0x7f64540b1910 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.287+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541e8a20 con 0x7f64540b1910 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.287+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6444017070 con 0x7f64540b1910 2024-09-13T08:41:23.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.287+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541e8fe0 con 0x7f64540b1910 2024-09-13T08:41:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.287+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444011b70 con 0x7f64540b1910 2024-09-13T08:41:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.288+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f642c005290 con 0x7f64540b1910 2024-09-13T08:41:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.289+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f6444011d10 con 0x7f64540b1910 2024-09-13T08:41:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.290+0000 7f644be00640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 0x7f643007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.290+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7f644409a790 con 0x7f64540b1910 2024-09-13T08:41:23.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.290+0000 7f644aa00640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 0x7f643007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.291+0000 7f644aa00640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 0x7f643007b070 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f643c034000 tx=0x7f643c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.294+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f64540cbe40 con 0x7f64540b1910 2024-09-13T08:41:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.490+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f642c005850 con 0x7f64540b1910 2024-09-13T08:41:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.490+0000 7f644be00640 1 -- 172.21.15.31:0/132869394 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+1998 (secure 0 0 0) 0x7f64440625e0 con 0x7f64540b1910 2024-09-13T08:41:23.491 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:41:23.491 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:41:23.491 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:41:23.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:41:23.492 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:41:23.493 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:41:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 msgr2=0x7f643007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 0x7f643007b070 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f643c034000 tx=0x7f643c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 msgr2=0x7f64540b5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f644400c3d0 tx=0x7f644400c890 comp rx=0 tx=0).stop 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 shutdown_connections 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f6430078bb0 0x7f643007b070 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f645415dfc0 0x7f645415b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64540ca5c0 0x7f645415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 --2- 172.21.15.31:0/132869394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64540b1910 0x7f64540b5440 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 >> 172.21.15.31:0/132869394 conn(0x7f64540cc910 msgr2=0x7f645415d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 shutdown_connections 2024-09-13T08:41:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.493+0000 7f645a000640 1 -- 172.21.15.31:0/132869394 wait complete. 2024-09-13T08:41:23.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:41:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3148283321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac0b5300 msgr2=0x7fb2ac0cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3148283321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac0b5300 0x7fb2ac0cc3e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb294009830 tx=0x7fb29402a480 comp rx=0 tx=0).stop 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3148283321 shutdown_connections 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3148283321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac0b5300 0x7fb2ac0cc3e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3148283321 >> v2:172.21.15.45:3300/0 conn(0x7fb2ac15bb70 0x7fb2ac06fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3148283321 >> 172.21.15.31:0/3148283321 conn(0x7fb2ac0b1520 msgr2=0x7fb2ac0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3148283321 shutdown_connections 2024-09-13T08:41:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.675+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3148283321 wait complete. 2024-09-13T08:41:23.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.677+0000 7fb2b1c00640 1 Processor -- start 2024-09-13T08:41:23.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.677+0000 7fb2b1c00640 1 -- start start 2024-09-13T08:41:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac0cc920 0x7fb2ac15c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bb70 0x7fb2ac1f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac0b29a0 con 0x7fb2ac0b5300 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac0b2820 con 0x7fb2ac0cc920 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ac0ca730 con 0x7fb2ac15bb70 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2abe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2abe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40236/0 (socket says 172.21.15.31:40236) 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2abe00640 1 -- 172.21.15.31:0/3991395259 learned_addr learned my addr 172.21.15.31:0/3991395259 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2b1200640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bb70 0x7fb2ac1f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.678+0000 7fb2a3e00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac0cc920 0x7fb2ac15c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 -- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bb70 msgr2=0x7fb2ac1f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bb70 0x7fb2ac1f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 -- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac0cc920 msgr2=0x7fb2ac15c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac0cc920 0x7fb2ac15c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 -- 172.21.15.31:0/3991395259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac15cc60 con 0x7fb2ac0b5300 2024-09-13T08:41:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2abe00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e790 tx=0x7fb29c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00eea0 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac1f8410 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb29c00cdb0 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c010690 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.679+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac1f89d0 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.681+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ac0cabb0 con 0x7fb2ac0b5300 2024-09-13T08:41:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.682+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fb29c010830 con 0x7fb2ac0b5300 2024-09-13T08:41:23.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.683+0000 7fb2a8c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 0x7fb28807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:23.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.683+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7fb29c09b600 con 0x7fb2ac0b5300 2024-09-13T08:41:23.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.684+0000 7fb2a3e00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 0x7fb28807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.684+0000 7fb2a3e00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 0x7fb28807b070 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb294034000 tx=0x7fb294004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:23.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.687+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb29c063340 con 0x7fb2ac0b5300 2024-09-13T08:41:23.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.882+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb2ac0cc550 con 0x7fb2ac0b5300 2024-09-13T08:41:23.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.886+0000 7fb2a8c00640 1 -- 172.21.15.31:0/3991395259 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7fb29c067d70 con 0x7fb2ac0b5300 2024-09-13T08:41:23.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 msgr2=0x7fb28807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 0x7fb28807b070 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb294034000 tx=0x7fb294004730 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 msgr2=0x7fb2ac0ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e790 tx=0x7fb29c00ec50 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 shutdown_connections 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb288078bb0 0x7fb28807b070 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb2ac15bb70 0x7fb2ac1f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2ac0cc920 0x7fb2ac15c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 --2- 172.21.15.31:0/3991395259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2ac0b5300 0x7fb2ac0ae0b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 >> 172.21.15.31:0/3991395259 conn(0x7fb2ac0b1520 msgr2=0x7fb2ac0b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 shutdown_connections 2024-09-13T08:41:23.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:23.888+0000 7fb2b1c00640 1 -- 172.21.15.31:0/3991395259 wait complete. 2024-09-13T08:41:23.906 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:41:23.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:23.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:24.096 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:41:24.096 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:24.096 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:24.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:41:24.133 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:24.133 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:24.171 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:41:24.171 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:24.171 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:24.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:41:24.210 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:24.211 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:24.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:41:24.247 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:41:24.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 -- 172.21.15.31:0/2986107653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 msgr2=0x7f02fc06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 --2- 172.21.15.31:0/2986107653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc06f9b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009830 tx=0x7f02e402a510 comp rx=0 tx=0).stop 2024-09-13T08:41:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 -- 172.21.15.31:0/2986107653 shutdown_connections 2024-09-13T08:41:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 --2- 172.21.15.31:0/2986107653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 0x7f02fc152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 --2- 172.21.15.31:0/2986107653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc06f9b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 -- 172.21.15.31:0/2986107653 >> 172.21.15.31:0/2986107653 conn(0x7f02fc06cb00 msgr2=0x7f02fc06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 -- 172.21.15.31:0/2986107653 shutdown_connections 2024-09-13T08:41:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.412+0000 7f0301e00640 1 -- 172.21.15.31:0/2986107653 wait complete. 2024-09-13T08:41:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.414+0000 7f0301e00640 1 Processor -- start 2024-09-13T08:41:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.414+0000 7f0301e00640 1 -- start start 2024-09-13T08:41:24.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 0x7f02fc0af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc152bc0 con 0x7f02fc0cc720 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc0703e0 con 0x7f02fc0b52d0 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f02fc152d40 con 0x7f02fc0ccb00 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f02f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f02f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40252/0 (socket says 172.21.15.31:40252) 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f02f3e00640 1 -- 172.21.15.31:0/1975770095 learned_addr learned my addr 172.21.15.31:0/1975770095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0300a00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.415+0000 7f0301400640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 0x7f02fc0af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 -- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 msgr2=0x7f02fc0af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 0x7f02fc0af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 -- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 msgr2=0x7f02fc0ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc0ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 -- 172.21.15.31:0/1975770095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc1f7e00 con 0x7f02fc0cc720 2024-09-13T08:41:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f3e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009950 tx=0x7f02e4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4040070 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02e4005310 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc1f80e0 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.416+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4005c90 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.417+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1f86a0 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.418+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc1542b0 con 0x7f02fc0cc720 2024-09-13T08:41:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.419+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f02e4039040 con 0x7f02fc0cc720 2024-09-13T08:41:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.420+0000 7f02f9600640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 0x7f02d807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.421+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7f02e40bbcc0 con 0x7f02fc0cc720 2024-09-13T08:41:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.421+0000 7f0300a00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 0x7f02d807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.422+0000 7f0300a00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 0x7f02d807b070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02ec007fd0 tx=0x7f02ec001410 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:24.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.425+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f02fc1542b0 con 0x7f02fc0cc720 2024-09-13T08:41:24.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.662+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02fc06f9b0 con 0x7f02fc0cc720 2024-09-13T08:41:24.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.663+0000 7f02f9600640 1 -- 172.21.15.31:0/1975770095 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+3905 (secure 0 0 0) 0x7f02e40c2080 con 0x7f02fc0cc720 2024-09-13T08:41:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:24.664 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"btime":"2024-09-13T08:41:18:070476+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/558916845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":558916845},{"type":"v1","addr":"172.21.15.31:6834","nonce":558916845}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1497446494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1497446494},{"type":"v1","addr":"172.21.15.45:6837","nonce":1497446494}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1649811324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1649811324},{"type":"v1","addr":"172.21.15.45:6836","nonce":1649811324}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/336443930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":336443930},{"type":"v1","addr":"172.21.15.31:6835","nonce":336443930}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[]} 2024-09-13T08:41:24.664 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2024-09-13T08:41:24.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 msgr2=0x7f02d807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 0x7f02d807b070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02ec007fd0 tx=0x7f02ec001410 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 msgr2=0x7f02fc0cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009950 tx=0x7f02e4032c10 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 shutdown_connections 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f02d8078bb0 0x7f02d807b070 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f02fc0ccb00 0x7f02fc0af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02fc0cc720 0x7f02fc0cb5f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 --2- 172.21.15.31:0/1975770095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02fc0b52d0 0x7f02fc0ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 >> 172.21.15.31:0/1975770095 conn(0x7f02fc06cb00 msgr2=0x7f02fc153ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.665+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 shutdown_connections 2024-09-13T08:41:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.666+0000 7f0301e00640 1 -- 172.21.15.31:0/1975770095 wait complete. 2024-09-13T08:41:24.683 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:41:24.683 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:41:24.683 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:41:24.683 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:41:24.684 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:41:24.684 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:41:24.684 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:41:24.684 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:41:24.684 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:41:24.684 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:41:24.685 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:41:24.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:41:24.685 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:41:24.685 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:41:24.685 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:41:24.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:41:24.697 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:41:24.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:41:24.700 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:41:24.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:41:24.702 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:41:24.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:41:24.703 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:41:24.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:41:24.744 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:41:24.745 DEBUG:teuthology.parallel:result is None 2024-09-13T08:41:24.747 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:41:24.748 DEBUG:teuthology.parallel:result is None 2024-09-13T08:41:24.749 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:41:24.749 DEBUG:teuthology.parallel:result is None 2024-09-13T08:41:24.751 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:41:24.751 DEBUG:teuthology.parallel:result is None 2024-09-13T08:41:24.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:41:24.881 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:41:24.882 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:41:24.886 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:41:24.886 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:41:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.927+0000 7f423be00640 1 -- 172.21.15.31:0/3063687252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4234164dd0 msgr2=0x7f42341671c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.927+0000 7f423be00640 1 --2- 172.21.15.31:0/3063687252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4234164dd0 0x7f42341671c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f422c005fd0 tx=0x7f422c02a8d0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.927+0000 7f423be00640 1 -- 172.21.15.31:0/3063687252 shutdown_connections 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.927+0000 7f423be00640 1 --2- 172.21.15.31:0/3063687252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4234164dd0 0x7f42341671c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.927+0000 7f423be00640 1 -- 172.21.15.31:0/3063687252 >> 172.21.15.31:0/3063687252 conn(0x7f42340b18f0 msgr2=0x7f42340b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.928+0000 7f423be00640 1 -- 172.21.15.31:0/3063687252 shutdown_connections 2024-09-13T08:41:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.928+0000 7f423be00640 1 -- 172.21.15.31:0/3063687252 wait complete. 2024-09-13T08:41:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.929+0000 7f423be00640 1 Processor -- start 2024-09-13T08:41:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.929+0000 7f423be00640 1 -- start start 2024-09-13T08:41:24.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42340ced20 0x7f42340b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f423415be30 0x7f42340b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42340749f0 con 0x7f4234164dd0 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4234074870 con 0x7f42340ced20 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.930+0000 7f423be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4234167700 con 0x7f423415be30 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56996/0 (socket says 172.21.15.31:56996) 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 -- 172.21.15.31:0/2819493603 learned_addr learned my addr 172.21.15.31:0/2819493603 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4238a00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f423415be30 0x7f42340b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239400640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42340ced20 0x7f42340b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 -- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f423415be30 msgr2=0x7f42340b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f423415be30 0x7f42340b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 -- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42340ced20 msgr2=0x7f42340b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:24.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42340ced20 0x7f42340b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.931+0000 7f4239e00640 1 -- 172.21.15.31:0/2819493603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423414e100 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f4239e00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f422c035040 tx=0x7f422c032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c041070 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f422c03ca20 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c037040 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423414e370 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.932+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423414e610 con 0x7f4234164dd0 2024-09-13T08:41:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.934+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42340ce340 con 0x7f4234164dd0 2024-09-13T08:41:24.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.935+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f422c044070 con 0x7f4234164dd0 2024-09-13T08:41:24.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.936+0000 7f4231600640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 0x7f421407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:24.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.936+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4943+0+0 (secure 0 0 0) 0x7f422c0bbd10 con 0x7f4234164dd0 2024-09-13T08:41:24.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.937+0000 7f4239400640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 0x7f421407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:24.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.937+0000 7f4239400640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 0x7f421407b070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f42340af4b0 tx=0x7f4228001720 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:24.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:24.939+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f422c083ba0 con 0x7f4234164dd0 2024-09-13T08:41:25.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:25.154+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f42340b25f0 con 0x7f4234164dd0 2024-09-13T08:41:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.124+0000 7f4231600640 1 -- 172.21.15.31:0/2819493603 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v72) ==== 97+0+0 (secure 0 0 0) 0x7f422c0885d0 con 0x7f4234164dd0 2024-09-13T08:41:26.125 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:41:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 msgr2=0x7f421407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 0x7f421407b070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f42340af4b0 tx=0x7f4228001720 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 msgr2=0x7f4234150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f422c035040 tx=0x7f422c032eb0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 shutdown_connections 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4234164dd0 0x7f4234150830 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4214078bb0 0x7f421407b070 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f423415be30 0x7f42340b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 --2- 172.21.15.31:0/2819493603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42340ced20 0x7f42340b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 >> 172.21.15.31:0/2819493603 conn(0x7f42340b18f0 msgr2=0x7f42341545e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 shutdown_connections 2024-09-13T08:41:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.127+0000 7f423be00640 1 -- 172.21.15.31:0/2819493603 wait complete. 2024-09-13T08:41:26.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:41:26.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 -- 172.21.15.31:0/315110476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15bd70 msgr2=0x7fb3ac0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/315110476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac0cf850 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009830 tx=0x7fb39c02a510 comp rx=0 tx=0).stop 2024-09-13T08:41:26.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 -- 172.21.15.31:0/315110476 shutdown_connections 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/315110476 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac15f9e0 0x7fb3ac15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/315110476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac0cf850 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 -- 172.21.15.31:0/315110476 >> 172.21.15.31:0/315110476 conn(0x7fb3ac0cc910 msgr2=0x7fb3ac0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 -- 172.21.15.31:0/315110476 shutdown_connections 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.311+0000 7fb3b1c00640 1 -- 172.21.15.31:0/315110476 wait complete. 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.313+0000 7fb3b1c00640 1 Processor -- start 2024-09-13T08:41:26.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.313+0000 7fb3b1c00640 1 -- start start 2024-09-13T08:41:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac0ce3c0 0x7fb3ac15da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac0cb9d0 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac0cb850 con 0x7fb3ac15bd70 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3b1c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ac0cc360 con 0x7fb3ac0ce3c0 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57012/0 (socket says 172.21.15.31:57012) 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3ab400640 1 -- 172.21.15.31:0/357632866 learned_addr learned my addr 172.21.15.31:0/357632866 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3a2000640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.314+0000 7fb3aaa00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac0ce3c0 0x7fb3ac15da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 -- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac0ce3c0 msgr2=0x7fb3ac15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac0ce3c0 0x7fb3ac15da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 -- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac15bd70 msgr2=0x7fb3ac15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 -- 172.21.15.31:0/357632866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac154840 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3ab400640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c770 tx=0x7fb3a400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4017070 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3a4005070 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac15b450 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40054c0 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.315+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac15cfb0 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.317+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ac0cf850 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.318+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fb3a401a070 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.320+0000 7fb3a3400640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 0x7fb38807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.320+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4857+0+0 (secure 0 0 0) 0x7fb3a409a310 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.320+0000 7fb3aaa00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 0x7fb38807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.321+0000 7fb3aaa00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 0x7fb38807b070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb39c034000 tx=0x7fb39c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:26.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.324+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb3a40627b0 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.590+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb3ac069a50 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.590+0000 7fb3a3400640 1 -- 172.21.15.31:0/357632866 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5617 (secure 0 0 0) 0x7fb3a40620b0 con 0x7fb3ac15f9e0 2024-09-13T08:41:26.591 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:26.591 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:41:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:41:26.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.596 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:41:26.598 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:41:26.600 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:41:26.601 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.602 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.603 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.alice", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCe+uNmre3GNhAAhD2EPuS8cMbQjF+Ssa06fg==", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol42010", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-13T08:41:26.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:41:26.605 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:26.606 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 msgr2=0x7fb38807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 0x7fb38807b070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb39c034000 tx=0x7fb39c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 msgr2=0x7fb3ac154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c770 tx=0x7fb3a400cc30 comp rx=0 tx=0).stop 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 shutdown_connections 2024-09-13T08:41:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3ac15f9e0 0x7fb3ac154300 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac15bd70 0x7fb3ac15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb388078bb0 0x7fb38807b070 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 --2- 172.21.15.31:0/357632866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3ac0ce3c0 0x7fb3ac15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 >> 172.21.15.31:0/357632866 conn(0x7fb3ac0cc910 msgr2=0x7fb3ac0acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 shutdown_connections 2024-09-13T08:41:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.593+0000 7fb3b1c00640 1 -- 172.21.15.31:0/357632866 wait complete. 2024-09-13T08:41:26.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.alice 2024-09-13T08:41:26.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 -- 172.21.15.31:0/3801691635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b0073860 msgr2=0x7f46b0073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3801691635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b0073860 0x7f46b0073cf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f46a8005fd0 tx=0x7f46a802aa90 comp rx=0 tx=0).stop 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 -- 172.21.15.31:0/3801691635 shutdown_connections 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3801691635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b0073860 0x7f46b0073cf0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3801691635 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b00ced20 0x7f46b00ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 -- 172.21.15.31:0/3801691635 >> 172.21.15.31:0/3801691635 conn(0x7f46b00b18f0 msgr2=0x7f46b00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 -- 172.21.15.31:0/3801691635 shutdown_connections 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.769+0000 7f46b4a00640 1 -- 172.21.15.31:0/3801691635 wait complete. 2024-09-13T08:41:26.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.771+0000 7f46b4a00640 1 Processor -- start 2024-09-13T08:41:26.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.771+0000 7f46b4a00640 1 -- start start 2024-09-13T08:41:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46b0073860 0x7f46b00b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b015be30 0x7f46b00c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0074230 con 0x7f46b00ced20 2024-09-13T08:41:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46b00ce9e0 con 0x7f46b0073860 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46b4a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f46b00743b0 con 0x7f46b015be30 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46ad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46ad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57024/0 (socket says 172.21.15.31:57024) 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46ad400640 1 -- 172.21.15.31:0/3041390530 learned_addr learned my addr 172.21.15.31:0/3041390530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46ae800640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b015be30 0x7f46b00c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.772+0000 7f46ade00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46b0073860 0x7f46b00b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 -- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b015be30 msgr2=0x7f46b00c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b015be30 0x7f46b00c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 -- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46b0073860 msgr2=0x7f46b00b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46b0073860 0x7f46b00b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 -- 172.21.15.31:0/3041390530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b00b3de0 con 0x7f46b00ced20 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46ad400640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f469c00d950 tx=0x7f469c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:26.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c017070 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f469c009d20 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b00b40c0 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c0106c0 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.773+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b00b4680 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.775+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b00748a0 con 0x7f46b00ced20 2024-09-13T08:41:26.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.776+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f469c010860 con 0x7f46b00ced20 2024-09-13T08:41:26.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.778+0000 7f46a6000640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 0x7f469007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:26.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.778+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4857+0+0 (secure 0 0 0) 0x7f469c09b040 con 0x7f46b00ced20 2024-09-13T08:41:26.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.778+0000 7f46ade00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 0x7f469007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:26.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.779+0000 7f46ade00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 0x7f469007b0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46a0001720 tx=0x7f46a0009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:26.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:26.782+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f469c062d50 con 0x7f46b00ced20 2024-09-13T08:41:27.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.051+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.alice"} v 0) -- 0x7f46b00cc6b0 con 0x7f46b00ced20 2024-09-13T08:41:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.053+0000 7f46a6000640 1 -- 172.21.15.31:0/3041390530 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.alice"}]=0 v13) ==== 82+0+0 (secure 0 0 0) 0x7f469c067780 con 0x7f46b00ced20 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.056+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 msgr2=0x7f469007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.056+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 0x7f469007b0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46a0001720 tx=0x7f46a0009040 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.056+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 msgr2=0x7f46b014d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.056+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f469c00d950 tx=0x7f469c00de10 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 shutdown_connections 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4690078be0 0x7f469007b0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46b015be30 0x7f46b00c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46b00ced20 0x7f46b014d5d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 --2- 172.21.15.31:0/3041390530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46b0073860 0x7f46b00b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 >> 172.21.15.31:0/3041390530 conn(0x7f46b00b18f0 msgr2=0x7f46b006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 shutdown_connections 2024-09-13T08:41:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.057+0000 7f46b4a00640 1 -- 172.21.15.31:0/3041390530 wait complete. 2024-09-13T08:41:27.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:41:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 -- 172.21.15.31:0/700095331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba540d0e80 msgr2=0x7fba540cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 --2- 172.21.15.31:0/700095331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba540d0e80 0x7fba540cbb00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fba40009830 tx=0x7fba4002a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 -- 172.21.15.31:0/700095331 shutdown_connections 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 --2- 172.21.15.31:0/700095331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba540d0e80 0x7fba540cbb00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 -- 172.21.15.31:0/700095331 >> 172.21.15.31:0/700095331 conn(0x7fba540d0360 msgr2=0x7fba540d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 -- 172.21.15.31:0/700095331 shutdown_connections 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.235+0000 7fba5a200640 1 -- 172.21.15.31:0/700095331 wait complete. 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.237+0000 7fba5a200640 1 Processor -- start 2024-09-13T08:41:27.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.237+0000 7fba5a200640 1 -- start start 2024-09-13T08:41:27.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba540cc040 0x7fba540ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba5415e2d0 0x7fba540c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba5415e830 con 0x7fba540d0e80 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba5415e6b0 con 0x7fba5415e2d0 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba5a200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fba540cca20 con 0x7fba540cc040 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57026/0 (socket says 172.21.15.31:57026) 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 -- 172.21.15.31:0/3369442483 learned_addr learned my addr 172.21.15.31:0/3369442483 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba59800640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba5415e2d0 0x7fba540c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba58e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba540cc040 0x7fba540ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 -- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba540cc040 msgr2=0x7fba540ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba540cc040 0x7fba540ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 -- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba5415e2d0 msgr2=0x7fba540c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba5415e2d0 0x7fba540c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.238+0000 7fba53e00640 1 -- 172.21.15.31:0/3369442483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba540c4ae0 con 0x7fba540d0e80 2024-09-13T08:41:27.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba53e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fba40000c00 tx=0x7fba40032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40040070 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba40003af0 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5406fa80 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40032e40 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.239+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba54070040 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.240+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba540b4470 con 0x7fba540d0e80 2024-09-13T08:41:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.241+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fba40039040 con 0x7fba540d0e80 2024-09-13T08:41:27.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.242+0000 7fba50c00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 0x7fba3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.242+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4857+0+0 (secure 0 0 0) 0x7fba400bbff0 con 0x7fba540d0e80 2024-09-13T08:41:27.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.243+0000 7fba58e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 0x7fba3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.244+0000 7fba58e00640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 0x7fba3407b0a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fba48000f60 tx=0x7fba48009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:27.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.246+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fba400844e0 con 0x7fba540d0e80 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.445+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fba5406f850 con 0x7fba540d0e80 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.445+0000 7fba50c00640 1 -- 172.21.15.31:0/3369442483 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7fba40083de0 con 0x7fba540d0e80 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:41:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:41:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:27.447 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.447+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 msgr2=0x7fba3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.447+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 0x7fba3407b0a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fba48000f60 tx=0x7fba48009040 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.447+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 msgr2=0x7fba540c3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.447+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fba40000c00 tx=0x7fba40032b90 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 shutdown_connections 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba5415e2d0 0x7fba540c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba540d0e80 0x7fba540c3e90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fba34078be0 0x7fba3407b0a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 --2- 172.21.15.31:0/3369442483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba540cc040 0x7fba540ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 >> 172.21.15.31:0/3369442483 conn(0x7fba540d0360 msgr2=0x7fba5415d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:27.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 shutdown_connections 2024-09-13T08:41:27.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.448+0000 7fba5a200640 1 -- 172.21.15.31:0/3369442483 wait complete. 2024-09-13T08:41:27.464 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:41:27.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:41:27.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.628+0000 7fef84600640 1 -- 172.21.15.31:0/2475901571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef64005180 msgr2=0x7fef6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.628+0000 7fef84600640 1 --2- 172.21.15.31:0/2475901571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef64005180 0x7fef6400b9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fef74009ed0 tx=0x7fef64005830 comp rx=0 tx=0).stop 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.628+0000 7fef82600640 1 -- 172.21.15.31:0/2475901571 reap_dead start 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.628+0000 7fef84600640 1 -- 172.21.15.31:0/2475901571 shutdown_connections 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.628+0000 7fef84600640 1 -- 172.21.15.31:0/2475901571 >> 172.21.15.31:0/2475901571 conn(0x7fef7c0cfbb0 msgr2=0x7fef7c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.629+0000 7fef84600640 1 -- 172.21.15.31:0/2475901571 shutdown_connections 2024-09-13T08:41:27.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.629+0000 7fef84600640 1 -- 172.21.15.31:0/2475901571 wait complete. 2024-09-13T08:41:27.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.630+0000 7fef84600640 1 Processor -- start 2024-09-13T08:41:27.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.630+0000 7fef84600640 1 -- start start 2024-09-13T08:41:27.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 0x7fef7c0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 0x7fef7c0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c0cbfd0 con 0x7fef7c1f5e00 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c0cbe50 con 0x7fef7c1572b0 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.631+0000 7fef84600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fef7c0b29e0 con 0x7fef64005180 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57048/0 (socket says 172.21.15.31:57048) 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 -- 172.21.15.31:0/2028435979 learned_addr learned my addr 172.21.15.31:0/2028435979 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef81200640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 0x7fef7c0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef81c00640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 0x7fef7c0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 -- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 msgr2=0x7fef7c0a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 0x7fef7c0a9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 -- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 msgr2=0x7fef7c0c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:27.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 0x7fef7c0c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 -- 172.21.15.31:0/2028435979 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7c1f6770 con 0x7fef7c1f5e00 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef81200640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 0x7fef7c0c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef81c00640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 0x7fef7c0a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef82600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fef74009ed0 tx=0x7fef74034c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74043070 con 0x7fef7c1f5e00 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.632+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef74034d10 con 0x7fef7c1f5e00 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.633+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74038040 con 0x7fef7c1f5e00 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.633+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7c1f6a50 con 0x7fef7c1f5e00 2024-09-13T08:41:27.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.633+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef7c0aa2f0 con 0x7fef7c1f5e00 2024-09-13T08:41:27.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.634+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7c06fec0 con 0x7fef7c1f5e00 2024-09-13T08:41:27.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.635+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fef74003bd0 con 0x7fef7c1f5e00 2024-09-13T08:41:27.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.636+0000 7fef79600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 0x7fef5c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:27.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.636+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4857+0+0 (secure 0 0 0) 0x7fef740baa50 con 0x7fef7c1f5e00 2024-09-13T08:41:27.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.637+0000 7fef81c00640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 0x7fef5c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:27.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.637+0000 7fef81c00640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 0x7fef5c07b070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fef70009b40 tx=0x7fef70005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:27.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.641+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fef740828f0 con 0x7fef7c1f5e00 2024-09-13T08:41:27.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:27.838+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fef7c0c9fc0 con 0x7fef7c1f5e00 2024-09-13T08:41:28.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.118+0000 7fef79600640 1 -- 172.21.15.31:0/2028435979 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fef74087320 con 0x7fef7c1f5e00 2024-09-13T08:41:28.119 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:41:28.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 msgr2=0x7fef5c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:28.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 0x7fef5c07b070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fef70009b40 tx=0x7fef70005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:28.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 msgr2=0x7fef7c0ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:28.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fef74009ed0 tx=0x7fef74034c20 comp rx=0 tx=0).stop 2024-09-13T08:41:28.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 shutdown_connections 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1f5e00 0x7fef7c0ad490 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7c1572b0 0x7fef7c0c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef5c078bb0 0x7fef5c07b070 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 --2- 172.21.15.31:0/2028435979 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef64005180 0x7fef7c0a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.123+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 >> 172.21.15.31:0/2028435979 conn(0x7fef7c0cfbb0 msgr2=0x7fef7c0b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.124+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 shutdown_connections 2024-09-13T08:41:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.124+0000 7fef84600640 1 -- 172.21.15.31:0/2028435979 wait complete. 2024-09-13T08:41:28.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:41:28.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 -- 172.21.15.31:0/3806025634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc64004000 msgr2=0x7efc64004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:28.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 --2- 172.21.15.31:0/3806025634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc64004000 0x7efc64004800 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7efc68009b20 tx=0x7efc64005830 comp rx=0 tx=0).stop 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7e800640 1 -- 172.21.15.31:0/3806025634 reap_dead start 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 -- 172.21.15.31:0/3806025634 shutdown_connections 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 -- 172.21.15.31:0/3806025634 >> 172.21.15.31:0/3806025634 conn(0x7efc800b1520 msgr2=0x7efc800b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 -- 172.21.15.31:0/3806025634 shutdown_connections 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.313+0000 7efc7f200640 1 -- 172.21.15.31:0/3806025634 wait complete. 2024-09-13T08:41:28.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.315+0000 7efc7f200640 1 Processor -- start 2024-09-13T08:41:28.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.315+0000 7efc7f200640 1 -- start start 2024-09-13T08:41:28.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc64004000 0x7efc800c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:28.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc8014d490 0x7efc8015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:28.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc80073e50 con 0x7efc8014d910 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efc80073cd0 con 0x7efc8014d490 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efc800cc850 con 0x7efc64004000 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57078/0 (socket says 172.21.15.31:57078) 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7e800640 1 -- 172.21.15.31:0/2039926862 learned_addr learned my addr 172.21.15.31:0/2039926862 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc7de00640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc64004000 0x7efc800c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.316+0000 7efc77e00640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc8014d490 0x7efc8015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 -- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc64004000 msgr2=0x7efc800c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc64004000 0x7efc800c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 -- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc8014d490 msgr2=0x7efc8015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc8014d490 0x7efc8015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:28.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 -- 172.21.15.31:0/2039926862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8015a1c0 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7e800640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7efc78001330 tx=0x7efc78006650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc78015070 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc7800eb00 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc800b5eb0 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc78011460 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.317+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8014e050 con 0x7efc8014d910 2024-09-13T08:41:28.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.319+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8006fc80 con 0x7efc8014d910 2024-09-13T08:41:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.321+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7efc78018070 con 0x7efc8014d910 2024-09-13T08:41:28.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.322+0000 7efc76000640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 0x7efc5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:28.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.322+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5227+0+0 (secure 0 0 0) 0x7efc7809b910 con 0x7efc8014d910 2024-09-13T08:41:28.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.322+0000 7efc7de00640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 0x7efc5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:28.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.323+0000 7efc7de00640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 0x7efc5c07b0a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efc70009b70 tx=0x7efc70005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:28.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.325+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efc8015a1c0 con 0x7efc8014d910 2024-09-13T08:41:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:28.522+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efc800aed50 con 0x7efc8014d910 2024-09-13T08:41:29.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.122+0000 7efc76000640 1 -- 172.21.15.31:0/2039926862 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) ==== 170+0+0 (secure 0 0 0) 0x7efc8015bfe0 con 0x7efc8014d910 2024-09-13T08:41:29.123 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:41:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 msgr2=0x7efc5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 0x7efc5c07b0a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efc70009b70 tx=0x7efc70005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 msgr2=0x7efc8015d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7efc78001330 tx=0x7efc78006650 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 shutdown_connections 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc8014d910 0x7efc8015d750 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc8014d490 0x7efc8015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7efc5c078be0 0x7efc5c07b0a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 --2- 172.21.15.31:0/2039926862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc64004000 0x7efc800c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 >> 172.21.15.31:0/2039926862 conn(0x7efc800b1520 msgr2=0x7efc8006eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.126+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 shutdown_connections 2024-09-13T08:41:29.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.127+0000 7efc7f200640 1 -- 172.21.15.31:0/2039926862 wait complete. 2024-09-13T08:41:29.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:41:29.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 -- 172.21.15.31:0/3245396313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf74004d40 msgr2=0x7fdf740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:29.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 --2- 172.21.15.31:0/3245396313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf74004d40 0x7fdf740051c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009f00 tx=0x7fdf74005830 comp rx=0 tx=0).stop 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf8fe00640 1 -- 172.21.15.31:0/3245396313 reap_dead start 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 -- 172.21.15.31:0/3245396313 shutdown_connections 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 -- 172.21.15.31:0/3245396313 >> 172.21.15.31:0/3245396313 conn(0x7fdf900d3e90 msgr2=0x7fdf900d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 -- 172.21.15.31:0/3245396313 shutdown_connections 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.331+0000 7fdf94a00640 1 -- 172.21.15.31:0/3245396313 wait complete. 2024-09-13T08:41:29.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.333+0000 7fdf94a00640 1 Processor -- start 2024-09-13T08:41:29.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.333+0000 7fdf94a00640 1 -- start start 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 0x7fdf901f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 0x7fdf900cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900d2620 con 0x7fdf9006e0f0 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900d24a0 con 0x7fdf900cc250 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf94a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdf900d27a0 con 0x7fdf74004d40 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57096/0 (socket says 172.21.15.31:57096) 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 -- 172.21.15.31:0/1481968436 learned_addr learned my addr 172.21.15.31:0/1481968436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8f400640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 0x7fdf901f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 -- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 msgr2=0x7fdf901f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 0x7fdf901f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 -- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 msgr2=0x7fdf900cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8fe00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 0x7fdf900cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 0x7fdf900cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8ea00640 1 -- 172.21.15.31:0/1481968436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf901f5690 con 0x7fdf9006e0f0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8f400640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 0x7fdf901f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.334+0000 7fdf8fe00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 0x7fdf900cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf8ea00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c011ab0 tx=0x7fdf7c011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c01e070 con 0x7fdf9006e0f0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf7c00fe70 con 0x7fdf9006e0f0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0042b0 con 0x7fdf9006e0f0 2024-09-13T08:41:29.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf900cb1b0 con 0x7fdf9006e0f0 2024-09-13T08:41:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.335+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf900cb770 con 0x7fdf9006e0f0 2024-09-13T08:41:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.336+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf900d1d90 con 0x7fdf9006e0f0 2024-09-13T08:41:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.337+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fdf7c021070 con 0x7fdf9006e0f0 2024-09-13T08:41:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.338+0000 7fdf87400640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 0x7fdf7007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.339+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5609+0+0 (secure 0 0 0) 0x7fdf7c09b930 con 0x7fdf9006e0f0 2024-09-13T08:41:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.339+0000 7fdf8f400640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 0x7fdf7007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.340+0000 7fdf8f400640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 0x7fdf7007b070 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001160 tx=0x7fdf800067a0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.343+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdf901f5690 con 0x7fdf9006e0f0 2024-09-13T08:41:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:29.576+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdf900b3dd0 con 0x7fdf9006e0f0 2024-09-13T08:41:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.145+0000 7fdf87400640 1 -- 172.21.15.31:0/1481968436 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v43) ==== 174+0+0 (secure 0 0 0) 0x7fdf7c063420 con 0x7fdf9006e0f0 2024-09-13T08:41:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 10 and data pool 11 2024-09-13T08:41:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 msgr2=0x7fdf7007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 0x7fdf7007b070 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001160 tx=0x7fdf800067a0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 msgr2=0x7fdf9015c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c011ab0 tx=0x7fdf7c011f70 comp rx=0 tx=0).stop 2024-09-13T08:41:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 shutdown_connections 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf900cc250 0x7fdf900cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9006e0f0 0x7fdf9015c6d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf70078bb0 0x7fdf7007b070 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 --2- 172.21.15.31:0/1481968436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf74004d40 0x7fdf901f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.149+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 >> 172.21.15.31:0/1481968436 conn(0x7fdf900d3e90 msgr2=0x7fdf9006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.150+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 shutdown_connections 2024-09-13T08:41:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.150+0000 7fdf94a00640 1 -- 172.21.15.31:0/1481968436 wait complete. 2024-09-13T08:41:30.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 -- 172.21.15.31:0/3562568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815bd70 msgr2=0x7fcae80cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 --2- 172.21.15.31:0/3562568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815bd70 0x7fcae80cf850 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009830 tx=0x7fcadc02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 -- 172.21.15.31:0/3562568504 shutdown_connections 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 --2- 172.21.15.31:0/3562568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815bd70 0x7fcae80cf850 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 -- 172.21.15.31:0/3562568504 >> 172.21.15.31:0/3562568504 conn(0x7fcae80cc910 msgr2=0x7fcae80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 -- 172.21.15.31:0/3562568504 shutdown_connections 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.352+0000 7fcaef600640 1 -- 172.21.15.31:0/3562568504 wait complete. 2024-09-13T08:41:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.354+0000 7fcaef600640 1 Processor -- start 2024-09-13T08:41:30.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.354+0000 7fcaef600640 1 -- start start 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 0x7fcae81551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815f9e0 0x7fcae815df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcae80cec50 con 0x7fcae815fe20 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcae80cead0 con 0x7fcae815f9e0 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaef600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcae80cb850 con 0x7fcae815bd70 2024-09-13T08:41:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaed600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaed600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57108/0 (socket says 172.21.15.31:57108) 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaed600640 1 -- 172.21.15.31:0/2088895342 learned_addr learned my addr 172.21.15.31:0/2088895342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcae7e00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815f9e0 0x7fcae815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.355+0000 7fcaecc00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 0x7fcae81551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 -- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 msgr2=0x7fcae81551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 0x7fcae81551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 -- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815f9e0 msgr2=0x7fcae815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815f9e0 0x7fcae815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 -- 172.21.15.31:0/2088895342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae8155700 con 0x7fcae815fe20 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaecc00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 0x7fcae81551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaed600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcae000cac0 tx=0x7fcae000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0016020 con 0x7fcae815fe20 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae815e4c0 con 0x7fcae815fe20 2024-09-13T08:41:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcae000ad40 con 0x7fcae815fe20 2024-09-13T08:41:30.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0012680 con 0x7fcae815fe20 2024-09-13T08:41:30.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.356+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae815c870 con 0x7fcae815fe20 2024-09-13T08:41:30.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.357+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae80cf850 con 0x7fcae815fe20 2024-09-13T08:41:30.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.359+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fcae0019020 con 0x7fcae815fe20 2024-09-13T08:41:30.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.359+0000 7fcae4c00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 0x7fcac807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.360+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5709+0+0 (secure 0 0 0) 0x7fcae009b5c0 con 0x7fcae815fe20 2024-09-13T08:41:30.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.360+0000 7fcaecc00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 0x7fcac807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.361+0000 7fcaecc00640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 0x7fcac807b070 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcadc034000 tx=0x7fcadc004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:30.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.364+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcae0020020 con 0x7fcae815fe20 2024-09-13T08:41:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.561+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcae815f210 con 0x7fcae815fe20 2024-09-13T08:41:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.562+0000 7fcae4c00640 1 -- 172.21.15.31:0/2088895342 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+16633 (secure 0 0 0) 0x7fcae815e4c0 con 0x7fcae815fe20 2024-09-13T08:41:30.563 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:30.563 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:29.577519+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:41:27.848143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T08:41:28.532134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.564+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 msgr2=0x7fcac807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.564+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 0x7fcac807b070 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcadc034000 tx=0x7fcadc004730 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.564+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 msgr2=0x7fcae806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.564+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcae000cac0 tx=0x7fcae000cf80 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.564+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 shutdown_connections 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae815fe20 0x7fcae806f540 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae815f9e0 0x7fcae815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fcac8078bb0 0x7fcac807b070 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 --2- 172.21.15.31:0/2088895342 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcae815bd70 0x7fcae81551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 >> 172.21.15.31:0/2088895342 conn(0x7fcae80cc910 msgr2=0x7fcae80af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 shutdown_connections 2024-09-13T08:41:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.565+0000 7fcaef600640 1 -- 172.21.15.31:0/2088895342 wait complete. 2024-09-13T08:41:30.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 -- 172.21.15.31:0/190517614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a10073fc0 msgr2=0x7f6a100743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 --2- 172.21.15.31:0/190517614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a10073fc0 0x7f6a100743f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6a1006daa0 tx=0x7f6a0802aac0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 -- 172.21.15.31:0/190517614 shutdown_connections 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 --2- 172.21.15.31:0/190517614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a10073fc0 0x7f6a100743f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 --2- 172.21.15.31:0/190517614 >> v2:172.21.15.45:3300/0 conn(0x7f6a10152a70 0x7f6a10152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 -- 172.21.15.31:0/190517614 >> 172.21.15.31:0/190517614 conn(0x7f6a100cc910 msgr2=0x7f6a100cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:30.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 -- 172.21.15.31:0/190517614 shutdown_connections 2024-09-13T08:41:30.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.742+0000 7f6a15800640 1 -- 172.21.15.31:0/190517614 wait complete. 2024-09-13T08:41:30.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.743+0000 7f6a15800640 1 Processor -- start 2024-09-13T08:41:30.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 -- start start 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a10073fc0 0x7f6a10158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a10152a70 0x7f6a101f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100b1ca0 con 0x7f6a100cf980 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100b1b20 con 0x7f6a10152a70 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.744+0000 7f6a15800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6a100b1e20 con 0x7f6a10073fc0 2024-09-13T08:41:30.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57126/0 (socket says 172.21.15.31:57126) 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 -- 172.21.15.31:0/2764734420 learned_addr learned my addr 172.21.15.31:0/2764734420 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0ea00640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a10073fc0 0x7f6a10158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0f400640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a10152a70 0x7f6a101f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 -- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a10073fc0 msgr2=0x7f6a10158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a10073fc0 0x7f6a10158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 -- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a10152a70 msgr2=0x7f6a101f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a10152a70 0x7f6a101f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 -- 172.21.15.31:0/2764734420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a10158f30 con 0x7f6a100cf980 2024-09-13T08:41:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.745+0000 7f6a0e000640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c500 tx=0x7f69fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:30.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.746+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0052d0 con 0x7f6a100cf980 2024-09-13T08:41:30.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.746+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69fc005470 con 0x7f6a100cf980 2024-09-13T08:41:30.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.746+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a10159210 con 0x7f6a100cf980 2024-09-13T08:41:30.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.746+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc012870 con 0x7f6a100cf980 2024-09-13T08:41:30.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.746+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a100b3eb0 con 0x7f6a100cf980 2024-09-13T08:41:30.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.747+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a100b22d0 con 0x7f6a100cf980 2024-09-13T08:41:30.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.748+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f69fc0035f0 con 0x7f6a100cf980 2024-09-13T08:41:30.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.749+0000 7f6a06a00640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 0x7f69f007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:30.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.749+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5709+0+0 (secure 0 0 0) 0x7f69fc09aad0 con 0x7f6a100cf980 2024-09-13T08:41:30.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.750+0000 7f6a0ea00640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 0x7f69f007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:30.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.750+0000 7f6a0ea00640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 0x7f69f007b070 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009bd0 tx=0x7f6a00005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:30.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.753+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f69fc0624a0 con 0x7f6a100cf980 2024-09-13T08:41:30.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.948+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a10159730 con 0x7f6a100cf980 2024-09-13T08:41:30.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.949+0000 7f6a06a00640 1 -- 172.21.15.31:0/2764734420 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+16633 (secure 0 0 0) 0x7f69fc066ed0 con 0x7f6a100cf980 2024-09-13T08:41:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:30.950 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:29.577519+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:41:27.848143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T08:41:28.532134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 msgr2=0x7f69f007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 0x7f69f007b070 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009bd0 tx=0x7f6a00005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 msgr2=0x7f6a101f7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c500 tx=0x7f69fc00c9c0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 shutdown_connections 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a10152a70 0x7f6a101f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a100cf980 0x7f6a101f7bf0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f69f0078bb0 0x7f69f007b070 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 --2- 172.21.15.31:0/2764734420 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a10073fc0 0x7f6a10158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 >> 172.21.15.31:0/2764734420 conn(0x7f6a100cc910 msgr2=0x7f6a100ca730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 shutdown_connections 2024-09-13T08:41:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:30.952+0000 7f6a15800640 1 -- 172.21.15.31:0/2764734420 wait complete. 2024-09-13T08:41:30.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:41:31.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 -- 172.21.15.31:0/1339720481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87840cc920 msgr2=0x7f878406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 --2- 172.21.15.31:0/1339720481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87840cc920 0x7f878406e660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f877c005fd0 tx=0x7f877c02aa10 comp rx=0 tx=0).stop 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 -- 172.21.15.31:0/1339720481 shutdown_connections 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 --2- 172.21.15.31:0/1339720481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87840cc920 0x7f878406e660 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 --2- 172.21.15.31:0/1339720481 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 0x7f87840cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 -- 172.21.15.31:0/1339720481 >> 172.21.15.31:0/1339720481 conn(0x7f87840b1520 msgr2=0x7f87840b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 -- 172.21.15.31:0/1339720481 shutdown_connections 2024-09-13T08:41:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.131+0000 7f8783200640 1 -- 172.21.15.31:0/1339720481 wait complete. 2024-09-13T08:41:31.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.133+0000 7f8783200640 1 Processor -- start 2024-09-13T08:41:31.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.133+0000 7f8783200640 1 -- start start 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 0x7f87840c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878415bb70 0x7f878415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f878406ed20 con 0x7f87840cc920 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f878406eba0 con 0x7f878415bb70 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8783200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f878406eea0 con 0x7f87840b5300 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:31.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57142/0 (socket says 172.21.15.31:57142) 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 -- 172.21.15.31:0/4025287816 learned_addr learned my addr 172.21.15.31:0/4025287816 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8782800640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878415bb70 0x7f878415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781e00640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 0x7f87840c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 -- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 msgr2=0x7f87840c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 0x7f87840c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 -- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878415bb70 msgr2=0x7f878415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878415bb70 0x7f878415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.134+0000 7f8781400640 1 -- 172.21.15.31:0/4025287816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841f5680 con 0x7f87840cc920 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f8781400640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f877000e9f0 tx=0x7f877000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877000cd80 con 0x7f87840cc920 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8770010430 con 0x7f87840cc920 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770010840 con 0x7f87840cc920 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87841f5960 con 0x7f87840cc920 2024-09-13T08:41:31.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.135+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841f5f20 con 0x7f87840cc920 2024-09-13T08:41:31.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.136+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f878406f3c0 con 0x7f87840cc920 2024-09-13T08:41:31.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.137+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f87700109e0 con 0x7f87840cc920 2024-09-13T08:41:31.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.139+0000 7f877a000640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 0x7f876407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:31.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.139+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5709+0+0 (secure 0 0 0) 0x7f877009ba90 con 0x7f87840cc920 2024-09-13T08:41:31.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.140+0000 7f8781e00640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 0x7f876407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:31.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.140+0000 7f8781e00640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 0x7f876407b070 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f87740015e0 tx=0x7f8774009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:31.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.143+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8770063610 con 0x7f87840cc920 2024-09-13T08:41:31.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:31.377+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8784069700 con 0x7f87840cc920 2024-09-13T08:41:32.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.168+0000 7f877a000640 1 -- 172.21.15.31:0/4025287816 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v46) ==== 118+0+0 (secure 0 0 0) 0x7f8770068040 con 0x7f87840cc920 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 msgr2=0x7f876407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 0x7f876407b070 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f87740015e0 tx=0x7f8774009040 comp rx=0 tx=0).stop 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 msgr2=0x7f87841f2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f877000e9f0 tx=0x7f877000eeb0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 shutdown_connections 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f878415bb70 0x7f878415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87840cc920 0x7f87841f2990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8764078bb0 0x7f876407b070 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 --2- 172.21.15.31:0/4025287816 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87840b5300 0x7f87840c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 >> 172.21.15.31:0/4025287816 conn(0x7f87840b1520 msgr2=0x7f87840afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:32.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 shutdown_connections 2024-09-13T08:41:32.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.172+0000 7f8783200640 1 -- 172.21.15.31:0/4025287816 wait complete. 2024-09-13T08:41:32.190 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:41:32.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:41:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 -- 172.21.15.31:0/1416693609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac15bfc0 msgr2=0x7f2fac072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1416693609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac15bfc0 0x7f2fac072560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009860 tx=0x7f2f9c02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 -- 172.21.15.31:0/1416693609 shutdown_connections 2024-09-13T08:41:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1416693609 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac15bfc0 0x7f2fac072560 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 -- 172.21.15.31:0/1416693609 >> 172.21.15.31:0/1416693609 conn(0x7f2fac0cfbb0 msgr2=0x7f2fac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 -- 172.21.15.31:0/1416693609 shutdown_connections 2024-09-13T08:41:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.358+0000 7f2fb3400640 1 -- 172.21.15.31:0/1416693609 wait complete. 2024-09-13T08:41:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.360+0000 7f2fb3400640 1 Processor -- start 2024-09-13T08:41:32.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.360+0000 7f2fb3400640 1 -- start start 2024-09-13T08:41:32.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac06fa60 0x7f2fac1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:32.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fac15bfc0 0x7f2fac159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:32.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac06e760 con 0x7f2fac159f90 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0cc060 con 0x7f2fac15bfc0 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb3400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2fac0b30f0 con 0x7f2fac06fa60 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57148/0 (socket says 172.21.15.31:57148) 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb1400640 1 -- 172.21.15.31:0/1220696022 learned_addr learned my addr 172.21.15.31:0/1220696022 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.361+0000 7f2fb0a00640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac06fa60 0x7f2fac1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fa3e00640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fac15bfc0 0x7f2fac159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 -- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac06fa60 msgr2=0x7f2fac1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac06fa60 0x7f2fac1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 -- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fac15bfc0 msgr2=0x7f2fac159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fac15bfc0 0x7f2fac159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 -- 172.21.15.31:0/1220696022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac146e90 con 0x7f2fac159f90 2024-09-13T08:41:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb1400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c7d0 tx=0x7f2fa400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4003b70 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fa4003d10 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400aab0 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.362+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fac144290 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.363+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fac146660 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.364+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fac072560 con 0x7f2fac159f90 2024-09-13T08:41:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.365+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f2fa400ac50 con 0x7f2fac159f90 2024-09-13T08:41:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.366+0000 7f2fa9600640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 0x7f2f8807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.366+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f2fa409ada0 con 0x7f2fac159f90 2024-09-13T08:41:32.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.366+0000 7f2fb0a00640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 0x7f2f8807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:32.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.367+0000 7f2fb0a00640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 0x7f2f8807b070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c034000 tx=0x7f2f9c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:32.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.370+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2fa4062e40 con 0x7f2fac159f90 2024-09-13T08:41:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:32.606+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2fac0cb310 con 0x7f2fac159f90 2024-09-13T08:41:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.174+0000 7f2fa9600640 1 -- 172.21.15.31:0/1220696022 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v47) ==== 122+0+0 (secure 0 0 0) 0x7f2fa4062740 con 0x7f2fac159f90 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.176+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 msgr2=0x7f2f8807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.176+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 0x7f2f8807b070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c034000 tx=0x7f2f9c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 msgr2=0x7f2fac146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c7d0 tx=0x7f2fa400cc90 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 shutdown_connections 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fac159f90 0x7f2fac146120 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fac15bfc0 0x7f2fac159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f2f88078bb0 0x7f2f8807b070 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 --2- 172.21.15.31:0/1220696022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fac06fa60 0x7f2fac1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 >> 172.21.15.31:0/1220696022 conn(0x7f2fac0cfbb0 msgr2=0x7f2fac153000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 shutdown_connections 2024-09-13T08:41:33.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.177+0000 7f2fb3400640 1 -- 172.21.15.31:0/1220696022 wait complete. 2024-09-13T08:41:33.196 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:41:33.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:33.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 -- 172.21.15.31:0/3554585210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 msgr2=0x7fdf9c0cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/3554585210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c0cced0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0cf950 tx=0x7fdf8c02a560 comp rx=0 tx=0).stop 2024-09-13T08:41:33.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 -- 172.21.15.31:0/3554585210 shutdown_connections 2024-09-13T08:41:33.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/3554585210 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0ceab0 0x7fdf9c0ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/3554585210 >> v2:172.21.15.45:3300/0 conn(0x7fdf9c0cad40 0x7fdf9c0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/3554585210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c0cced0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 -- 172.21.15.31:0/3554585210 >> 172.21.15.31:0/3554585210 conn(0x7fdf9c06cb00 msgr2=0x7fdf9c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 -- 172.21.15.31:0/3554585210 shutdown_connections 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.359+0000 7fdfa1a00640 1 -- 172.21.15.31:0/3554585210 wait complete. 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.361+0000 7fdfa1a00640 1 Processor -- start 2024-09-13T08:41:33.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.361+0000 7fdfa1a00640 1 -- start start 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0cad40 0x7fdf9c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf9c0ceab0 0x7fdf9c14fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c06e870 con 0x7fdf9c151a80 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c06e6f0 con 0x7fdf9c0ceab0 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9c06e9f0 con 0x7fdf9c0cad40 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57164/0 (socket says 172.21.15.31:57164) 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdfa1000640 1 -- 172.21.15.31:0/2873586339 learned_addr learned my addr 172.21.15.31:0/2873586339 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdf9b400640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf9c0ceab0 0x7fdf9c14fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.362+0000 7fdf9be00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0cad40 0x7fdf9c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 -- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0cad40 msgr2=0x7fdf9c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0cad40 0x7fdf9c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 -- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf9c0ceab0 msgr2=0x7fdf9c14fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf9c0ceab0 0x7fdf9c14fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 -- 172.21.15.31:0/2873586339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c0c6160 con 0x7fdf9c151a80 2024-09-13T08:41:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1000640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400b9a0 tx=0x7fdf9400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94019070 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf9400cbb0 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94014440 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.363+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c159b90 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.364+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c14be60 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.365+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9c0ca180 con 0x7fdf9c151a80 2024-09-13T08:41:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.366+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fdf9401c070 con 0x7fdf9c151a80 2024-09-13T08:41:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.367+0000 7fdf93e00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 0x7fdf7c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.367+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fdf9409ab30 con 0x7fdf9c151a80 2024-09-13T08:41:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.367+0000 7fdf9be00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 0x7fdf7c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.368+0000 7fdf9be00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 0x7fdf7c07b070 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c033000 tx=0x7fdf8c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:33.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.371+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdf94062680 con 0x7fdf9c151a80 2024-09-13T08:41:33.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.603+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf9c0ca860 con 0x7fdf9c151a80 2024-09-13T08:41:33.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.604+0000 7fdf93e00640 1 -- 172.21.15.31:0/2873586339 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5222 (secure 0 0 0) 0x7fdf940670b0 con 0x7fdf9c151a80 2024-09-13T08:41:33.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:33.605 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"btime":"2024-09-13T08:41:33:169589+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:33.605 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2024-09-13T08:41:33.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 msgr2=0x7fdf7c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 0x7fdf7c07b070 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c033000 tx=0x7fdf8c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 msgr2=0x7fdf9c1500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400b9a0 tx=0x7fdf9400bda0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 shutdown_connections 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf9c151a80 0x7fdf9c1500c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf9c0ceab0 0x7fdf9c14fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fdf7c078bb0 0x7fdf7c07b070 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.606+0000 7fdfa1a00640 1 --2- 172.21.15.31:0/2873586339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf9c0cad40 0x7fdf9c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.607+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 >> 172.21.15.31:0/2873586339 conn(0x7fdf9c06cb00 msgr2=0x7fdf9c0b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.607+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 shutdown_connections 2024-09-13T08:41:33.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.607+0000 7fdfa1a00640 1 -- 172.21.15.31:0/2873586339 wait complete. 2024-09-13T08:41:33.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 -- 172.21.15.31:0/1158127066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa80062d0 msgr2=0x7f0fa800cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/1158127066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa80062d0 0x7f0fa800cad0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009ed0 tx=0x7f0fa8006940 comp rx=0 tx=0).stop 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 -- 172.21.15.31:0/1158127066 shutdown_connections 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/1158127066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fa80062d0 0x7f0fa800cad0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 -- 172.21.15.31:0/1158127066 >> 172.21.15.31:0/1158127066 conn(0x7f0fc40cc910 msgr2=0x7f0fc40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:33.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 -- 172.21.15.31:0/1158127066 shutdown_connections 2024-09-13T08:41:33.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.785+0000 7f0fc8e00640 1 -- 172.21.15.31:0/1158127066 wait complete. 2024-09-13T08:41:33.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.787+0000 7f0fc8e00640 1 Processor -- start 2024-09-13T08:41:33.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.787+0000 7f0fc8e00640 1 -- start start 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fa80062d0 0x7f0fc40c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40c9a40 0x7f0fc40b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc415a010 con 0x7f0fc40b57f0 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc40cc690 con 0x7f0fc40c9a40 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc8e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc415a190 con 0x7f0fa80062d0 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57172/0 (socket says 172.21.15.31:57172) 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 -- 172.21.15.31:0/2571415394 learned_addr learned my addr 172.21.15.31:0/2571415394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc1e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fa80062d0 0x7f0fc40c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fc2800640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40c9a40 0x7f0fc40b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 -- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fa80062d0 msgr2=0x7f0fc40c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fa80062d0 0x7f0fc40c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 -- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40c9a40 msgr2=0x7f0fc40b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40c9a40 0x7f0fc40b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.788+0000 7f0fb9400640 1 -- 172.21.15.31:0/2571415394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc41f55b0 con 0x7f0fc40b57f0 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fb9400640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00e590 tx=0x7f0fac00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac015070 con 0x7f0fc40b57f0 2024-09-13T08:41:33.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fac010980 con 0x7f0fc40b57f0 2024-09-13T08:41:33.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac010dd0 con 0x7f0fc40b57f0 2024-09-13T08:41:33.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc41f5890 con 0x7f0fc40b57f0 2024-09-13T08:41:33.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.789+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc40c9520 con 0x7f0fc40b57f0 2024-09-13T08:41:33.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.790+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc40b1910 con 0x7f0fc40b57f0 2024-09-13T08:41:33.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.791+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f0fac018070 con 0x7f0fc40b57f0 2024-09-13T08:41:33.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.793+0000 7f0fbaa00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 0x7f0fa007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:33.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.793+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f0fac09b3b0 con 0x7f0fc40b57f0 2024-09-13T08:41:33.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.793+0000 7f0fc1e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 0x7f0fa007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:33.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.794+0000 7f0fc1e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 0x7f0fa007b070 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4000c00 tx=0x7f0fb4006310 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:33.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.797+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0fac062d70 con 0x7f0fc40b57f0 2024-09-13T08:41:33.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.994+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fc40c96f0 con 0x7f0fc40b57f0 2024-09-13T08:41:33.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.995+0000 7f0fbaa00640 1 -- 172.21.15.31:0/2571415394 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+16620 (secure 0 0 0) 0x7f0fac0677a0 con 0x7f0fc40b57f0 2024-09-13T08:41:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:33.996 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:31.145913+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:41:27.848143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T08:41:28.532134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 msgr2=0x7f0fa007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 0x7f0fa007b070 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4000c00 tx=0x7f0fb4006310 comp rx=0 tx=0).stop 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 msgr2=0x7f0fc41f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00e590 tx=0x7f0fac00ea50 comp rx=0 tx=0).stop 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 shutdown_connections 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40c9a40 0x7f0fc40b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc40b57f0 0x7f0fc41f4cf0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0fa0078bb0 0x7f0fa007b070 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:33.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 --2- 172.21.15.31:0/2571415394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fa80062d0 0x7f0fc40c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.997+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 >> 172.21.15.31:0/2571415394 conn(0x7f0fc40cc910 msgr2=0x7f0fc409d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:34.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.998+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 shutdown_connections 2024-09-13T08:41:34.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:33.998+0000 7f0fc8e00640 1 -- 172.21.15.31:0/2571415394 wait complete. 2024-09-13T08:41:34.014 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:41:34.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.173+0000 7fc2f0e00640 1 -- 172.21.15.31:0/530880535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec074500 msgr2=0x7fc2ec0b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.173+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/530880535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec074500 0x7fc2ec0b1910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005fd0 tx=0x7fc2e402aa10 comp rx=0 tx=0).stop 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 -- 172.21.15.31:0/530880535 shutdown_connections 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/530880535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec074500 0x7fc2ec0b1910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/530880535 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec072850 0x7fc2ec073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 -- 172.21.15.31:0/530880535 >> 172.21.15.31:0/530880535 conn(0x7fc2ec0cc910 msgr2=0x7fc2ec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 -- 172.21.15.31:0/530880535 shutdown_connections 2024-09-13T08:41:34.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.174+0000 7fc2f0e00640 1 -- 172.21.15.31:0/530880535 wait complete. 2024-09-13T08:41:34.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.175+0000 7fc2f0e00640 1 Processor -- start 2024-09-13T08:41:34.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.176+0000 7fc2f0e00640 1 -- start start 2024-09-13T08:41:34.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec072850 0x7fc2ec09ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec074500 0x7fc2ec09b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ec15c490 con 0x7fc2ec152f10 2024-09-13T08:41:34.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ec0b1e50 con 0x7fc2ec072850 2024-09-13T08:41:34.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2f0e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec0b1fd0 con 0x7fc2ec074500 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2e9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec072850 0x7fc2ec09ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57176/0 (socket says 172.21.15.31:57176) 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 -- 172.21.15.31:0/2495622276 learned_addr learned my addr 172.21.15.31:0/2495622276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2e1400640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec074500 0x7fc2ec09b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 -- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec074500 msgr2=0x7fc2ec09b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec074500 0x7fc2ec09b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 -- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec072850 msgr2=0x7fc2ec09ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec072850 0x7fc2ec09ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.177+0000 7fc2ea800640 1 -- 172.21.15.31:0/2495622276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec09bdc0 con 0x7fc2ec152f10 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2ea800640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e4003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4047070 con 0x7fc2ec152f10 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e402f670 con 0x7fc2ec152f10 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4033050 con 0x7fc2ec152f10 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ec09c0a0 con 0x7fc2ec152f10 2024-09-13T08:41:34.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.178+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2ec09c660 con 0x7fc2ec152f10 2024-09-13T08:41:34.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.180+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2ec073fc0 con 0x7fc2ec152f10 2024-09-13T08:41:34.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.181+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fc2e404a070 con 0x7fc2ec152f10 2024-09-13T08:41:34.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.182+0000 7fc2e2a00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 0x7fc2c807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.182+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fc2e40bc150 con 0x7fc2ec152f10 2024-09-13T08:41:34.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.183+0000 7fc2e9e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 0x7fc2c807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.183+0000 7fc2e9e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 0x7fc2c807b070 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc001720 tx=0x7fc2dc009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:34.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.187+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc2e4083b80 con 0x7fc2ec152f10 2024-09-13T08:41:34.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.377+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2ec15bbf0 con 0x7fc2c8078bb0 2024-09-13T08:41:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.382+0000 7fc2e2a00640 1 -- 172.21.15.31:0/2495622276 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241258 (secure 0 0 0) 0x7fc2ec15bbf0 con 0x7fc2c8078bb0 2024-09-13T08:41:34.386 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:34.388 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 msgr2=0x7fc2c807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 0x7fc2c807b070 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc001720 tx=0x7fc2dc009040 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 msgr2=0x7fc2ec0c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e4003ef0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 shutdown_connections 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2ec152f10 0x7fc2ec0c4dc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc2c8078bb0 0x7fc2c807b070 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2ec074500 0x7fc2ec09b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 --2- 172.21.15.31:0/2495622276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ec072850 0x7fc2ec09ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 >> 172.21.15.31:0/2495622276 conn(0x7fc2ec0cc910 msgr2=0x7fc2ec06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:34.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 shutdown_connections 2024-09-13T08:41:34.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.386+0000 7fc2f0e00640 1 -- 172.21.15.31:0/2495622276 wait complete. 2024-09-13T08:41:34.402 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-13T08:41:33.103155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":105,"num_write_kb":1629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":76,"acting":76,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":257548,"kb_used_data":8016,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717172,"statfs":{"total":773094113280,"available":772830384128,"internally_reserved":0,"allocated":8208384,"data_stored":4103806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.982612"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475051+0000","last_change":"2024-09-13T08:41:30.167545+0000","last_active":"2024-09-13T08:41:31.475051+0000","last_peered":"2024-09-13T08:41:31.475051+0000","last_clean":"2024-09-13T08:41:31.475051+0000","last_became_active":"2024-09-13T08:41:30.167268+0000","last_became_peered":"2024-09-13T08:41:30.167268+0000","last_unstale":"2024-09-13T08:41:31.475051+0000","last_undegraded":"2024-09-13T08:41:31.475051+0000","last_fullsized":"2024-09-13T08:41:31.475051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:01.162295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474710+0000","last_change":"2024-09-13T08:41:30.172201+0000","last_active":"2024-09-13T08:41:31.474710+0000","last_peered":"2024-09-13T08:41:31.474710+0000","last_clean":"2024-09-13T08:41:31.474710+0000","last_became_active":"2024-09-13T08:41:30.171269+0000","last_became_peered":"2024-09-13T08:41:30.171269+0000","last_unstale":"2024-09-13T08:41:31.474710+0000","last_undegraded":"2024-09-13T08:41:31.474710+0000","last_fullsized":"2024-09-13T08:41:31.474710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.336999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474006+0000","last_change":"2024-09-13T08:41:30.169784+0000","last_active":"2024-09-13T08:41:31.474006+0000","last_peered":"2024-09-13T08:41:31.474006+0000","last_clean":"2024-09-13T08:41:31.474006+0000","last_became_active":"2024-09-13T08:41:30.169508+0000","last_became_peered":"2024-09-13T08:41:30.169508+0000","last_unstale":"2024-09-13T08:41:31.474006+0000","last_undegraded":"2024-09-13T08:41:31.474006+0000","last_fullsized":"2024-09-13T08:41:31.474006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:32.762281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125668+0000","last_change":"2024-09-13T08:41:28.125668+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125668+0000","last_undegraded":"2024-09-13T08:41:28.125668+0000","last_fullsized":"2024-09-13T08:41:28.125668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126514+0000","last_change":"2024-09-13T08:41:28.126514+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126514+0000","last_undegraded":"2024-09-13T08:41:28.126514+0000","last_fullsized":"2024-09-13T08:41:28.126514+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130480+0000","last_change":"2024-09-13T08:41:28.130480+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130480+0000","last_undegraded":"2024-09-13T08:41:28.130480+0000","last_fullsized":"2024-09-13T08:41:28.130480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126050+0000","last_change":"2024-09-13T08:41:28.126050+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126050+0000","last_undegraded":"2024-09-13T08:41:28.126050+0000","last_fullsized":"2024-09-13T08:41:28.126050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474813+0000","last_change":"2024-09-13T08:41:29.152256+0000","last_active":"2024-09-13T08:41:31.474813+0000","last_peered":"2024-09-13T08:41:31.474813+0000","last_clean":"2024-09-13T08:41:31.474813+0000","last_became_active":"2024-09-13T08:41:29.151880+0000","last_became_peered":"2024-09-13T08:41:29.151880+0000","last_unstale":"2024-09-13T08:41:31.474813+0000","last_undegraded":"2024-09-13T08:41:31.474813+0000","last_fullsized":"2024-09-13T08:41:31.474813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:45.557782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.124421+0000","last_change":"2024-09-13T08:41:28.124421+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.124421+0000","last_undegraded":"2024-09-13T08:41:28.124421+0000","last_fullsized":"2024-09-13T08:41:28.124421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126135+0000","last_change":"2024-09-13T08:41:28.126135+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126135+0000","last_undegraded":"2024-09-13T08:41:28.126135+0000","last_fullsized":"2024-09-13T08:41:28.126135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474073+0000","last_change":"2024-09-13T08:41:30.171632+0000","last_active":"2024-09-13T08:41:31.474073+0000","last_peered":"2024-09-13T08:41:31.474073+0000","last_clean":"2024-09-13T08:41:31.474073+0000","last_became_active":"2024-09-13T08:41:30.171351+0000","last_became_peered":"2024-09-13T08:41:30.171351+0000","last_unstale":"2024-09-13T08:41:31.474073+0000","last_undegraded":"2024-09-13T08:41:31.474073+0000","last_fullsized":"2024-09-13T08:41:31.474073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:36.601619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.473982+0000","last_change":"2024-09-13T08:41:29.141838+0000","last_active":"2024-09-13T08:41:31.473982+0000","last_peered":"2024-09-13T08:41:31.473982+0000","last_clean":"2024-09-13T08:41:31.473982+0000","last_became_active":"2024-09-13T08:41:29.141458+0000","last_became_peered":"2024-09-13T08:41:29.141458+0000","last_unstale":"2024-09-13T08:41:31.473982+0000","last_undegraded":"2024-09-13T08:41:31.473982+0000","last_fullsized":"2024-09-13T08:41:31.473982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:57.962170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.133318+0000","last_change":"2024-09-13T08:41:28.133318+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.133318+0000","last_undegraded":"2024-09-13T08:41:28.133318+0000","last_fullsized":"2024-09-13T08:41:28.133318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.131009+0000","last_change":"2024-09-13T08:41:28.131009+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.131009+0000","last_undegraded":"2024-09-13T08:41:28.131009+0000","last_fullsized":"2024-09-13T08:41:28.131009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474613+0000","last_change":"2024-09-13T08:41:29.147972+0000","last_active":"2024-09-13T08:41:31.474613+0000","last_peered":"2024-09-13T08:41:31.474613+0000","last_clean":"2024-09-13T08:41:31.474613+0000","last_became_active":"2024-09-13T08:41:29.147657+0000","last_became_peered":"2024-09-13T08:41:29.147657+0000","last_unstale":"2024-09-13T08:41:31.474613+0000","last_undegraded":"2024-09-13T08:41:31.474613+0000","last_fullsized":"2024-09-13T08:41:31.474613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:37.087395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126249+0000","last_change":"2024-09-13T08:41:28.126249+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126249+0000","last_undegraded":"2024-09-13T08:41:28.126249+0000","last_fullsized":"2024-09-13T08:41:28.126249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126123+0000","last_change":"2024-09-13T08:41:28.126123+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126123+0000","last_undegraded":"2024-09-13T08:41:28.126123+0000","last_fullsized":"2024-09-13T08:41:28.126123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123281+0000","last_change":"2024-09-13T08:41:28.123281+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123281+0000","last_undegraded":"2024-09-13T08:41:28.123281+0000","last_fullsized":"2024-09-13T08:41:28.123281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123170+0000","last_change":"2024-09-13T08:41:28.123170+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123170+0000","last_undegraded":"2024-09-13T08:41:28.123170+0000","last_fullsized":"2024-09-13T08:41:28.123170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123203+0000","last_change":"2024-09-13T08:41:28.123203+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123203+0000","last_undegraded":"2024-09-13T08:41:28.123203+0000","last_fullsized":"2024-09-13T08:41:28.123203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.132982+0000","last_change":"2024-09-13T08:41:28.132982+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.132982+0000","last_undegraded":"2024-09-13T08:41:28.132982+0000","last_fullsized":"2024-09-13T08:41:28.132982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130540+0000","last_change":"2024-09-13T08:41:28.130540+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130540+0000","last_undegraded":"2024-09-13T08:41:28.130540+0000","last_fullsized":"2024-09-13T08:41:28.130540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130589+0000","last_change":"2024-09-13T08:41:28.130589+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130589+0000","last_undegraded":"2024-09-13T08:41:28.130589+0000","last_fullsized":"2024-09-13T08:41:28.130589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125603+0000","last_change":"2024-09-13T08:41:28.125603+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125603+0000","last_undegraded":"2024-09-13T08:41:28.125603+0000","last_fullsized":"2024-09-13T08:41:28.125603+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125589+0000","last_change":"2024-09-13T08:41:28.125589+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125589+0000","last_undegraded":"2024-09-13T08:41:28.125589+0000","last_fullsized":"2024-09-13T08:41:28.125589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.131216+0000","last_change":"2024-09-13T08:41:28.131216+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.131216+0000","last_undegraded":"2024-09-13T08:41:28.131216+0000","last_fullsized":"2024-09-13T08:41:28.131216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474394+0000","last_change":"2024-09-13T08:41:30.158406+0000","last_active":"2024-09-13T08:41:31.474394+0000","last_peered":"2024-09-13T08:41:31.474394+0000","last_clean":"2024-09-13T08:41:31.474394+0000","last_became_active":"2024-09-13T08:41:30.157975+0000","last_became_peered":"2024-09-13T08:41:30.157975+0000","last_unstale":"2024-09-13T08:41:31.474394+0000","last_undegraded":"2024-09-13T08:41:31.474394+0000","last_fullsized":"2024-09-13T08:41:31.474394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:54.914413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.127296+0000","last_change":"2024-09-13T08:41:28.127296+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.127296+0000","last_undegraded":"2024-09-13T08:41:28.127296+0000","last_fullsized":"2024-09-13T08:41:28.127296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.129945+0000","last_change":"2024-09-13T08:41:28.129945+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.129945+0000","last_undegraded":"2024-09-13T08:41:28.129945+0000","last_fullsized":"2024-09-13T08:41:28.129945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.129993+0000","last_change":"2024-09-13T08:41:28.129993+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.129993+0000","last_undegraded":"2024-09-13T08:41:28.129993+0000","last_fullsized":"2024-09-13T08:41:28.129993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474730+0000","last_change":"2024-09-13T08:41:29.148076+0000","last_active":"2024-09-13T08:41:31.474730+0000","last_peered":"2024-09-13T08:41:31.474730+0000","last_clean":"2024-09-13T08:41:31.474730+0000","last_became_active":"2024-09-13T08:41:29.147012+0000","last_became_peered":"2024-09-13T08:41:29.147012+0000","last_unstale":"2024-09-13T08:41:31.474730+0000","last_undegraded":"2024-09-13T08:41:31.474730+0000","last_fullsized":"2024-09-13T08:41:31.474730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:18.467245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475003+0000","last_change":"2024-09-13T08:41:29.148851+0000","last_active":"2024-09-13T08:41:31.475003+0000","last_peered":"2024-09-13T08:41:31.475003+0000","last_clean":"2024-09-13T08:41:31.475003+0000","last_became_active":"2024-09-13T08:41:29.148567+0000","last_became_peered":"2024-09-13T08:41:29.148567+0000","last_unstale":"2024-09-13T08:41:31.475003+0000","last_undegraded":"2024-09-13T08:41:31.475003+0000","last_fullsized":"2024-09-13T08:41:31.475003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.901791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474879+0000","last_change":"2024-09-13T08:41:29.147550+0000","last_active":"2024-09-13T08:41:31.474879+0000","last_peered":"2024-09-13T08:41:31.474879+0000","last_clean":"2024-09-13T08:41:31.474879+0000","last_became_active":"2024-09-13T08:41:29.147183+0000","last_became_peered":"2024-09-13T08:41:29.147183+0000","last_unstale":"2024-09-13T08:41:31.474879+0000","last_undegraded":"2024-09-13T08:41:31.474879+0000","last_fullsized":"2024-09-13T08:41:31.474879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:09.961905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126086+0000","last_change":"2024-09-13T08:41:28.126086+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126086+0000","last_undegraded":"2024-09-13T08:41:28.126086+0000","last_fullsized":"2024-09-13T08:41:28.126086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130783+0000","last_change":"2024-09-13T08:41:28.130783+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130783+0000","last_undegraded":"2024-09-13T08:41:28.130783+0000","last_fullsized":"2024-09-13T08:41:28.130783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125578+0000","last_change":"2024-09-13T08:41:28.125578+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125578+0000","last_undegraded":"2024-09-13T08:41:28.125578+0000","last_fullsized":"2024-09-13T08:41:28.125578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123188+0000","last_change":"2024-09-13T08:41:28.123188+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123188+0000","last_undegraded":"2024-09-13T08:41:28.123188+0000","last_fullsized":"2024-09-13T08:41:28.123188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":201,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T08:41:28.121362+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:41:28.121362+0000","last_peered":"2024-09-13T08:41:28.121362+0000","last_clean":"2024-09-13T08:41:28.121362+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:41:28.121362+0000","last_undegraded":"2024-09-13T08:41:28.121362+0000","last_fullsized":"2024-09-13T08:41:28.121362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":1},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32048,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":880640,"data_stored":367242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509497,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1462272,"data_stored":951793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509498,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1462272,"data_stored":951793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":929792,"data_stored":387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:41:34.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:41:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 -- 172.21.15.31:0/2342454434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015bd70 msgr2=0x7fb9100cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 --2- 172.21.15.31:0/2342454434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015bd70 0x7fb9100cf850 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb900005fd0 tx=0x7fb90002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 -- 172.21.15.31:0/2342454434 shutdown_connections 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 --2- 172.21.15.31:0/2342454434 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 0x7fb9100cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 --2- 172.21.15.31:0/2342454434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015bd70 0x7fb9100cf850 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 -- 172.21.15.31:0/2342454434 >> 172.21.15.31:0/2342454434 conn(0x7fb9100cc910 msgr2=0x7fb9100cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 -- 172.21.15.31:0/2342454434 shutdown_connections 2024-09-13T08:41:34.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.608+0000 7fb915600640 1 -- 172.21.15.31:0/2342454434 wait complete. 2024-09-13T08:41:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.610+0000 7fb915600640 1 Processor -- start 2024-09-13T08:41:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.610+0000 7fb915600640 1 -- start start 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb91015bd70 0x7fb9100c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 0x7fb9100ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9100cec50 con 0x7fb91015f9e0 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9100cead0 con 0x7fb91015bd70 2024-09-13T08:41:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb915600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb9100cb850 con 0x7fb91015fe20 2024-09-13T08:41:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb906000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb906000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57192/0 (socket says 172.21.15.31:57192) 2024-09-13T08:41:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb906000640 1 -- 172.21.15.31:0/2138194427 learned_addr learned my addr 172.21.15.31:0/2138194427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb90ea00640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb91015bd70 0x7fb9100c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.611+0000 7fb90f400640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 0x7fb9100ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 -- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 msgr2=0x7fb9100ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 0x7fb9100ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 -- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb91015bd70 msgr2=0x7fb9100c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb91015bd70 0x7fb9100c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 -- 172.21.15.31:0/2138194427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9101f88c0 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb906000640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c450 tx=0x7fb8f800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80052d0 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8f8005470 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8012870 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9101f47d0 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.612+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101f4d90 con 0x7fb91015f9e0 2024-09-13T08:41:34.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.614+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9101f4f90 con 0x7fb91015f9e0 2024-09-13T08:41:34.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.615+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fb8f80035f0 con 0x7fb91015f9e0 2024-09-13T08:41:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.616+0000 7fb907400640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 0x7fb8ec07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.616+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fb8f809aa30 con 0x7fb91015f9e0 2024-09-13T08:41:34.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.616+0000 7fb90ea00640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 0x7fb8ec07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:34.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.617+0000 7fb90ea00640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 0x7fb8ec07b070 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb900033000 tx=0x7fb900004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:34.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.620+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb9101f4f90 con 0x7fb91015f9e0 2024-09-13T08:41:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.813+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9100cce70 con 0x7fb8ec078bb0 2024-09-13T08:41:34.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.818+0000 7fb907400640 1 -- 172.21.15.31:0/2138194427 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241258 (secure 0 0 0) 0x7fb9100cce70 con 0x7fb8ec078bb0 2024-09-13T08:41:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:34.822 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 msgr2=0x7fb8ec07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 0x7fb8ec07b070 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb900033000 tx=0x7fb900004730 comp rx=0 tx=0).stop 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 msgr2=0x7fb9100ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c450 tx=0x7fb8f800c910 comp rx=0 tx=0).stop 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 shutdown_connections 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb8ec078bb0 0x7fb8ec07b070 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb91015fe20 0x7fb9100ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb91015f9e0 0x7fb9100ade80 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 --2- 172.21.15.31:0/2138194427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb91015bd70 0x7fb9100c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 >> 172.21.15.31:0/2138194427 conn(0x7fb9100cc910 msgr2=0x7fb9100af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 shutdown_connections 2024-09-13T08:41:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:34.822+0000 7fb915600640 1 -- 172.21.15.31:0/2138194427 wait complete. 2024-09-13T08:41:34.839 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-13T08:41:33.103155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":105,"num_write_kb":1629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":76,"acting":76,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":257548,"kb_used_data":8016,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717172,"statfs":{"total":773094113280,"available":772830384128,"internally_reserved":0,"allocated":8208384,"data_stored":4103806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.982612"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475051+0000","last_change":"2024-09-13T08:41:30.167545+0000","last_active":"2024-09-13T08:41:31.475051+0000","last_peered":"2024-09-13T08:41:31.475051+0000","last_clean":"2024-09-13T08:41:31.475051+0000","last_became_active":"2024-09-13T08:41:30.167268+0000","last_became_peered":"2024-09-13T08:41:30.167268+0000","last_unstale":"2024-09-13T08:41:31.475051+0000","last_undegraded":"2024-09-13T08:41:31.475051+0000","last_fullsized":"2024-09-13T08:41:31.475051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:01.162295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474710+0000","last_change":"2024-09-13T08:41:30.172201+0000","last_active":"2024-09-13T08:41:31.474710+0000","last_peered":"2024-09-13T08:41:31.474710+0000","last_clean":"2024-09-13T08:41:31.474710+0000","last_became_active":"2024-09-13T08:41:30.171269+0000","last_became_peered":"2024-09-13T08:41:30.171269+0000","last_unstale":"2024-09-13T08:41:31.474710+0000","last_undegraded":"2024-09-13T08:41:31.474710+0000","last_fullsized":"2024-09-13T08:41:31.474710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.336999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474006+0000","last_change":"2024-09-13T08:41:30.169784+0000","last_active":"2024-09-13T08:41:31.474006+0000","last_peered":"2024-09-13T08:41:31.474006+0000","last_clean":"2024-09-13T08:41:31.474006+0000","last_became_active":"2024-09-13T08:41:30.169508+0000","last_became_peered":"2024-09-13T08:41:30.169508+0000","last_unstale":"2024-09-13T08:41:31.474006+0000","last_undegraded":"2024-09-13T08:41:31.474006+0000","last_fullsized":"2024-09-13T08:41:31.474006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:32.762281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125668+0000","last_change":"2024-09-13T08:41:28.125668+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125668+0000","last_undegraded":"2024-09-13T08:41:28.125668+0000","last_fullsized":"2024-09-13T08:41:28.125668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126514+0000","last_change":"2024-09-13T08:41:28.126514+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126514+0000","last_undegraded":"2024-09-13T08:41:28.126514+0000","last_fullsized":"2024-09-13T08:41:28.126514+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130480+0000","last_change":"2024-09-13T08:41:28.130480+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130480+0000","last_undegraded":"2024-09-13T08:41:28.130480+0000","last_fullsized":"2024-09-13T08:41:28.130480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126050+0000","last_change":"2024-09-13T08:41:28.126050+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126050+0000","last_undegraded":"2024-09-13T08:41:28.126050+0000","last_fullsized":"2024-09-13T08:41:28.126050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474813+0000","last_change":"2024-09-13T08:41:29.152256+0000","last_active":"2024-09-13T08:41:31.474813+0000","last_peered":"2024-09-13T08:41:31.474813+0000","last_clean":"2024-09-13T08:41:31.474813+0000","last_became_active":"2024-09-13T08:41:29.151880+0000","last_became_peered":"2024-09-13T08:41:29.151880+0000","last_unstale":"2024-09-13T08:41:31.474813+0000","last_undegraded":"2024-09-13T08:41:31.474813+0000","last_fullsized":"2024-09-13T08:41:31.474813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:45.557782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.124421+0000","last_change":"2024-09-13T08:41:28.124421+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.124421+0000","last_undegraded":"2024-09-13T08:41:28.124421+0000","last_fullsized":"2024-09-13T08:41:28.124421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126135+0000","last_change":"2024-09-13T08:41:28.126135+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126135+0000","last_undegraded":"2024-09-13T08:41:28.126135+0000","last_fullsized":"2024-09-13T08:41:28.126135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474073+0000","last_change":"2024-09-13T08:41:30.171632+0000","last_active":"2024-09-13T08:41:31.474073+0000","last_peered":"2024-09-13T08:41:31.474073+0000","last_clean":"2024-09-13T08:41:31.474073+0000","last_became_active":"2024-09-13T08:41:30.171351+0000","last_became_peered":"2024-09-13T08:41:30.171351+0000","last_unstale":"2024-09-13T08:41:31.474073+0000","last_undegraded":"2024-09-13T08:41:31.474073+0000","last_fullsized":"2024-09-13T08:41:31.474073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:36.601619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.473982+0000","last_change":"2024-09-13T08:41:29.141838+0000","last_active":"2024-09-13T08:41:31.473982+0000","last_peered":"2024-09-13T08:41:31.473982+0000","last_clean":"2024-09-13T08:41:31.473982+0000","last_became_active":"2024-09-13T08:41:29.141458+0000","last_became_peered":"2024-09-13T08:41:29.141458+0000","last_unstale":"2024-09-13T08:41:31.473982+0000","last_undegraded":"2024-09-13T08:41:31.473982+0000","last_fullsized":"2024-09-13T08:41:31.473982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:57.962170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.133318+0000","last_change":"2024-09-13T08:41:28.133318+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.133318+0000","last_undegraded":"2024-09-13T08:41:28.133318+0000","last_fullsized":"2024-09-13T08:41:28.133318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.131009+0000","last_change":"2024-09-13T08:41:28.131009+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.131009+0000","last_undegraded":"2024-09-13T08:41:28.131009+0000","last_fullsized":"2024-09-13T08:41:28.131009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474613+0000","last_change":"2024-09-13T08:41:29.147972+0000","last_active":"2024-09-13T08:41:31.474613+0000","last_peered":"2024-09-13T08:41:31.474613+0000","last_clean":"2024-09-13T08:41:31.474613+0000","last_became_active":"2024-09-13T08:41:29.147657+0000","last_became_peered":"2024-09-13T08:41:29.147657+0000","last_unstale":"2024-09-13T08:41:31.474613+0000","last_undegraded":"2024-09-13T08:41:31.474613+0000","last_fullsized":"2024-09-13T08:41:31.474613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:37.087395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126249+0000","last_change":"2024-09-13T08:41:28.126249+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126249+0000","last_undegraded":"2024-09-13T08:41:28.126249+0000","last_fullsized":"2024-09-13T08:41:28.126249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126123+0000","last_change":"2024-09-13T08:41:28.126123+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126123+0000","last_undegraded":"2024-09-13T08:41:28.126123+0000","last_fullsized":"2024-09-13T08:41:28.126123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123281+0000","last_change":"2024-09-13T08:41:28.123281+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123281+0000","last_undegraded":"2024-09-13T08:41:28.123281+0000","last_fullsized":"2024-09-13T08:41:28.123281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123170+0000","last_change":"2024-09-13T08:41:28.123170+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123170+0000","last_undegraded":"2024-09-13T08:41:28.123170+0000","last_fullsized":"2024-09-13T08:41:28.123170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123203+0000","last_change":"2024-09-13T08:41:28.123203+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123203+0000","last_undegraded":"2024-09-13T08:41:28.123203+0000","last_fullsized":"2024-09-13T08:41:28.123203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.132982+0000","last_change":"2024-09-13T08:41:28.132982+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.132982+0000","last_undegraded":"2024-09-13T08:41:28.132982+0000","last_fullsized":"2024-09-13T08:41:28.132982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130540+0000","last_change":"2024-09-13T08:41:28.130540+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130540+0000","last_undegraded":"2024-09-13T08:41:28.130540+0000","last_fullsized":"2024-09-13T08:41:28.130540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130589+0000","last_change":"2024-09-13T08:41:28.130589+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130589+0000","last_undegraded":"2024-09-13T08:41:28.130589+0000","last_fullsized":"2024-09-13T08:41:28.130589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125603+0000","last_change":"2024-09-13T08:41:28.125603+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125603+0000","last_undegraded":"2024-09-13T08:41:28.125603+0000","last_fullsized":"2024-09-13T08:41:28.125603+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125589+0000","last_change":"2024-09-13T08:41:28.125589+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125589+0000","last_undegraded":"2024-09-13T08:41:28.125589+0000","last_fullsized":"2024-09-13T08:41:28.125589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.131216+0000","last_change":"2024-09-13T08:41:28.131216+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.131216+0000","last_undegraded":"2024-09-13T08:41:28.131216+0000","last_fullsized":"2024-09-13T08:41:28.131216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474394+0000","last_change":"2024-09-13T08:41:30.158406+0000","last_active":"2024-09-13T08:41:31.474394+0000","last_peered":"2024-09-13T08:41:31.474394+0000","last_clean":"2024-09-13T08:41:31.474394+0000","last_became_active":"2024-09-13T08:41:30.157975+0000","last_became_peered":"2024-09-13T08:41:30.157975+0000","last_unstale":"2024-09-13T08:41:31.474394+0000","last_undegraded":"2024-09-13T08:41:31.474394+0000","last_fullsized":"2024-09-13T08:41:31.474394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:54.914413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.127296+0000","last_change":"2024-09-13T08:41:28.127296+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.127296+0000","last_undegraded":"2024-09-13T08:41:28.127296+0000","last_fullsized":"2024-09-13T08:41:28.127296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.129945+0000","last_change":"2024-09-13T08:41:28.129945+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.129945+0000","last_undegraded":"2024-09-13T08:41:28.129945+0000","last_fullsized":"2024-09-13T08:41:28.129945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.129993+0000","last_change":"2024-09-13T08:41:28.129993+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.129993+0000","last_undegraded":"2024-09-13T08:41:28.129993+0000","last_fullsized":"2024-09-13T08:41:28.129993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474730+0000","last_change":"2024-09-13T08:41:29.148076+0000","last_active":"2024-09-13T08:41:31.474730+0000","last_peered":"2024-09-13T08:41:31.474730+0000","last_clean":"2024-09-13T08:41:31.474730+0000","last_became_active":"2024-09-13T08:41:29.147012+0000","last_became_peered":"2024-09-13T08:41:29.147012+0000","last_unstale":"2024-09-13T08:41:31.474730+0000","last_undegraded":"2024-09-13T08:41:31.474730+0000","last_fullsized":"2024-09-13T08:41:31.474730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:18.467245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475003+0000","last_change":"2024-09-13T08:41:29.148851+0000","last_active":"2024-09-13T08:41:31.475003+0000","last_peered":"2024-09-13T08:41:31.475003+0000","last_clean":"2024-09-13T08:41:31.475003+0000","last_became_active":"2024-09-13T08:41:29.148567+0000","last_became_peered":"2024-09-13T08:41:29.148567+0000","last_unstale":"2024-09-13T08:41:31.475003+0000","last_undegraded":"2024-09-13T08:41:31.475003+0000","last_fullsized":"2024-09-13T08:41:31.475003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.901791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474879+0000","last_change":"2024-09-13T08:41:29.147550+0000","last_active":"2024-09-13T08:41:31.474879+0000","last_peered":"2024-09-13T08:41:31.474879+0000","last_clean":"2024-09-13T08:41:31.474879+0000","last_became_active":"2024-09-13T08:41:29.147183+0000","last_became_peered":"2024-09-13T08:41:29.147183+0000","last_unstale":"2024-09-13T08:41:31.474879+0000","last_undegraded":"2024-09-13T08:41:31.474879+0000","last_fullsized":"2024-09-13T08:41:31.474879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:09.961905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.126086+0000","last_change":"2024-09-13T08:41:28.126086+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.126086+0000","last_undegraded":"2024-09-13T08:41:28.126086+0000","last_fullsized":"2024-09-13T08:41:28.126086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.130783+0000","last_change":"2024-09-13T08:41:28.130783+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.130783+0000","last_undegraded":"2024-09-13T08:41:28.130783+0000","last_fullsized":"2024-09-13T08:41:28.130783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.125578+0000","last_change":"2024-09-13T08:41:28.125578+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.125578+0000","last_undegraded":"2024-09-13T08:41:28.125578+0000","last_fullsized":"2024-09-13T08:41:28.125578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:41:29.119254+0000","last_change":"2024-09-13T08:41:29.119254+0000","last_active":"2024-09-13T08:41:29.119254+0000","last_peered":"2024-09-13T08:41:29.119254+0000","last_clean":"2024-09-13T08:41:29.119254+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:29.119254+0000","last_undegraded":"2024-09-13T08:41:29.119254+0000","last_fullsized":"2024-09-13T08:41:29.119254+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-09-13T08:41:28.123188+0000","last_change":"2024-09-13T08:41:28.123188+0000","last_active":"2024-09-13T08:41:28.117026+0000","last_peered":"2024-09-13T08:41:28.117026+0000","last_clean":"2024-09-13T08:41:28.117026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:41:28.123188+0000","last_undegraded":"2024-09-13T08:41:28.123188+0000","last_fullsized":"2024-09-13T08:41:28.123188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":201,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T08:41:28.121362+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:41:28.121362+0000","last_peered":"2024-09-13T08:41:28.121362+0000","last_clean":"2024-09-13T08:41:28.121362+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:41:28.121362+0000","last_undegraded":"2024-09-13T08:41:28.121362+0000","last_fullsized":"2024-09-13T08:41:28.121362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":1},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32048,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":880640,"data_stored":367242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509497,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1462272,"data_stored":951793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509498,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1462272,"data_stored":951793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":361425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":929792,"data_stored":387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:41:34.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:41:35.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 -- 172.21.15.31:0/3695772263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38540cbc50 msgr2=0x7f38540cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 --2- 172.21.15.31:0/3695772263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38540cbc50 0x7f38540cc060 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f38540cfe60 tx=0x7f384c02a920 comp rx=0 tx=0).stop 2024-09-13T08:41:35.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 -- 172.21.15.31:0/3695772263 shutdown_connections 2024-09-13T08:41:35.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 --2- 172.21.15.31:0/3695772263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38540cbc50 0x7f38540cc060 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 -- 172.21.15.31:0/3695772263 >> 172.21.15.31:0/3695772263 conn(0x7f38540cc910 msgr2=0x7f38540cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 -- 172.21.15.31:0/3695772263 shutdown_connections 2024-09-13T08:41:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.040+0000 7f385aa00640 1 -- 172.21.15.31:0/3695772263 wait complete. 2024-09-13T08:41:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.042+0000 7f385aa00640 1 Processor -- start 2024-09-13T08:41:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.042+0000 7f385aa00640 1 -- start start 2024-09-13T08:41:35.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38540cbc50 0x7f38541502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38540cf850 0x7f385409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38541600f0 con 0x7f385415bd70 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f38540cead0 con 0x7f38540cbc50 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f385aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f38541507e0 con 0x7f38540cf850 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59674/0 (socket says 172.21.15.31:59674) 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 -- 172.21.15.31:0/498052009 learned_addr learned my addr 172.21.15.31:0/498052009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3853e00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38540cbc50 0x7f38541502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f384be00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38540cf850 0x7f385409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 -- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38540cf850 msgr2=0x7f385409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38540cf850 0x7f385409aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 -- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38540cbc50 msgr2=0x7f38541502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38540cbc50 0x7f38541502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.043+0000 7f3858a00640 1 -- 172.21.15.31:0/498052009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38541f3220 con 0x7f385415bd70 2024-09-13T08:41:35.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f3858a00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f384c009ca0 tx=0x7f384c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c004240 con 0x7f385415bd70 2024-09-13T08:41:35.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f384c0043e0 con 0x7f385415bd70 2024-09-13T08:41:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38541f3500 con 0x7f385415bd70 2024-09-13T08:41:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c0047b0 con 0x7f385415bd70 2024-09-13T08:41:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.044+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38540b2d60 con 0x7f385415bd70 2024-09-13T08:41:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.045+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38540cb850 con 0x7f385415bd70 2024-09-13T08:41:35.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.046+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f384c03b450 con 0x7f385415bd70 2024-09-13T08:41:35.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.047+0000 7f3850c00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 0x7f383007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.047+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f384c0bce60 con 0x7f385415bd70 2024-09-13T08:41:35.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.048+0000 7f3853e00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 0x7f383007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.049+0000 7f3853e00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 0x7f383007b0a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3844000f60 tx=0x7f3844009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.052+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f384c02d070 con 0x7f385415bd70 2024-09-13T08:41:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.249+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f38540af080 con 0x7f3830078be0 2024-09-13T08:41:35.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.254+0000 7f3850c00640 1 -- 172.21.15.31:0/498052009 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7f38540af080 con 0x7f3830078be0 2024-09-13T08:41:35.258 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:41:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 msgr2=0x7f383007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 0x7f383007b0a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3844000f60 tx=0x7f3844009040 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 msgr2=0x7f38541f2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f384c009ca0 tx=0x7f384c003ce0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 shutdown_connections 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f385415bd70 0x7f38541f2880 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3830078be0 0x7f383007b0a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38540cf850 0x7f385409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 --2- 172.21.15.31:0/498052009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38540cbc50 0x7f38541502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 >> 172.21.15.31:0/498052009 conn(0x7f38540cc910 msgr2=0x7f38540add90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 shutdown_connections 2024-09-13T08:41:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.258+0000 7f385aa00640 1 -- 172.21.15.31:0/498052009 wait complete. 2024-09-13T08:41:35.275 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-13T08:41:35.103549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262072,"kb_used_data":8392,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712648,"statfs":{"total":773094113280,"available":772825751552,"internally_reserved":0,"allocated":8593408,"data_stored":4278228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150712+0000","last_change":"2024-09-13T08:41:30.177953+0000","last_active":"2024-09-13T08:41:31.150712+0000","last_peered":"2024-09-13T08:41:31.150712+0000","last_clean":"2024-09-13T08:41:31.150712+0000","last_became_active":"2024-09-13T08:41:30.177705+0000","last_became_peered":"2024-09-13T08:41:30.177705+0000","last_unstale":"2024-09-13T08:41:31.150712+0000","last_undegraded":"2024-09-13T08:41:31.150712+0000","last_fullsized":"2024-09-13T08:41:31.150712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:51.644500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.171555+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.171277+0000","last_became_peered":"2024-09-13T08:41:30.171277+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152712+0000","last_change":"2024-09-13T08:41:30.184527+0000","last_active":"2024-09-13T08:41:31.152712+0000","last_peered":"2024-09-13T08:41:31.152712+0000","last_clean":"2024-09-13T08:41:31.152712+0000","last_became_active":"2024-09-13T08:41:30.184178+0000","last_became_peered":"2024-09-13T08:41:30.184178+0000","last_unstale":"2024-09-13T08:41:31.152712+0000","last_undegraded":"2024-09-13T08:41:31.152712+0000","last_fullsized":"2024-09-13T08:41:31.152712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:11.304643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152599+0000","last_change":"2024-09-13T08:41:30.177541+0000","last_active":"2024-09-13T08:41:31.152599+0000","last_peered":"2024-09-13T08:41:31.152599+0000","last_clean":"2024-09-13T08:41:31.152599+0000","last_became_active":"2024-09-13T08:41:30.177252+0000","last_became_peered":"2024-09-13T08:41:30.177252+0000","last_unstale":"2024-09-13T08:41:31.152599+0000","last_undegraded":"2024-09-13T08:41:31.152599+0000","last_fullsized":"2024-09-13T08:41:31.152599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:05.041863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152229+0000","last_change":"2024-09-13T08:41:30.176493+0000","last_active":"2024-09-13T08:41:31.152229+0000","last_peered":"2024-09-13T08:41:31.152229+0000","last_clean":"2024-09-13T08:41:31.152229+0000","last_became_active":"2024-09-13T08:41:30.176063+0000","last_became_peered":"2024-09-13T08:41:30.176063+0000","last_unstale":"2024-09-13T08:41:31.152229+0000","last_undegraded":"2024-09-13T08:41:31.152229+0000","last_fullsized":"2024-09-13T08:41:31.152229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:10.675747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475149+0000","last_change":"2024-09-13T08:41:30.180783+0000","last_active":"2024-09-13T08:41:31.475149+0000","last_peered":"2024-09-13T08:41:31.475149+0000","last_clean":"2024-09-13T08:41:31.475149+0000","last_became_active":"2024-09-13T08:41:30.180485+0000","last_became_peered":"2024-09-13T08:41:30.180485+0000","last_unstale":"2024-09-13T08:41:31.475149+0000","last_undegraded":"2024-09-13T08:41:31.475149+0000","last_fullsized":"2024-09-13T08:41:31.475149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474644+0000","last_change":"2024-09-13T08:41:30.178989+0000","last_active":"2024-09-13T08:41:31.474644+0000","last_peered":"2024-09-13T08:41:31.474644+0000","last_clean":"2024-09-13T08:41:31.474644+0000","last_became_active":"2024-09-13T08:41:30.178708+0000","last_became_peered":"2024-09-13T08:41:30.178708+0000","last_unstale":"2024-09-13T08:41:31.474644+0000","last_undegraded":"2024-09-13T08:41:31.474644+0000","last_fullsized":"2024-09-13T08:41:31.474644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:02.877430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149574+0000","last_change":"2024-09-13T08:41:30.176643+0000","last_active":"2024-09-13T08:41:31.149574+0000","last_peered":"2024-09-13T08:41:31.149574+0000","last_clean":"2024-09-13T08:41:31.149574+0000","last_became_active":"2024-09-13T08:41:30.176347+0000","last_became_peered":"2024-09-13T08:41:30.176347+0000","last_unstale":"2024-09-13T08:41:31.149574+0000","last_undegraded":"2024-09-13T08:41:31.149574+0000","last_fullsized":"2024-09-13T08:41:31.149574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:45.146060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152963+0000","last_change":"2024-09-13T08:41:30.156159+0000","last_active":"2024-09-13T08:41:31.152963+0000","last_peered":"2024-09-13T08:41:31.152963+0000","last_clean":"2024-09-13T08:41:31.152963+0000","last_became_active":"2024-09-13T08:41:30.155807+0000","last_became_peered":"2024-09-13T08:41:30.155807+0000","last_unstale":"2024-09-13T08:41:31.152963+0000","last_undegraded":"2024-09-13T08:41:31.152963+0000","last_fullsized":"2024-09-13T08:41:31.152963+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:11.356687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475634+0000","last_change":"2024-09-13T08:41:30.181209+0000","last_active":"2024-09-13T08:41:31.475634+0000","last_peered":"2024-09-13T08:41:31.475634+0000","last_clean":"2024-09-13T08:41:31.475634+0000","last_became_active":"2024-09-13T08:41:30.179580+0000","last_became_peered":"2024-09-13T08:41:30.179580+0000","last_unstale":"2024-09-13T08:41:31.475634+0000","last_undegraded":"2024-09-13T08:41:31.475634+0000","last_fullsized":"2024-09-13T08:41:31.475634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:36.230665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150715+0000","last_change":"2024-09-13T08:41:30.175255+0000","last_active":"2024-09-13T08:41:31.150715+0000","last_peered":"2024-09-13T08:41:31.150715+0000","last_clean":"2024-09-13T08:41:31.150715+0000","last_became_active":"2024-09-13T08:41:30.174389+0000","last_became_peered":"2024-09-13T08:41:30.174389+0000","last_unstale":"2024-09-13T08:41:31.150715+0000","last_undegraded":"2024-09-13T08:41:31.150715+0000","last_fullsized":"2024-09-13T08:41:31.150715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:57:33.926281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475051+0000","last_change":"2024-09-13T08:41:30.167545+0000","last_active":"2024-09-13T08:41:31.475051+0000","last_peered":"2024-09-13T08:41:31.475051+0000","last_clean":"2024-09-13T08:41:31.475051+0000","last_became_active":"2024-09-13T08:41:30.167268+0000","last_became_peered":"2024-09-13T08:41:30.167268+0000","last_unstale":"2024-09-13T08:41:31.475051+0000","last_undegraded":"2024-09-13T08:41:31.475051+0000","last_fullsized":"2024-09-13T08:41:31.475051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:01.162295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152039+0000","last_change":"2024-09-13T08:41:30.177514+0000","last_active":"2024-09-13T08:41:31.152039+0000","last_peered":"2024-09-13T08:41:31.152039+0000","last_clean":"2024-09-13T08:41:31.152039+0000","last_became_active":"2024-09-13T08:41:30.177234+0000","last_became_peered":"2024-09-13T08:41:30.177234+0000","last_unstale":"2024-09-13T08:41:31.152039+0000","last_undegraded":"2024-09-13T08:41:31.152039+0000","last_fullsized":"2024-09-13T08:41:31.152039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:59.795415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474710+0000","last_change":"2024-09-13T08:41:30.172201+0000","last_active":"2024-09-13T08:41:31.474710+0000","last_peered":"2024-09-13T08:41:31.474710+0000","last_clean":"2024-09-13T08:41:31.474710+0000","last_became_active":"2024-09-13T08:41:30.171269+0000","last_became_peered":"2024-09-13T08:41:30.171269+0000","last_unstale":"2024-09-13T08:41:31.474710+0000","last_undegraded":"2024-09-13T08:41:31.474710+0000","last_fullsized":"2024-09-13T08:41:31.474710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.336999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474006+0000","last_change":"2024-09-13T08:41:30.169784+0000","last_active":"2024-09-13T08:41:31.474006+0000","last_peered":"2024-09-13T08:41:31.474006+0000","last_clean":"2024-09-13T08:41:31.474006+0000","last_became_active":"2024-09-13T08:41:30.169508+0000","last_became_peered":"2024-09-13T08:41:30.169508+0000","last_unstale":"2024-09-13T08:41:31.474006+0000","last_undegraded":"2024-09-13T08:41:31.474006+0000","last_fullsized":"2024-09-13T08:41:31.474006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:32.762281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149585+0000","last_change":"2024-09-13T08:41:30.168753+0000","last_active":"2024-09-13T08:41:31.149585+0000","last_peered":"2024-09-13T08:41:31.149585+0000","last_clean":"2024-09-13T08:41:31.149585+0000","last_became_active":"2024-09-13T08:41:30.168428+0000","last_became_peered":"2024-09-13T08:41:30.168428+0000","last_unstale":"2024-09-13T08:41:31.149585+0000","last_undegraded":"2024-09-13T08:41:31.149585+0000","last_fullsized":"2024-09-13T08:41:31.149585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:28.857492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152183+0000","last_change":"2024-09-13T08:41:30.174176+0000","last_active":"2024-09-13T08:41:31.152183+0000","last_peered":"2024-09-13T08:41:31.152183+0000","last_clean":"2024-09-13T08:41:31.152183+0000","last_became_active":"2024-09-13T08:41:30.173898+0000","last_became_peered":"2024-09-13T08:41:30.173898+0000","last_unstale":"2024-09-13T08:41:31.152183+0000","last_undegraded":"2024-09-13T08:41:31.152183+0000","last_fullsized":"2024-09-13T08:41:31.152183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:43:41.193530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152346+0000","last_change":"2024-09-13T08:41:30.179501+0000","last_active":"2024-09-13T08:41:31.152346+0000","last_peered":"2024-09-13T08:41:31.152346+0000","last_clean":"2024-09-13T08:41:31.152346+0000","last_became_active":"2024-09-13T08:41:30.179215+0000","last_became_peered":"2024-09-13T08:41:30.179215+0000","last_unstale":"2024-09-13T08:41:31.152346+0000","last_undegraded":"2024-09-13T08:41:31.152346+0000","last_fullsized":"2024-09-13T08:41:31.152346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:41:33.182476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150478+0000","last_change":"2024-09-13T08:41:30.176585+0000","last_active":"2024-09-13T08:41:31.150478+0000","last_peered":"2024-09-13T08:41:31.150478+0000","last_clean":"2024-09-13T08:41:31.150478+0000","last_became_active":"2024-09-13T08:41:30.173677+0000","last_became_peered":"2024-09-13T08:41:30.173677+0000","last_unstale":"2024-09-13T08:41:31.150478+0000","last_undegraded":"2024-09-13T08:41:31.150478+0000","last_fullsized":"2024-09-13T08:41:31.150478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:18.526089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152370+0000","last_change":"2024-09-13T08:41:30.179532+0000","last_active":"2024-09-13T08:41:31.152370+0000","last_peered":"2024-09-13T08:41:31.152370+0000","last_clean":"2024-09-13T08:41:31.152370+0000","last_became_active":"2024-09-13T08:41:30.179182+0000","last_became_peered":"2024-09-13T08:41:30.179182+0000","last_unstale":"2024-09-13T08:41:31.152370+0000","last_undegraded":"2024-09-13T08:41:31.152370+0000","last_fullsized":"2024-09-13T08:41:31.152370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:03.650279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475052+0000","last_change":"2024-09-13T08:41:30.155465+0000","last_active":"2024-09-13T08:41:31.475052+0000","last_peered":"2024-09-13T08:41:31.475052+0000","last_clean":"2024-09-13T08:41:31.475052+0000","last_became_active":"2024-09-13T08:41:30.155056+0000","last_became_peered":"2024-09-13T08:41:30.155056+0000","last_unstale":"2024-09-13T08:41:31.475052+0000","last_undegraded":"2024-09-13T08:41:31.475052+0000","last_fullsized":"2024-09-13T08:41:31.475052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475329+0000","last_change":"2024-09-13T08:41:30.178147+0000","last_active":"2024-09-13T08:41:31.475329+0000","last_peered":"2024-09-13T08:41:31.475329+0000","last_clean":"2024-09-13T08:41:31.475329+0000","last_became_active":"2024-09-13T08:41:30.175910+0000","last_became_peered":"2024-09-13T08:41:30.175910+0000","last_unstale":"2024-09-13T08:41:31.475329+0000","last_undegraded":"2024-09-13T08:41:31.475329+0000","last_fullsized":"2024-09-13T08:41:31.475329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474864+0000","last_change":"2024-09-13T08:41:30.190277+0000","last_active":"2024-09-13T08:41:31.474864+0000","last_peered":"2024-09-13T08:41:31.474864+0000","last_clean":"2024-09-13T08:41:31.474864+0000","last_became_active":"2024-09-13T08:41:30.189911+0000","last_became_peered":"2024-09-13T08:41:30.189911+0000","last_unstale":"2024-09-13T08:41:31.474864+0000","last_undegraded":"2024-09-13T08:41:31.474864+0000","last_fullsized":"2024-09-13T08:41:31.474864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:46.494807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149710+0000","last_change":"2024-09-13T08:41:30.179319+0000","last_active":"2024-09-13T08:41:31.149710+0000","last_peered":"2024-09-13T08:41:31.149710+0000","last_clean":"2024-09-13T08:41:31.149710+0000","last_became_active":"2024-09-13T08:41:30.179050+0000","last_became_peered":"2024-09-13T08:41:30.179050+0000","last_unstale":"2024-09-13T08:41:31.149710+0000","last_undegraded":"2024-09-13T08:41:31.149710+0000","last_fullsized":"2024-09-13T08:41:31.149710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:01.953440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475362+0000","last_change":"2024-09-13T08:41:30.190025+0000","last_active":"2024-09-13T08:41:31.475362+0000","last_peered":"2024-09-13T08:41:31.475362+0000","last_clean":"2024-09-13T08:41:31.475362+0000","last_became_active":"2024-09-13T08:41:30.189682+0000","last_became_peered":"2024-09-13T08:41:30.189682+0000","last_unstale":"2024-09-13T08:41:31.475362+0000","last_undegraded":"2024-09-13T08:41:31.475362+0000","last_fullsized":"2024-09-13T08:41:31.475362+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:08.385273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150264+0000","last_change":"2024-09-13T08:41:30.190500+0000","last_active":"2024-09-13T08:41:31.150264+0000","last_peered":"2024-09-13T08:41:31.150264+0000","last_clean":"2024-09-13T08:41:31.150264+0000","last_became_active":"2024-09-13T08:41:30.190211+0000","last_became_peered":"2024-09-13T08:41:30.190211+0000","last_unstale":"2024-09-13T08:41:31.150264+0000","last_undegraded":"2024-09-13T08:41:31.150264+0000","last_fullsized":"2024-09-13T08:41:31.150264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:55:48.809149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151986+0000","last_change":"2024-09-13T08:41:30.168525+0000","last_active":"2024-09-13T08:41:31.151986+0000","last_peered":"2024-09-13T08:41:31.151986+0000","last_clean":"2024-09-13T08:41:31.151986+0000","last_became_active":"2024-09-13T08:41:30.167750+0000","last_became_peered":"2024-09-13T08:41:30.167750+0000","last_unstale":"2024-09-13T08:41:31.151986+0000","last_undegraded":"2024-09-13T08:41:31.151986+0000","last_fullsized":"2024-09-13T08:41:31.151986+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:37.422258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474553+0000","last_change":"2024-09-13T08:41:30.165606+0000","last_active":"2024-09-13T08:41:31.474553+0000","last_peered":"2024-09-13T08:41:31.474553+0000","last_clean":"2024-09-13T08:41:31.474553+0000","last_became_active":"2024-09-13T08:41:30.165277+0000","last_became_peered":"2024-09-13T08:41:30.165277+0000","last_unstale":"2024-09-13T08:41:31.474553+0000","last_undegraded":"2024-09-13T08:41:31.474553+0000","last_fullsized":"2024-09-13T08:41:31.474553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152266+0000","last_change":"2024-09-13T08:41:30.170008+0000","last_active":"2024-09-13T08:41:31.152266+0000","last_peered":"2024-09-13T08:41:31.152266+0000","last_clean":"2024-09-13T08:41:31.152266+0000","last_became_active":"2024-09-13T08:41:30.169516+0000","last_became_peered":"2024-09-13T08:41:30.169516+0000","last_unstale":"2024-09-13T08:41:31.152266+0000","last_undegraded":"2024-09-13T08:41:31.152266+0000","last_fullsized":"2024-09-13T08:41:31.152266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:12:20.327460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149957+0000","last_change":"2024-09-13T08:41:30.171602+0000","last_active":"2024-09-13T08:41:31.149957+0000","last_peered":"2024-09-13T08:41:31.149957+0000","last_clean":"2024-09-13T08:41:31.149957+0000","last_became_active":"2024-09-13T08:41:30.170882+0000","last_became_peered":"2024-09-13T08:41:30.170882+0000","last_unstale":"2024-09-13T08:41:31.149957+0000","last_undegraded":"2024-09-13T08:41:31.149957+0000","last_fullsized":"2024-09-13T08:41:31.149957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:17.491430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474487+0000","last_change":"2024-09-13T08:41:30.165505+0000","last_active":"2024-09-13T08:41:31.474487+0000","last_peered":"2024-09-13T08:41:31.474487+0000","last_clean":"2024-09-13T08:41:31.474487+0000","last_became_active":"2024-09-13T08:41:30.165205+0000","last_became_peered":"2024-09-13T08:41:30.165205+0000","last_unstale":"2024-09-13T08:41:31.474487+0000","last_undegraded":"2024-09-13T08:41:31.474487+0000","last_fullsized":"2024-09-13T08:41:31.474487+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:53.060236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474225+0000","last_change":"2024-09-13T08:41:30.166147+0000","last_active":"2024-09-13T08:41:31.474225+0000","last_peered":"2024-09-13T08:41:31.474225+0000","last_clean":"2024-09-13T08:41:31.474225+0000","last_became_active":"2024-09-13T08:41:30.165114+0000","last_became_peered":"2024-09-13T08:41:30.165114+0000","last_unstale":"2024-09-13T08:41:31.474225+0000","last_undegraded":"2024-09-13T08:41:31.474225+0000","last_fullsized":"2024-09-13T08:41:31.474225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:53.742788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150526+0000","last_change":"2024-09-13T08:41:30.184240+0000","last_active":"2024-09-13T08:41:31.150526+0000","last_peered":"2024-09-13T08:41:31.150526+0000","last_clean":"2024-09-13T08:41:31.150526+0000","last_became_active":"2024-09-13T08:41:30.183893+0000","last_became_peered":"2024-09-13T08:41:30.183893+0000","last_unstale":"2024-09-13T08:41:31.150526+0000","last_undegraded":"2024-09-13T08:41:31.150526+0000","last_fullsized":"2024-09-13T08:41:31.150526+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:51.357207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"75'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474599+0000","last_change":"2024-09-13T08:41:29.151268+0000","last_active":"2024-09-13T08:41:31.474599+0000","last_peered":"2024-09-13T08:41:31.474599+0000","last_clean":"2024-09-13T08:41:31.474599+0000","last_became_active":"2024-09-13T08:41:29.150960+0000","last_became_peered":"2024-09-13T08:41:29.150960+0000","last_unstale":"2024-09-13T08:41:31.474599+0000","last_undegraded":"2024-09-13T08:41:31.474599+0000","last_fullsized":"2024-09-13T08:41:31.474599+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:42.453175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152927+0000","last_change":"2024-09-13T08:41:30.164561+0000","last_active":"2024-09-13T08:41:31.152927+0000","last_peered":"2024-09-13T08:41:31.152927+0000","last_clean":"2024-09-13T08:41:31.152927+0000","last_became_active":"2024-09-13T08:41:30.164290+0000","last_became_peered":"2024-09-13T08:41:30.164290+0000","last_unstale":"2024-09-13T08:41:31.152927+0000","last_undegraded":"2024-09-13T08:41:31.152927+0000","last_fullsized":"2024-09-13T08:41:31.152927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:33.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152403+0000","last_change":"2024-09-13T08:41:29.144190+0000","last_active":"2024-09-13T08:41:31.152403+0000","last_peered":"2024-09-13T08:41:31.152403+0000","last_clean":"2024-09-13T08:41:31.152403+0000","last_became_active":"2024-09-13T08:41:29.143622+0000","last_became_peered":"2024-09-13T08:41:29.143622+0000","last_unstale":"2024-09-13T08:41:31.152403+0000","last_undegraded":"2024-09-13T08:41:31.152403+0000","last_fullsized":"2024-09-13T08:41:31.152403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:37.575695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474834+0000","last_change":"2024-09-13T08:41:30.176252+0000","last_active":"2024-09-13T08:41:31.474834+0000","last_peered":"2024-09-13T08:41:31.474834+0000","last_clean":"2024-09-13T08:41:31.474834+0000","last_became_active":"2024-09-13T08:41:30.174505+0000","last_became_peered":"2024-09-13T08:41:30.174505+0000","last_unstale":"2024-09-13T08:41:31.474834+0000","last_undegraded":"2024-09-13T08:41:31.474834+0000","last_fullsized":"2024-09-13T08:41:31.474834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150540+0000","last_change":"2024-09-13T08:41:29.153502+0000","last_active":"2024-09-13T08:41:31.150540+0000","last_peered":"2024-09-13T08:41:31.150540+0000","last_clean":"2024-09-13T08:41:31.150540+0000","last_became_active":"2024-09-13T08:41:29.153256+0000","last_became_peered":"2024-09-13T08:41:29.153256+0000","last_unstale":"2024-09-13T08:41:31.150540+0000","last_undegraded":"2024-09-13T08:41:31.150540+0000","last_fullsized":"2024-09-13T08:41:31.150540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:11.750149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475366+0000","last_change":"2024-09-13T08:41:30.180280+0000","last_active":"2024-09-13T08:41:31.475366+0000","last_peered":"2024-09-13T08:41:31.475366+0000","last_clean":"2024-09-13T08:41:31.475366+0000","last_became_active":"2024-09-13T08:41:30.179979+0000","last_became_peered":"2024-09-13T08:41:30.179979+0000","last_unstale":"2024-09-13T08:41:31.475366+0000","last_undegraded":"2024-09-13T08:41:31.475366+0000","last_fullsized":"2024-09-13T08:41:31.475366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:39.299088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474892+0000","last_change":"2024-09-13T08:41:29.142380+0000","last_active":"2024-09-13T08:41:31.474892+0000","last_peered":"2024-09-13T08:41:31.474892+0000","last_clean":"2024-09-13T08:41:31.474892+0000","last_became_active":"2024-09-13T08:41:29.142093+0000","last_became_peered":"2024-09-13T08:41:29.142093+0000","last_unstale":"2024-09-13T08:41:31.474892+0000","last_undegraded":"2024-09-13T08:41:31.474892+0000","last_fullsized":"2024-09-13T08:41:31.474892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:05.945048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149958+0000","last_change":"2024-09-13T08:41:30.175452+0000","last_active":"2024-09-13T08:41:31.149958+0000","last_peered":"2024-09-13T08:41:31.149958+0000","last_clean":"2024-09-13T08:41:31.149958+0000","last_became_active":"2024-09-13T08:41:30.175101+0000","last_became_peered":"2024-09-13T08:41:30.175101+0000","last_unstale":"2024-09-13T08:41:31.149958+0000","last_undegraded":"2024-09-13T08:41:31.149958+0000","last_fullsized":"2024-09-13T08:41:31.149958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:35.678193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474813+0000","last_change":"2024-09-13T08:41:29.152256+0000","last_active":"2024-09-13T08:41:31.474813+0000","last_peered":"2024-09-13T08:41:31.474813+0000","last_clean":"2024-09-13T08:41:31.474813+0000","last_became_active":"2024-09-13T08:41:29.151880+0000","last_became_peered":"2024-09-13T08:41:29.151880+0000","last_unstale":"2024-09-13T08:41:31.474813+0000","last_undegraded":"2024-09-13T08:41:31.474813+0000","last_fullsized":"2024-09-13T08:41:31.474813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:45.557782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474596+0000","last_change":"2024-09-13T08:41:30.179754+0000","last_active":"2024-09-13T08:41:31.474596+0000","last_peered":"2024-09-13T08:41:31.474596+0000","last_clean":"2024-09-13T08:41:31.474596+0000","last_became_active":"2024-09-13T08:41:30.176002+0000","last_became_peered":"2024-09-13T08:41:30.176002+0000","last_unstale":"2024-09-13T08:41:31.474596+0000","last_undegraded":"2024-09-13T08:41:31.474596+0000","last_fullsized":"2024-09-13T08:41:31.474596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474629+0000","last_change":"2024-09-13T08:41:29.144056+0000","last_active":"2024-09-13T08:41:31.474629+0000","last_peered":"2024-09-13T08:41:31.474629+0000","last_clean":"2024-09-13T08:41:31.474629+0000","last_became_active":"2024-09-13T08:41:29.143787+0000","last_became_peered":"2024-09-13T08:41:29.143787+0000","last_unstale":"2024-09-13T08:41:31.474629+0000","last_undegraded":"2024-09-13T08:41:31.474629+0000","last_fullsized":"2024-09-13T08:41:31.474629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:07.698273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474317+0000","last_change":"2024-09-13T08:41:30.176412+0000","last_active":"2024-09-13T08:41:31.474317+0000","last_peered":"2024-09-13T08:41:31.474317+0000","last_clean":"2024-09-13T08:41:31.474317+0000","last_became_active":"2024-09-13T08:41:30.176124+0000","last_became_peered":"2024-09-13T08:41:30.176124+0000","last_unstale":"2024-09-13T08:41:31.474317+0000","last_undegraded":"2024-09-13T08:41:31.474317+0000","last_fullsized":"2024-09-13T08:41:31.474317+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:19:31.333677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474680+0000","last_change":"2024-09-13T08:41:29.142404+0000","last_active":"2024-09-13T08:41:31.474680+0000","last_peered":"2024-09-13T08:41:31.474680+0000","last_clean":"2024-09-13T08:41:31.474680+0000","last_became_active":"2024-09-13T08:41:29.142078+0000","last_became_peered":"2024-09-13T08:41:29.142078+0000","last_unstale":"2024-09-13T08:41:31.474680+0000","last_undegraded":"2024-09-13T08:41:31.474680+0000","last_fullsized":"2024-09-13T08:41:31.474680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:42.389929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474073+0000","last_change":"2024-09-13T08:41:30.171632+0000","last_active":"2024-09-13T08:41:31.474073+0000","last_peered":"2024-09-13T08:41:31.474073+0000","last_clean":"2024-09-13T08:41:31.474073+0000","last_became_active":"2024-09-13T08:41:30.171351+0000","last_became_peered":"2024-09-13T08:41:30.171351+0000","last_unstale":"2024-09-13T08:41:31.474073+0000","last_undegraded":"2024-09-13T08:41:31.474073+0000","last_fullsized":"2024-09-13T08:41:31.474073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:36.601619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.473982+0000","last_change":"2024-09-13T08:41:29.141838+0000","last_active":"2024-09-13T08:41:31.473982+0000","last_peered":"2024-09-13T08:41:31.473982+0000","last_clean":"2024-09-13T08:41:31.473982+0000","last_became_active":"2024-09-13T08:41:29.141458+0000","last_became_peered":"2024-09-13T08:41:29.141458+0000","last_unstale":"2024-09-13T08:41:31.473982+0000","last_undegraded":"2024-09-13T08:41:31.473982+0000","last_fullsized":"2024-09-13T08:41:31.473982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:57.962170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475749+0000","last_change":"2024-09-13T08:41:30.186535+0000","last_active":"2024-09-13T08:41:31.475749+0000","last_peered":"2024-09-13T08:41:31.475749+0000","last_clean":"2024-09-13T08:41:31.475749+0000","last_became_active":"2024-09-13T08:41:30.186257+0000","last_became_peered":"2024-09-13T08:41:30.186257+0000","last_unstale":"2024-09-13T08:41:31.475749+0000","last_undegraded":"2024-09-13T08:41:31.475749+0000","last_fullsized":"2024-09-13T08:41:31.475749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:05.191523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150332+0000","last_change":"2024-09-13T08:41:29.149880+0000","last_active":"2024-09-13T08:41:31.150332+0000","last_peered":"2024-09-13T08:41:31.150332+0000","last_clean":"2024-09-13T08:41:31.150332+0000","last_became_active":"2024-09-13T08:41:29.149556+0000","last_became_peered":"2024-09-13T08:41:29.149556+0000","last_unstale":"2024-09-13T08:41:31.150332+0000","last_undegraded":"2024-09-13T08:41:31.150332+0000","last_fullsized":"2024-09-13T08:41:31.150332+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:39.387524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475671+0000","last_change":"2024-09-13T08:41:30.171210+0000","last_active":"2024-09-13T08:41:31.475671+0000","last_peered":"2024-09-13T08:41:31.475671+0000","last_clean":"2024-09-13T08:41:31.475671+0000","last_became_active":"2024-09-13T08:41:30.170494+0000","last_became_peered":"2024-09-13T08:41:30.170494+0000","last_unstale":"2024-09-13T08:41:31.475671+0000","last_undegraded":"2024-09-13T08:41:31.475671+0000","last_fullsized":"2024-09-13T08:41:31.475671+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475717+0000","last_change":"2024-09-13T08:41:29.154606+0000","last_active":"2024-09-13T08:41:31.475717+0000","last_peered":"2024-09-13T08:41:31.475717+0000","last_clean":"2024-09-13T08:41:31.475717+0000","last_became_active":"2024-09-13T08:41:29.154326+0000","last_became_peered":"2024-09-13T08:41:29.154326+0000","last_unstale":"2024-09-13T08:41:31.475717+0000","last_undegraded":"2024-09-13T08:41:31.475717+0000","last_fullsized":"2024-09-13T08:41:31.475717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:52.695133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150201+0000","last_change":"2024-09-13T08:41:30.191552+0000","last_active":"2024-09-13T08:41:31.150201+0000","last_peered":"2024-09-13T08:41:31.150201+0000","last_clean":"2024-09-13T08:41:31.150201+0000","last_became_active":"2024-09-13T08:41:30.191295+0000","last_became_peered":"2024-09-13T08:41:30.191295+0000","last_unstale":"2024-09-13T08:41:31.150201+0000","last_undegraded":"2024-09-13T08:41:31.150201+0000","last_fullsized":"2024-09-13T08:41:31.150201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:34.894273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474613+0000","last_change":"2024-09-13T08:41:29.147972+0000","last_active":"2024-09-13T08:41:31.474613+0000","last_peered":"2024-09-13T08:41:31.474613+0000","last_clean":"2024-09-13T08:41:31.474613+0000","last_became_active":"2024-09-13T08:41:29.147657+0000","last_became_peered":"2024-09-13T08:41:29.147657+0000","last_unstale":"2024-09-13T08:41:31.474613+0000","last_undegraded":"2024-09-13T08:41:31.474613+0000","last_fullsized":"2024-09-13T08:41:31.474613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:37.087395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152158+0000","last_change":"2024-09-13T08:41:30.168763+0000","last_active":"2024-09-13T08:41:31.152158+0000","last_peered":"2024-09-13T08:41:31.152158+0000","last_clean":"2024-09-13T08:41:31.152158+0000","last_became_active":"2024-09-13T08:41:30.168493+0000","last_became_peered":"2024-09-13T08:41:30.168493+0000","last_unstale":"2024-09-13T08:41:31.152158+0000","last_undegraded":"2024-09-13T08:41:31.152158+0000","last_fullsized":"2024-09-13T08:41:31.152158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:27.265527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"75'6","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474699+0000","last_change":"2024-09-13T08:41:29.142862+0000","last_active":"2024-09-13T08:41:31.474699+0000","last_peered":"2024-09-13T08:41:31.474699+0000","last_clean":"2024-09-13T08:41:31.474699+0000","last_became_active":"2024-09-13T08:41:29.140928+0000","last_became_peered":"2024-09-13T08:41:29.140928+0000","last_unstale":"2024-09-13T08:41:31.474699+0000","last_undegraded":"2024-09-13T08:41:31.474699+0000","last_fullsized":"2024-09-13T08:41:31.474699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:29:54.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475544+0000","last_change":"2024-09-13T08:41:30.166884+0000","last_active":"2024-09-13T08:41:31.475544+0000","last_peered":"2024-09-13T08:41:31.475544+0000","last_clean":"2024-09-13T08:41:31.475544+0000","last_became_active":"2024-09-13T08:41:30.166578+0000","last_became_peered":"2024-09-13T08:41:30.166578+0000","last_unstale":"2024-09-13T08:41:31.475544+0000","last_undegraded":"2024-09-13T08:41:31.475544+0000","last_fullsized":"2024-09-13T08:41:31.475544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474519+0000","last_change":"2024-09-13T08:41:29.144444+0000","last_active":"2024-09-13T08:41:31.474519+0000","last_peered":"2024-09-13T08:41:31.474519+0000","last_clean":"2024-09-13T08:41:31.474519+0000","last_became_active":"2024-09-13T08:41:29.144236+0000","last_became_peered":"2024-09-13T08:41:29.144236+0000","last_unstale":"2024-09-13T08:41:31.474519+0000","last_undegraded":"2024-09-13T08:41:31.474519+0000","last_fullsized":"2024-09-13T08:41:31.474519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:31.977262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475053+0000","last_change":"2024-09-13T08:41:30.178971+0000","last_active":"2024-09-13T08:41:31.475053+0000","last_peered":"2024-09-13T08:41:31.475053+0000","last_clean":"2024-09-13T08:41:31.475053+0000","last_became_active":"2024-09-13T08:41:30.178662+0000","last_became_peered":"2024-09-13T08:41:30.178662+0000","last_unstale":"2024-09-13T08:41:31.475053+0000","last_undegraded":"2024-09-13T08:41:31.475053+0000","last_fullsized":"2024-09-13T08:41:31.475053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.425812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474733+0000","last_change":"2024-09-13T08:41:29.148940+0000","last_active":"2024-09-13T08:41:31.474733+0000","last_peered":"2024-09-13T08:41:31.474733+0000","last_clean":"2024-09-13T08:41:31.474733+0000","last_became_active":"2024-09-13T08:41:29.148677+0000","last_became_peered":"2024-09-13T08:41:29.148677+0000","last_unstale":"2024-09-13T08:41:31.474733+0000","last_undegraded":"2024-09-13T08:41:31.474733+0000","last_fullsized":"2024-09-13T08:41:31.474733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:36:49.700010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151295+0000","last_change":"2024-09-13T08:41:30.176836+0000","last_active":"2024-09-13T08:41:31.151295+0000","last_peered":"2024-09-13T08:41:31.151295+0000","last_clean":"2024-09-13T08:41:31.151295+0000","last_became_active":"2024-09-13T08:41:30.176561+0000","last_became_peered":"2024-09-13T08:41:30.176561+0000","last_unstale":"2024-09-13T08:41:31.151295+0000","last_undegraded":"2024-09-13T08:41:31.151295+0000","last_fullsized":"2024-09-13T08:41:31.151295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:27.201322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474722+0000","last_change":"2024-09-13T08:41:29.137115+0000","last_active":"2024-09-13T08:41:31.474722+0000","last_peered":"2024-09-13T08:41:31.474722+0000","last_clean":"2024-09-13T08:41:31.474722+0000","last_became_active":"2024-09-13T08:41:29.136800+0000","last_became_peered":"2024-09-13T08:41:29.136800+0000","last_unstale":"2024-09-13T08:41:31.474722+0000","last_undegraded":"2024-09-13T08:41:31.474722+0000","last_fullsized":"2024-09-13T08:41:31.474722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:53.975223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151142+0000","last_change":"2024-09-13T08:41:30.164645+0000","last_active":"2024-09-13T08:41:31.151142+0000","last_peered":"2024-09-13T08:41:31.151142+0000","last_clean":"2024-09-13T08:41:31.151142+0000","last_became_active":"2024-09-13T08:41:30.164362+0000","last_became_peered":"2024-09-13T08:41:30.164362+0000","last_unstale":"2024-09-13T08:41:31.151142+0000","last_undegraded":"2024-09-13T08:41:31.151142+0000","last_fullsized":"2024-09-13T08:41:31.151142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:39.424459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"75'8","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474024+0000","last_change":"2024-09-13T08:41:29.141724+0000","last_active":"2024-09-13T08:41:31.474024+0000","last_peered":"2024-09-13T08:41:31.474024+0000","last_clean":"2024-09-13T08:41:31.474024+0000","last_became_active":"2024-09-13T08:41:29.141410+0000","last_became_peered":"2024-09-13T08:41:29.141410+0000","last_unstale":"2024-09-13T08:41:31.474024+0000","last_undegraded":"2024-09-13T08:41:31.474024+0000","last_fullsized":"2024-09-13T08:41:31.474024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:36.990282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475780+0000","last_change":"2024-09-13T08:41:30.170258+0000","last_active":"2024-09-13T08:41:31.475780+0000","last_peered":"2024-09-13T08:41:31.475780+0000","last_clean":"2024-09-13T08:41:31.475780+0000","last_became_active":"2024-09-13T08:41:30.169900+0000","last_became_peered":"2024-09-13T08:41:30.169900+0000","last_unstale":"2024-09-13T08:41:31.475780+0000","last_undegraded":"2024-09-13T08:41:31.475780+0000","last_fullsized":"2024-09-13T08:41:31.475780+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.577924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150429+0000","last_change":"2024-09-13T08:41:29.152478+0000","last_active":"2024-09-13T08:41:31.150429+0000","last_peered":"2024-09-13T08:41:31.150429+0000","last_clean":"2024-09-13T08:41:31.150429+0000","last_became_active":"2024-09-13T08:41:29.152174+0000","last_became_peered":"2024-09-13T08:41:29.152174+0000","last_unstale":"2024-09-13T08:41:31.150429+0000","last_undegraded":"2024-09-13T08:41:31.150429+0000","last_fullsized":"2024-09-13T08:41:31.150429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:27.809813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475948+0000","last_change":"2024-09-13T08:41:30.157129+0000","last_active":"2024-09-13T08:41:31.475948+0000","last_peered":"2024-09-13T08:41:31.475948+0000","last_clean":"2024-09-13T08:41:31.475948+0000","last_became_active":"2024-09-13T08:41:30.156648+0000","last_became_peered":"2024-09-13T08:41:30.156648+0000","last_unstale":"2024-09-13T08:41:31.475948+0000","last_undegraded":"2024-09-13T08:41:31.475948+0000","last_fullsized":"2024-09-13T08:41:31.475948+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:28.126509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475814+0000","last_change":"2024-09-13T08:41:29.156147+0000","last_active":"2024-09-13T08:41:31.475814+0000","last_peered":"2024-09-13T08:41:31.475814+0000","last_clean":"2024-09-13T08:41:31.475814+0000","last_became_active":"2024-09-13T08:41:29.155857+0000","last_became_peered":"2024-09-13T08:41:29.155857+0000","last_unstale":"2024-09-13T08:41:31.475814+0000","last_undegraded":"2024-09-13T08:41:31.475814+0000","last_fullsized":"2024-09-13T08:41:31.475814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:27.046693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152642+0000","last_change":"2024-09-13T08:41:30.160794+0000","last_active":"2024-09-13T08:41:31.152642+0000","last_peered":"2024-09-13T08:41:31.152642+0000","last_clean":"2024-09-13T08:41:31.152642+0000","last_became_active":"2024-09-13T08:41:30.160339+0000","last_became_peered":"2024-09-13T08:41:30.160339+0000","last_unstale":"2024-09-13T08:41:31.152642+0000","last_undegraded":"2024-09-13T08:41:31.152642+0000","last_fullsized":"2024-09-13T08:41:31.152642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:56.686232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474046+0000","last_change":"2024-09-13T08:41:29.154023+0000","last_active":"2024-09-13T08:41:31.474046+0000","last_peered":"2024-09-13T08:41:31.474046+0000","last_clean":"2024-09-13T08:41:31.474046+0000","last_became_active":"2024-09-13T08:41:29.152087+0000","last_became_peered":"2024-09-13T08:41:29.152087+0000","last_unstale":"2024-09-13T08:41:31.474046+0000","last_undegraded":"2024-09-13T08:41:31.474046+0000","last_fullsized":"2024-09-13T08:41:31.474046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:38:02.653402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152683+0000","last_change":"2024-09-13T08:41:30.170644+0000","last_active":"2024-09-13T08:41:31.152683+0000","last_peered":"2024-09-13T08:41:31.152683+0000","last_clean":"2024-09-13T08:41:31.152683+0000","last_became_active":"2024-09-13T08:41:30.170350+0000","last_became_peered":"2024-09-13T08:41:30.170350+0000","last_unstale":"2024-09-13T08:41:31.152683+0000","last_undegraded":"2024-09-13T08:41:31.152683+0000","last_fullsized":"2024-09-13T08:41:31.152683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:58.357807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474990+0000","last_change":"2024-09-13T08:41:29.143199+0000","last_active":"2024-09-13T08:41:31.474990+0000","last_peered":"2024-09-13T08:41:31.474990+0000","last_clean":"2024-09-13T08:41:31.474990+0000","last_became_active":"2024-09-13T08:41:29.142930+0000","last_became_peered":"2024-09-13T08:41:29.142930+0000","last_unstale":"2024-09-13T08:41:31.474990+0000","last_undegraded":"2024-09-13T08:41:31.474990+0000","last_fullsized":"2024-09-13T08:41:31.474990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:36.019466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474439+0000","last_change":"2024-09-13T08:41:30.168382+0000","last_active":"2024-09-13T08:41:31.474439+0000","last_peered":"2024-09-13T08:41:31.474439+0000","last_clean":"2024-09-13T08:41:31.474439+0000","last_became_active":"2024-09-13T08:41:30.167748+0000","last_became_peered":"2024-09-13T08:41:30.167748+0000","last_unstale":"2024-09-13T08:41:31.474439+0000","last_undegraded":"2024-09-13T08:41:31.474439+0000","last_fullsized":"2024-09-13T08:41:31.474439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:43.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475680+0000","last_change":"2024-09-13T08:41:29.148961+0000","last_active":"2024-09-13T08:41:31.475680+0000","last_peered":"2024-09-13T08:41:31.475680+0000","last_clean":"2024-09-13T08:41:31.475680+0000","last_became_active":"2024-09-13T08:41:29.146591+0000","last_became_peered":"2024-09-13T08:41:29.146591+0000","last_unstale":"2024-09-13T08:41:31.475680+0000","last_undegraded":"2024-09-13T08:41:31.475680+0000","last_fullsized":"2024-09-13T08:41:31.475680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:25.694758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474398+0000","last_change":"2024-09-13T08:41:30.175697+0000","last_active":"2024-09-13T08:41:31.474398+0000","last_peered":"2024-09-13T08:41:31.474398+0000","last_clean":"2024-09-13T08:41:31.474398+0000","last_became_active":"2024-09-13T08:41:30.174133+0000","last_became_peered":"2024-09-13T08:41:30.174133+0000","last_unstale":"2024-09-13T08:41:31.474398+0000","last_undegraded":"2024-09-13T08:41:31.474398+0000","last_fullsized":"2024-09-13T08:41:31.474398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:45.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474547+0000","last_change":"2024-09-13T08:41:29.154197+0000","last_active":"2024-09-13T08:41:31.474547+0000","last_peered":"2024-09-13T08:41:31.474547+0000","last_clean":"2024-09-13T08:41:31.474547+0000","last_became_active":"2024-09-13T08:41:29.152859+0000","last_became_peered":"2024-09-13T08:41:29.152859+0000","last_unstale":"2024-09-13T08:41:31.474547+0000","last_undegraded":"2024-09-13T08:41:31.474547+0000","last_fullsized":"2024-09-13T08:41:31.474547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:35.228437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474394+0000","last_change":"2024-09-13T08:41:30.158406+0000","last_active":"2024-09-13T08:41:31.474394+0000","last_peered":"2024-09-13T08:41:31.474394+0000","last_clean":"2024-09-13T08:41:31.474394+0000","last_became_active":"2024-09-13T08:41:30.157975+0000","last_became_peered":"2024-09-13T08:41:30.157975+0000","last_unstale":"2024-09-13T08:41:31.474394+0000","last_undegraded":"2024-09-13T08:41:31.474394+0000","last_fullsized":"2024-09-13T08:41:31.474394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:54.914413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150950+0000","last_change":"2024-09-13T08:41:29.154649+0000","last_active":"2024-09-13T08:41:31.150950+0000","last_peered":"2024-09-13T08:41:31.150950+0000","last_clean":"2024-09-13T08:41:31.150950+0000","last_became_active":"2024-09-13T08:41:29.154327+0000","last_became_peered":"2024-09-13T08:41:29.154327+0000","last_unstale":"2024-09-13T08:41:31.150950+0000","last_undegraded":"2024-09-13T08:41:31.150950+0000","last_fullsized":"2024-09-13T08:41:31.150950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:18.313976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152456+0000","last_change":"2024-09-13T08:41:30.173642+0000","last_active":"2024-09-13T08:41:31.152456+0000","last_peered":"2024-09-13T08:41:31.152456+0000","last_clean":"2024-09-13T08:41:31.152456+0000","last_became_active":"2024-09-13T08:41:30.173362+0000","last_became_peered":"2024-09-13T08:41:30.173362+0000","last_unstale":"2024-09-13T08:41:31.152456+0000","last_undegraded":"2024-09-13T08:41:31.152456+0000","last_fullsized":"2024-09-13T08:41:31.152456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.753257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150731+0000","last_change":"2024-09-13T08:41:29.150523+0000","last_active":"2024-09-13T08:41:31.150731+0000","last_peered":"2024-09-13T08:41:31.150731+0000","last_clean":"2024-09-13T08:41:31.150731+0000","last_became_active":"2024-09-13T08:41:29.149999+0000","last_became_peered":"2024-09-13T08:41:29.149999+0000","last_unstale":"2024-09-13T08:41:31.150731+0000","last_undegraded":"2024-09-13T08:41:31.150731+0000","last_fullsized":"2024-09-13T08:41:31.150731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:32.616347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475559+0000","last_change":"2024-09-13T08:41:30.172825+0000","last_active":"2024-09-13T08:41:31.475559+0000","last_peered":"2024-09-13T08:41:31.475559+0000","last_clean":"2024-09-13T08:41:31.475559+0000","last_became_active":"2024-09-13T08:41:30.172459+0000","last_became_peered":"2024-09-13T08:41:30.172459+0000","last_unstale":"2024-09-13T08:41:31.475559+0000","last_undegraded":"2024-09-13T08:41:31.475559+0000","last_fullsized":"2024-09-13T08:41:31.475559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:08.614713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150663+0000","last_change":"2024-09-13T08:41:29.156767+0000","last_active":"2024-09-13T08:41:31.150663+0000","last_peered":"2024-09-13T08:41:31.150663+0000","last_clean":"2024-09-13T08:41:31.150663+0000","last_became_active":"2024-09-13T08:41:29.156127+0000","last_became_peered":"2024-09-13T08:41:29.156127+0000","last_unstale":"2024-09-13T08:41:31.150663+0000","last_undegraded":"2024-09-13T08:41:31.150663+0000","last_fullsized":"2024-09-13T08:41:31.150663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:26.199094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474754+0000","last_change":"2024-09-13T08:41:30.176974+0000","last_active":"2024-09-13T08:41:31.474754+0000","last_peered":"2024-09-13T08:41:31.474754+0000","last_clean":"2024-09-13T08:41:31.474754+0000","last_became_active":"2024-09-13T08:41:30.176715+0000","last_became_peered":"2024-09-13T08:41:30.176715+0000","last_unstale":"2024-09-13T08:41:31.474754+0000","last_undegraded":"2024-09-13T08:41:31.474754+0000","last_fullsized":"2024-09-13T08:41:31.474754+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474730+0000","last_change":"2024-09-13T08:41:29.148076+0000","last_active":"2024-09-13T08:41:31.474730+0000","last_peered":"2024-09-13T08:41:31.474730+0000","last_clean":"2024-09-13T08:41:31.474730+0000","last_became_active":"2024-09-13T08:41:29.147012+0000","last_became_peered":"2024-09-13T08:41:29.147012+0000","last_unstale":"2024-09-13T08:41:31.474730+0000","last_undegraded":"2024-09-13T08:41:31.474730+0000","last_fullsized":"2024-09-13T08:41:31.474730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:18.467245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.173815+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.173512+0000","last_became_peered":"2024-09-13T08:41:30.173512+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:17.533940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475003+0000","last_change":"2024-09-13T08:41:29.148851+0000","last_active":"2024-09-13T08:41:31.475003+0000","last_peered":"2024-09-13T08:41:31.475003+0000","last_clean":"2024-09-13T08:41:31.475003+0000","last_became_active":"2024-09-13T08:41:29.148567+0000","last_became_peered":"2024-09-13T08:41:29.148567+0000","last_unstale":"2024-09-13T08:41:31.475003+0000","last_undegraded":"2024-09-13T08:41:31.475003+0000","last_fullsized":"2024-09-13T08:41:31.475003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.901791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151894+0000","last_change":"2024-09-13T08:41:30.164444+0000","last_active":"2024-09-13T08:41:31.151894+0000","last_peered":"2024-09-13T08:41:31.151894+0000","last_clean":"2024-09-13T08:41:31.151894+0000","last_became_active":"2024-09-13T08:41:30.164177+0000","last_became_peered":"2024-09-13T08:41:30.164177+0000","last_unstale":"2024-09-13T08:41:31.151894+0000","last_undegraded":"2024-09-13T08:41:31.151894+0000","last_fullsized":"2024-09-13T08:41:31.151894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:10.013440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474879+0000","last_change":"2024-09-13T08:41:29.147550+0000","last_active":"2024-09-13T08:41:31.474879+0000","last_peered":"2024-09-13T08:41:31.474879+0000","last_clean":"2024-09-13T08:41:31.474879+0000","last_became_active":"2024-09-13T08:41:29.147183+0000","last_became_peered":"2024-09-13T08:41:29.147183+0000","last_unstale":"2024-09-13T08:41:31.474879+0000","last_undegraded":"2024-09-13T08:41:31.474879+0000","last_fullsized":"2024-09-13T08:41:31.474879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:09.961905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150069+0000","last_change":"2024-09-13T08:41:30.188192+0000","last_active":"2024-09-13T08:41:31.150069+0000","last_peered":"2024-09-13T08:41:31.150069+0000","last_clean":"2024-09-13T08:41:31.150069+0000","last_became_active":"2024-09-13T08:41:30.187402+0000","last_became_peered":"2024-09-13T08:41:30.187402+0000","last_unstale":"2024-09-13T08:41:31.150069+0000","last_undegraded":"2024-09-13T08:41:31.150069+0000","last_fullsized":"2024-09-13T08:41:31.150069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:23.226310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152027+0000","last_change":"2024-09-13T08:41:29.144692+0000","last_active":"2024-09-13T08:41:31.152027+0000","last_peered":"2024-09-13T08:41:31.152027+0000","last_clean":"2024-09-13T08:41:31.152027+0000","last_became_active":"2024-09-13T08:41:29.144396+0000","last_became_peered":"2024-09-13T08:41:29.144396+0000","last_unstale":"2024-09-13T08:41:31.152027+0000","last_undegraded":"2024-09-13T08:41:31.152027+0000","last_fullsized":"2024-09-13T08:41:31.152027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:05.544795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474559+0000","last_change":"2024-09-13T08:41:30.166665+0000","last_active":"2024-09-13T08:41:31.474559+0000","last_peered":"2024-09-13T08:41:31.474559+0000","last_clean":"2024-09-13T08:41:31.474559+0000","last_became_active":"2024-09-13T08:41:30.166364+0000","last_became_peered":"2024-09-13T08:41:30.166364+0000","last_unstale":"2024-09-13T08:41:31.474559+0000","last_undegraded":"2024-09-13T08:41:31.474559+0000","last_fullsized":"2024-09-13T08:41:31.474559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:37.021694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474651+0000","last_change":"2024-09-13T08:41:29.156770+0000","last_active":"2024-09-13T08:41:31.474651+0000","last_peered":"2024-09-13T08:41:31.474651+0000","last_clean":"2024-09-13T08:41:31.474651+0000","last_became_active":"2024-09-13T08:41:29.156466+0000","last_became_peered":"2024-09-13T08:41:29.156466+0000","last_unstale":"2024-09-13T08:41:31.474651+0000","last_undegraded":"2024-09-13T08:41:31.474651+0000","last_fullsized":"2024-09-13T08:41:31.474651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:04.189295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474295+0000","last_change":"2024-09-13T08:41:30.170922+0000","last_active":"2024-09-13T08:41:31.474295+0000","last_peered":"2024-09-13T08:41:31.474295+0000","last_clean":"2024-09-13T08:41:31.474295+0000","last_became_active":"2024-09-13T08:41:30.170663+0000","last_became_peered":"2024-09-13T08:41:30.170663+0000","last_unstale":"2024-09-13T08:41:31.474295+0000","last_undegraded":"2024-09-13T08:41:31.474295+0000","last_fullsized":"2024-09-13T08:41:31.474295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:07:26.254945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474197+0000","last_change":"2024-09-13T08:41:29.149340+0000","last_active":"2024-09-13T08:41:31.474197+0000","last_peered":"2024-09-13T08:41:31.474197+0000","last_clean":"2024-09-13T08:41:31.474197+0000","last_became_active":"2024-09-13T08:41:29.149060+0000","last_became_peered":"2024-09-13T08:41:29.149060+0000","last_unstale":"2024-09-13T08:41:31.474197+0000","last_undegraded":"2024-09-13T08:41:31.474197+0000","last_fullsized":"2024-09-13T08:41:31.474197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:44.441597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151653+0000","last_change":"2024-09-13T08:41:30.177150+0000","last_active":"2024-09-13T08:41:31.151653+0000","last_peered":"2024-09-13T08:41:31.151653+0000","last_clean":"2024-09-13T08:41:31.151653+0000","last_became_active":"2024-09-13T08:41:30.176850+0000","last_became_peered":"2024-09-13T08:41:30.176850+0000","last_unstale":"2024-09-13T08:41:31.151653+0000","last_undegraded":"2024-09-13T08:41:31.151653+0000","last_fullsized":"2024-09-13T08:41:31.151653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:44.657225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475011+0000","last_change":"2024-09-13T08:41:29.144964+0000","last_active":"2024-09-13T08:41:31.475011+0000","last_peered":"2024-09-13T08:41:31.475011+0000","last_clean":"2024-09-13T08:41:31.475011+0000","last_became_active":"2024-09-13T08:41:29.144673+0000","last_became_peered":"2024-09-13T08:41:29.144673+0000","last_unstale":"2024-09-13T08:41:31.475011+0000","last_undegraded":"2024-09-13T08:41:31.475011+0000","last_fullsized":"2024-09-13T08:41:31.475011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:12.107299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":207,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474058+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:41:31.474058+0000","last_peered":"2024-09-13T08:41:31.474058+0000","last_clean":"2024-09-13T08:41:31.474058+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:41:31.474058+0000","last_undegraded":"2024-09-13T08:41:31.474058+0000","last_fullsized":"2024-09-13T08:41:31.474058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":388196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":388230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509498,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":977042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509498,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":386710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509498,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":387166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509499,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1511424,"data_stored":976952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":921600,"data_stored":386654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":929792,"data_stored":387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:41:35.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 -- 172.21.15.31:0/2170234669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540ce900 msgr2=0x7f8d540cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 --2- 172.21.15.31:0/2170234669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540ce900 0x7f8d540cbc50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c005fd0 tx=0x7f8d4c02aa00 comp rx=0 tx=0).stop 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 -- 172.21.15.31:0/2170234669 shutdown_connections 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 --2- 172.21.15.31:0/2170234669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540ce900 0x7f8d540cbc50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 --2- 172.21.15.31:0/2170234669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d540cf850 0x7f8d540ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.477+0000 7f8d5a400640 1 -- 172.21.15.31:0/2170234669 >> 172.21.15.31:0/2170234669 conn(0x7f8d540cc910 msgr2=0x7f8d540cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.478+0000 7f8d5a400640 1 -- 172.21.15.31:0/2170234669 shutdown_connections 2024-09-13T08:41:35.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.478+0000 7f8d5a400640 1 -- 172.21.15.31:0/2170234669 wait complete. 2024-09-13T08:41:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.479+0000 7f8d5a400640 1 Processor -- start 2024-09-13T08:41:35.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.479+0000 7f8d5a400640 1 -- start start 2024-09-13T08:41:35.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 0x7f8d541f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 0x7f8d5406e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541600f0 con 0x7f8d540cf850 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8d540cb850 con 0x7f8d540ce900 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d5a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8d541f8480 con 0x7f8d5415bd70 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59682/0 (socket says 172.21.15.31:59682) 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.480+0000 7f8d4aa00640 1 -- 172.21.15.31:0/4240198862 learned_addr learned my addr 172.21.15.31:0/4240198862 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d53400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 0x7f8d541f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d53e00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 0x7f8d5406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 -- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 msgr2=0x7f8d5406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 0x7f8d5406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 -- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 msgr2=0x7f8d541f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 0x7f8d541f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 -- 172.21.15.31:0/4240198862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d541f5970 con 0x7f8d540cf850 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d53e00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 0x7f8d5406e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d53400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 0x7f8d541f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4aa00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00d950 tx=0x7f8d3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.481+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d540cf850 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.482+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d3c009d20 con 0x7f8d540cf850 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.482+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0106c0 con 0x7f8d540cf850 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.482+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d541f5c50 con 0x7f8d540cf850 2024-09-13T08:41:35.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.482+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d541f89b0 con 0x7f8d540cf850 2024-09-13T08:41:35.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.483+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d540ce3c0 con 0x7f8d540cf850 2024-09-13T08:41:35.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.484+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f8d3c01a070 con 0x7f8d540cf850 2024-09-13T08:41:35.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.485+0000 7f8d4be00640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 0x7f8d3007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.485+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f8d3c09b260 con 0x7f8d540cf850 2024-09-13T08:41:35.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.485+0000 7f8d53400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 0x7f8d3007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.486+0000 7f8d53400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 0x7f8d3007b070 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400a830 tx=0x7f8d44008040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.489+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8d3c063300 con 0x7f8d540cf850 2024-09-13T08:41:35.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.757+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8d540acec0 con 0x7f8d540cf850 2024-09-13T08:41:35.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.757+0000 7f8d4be00640 1 -- 172.21.15.31:0/4240198862 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1474 (secure 0 0 0) 0x7f8d3c062c00 con 0x7f8d540cf850 2024-09-13T08:41:35.758 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:35.759 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"0251b593-a02a-47fa-b938-c065b7284822","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":76,"num_osds":8,"num_up_osds":8,"osd_up_since":1726216617,"num_in_osds":8,"osd_in_since":1726216607,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":268361728,"bytes_avail":772825751552,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":47,"btime":"2024-09-13T08:41:33:169589+0000","id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":6162}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T08:41:17.100021+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 msgr2=0x7f8d3007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 0x7f8d3007b070 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400a830 tx=0x7f8d44008040 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 msgr2=0x7f8d541e8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00d950 tx=0x7f8d3c00de10 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 shutdown_connections 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f8d30078bb0 0x7f8d3007b070 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8d5415bd70 0x7f8d5406e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d540cf850 0x7f8d541e8620 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 --2- 172.21.15.31:0/4240198862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d540ce900 0x7f8d541f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 >> 172.21.15.31:0/4240198862 conn(0x7f8d540cc910 msgr2=0x7f8d540cbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 shutdown_connections 2024-09-13T08:41:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.760+0000 7f8d5a400640 1 -- 172.21.15.31:0/4240198862 wait complete. 2024-09-13T08:41:35.777 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T08:41:35.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:41:35.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 -- 172.21.15.31:0/180696962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 msgr2=0x7f08400cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 --2- 172.21.15.31:0/180696962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f08400cf850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0830009830 tx=0x7f083002a520 comp rx=0 tx=0).stop 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 -- 172.21.15.31:0/180696962 shutdown_connections 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 --2- 172.21.15.31:0/180696962 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f084015f9e0 0x7f084015fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 --2- 172.21.15.31:0/180696962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f08400cf850 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 -- 172.21.15.31:0/180696962 >> 172.21.15.31:0/180696962 conn(0x7f08400cc910 msgr2=0x7f08400cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 -- 172.21.15.31:0/180696962 shutdown_connections 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.937+0000 7f0847200640 1 -- 172.21.15.31:0/180696962 wait complete. 2024-09-13T08:41:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.939+0000 7f0847200640 1 Processor -- start 2024-09-13T08:41:35.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.939+0000 7f0847200640 1 -- start start 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 0x7f084006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 0x7f084009c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08400cb9d0 con 0x7f084015bd70 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f08400cb850 con 0x7f084015f9e0 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0847200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f08400cc360 con 0x7f08400ce3c0 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59706/0 (socket says 172.21.15.31:59706) 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 -- 172.21.15.31:0/1841668759 learned_addr learned my addr 172.21.15.31:0/1841668759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f083fe00640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 0x7f084006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 -- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 msgr2=0x7f084006f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 0x7f084006f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 -- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 msgr2=0x7f084009c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0845200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 0x7f084009c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 0x7f084009c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0837400640 1 -- 172.21.15.31:0/1841668759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084006f890 con 0x7f084015bd70 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f0845200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 0x7f084009c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.940+0000 7f083fe00640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 0x7f084006f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f0837400640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0828009fd0 tx=0x7f082800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082800ccc0 con 0x7f084015bd70 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f082800ee70 con 0x7f084015bd70 2024-09-13T08:41:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082800ce60 con 0x7f084015bd70 2024-09-13T08:41:35.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08401f58a0 con 0x7f084015bd70 2024-09-13T08:41:35.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.941+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084006e870 con 0x7f084015bd70 2024-09-13T08:41:35.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.942+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08400cf850 con 0x7f084015bd70 2024-09-13T08:41:35.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.943+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f0828004280 con 0x7f084015bd70 2024-09-13T08:41:35.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.944+0000 7f083cc00640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 0x7f081c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:35.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.944+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f082809baf0 con 0x7f084015bd70 2024-09-13T08:41:35.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.945+0000 7f083fe00640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 0x7f081c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:35.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.946+0000 7f083fe00640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 0x7f081c07b070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0830034000 tx=0x7f0830004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:35.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:35.949+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0828063a70 con 0x7f084015bd70 2024-09-13T08:41:36.124 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:41:36.123+0000 7fd149e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:41:36.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.143+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08400ceaa0 con 0x7f081c078bb0 2024-09-13T08:41:36.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.148+0000 7f083cc00640 1 -- 172.21.15.31:0/1841668759 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7f08400ceaa0 con 0x7f081c078bb0 2024-09-13T08:41:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:36.153 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.152+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 msgr2=0x7f081c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.152+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 0x7f081c07b070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0830034000 tx=0x7f0830004730 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.152+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 msgr2=0x7f084006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.152+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0828009fd0 tx=0x7f082800ec20 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 shutdown_connections 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f084015f9e0 0x7f084009c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f084015bd70 0x7f084006ff20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f081c078bb0 0x7f081c07b070 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 --2- 172.21.15.31:0/1841668759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08400ce3c0 0x7f084006f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 >> 172.21.15.31:0/1841668759 conn(0x7f08400cc910 msgr2=0x7f08400ae450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 shutdown_connections 2024-09-13T08:41:36.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:36.153+0000 7f0847200640 1 -- 172.21.15.31:0/1841668759 wait complete. 2024-09-13T08:41:36.170 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-13T08:41:35.103549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262072,"kb_used_data":8392,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712648,"statfs":{"total":773094113280,"available":772825751552,"internally_reserved":0,"allocated":8593408,"data_stored":4278228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150712+0000","last_change":"2024-09-13T08:41:30.177953+0000","last_active":"2024-09-13T08:41:31.150712+0000","last_peered":"2024-09-13T08:41:31.150712+0000","last_clean":"2024-09-13T08:41:31.150712+0000","last_became_active":"2024-09-13T08:41:30.177705+0000","last_became_peered":"2024-09-13T08:41:30.177705+0000","last_unstale":"2024-09-13T08:41:31.150712+0000","last_undegraded":"2024-09-13T08:41:31.150712+0000","last_fullsized":"2024-09-13T08:41:31.150712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:51.644500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.171555+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.171277+0000","last_became_peered":"2024-09-13T08:41:30.171277+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152712+0000","last_change":"2024-09-13T08:41:30.184527+0000","last_active":"2024-09-13T08:41:31.152712+0000","last_peered":"2024-09-13T08:41:31.152712+0000","last_clean":"2024-09-13T08:41:31.152712+0000","last_became_active":"2024-09-13T08:41:30.184178+0000","last_became_peered":"2024-09-13T08:41:30.184178+0000","last_unstale":"2024-09-13T08:41:31.152712+0000","last_undegraded":"2024-09-13T08:41:31.152712+0000","last_fullsized":"2024-09-13T08:41:31.152712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:11.304643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152599+0000","last_change":"2024-09-13T08:41:30.177541+0000","last_active":"2024-09-13T08:41:31.152599+0000","last_peered":"2024-09-13T08:41:31.152599+0000","last_clean":"2024-09-13T08:41:31.152599+0000","last_became_active":"2024-09-13T08:41:30.177252+0000","last_became_peered":"2024-09-13T08:41:30.177252+0000","last_unstale":"2024-09-13T08:41:31.152599+0000","last_undegraded":"2024-09-13T08:41:31.152599+0000","last_fullsized":"2024-09-13T08:41:31.152599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:05.041863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152229+0000","last_change":"2024-09-13T08:41:30.176493+0000","last_active":"2024-09-13T08:41:31.152229+0000","last_peered":"2024-09-13T08:41:31.152229+0000","last_clean":"2024-09-13T08:41:31.152229+0000","last_became_active":"2024-09-13T08:41:30.176063+0000","last_became_peered":"2024-09-13T08:41:30.176063+0000","last_unstale":"2024-09-13T08:41:31.152229+0000","last_undegraded":"2024-09-13T08:41:31.152229+0000","last_fullsized":"2024-09-13T08:41:31.152229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:10.675747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475149+0000","last_change":"2024-09-13T08:41:30.180783+0000","last_active":"2024-09-13T08:41:31.475149+0000","last_peered":"2024-09-13T08:41:31.475149+0000","last_clean":"2024-09-13T08:41:31.475149+0000","last_became_active":"2024-09-13T08:41:30.180485+0000","last_became_peered":"2024-09-13T08:41:30.180485+0000","last_unstale":"2024-09-13T08:41:31.475149+0000","last_undegraded":"2024-09-13T08:41:31.475149+0000","last_fullsized":"2024-09-13T08:41:31.475149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474644+0000","last_change":"2024-09-13T08:41:30.178989+0000","last_active":"2024-09-13T08:41:31.474644+0000","last_peered":"2024-09-13T08:41:31.474644+0000","last_clean":"2024-09-13T08:41:31.474644+0000","last_became_active":"2024-09-13T08:41:30.178708+0000","last_became_peered":"2024-09-13T08:41:30.178708+0000","last_unstale":"2024-09-13T08:41:31.474644+0000","last_undegraded":"2024-09-13T08:41:31.474644+0000","last_fullsized":"2024-09-13T08:41:31.474644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:02.877430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149574+0000","last_change":"2024-09-13T08:41:30.176643+0000","last_active":"2024-09-13T08:41:31.149574+0000","last_peered":"2024-09-13T08:41:31.149574+0000","last_clean":"2024-09-13T08:41:31.149574+0000","last_became_active":"2024-09-13T08:41:30.176347+0000","last_became_peered":"2024-09-13T08:41:30.176347+0000","last_unstale":"2024-09-13T08:41:31.149574+0000","last_undegraded":"2024-09-13T08:41:31.149574+0000","last_fullsized":"2024-09-13T08:41:31.149574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:45.146060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152963+0000","last_change":"2024-09-13T08:41:30.156159+0000","last_active":"2024-09-13T08:41:31.152963+0000","last_peered":"2024-09-13T08:41:31.152963+0000","last_clean":"2024-09-13T08:41:31.152963+0000","last_became_active":"2024-09-13T08:41:30.155807+0000","last_became_peered":"2024-09-13T08:41:30.155807+0000","last_unstale":"2024-09-13T08:41:31.152963+0000","last_undegraded":"2024-09-13T08:41:31.152963+0000","last_fullsized":"2024-09-13T08:41:31.152963+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:11.356687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475634+0000","last_change":"2024-09-13T08:41:30.181209+0000","last_active":"2024-09-13T08:41:31.475634+0000","last_peered":"2024-09-13T08:41:31.475634+0000","last_clean":"2024-09-13T08:41:31.475634+0000","last_became_active":"2024-09-13T08:41:30.179580+0000","last_became_peered":"2024-09-13T08:41:30.179580+0000","last_unstale":"2024-09-13T08:41:31.475634+0000","last_undegraded":"2024-09-13T08:41:31.475634+0000","last_fullsized":"2024-09-13T08:41:31.475634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:36.230665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150715+0000","last_change":"2024-09-13T08:41:30.175255+0000","last_active":"2024-09-13T08:41:31.150715+0000","last_peered":"2024-09-13T08:41:31.150715+0000","last_clean":"2024-09-13T08:41:31.150715+0000","last_became_active":"2024-09-13T08:41:30.174389+0000","last_became_peered":"2024-09-13T08:41:30.174389+0000","last_unstale":"2024-09-13T08:41:31.150715+0000","last_undegraded":"2024-09-13T08:41:31.150715+0000","last_fullsized":"2024-09-13T08:41:31.150715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:57:33.926281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475051+0000","last_change":"2024-09-13T08:41:30.167545+0000","last_active":"2024-09-13T08:41:31.475051+0000","last_peered":"2024-09-13T08:41:31.475051+0000","last_clean":"2024-09-13T08:41:31.475051+0000","last_became_active":"2024-09-13T08:41:30.167268+0000","last_became_peered":"2024-09-13T08:41:30.167268+0000","last_unstale":"2024-09-13T08:41:31.475051+0000","last_undegraded":"2024-09-13T08:41:31.475051+0000","last_fullsized":"2024-09-13T08:41:31.475051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:01.162295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152039+0000","last_change":"2024-09-13T08:41:30.177514+0000","last_active":"2024-09-13T08:41:31.152039+0000","last_peered":"2024-09-13T08:41:31.152039+0000","last_clean":"2024-09-13T08:41:31.152039+0000","last_became_active":"2024-09-13T08:41:30.177234+0000","last_became_peered":"2024-09-13T08:41:30.177234+0000","last_unstale":"2024-09-13T08:41:31.152039+0000","last_undegraded":"2024-09-13T08:41:31.152039+0000","last_fullsized":"2024-09-13T08:41:31.152039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:59.795415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474710+0000","last_change":"2024-09-13T08:41:30.172201+0000","last_active":"2024-09-13T08:41:31.474710+0000","last_peered":"2024-09-13T08:41:31.474710+0000","last_clean":"2024-09-13T08:41:31.474710+0000","last_became_active":"2024-09-13T08:41:30.171269+0000","last_became_peered":"2024-09-13T08:41:30.171269+0000","last_unstale":"2024-09-13T08:41:31.474710+0000","last_undegraded":"2024-09-13T08:41:31.474710+0000","last_fullsized":"2024-09-13T08:41:31.474710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.336999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474006+0000","last_change":"2024-09-13T08:41:30.169784+0000","last_active":"2024-09-13T08:41:31.474006+0000","last_peered":"2024-09-13T08:41:31.474006+0000","last_clean":"2024-09-13T08:41:31.474006+0000","last_became_active":"2024-09-13T08:41:30.169508+0000","last_became_peered":"2024-09-13T08:41:30.169508+0000","last_unstale":"2024-09-13T08:41:31.474006+0000","last_undegraded":"2024-09-13T08:41:31.474006+0000","last_fullsized":"2024-09-13T08:41:31.474006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:32.762281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149585+0000","last_change":"2024-09-13T08:41:30.168753+0000","last_active":"2024-09-13T08:41:31.149585+0000","last_peered":"2024-09-13T08:41:31.149585+0000","last_clean":"2024-09-13T08:41:31.149585+0000","last_became_active":"2024-09-13T08:41:30.168428+0000","last_became_peered":"2024-09-13T08:41:30.168428+0000","last_unstale":"2024-09-13T08:41:31.149585+0000","last_undegraded":"2024-09-13T08:41:31.149585+0000","last_fullsized":"2024-09-13T08:41:31.149585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:28.857492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152183+0000","last_change":"2024-09-13T08:41:30.174176+0000","last_active":"2024-09-13T08:41:31.152183+0000","last_peered":"2024-09-13T08:41:31.152183+0000","last_clean":"2024-09-13T08:41:31.152183+0000","last_became_active":"2024-09-13T08:41:30.173898+0000","last_became_peered":"2024-09-13T08:41:30.173898+0000","last_unstale":"2024-09-13T08:41:31.152183+0000","last_undegraded":"2024-09-13T08:41:31.152183+0000","last_fullsized":"2024-09-13T08:41:31.152183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:43:41.193530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152346+0000","last_change":"2024-09-13T08:41:30.179501+0000","last_active":"2024-09-13T08:41:31.152346+0000","last_peered":"2024-09-13T08:41:31.152346+0000","last_clean":"2024-09-13T08:41:31.152346+0000","last_became_active":"2024-09-13T08:41:30.179215+0000","last_became_peered":"2024-09-13T08:41:30.179215+0000","last_unstale":"2024-09-13T08:41:31.152346+0000","last_undegraded":"2024-09-13T08:41:31.152346+0000","last_fullsized":"2024-09-13T08:41:31.152346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:41:33.182476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150478+0000","last_change":"2024-09-13T08:41:30.176585+0000","last_active":"2024-09-13T08:41:31.150478+0000","last_peered":"2024-09-13T08:41:31.150478+0000","last_clean":"2024-09-13T08:41:31.150478+0000","last_became_active":"2024-09-13T08:41:30.173677+0000","last_became_peered":"2024-09-13T08:41:30.173677+0000","last_unstale":"2024-09-13T08:41:31.150478+0000","last_undegraded":"2024-09-13T08:41:31.150478+0000","last_fullsized":"2024-09-13T08:41:31.150478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:18.526089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152370+0000","last_change":"2024-09-13T08:41:30.179532+0000","last_active":"2024-09-13T08:41:31.152370+0000","last_peered":"2024-09-13T08:41:31.152370+0000","last_clean":"2024-09-13T08:41:31.152370+0000","last_became_active":"2024-09-13T08:41:30.179182+0000","last_became_peered":"2024-09-13T08:41:30.179182+0000","last_unstale":"2024-09-13T08:41:31.152370+0000","last_undegraded":"2024-09-13T08:41:31.152370+0000","last_fullsized":"2024-09-13T08:41:31.152370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:03.650279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475052+0000","last_change":"2024-09-13T08:41:30.155465+0000","last_active":"2024-09-13T08:41:31.475052+0000","last_peered":"2024-09-13T08:41:31.475052+0000","last_clean":"2024-09-13T08:41:31.475052+0000","last_became_active":"2024-09-13T08:41:30.155056+0000","last_became_peered":"2024-09-13T08:41:30.155056+0000","last_unstale":"2024-09-13T08:41:31.475052+0000","last_undegraded":"2024-09-13T08:41:31.475052+0000","last_fullsized":"2024-09-13T08:41:31.475052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475329+0000","last_change":"2024-09-13T08:41:30.178147+0000","last_active":"2024-09-13T08:41:31.475329+0000","last_peered":"2024-09-13T08:41:31.475329+0000","last_clean":"2024-09-13T08:41:31.475329+0000","last_became_active":"2024-09-13T08:41:30.175910+0000","last_became_peered":"2024-09-13T08:41:30.175910+0000","last_unstale":"2024-09-13T08:41:31.475329+0000","last_undegraded":"2024-09-13T08:41:31.475329+0000","last_fullsized":"2024-09-13T08:41:31.475329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474864+0000","last_change":"2024-09-13T08:41:30.190277+0000","last_active":"2024-09-13T08:41:31.474864+0000","last_peered":"2024-09-13T08:41:31.474864+0000","last_clean":"2024-09-13T08:41:31.474864+0000","last_became_active":"2024-09-13T08:41:30.189911+0000","last_became_peered":"2024-09-13T08:41:30.189911+0000","last_unstale":"2024-09-13T08:41:31.474864+0000","last_undegraded":"2024-09-13T08:41:31.474864+0000","last_fullsized":"2024-09-13T08:41:31.474864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:46.494807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149710+0000","last_change":"2024-09-13T08:41:30.179319+0000","last_active":"2024-09-13T08:41:31.149710+0000","last_peered":"2024-09-13T08:41:31.149710+0000","last_clean":"2024-09-13T08:41:31.149710+0000","last_became_active":"2024-09-13T08:41:30.179050+0000","last_became_peered":"2024-09-13T08:41:30.179050+0000","last_unstale":"2024-09-13T08:41:31.149710+0000","last_undegraded":"2024-09-13T08:41:31.149710+0000","last_fullsized":"2024-09-13T08:41:31.149710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:01.953440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475362+0000","last_change":"2024-09-13T08:41:30.190025+0000","last_active":"2024-09-13T08:41:31.475362+0000","last_peered":"2024-09-13T08:41:31.475362+0000","last_clean":"2024-09-13T08:41:31.475362+0000","last_became_active":"2024-09-13T08:41:30.189682+0000","last_became_peered":"2024-09-13T08:41:30.189682+0000","last_unstale":"2024-09-13T08:41:31.475362+0000","last_undegraded":"2024-09-13T08:41:31.475362+0000","last_fullsized":"2024-09-13T08:41:31.475362+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:08.385273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150264+0000","last_change":"2024-09-13T08:41:30.190500+0000","last_active":"2024-09-13T08:41:31.150264+0000","last_peered":"2024-09-13T08:41:31.150264+0000","last_clean":"2024-09-13T08:41:31.150264+0000","last_became_active":"2024-09-13T08:41:30.190211+0000","last_became_peered":"2024-09-13T08:41:30.190211+0000","last_unstale":"2024-09-13T08:41:31.150264+0000","last_undegraded":"2024-09-13T08:41:31.150264+0000","last_fullsized":"2024-09-13T08:41:31.150264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:55:48.809149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151986+0000","last_change":"2024-09-13T08:41:30.168525+0000","last_active":"2024-09-13T08:41:31.151986+0000","last_peered":"2024-09-13T08:41:31.151986+0000","last_clean":"2024-09-13T08:41:31.151986+0000","last_became_active":"2024-09-13T08:41:30.167750+0000","last_became_peered":"2024-09-13T08:41:30.167750+0000","last_unstale":"2024-09-13T08:41:31.151986+0000","last_undegraded":"2024-09-13T08:41:31.151986+0000","last_fullsized":"2024-09-13T08:41:31.151986+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:37.422258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474553+0000","last_change":"2024-09-13T08:41:30.165606+0000","last_active":"2024-09-13T08:41:31.474553+0000","last_peered":"2024-09-13T08:41:31.474553+0000","last_clean":"2024-09-13T08:41:31.474553+0000","last_became_active":"2024-09-13T08:41:30.165277+0000","last_became_peered":"2024-09-13T08:41:30.165277+0000","last_unstale":"2024-09-13T08:41:31.474553+0000","last_undegraded":"2024-09-13T08:41:31.474553+0000","last_fullsized":"2024-09-13T08:41:31.474553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152266+0000","last_change":"2024-09-13T08:41:30.170008+0000","last_active":"2024-09-13T08:41:31.152266+0000","last_peered":"2024-09-13T08:41:31.152266+0000","last_clean":"2024-09-13T08:41:31.152266+0000","last_became_active":"2024-09-13T08:41:30.169516+0000","last_became_peered":"2024-09-13T08:41:30.169516+0000","last_unstale":"2024-09-13T08:41:31.152266+0000","last_undegraded":"2024-09-13T08:41:31.152266+0000","last_fullsized":"2024-09-13T08:41:31.152266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:12:20.327460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149957+0000","last_change":"2024-09-13T08:41:30.171602+0000","last_active":"2024-09-13T08:41:31.149957+0000","last_peered":"2024-09-13T08:41:31.149957+0000","last_clean":"2024-09-13T08:41:31.149957+0000","last_became_active":"2024-09-13T08:41:30.170882+0000","last_became_peered":"2024-09-13T08:41:30.170882+0000","last_unstale":"2024-09-13T08:41:31.149957+0000","last_undegraded":"2024-09-13T08:41:31.149957+0000","last_fullsized":"2024-09-13T08:41:31.149957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:17.491430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474487+0000","last_change":"2024-09-13T08:41:30.165505+0000","last_active":"2024-09-13T08:41:31.474487+0000","last_peered":"2024-09-13T08:41:31.474487+0000","last_clean":"2024-09-13T08:41:31.474487+0000","last_became_active":"2024-09-13T08:41:30.165205+0000","last_became_peered":"2024-09-13T08:41:30.165205+0000","last_unstale":"2024-09-13T08:41:31.474487+0000","last_undegraded":"2024-09-13T08:41:31.474487+0000","last_fullsized":"2024-09-13T08:41:31.474487+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:53.060236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474225+0000","last_change":"2024-09-13T08:41:30.166147+0000","last_active":"2024-09-13T08:41:31.474225+0000","last_peered":"2024-09-13T08:41:31.474225+0000","last_clean":"2024-09-13T08:41:31.474225+0000","last_became_active":"2024-09-13T08:41:30.165114+0000","last_became_peered":"2024-09-13T08:41:30.165114+0000","last_unstale":"2024-09-13T08:41:31.474225+0000","last_undegraded":"2024-09-13T08:41:31.474225+0000","last_fullsized":"2024-09-13T08:41:31.474225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:53.742788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150526+0000","last_change":"2024-09-13T08:41:30.184240+0000","last_active":"2024-09-13T08:41:31.150526+0000","last_peered":"2024-09-13T08:41:31.150526+0000","last_clean":"2024-09-13T08:41:31.150526+0000","last_became_active":"2024-09-13T08:41:30.183893+0000","last_became_peered":"2024-09-13T08:41:30.183893+0000","last_unstale":"2024-09-13T08:41:31.150526+0000","last_undegraded":"2024-09-13T08:41:31.150526+0000","last_fullsized":"2024-09-13T08:41:31.150526+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:51.357207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"75'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474599+0000","last_change":"2024-09-13T08:41:29.151268+0000","last_active":"2024-09-13T08:41:31.474599+0000","last_peered":"2024-09-13T08:41:31.474599+0000","last_clean":"2024-09-13T08:41:31.474599+0000","last_became_active":"2024-09-13T08:41:29.150960+0000","last_became_peered":"2024-09-13T08:41:29.150960+0000","last_unstale":"2024-09-13T08:41:31.474599+0000","last_undegraded":"2024-09-13T08:41:31.474599+0000","last_fullsized":"2024-09-13T08:41:31.474599+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:42.453175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152927+0000","last_change":"2024-09-13T08:41:30.164561+0000","last_active":"2024-09-13T08:41:31.152927+0000","last_peered":"2024-09-13T08:41:31.152927+0000","last_clean":"2024-09-13T08:41:31.152927+0000","last_became_active":"2024-09-13T08:41:30.164290+0000","last_became_peered":"2024-09-13T08:41:30.164290+0000","last_unstale":"2024-09-13T08:41:31.152927+0000","last_undegraded":"2024-09-13T08:41:31.152927+0000","last_fullsized":"2024-09-13T08:41:31.152927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:33.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152403+0000","last_change":"2024-09-13T08:41:29.144190+0000","last_active":"2024-09-13T08:41:31.152403+0000","last_peered":"2024-09-13T08:41:31.152403+0000","last_clean":"2024-09-13T08:41:31.152403+0000","last_became_active":"2024-09-13T08:41:29.143622+0000","last_became_peered":"2024-09-13T08:41:29.143622+0000","last_unstale":"2024-09-13T08:41:31.152403+0000","last_undegraded":"2024-09-13T08:41:31.152403+0000","last_fullsized":"2024-09-13T08:41:31.152403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:37.575695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474834+0000","last_change":"2024-09-13T08:41:30.176252+0000","last_active":"2024-09-13T08:41:31.474834+0000","last_peered":"2024-09-13T08:41:31.474834+0000","last_clean":"2024-09-13T08:41:31.474834+0000","last_became_active":"2024-09-13T08:41:30.174505+0000","last_became_peered":"2024-09-13T08:41:30.174505+0000","last_unstale":"2024-09-13T08:41:31.474834+0000","last_undegraded":"2024-09-13T08:41:31.474834+0000","last_fullsized":"2024-09-13T08:41:31.474834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150540+0000","last_change":"2024-09-13T08:41:29.153502+0000","last_active":"2024-09-13T08:41:31.150540+0000","last_peered":"2024-09-13T08:41:31.150540+0000","last_clean":"2024-09-13T08:41:31.150540+0000","last_became_active":"2024-09-13T08:41:29.153256+0000","last_became_peered":"2024-09-13T08:41:29.153256+0000","last_unstale":"2024-09-13T08:41:31.150540+0000","last_undegraded":"2024-09-13T08:41:31.150540+0000","last_fullsized":"2024-09-13T08:41:31.150540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:11.750149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475366+0000","last_change":"2024-09-13T08:41:30.180280+0000","last_active":"2024-09-13T08:41:31.475366+0000","last_peered":"2024-09-13T08:41:31.475366+0000","last_clean":"2024-09-13T08:41:31.475366+0000","last_became_active":"2024-09-13T08:41:30.179979+0000","last_became_peered":"2024-09-13T08:41:30.179979+0000","last_unstale":"2024-09-13T08:41:31.475366+0000","last_undegraded":"2024-09-13T08:41:31.475366+0000","last_fullsized":"2024-09-13T08:41:31.475366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:39.299088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474892+0000","last_change":"2024-09-13T08:41:29.142380+0000","last_active":"2024-09-13T08:41:31.474892+0000","last_peered":"2024-09-13T08:41:31.474892+0000","last_clean":"2024-09-13T08:41:31.474892+0000","last_became_active":"2024-09-13T08:41:29.142093+0000","last_became_peered":"2024-09-13T08:41:29.142093+0000","last_unstale":"2024-09-13T08:41:31.474892+0000","last_undegraded":"2024-09-13T08:41:31.474892+0000","last_fullsized":"2024-09-13T08:41:31.474892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:05.945048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149958+0000","last_change":"2024-09-13T08:41:30.175452+0000","last_active":"2024-09-13T08:41:31.149958+0000","last_peered":"2024-09-13T08:41:31.149958+0000","last_clean":"2024-09-13T08:41:31.149958+0000","last_became_active":"2024-09-13T08:41:30.175101+0000","last_became_peered":"2024-09-13T08:41:30.175101+0000","last_unstale":"2024-09-13T08:41:31.149958+0000","last_undegraded":"2024-09-13T08:41:31.149958+0000","last_fullsized":"2024-09-13T08:41:31.149958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:35.678193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474813+0000","last_change":"2024-09-13T08:41:29.152256+0000","last_active":"2024-09-13T08:41:31.474813+0000","last_peered":"2024-09-13T08:41:31.474813+0000","last_clean":"2024-09-13T08:41:31.474813+0000","last_became_active":"2024-09-13T08:41:29.151880+0000","last_became_peered":"2024-09-13T08:41:29.151880+0000","last_unstale":"2024-09-13T08:41:31.474813+0000","last_undegraded":"2024-09-13T08:41:31.474813+0000","last_fullsized":"2024-09-13T08:41:31.474813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:45.557782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474596+0000","last_change":"2024-09-13T08:41:30.179754+0000","last_active":"2024-09-13T08:41:31.474596+0000","last_peered":"2024-09-13T08:41:31.474596+0000","last_clean":"2024-09-13T08:41:31.474596+0000","last_became_active":"2024-09-13T08:41:30.176002+0000","last_became_peered":"2024-09-13T08:41:30.176002+0000","last_unstale":"2024-09-13T08:41:31.474596+0000","last_undegraded":"2024-09-13T08:41:31.474596+0000","last_fullsized":"2024-09-13T08:41:31.474596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474629+0000","last_change":"2024-09-13T08:41:29.144056+0000","last_active":"2024-09-13T08:41:31.474629+0000","last_peered":"2024-09-13T08:41:31.474629+0000","last_clean":"2024-09-13T08:41:31.474629+0000","last_became_active":"2024-09-13T08:41:29.143787+0000","last_became_peered":"2024-09-13T08:41:29.143787+0000","last_unstale":"2024-09-13T08:41:31.474629+0000","last_undegraded":"2024-09-13T08:41:31.474629+0000","last_fullsized":"2024-09-13T08:41:31.474629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:07.698273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474317+0000","last_change":"2024-09-13T08:41:30.176412+0000","last_active":"2024-09-13T08:41:31.474317+0000","last_peered":"2024-09-13T08:41:31.474317+0000","last_clean":"2024-09-13T08:41:31.474317+0000","last_became_active":"2024-09-13T08:41:30.176124+0000","last_became_peered":"2024-09-13T08:41:30.176124+0000","last_unstale":"2024-09-13T08:41:31.474317+0000","last_undegraded":"2024-09-13T08:41:31.474317+0000","last_fullsized":"2024-09-13T08:41:31.474317+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:19:31.333677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474680+0000","last_change":"2024-09-13T08:41:29.142404+0000","last_active":"2024-09-13T08:41:31.474680+0000","last_peered":"2024-09-13T08:41:31.474680+0000","last_clean":"2024-09-13T08:41:31.474680+0000","last_became_active":"2024-09-13T08:41:29.142078+0000","last_became_peered":"2024-09-13T08:41:29.142078+0000","last_unstale":"2024-09-13T08:41:31.474680+0000","last_undegraded":"2024-09-13T08:41:31.474680+0000","last_fullsized":"2024-09-13T08:41:31.474680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:42.389929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474073+0000","last_change":"2024-09-13T08:41:30.171632+0000","last_active":"2024-09-13T08:41:31.474073+0000","last_peered":"2024-09-13T08:41:31.474073+0000","last_clean":"2024-09-13T08:41:31.474073+0000","last_became_active":"2024-09-13T08:41:30.171351+0000","last_became_peered":"2024-09-13T08:41:30.171351+0000","last_unstale":"2024-09-13T08:41:31.474073+0000","last_undegraded":"2024-09-13T08:41:31.474073+0000","last_fullsized":"2024-09-13T08:41:31.474073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:36.601619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.473982+0000","last_change":"2024-09-13T08:41:29.141838+0000","last_active":"2024-09-13T08:41:31.473982+0000","last_peered":"2024-09-13T08:41:31.473982+0000","last_clean":"2024-09-13T08:41:31.473982+0000","last_became_active":"2024-09-13T08:41:29.141458+0000","last_became_peered":"2024-09-13T08:41:29.141458+0000","last_unstale":"2024-09-13T08:41:31.473982+0000","last_undegraded":"2024-09-13T08:41:31.473982+0000","last_fullsized":"2024-09-13T08:41:31.473982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:57.962170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475749+0000","last_change":"2024-09-13T08:41:30.186535+0000","last_active":"2024-09-13T08:41:31.475749+0000","last_peered":"2024-09-13T08:41:31.475749+0000","last_clean":"2024-09-13T08:41:31.475749+0000","last_became_active":"2024-09-13T08:41:30.186257+0000","last_became_peered":"2024-09-13T08:41:30.186257+0000","last_unstale":"2024-09-13T08:41:31.475749+0000","last_undegraded":"2024-09-13T08:41:31.475749+0000","last_fullsized":"2024-09-13T08:41:31.475749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:05.191523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150332+0000","last_change":"2024-09-13T08:41:29.149880+0000","last_active":"2024-09-13T08:41:31.150332+0000","last_peered":"2024-09-13T08:41:31.150332+0000","last_clean":"2024-09-13T08:41:31.150332+0000","last_became_active":"2024-09-13T08:41:29.149556+0000","last_became_peered":"2024-09-13T08:41:29.149556+0000","last_unstale":"2024-09-13T08:41:31.150332+0000","last_undegraded":"2024-09-13T08:41:31.150332+0000","last_fullsized":"2024-09-13T08:41:31.150332+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:39.387524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475671+0000","last_change":"2024-09-13T08:41:30.171210+0000","last_active":"2024-09-13T08:41:31.475671+0000","last_peered":"2024-09-13T08:41:31.475671+0000","last_clean":"2024-09-13T08:41:31.475671+0000","last_became_active":"2024-09-13T08:41:30.170494+0000","last_became_peered":"2024-09-13T08:41:30.170494+0000","last_unstale":"2024-09-13T08:41:31.475671+0000","last_undegraded":"2024-09-13T08:41:31.475671+0000","last_fullsized":"2024-09-13T08:41:31.475671+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475717+0000","last_change":"2024-09-13T08:41:29.154606+0000","last_active":"2024-09-13T08:41:31.475717+0000","last_peered":"2024-09-13T08:41:31.475717+0000","last_clean":"2024-09-13T08:41:31.475717+0000","last_became_active":"2024-09-13T08:41:29.154326+0000","last_became_peered":"2024-09-13T08:41:29.154326+0000","last_unstale":"2024-09-13T08:41:31.475717+0000","last_undegraded":"2024-09-13T08:41:31.475717+0000","last_fullsized":"2024-09-13T08:41:31.475717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:52.695133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150201+0000","last_change":"2024-09-13T08:41:30.191552+0000","last_active":"2024-09-13T08:41:31.150201+0000","last_peered":"2024-09-13T08:41:31.150201+0000","last_clean":"2024-09-13T08:41:31.150201+0000","last_became_active":"2024-09-13T08:41:30.191295+0000","last_became_peered":"2024-09-13T08:41:30.191295+0000","last_unstale":"2024-09-13T08:41:31.150201+0000","last_undegraded":"2024-09-13T08:41:31.150201+0000","last_fullsized":"2024-09-13T08:41:31.150201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:34.894273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474613+0000","last_change":"2024-09-13T08:41:29.147972+0000","last_active":"2024-09-13T08:41:31.474613+0000","last_peered":"2024-09-13T08:41:31.474613+0000","last_clean":"2024-09-13T08:41:31.474613+0000","last_became_active":"2024-09-13T08:41:29.147657+0000","last_became_peered":"2024-09-13T08:41:29.147657+0000","last_unstale":"2024-09-13T08:41:31.474613+0000","last_undegraded":"2024-09-13T08:41:31.474613+0000","last_fullsized":"2024-09-13T08:41:31.474613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:37.087395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152158+0000","last_change":"2024-09-13T08:41:30.168763+0000","last_active":"2024-09-13T08:41:31.152158+0000","last_peered":"2024-09-13T08:41:31.152158+0000","last_clean":"2024-09-13T08:41:31.152158+0000","last_became_active":"2024-09-13T08:41:30.168493+0000","last_became_peered":"2024-09-13T08:41:30.168493+0000","last_unstale":"2024-09-13T08:41:31.152158+0000","last_undegraded":"2024-09-13T08:41:31.152158+0000","last_fullsized":"2024-09-13T08:41:31.152158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:27.265527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"75'6","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474699+0000","last_change":"2024-09-13T08:41:29.142862+0000","last_active":"2024-09-13T08:41:31.474699+0000","last_peered":"2024-09-13T08:41:31.474699+0000","last_clean":"2024-09-13T08:41:31.474699+0000","last_became_active":"2024-09-13T08:41:29.140928+0000","last_became_peered":"2024-09-13T08:41:29.140928+0000","last_unstale":"2024-09-13T08:41:31.474699+0000","last_undegraded":"2024-09-13T08:41:31.474699+0000","last_fullsized":"2024-09-13T08:41:31.474699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:29:54.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475544+0000","last_change":"2024-09-13T08:41:30.166884+0000","last_active":"2024-09-13T08:41:31.475544+0000","last_peered":"2024-09-13T08:41:31.475544+0000","last_clean":"2024-09-13T08:41:31.475544+0000","last_became_active":"2024-09-13T08:41:30.166578+0000","last_became_peered":"2024-09-13T08:41:30.166578+0000","last_unstale":"2024-09-13T08:41:31.475544+0000","last_undegraded":"2024-09-13T08:41:31.475544+0000","last_fullsized":"2024-09-13T08:41:31.475544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474519+0000","last_change":"2024-09-13T08:41:29.144444+0000","last_active":"2024-09-13T08:41:31.474519+0000","last_peered":"2024-09-13T08:41:31.474519+0000","last_clean":"2024-09-13T08:41:31.474519+0000","last_became_active":"2024-09-13T08:41:29.144236+0000","last_became_peered":"2024-09-13T08:41:29.144236+0000","last_unstale":"2024-09-13T08:41:31.474519+0000","last_undegraded":"2024-09-13T08:41:31.474519+0000","last_fullsized":"2024-09-13T08:41:31.474519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:31.977262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475053+0000","last_change":"2024-09-13T08:41:30.178971+0000","last_active":"2024-09-13T08:41:31.475053+0000","last_peered":"2024-09-13T08:41:31.475053+0000","last_clean":"2024-09-13T08:41:31.475053+0000","last_became_active":"2024-09-13T08:41:30.178662+0000","last_became_peered":"2024-09-13T08:41:30.178662+0000","last_unstale":"2024-09-13T08:41:31.475053+0000","last_undegraded":"2024-09-13T08:41:31.475053+0000","last_fullsized":"2024-09-13T08:41:31.475053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.425812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474733+0000","last_change":"2024-09-13T08:41:29.148940+0000","last_active":"2024-09-13T08:41:31.474733+0000","last_peered":"2024-09-13T08:41:31.474733+0000","last_clean":"2024-09-13T08:41:31.474733+0000","last_became_active":"2024-09-13T08:41:29.148677+0000","last_became_peered":"2024-09-13T08:41:29.148677+0000","last_unstale":"2024-09-13T08:41:31.474733+0000","last_undegraded":"2024-09-13T08:41:31.474733+0000","last_fullsized":"2024-09-13T08:41:31.474733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:36:49.700010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151295+0000","last_change":"2024-09-13T08:41:30.176836+0000","last_active":"2024-09-13T08:41:31.151295+0000","last_peered":"2024-09-13T08:41:31.151295+0000","last_clean":"2024-09-13T08:41:31.151295+0000","last_became_active":"2024-09-13T08:41:30.176561+0000","last_became_peered":"2024-09-13T08:41:30.176561+0000","last_unstale":"2024-09-13T08:41:31.151295+0000","last_undegraded":"2024-09-13T08:41:31.151295+0000","last_fullsized":"2024-09-13T08:41:31.151295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:27.201322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474722+0000","last_change":"2024-09-13T08:41:29.137115+0000","last_active":"2024-09-13T08:41:31.474722+0000","last_peered":"2024-09-13T08:41:31.474722+0000","last_clean":"2024-09-13T08:41:31.474722+0000","last_became_active":"2024-09-13T08:41:29.136800+0000","last_became_peered":"2024-09-13T08:41:29.136800+0000","last_unstale":"2024-09-13T08:41:31.474722+0000","last_undegraded":"2024-09-13T08:41:31.474722+0000","last_fullsized":"2024-09-13T08:41:31.474722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:53.975223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151142+0000","last_change":"2024-09-13T08:41:30.164645+0000","last_active":"2024-09-13T08:41:31.151142+0000","last_peered":"2024-09-13T08:41:31.151142+0000","last_clean":"2024-09-13T08:41:31.151142+0000","last_became_active":"2024-09-13T08:41:30.164362+0000","last_became_peered":"2024-09-13T08:41:30.164362+0000","last_unstale":"2024-09-13T08:41:31.151142+0000","last_undegraded":"2024-09-13T08:41:31.151142+0000","last_fullsized":"2024-09-13T08:41:31.151142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:39.424459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"75'8","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474024+0000","last_change":"2024-09-13T08:41:29.141724+0000","last_active":"2024-09-13T08:41:31.474024+0000","last_peered":"2024-09-13T08:41:31.474024+0000","last_clean":"2024-09-13T08:41:31.474024+0000","last_became_active":"2024-09-13T08:41:29.141410+0000","last_became_peered":"2024-09-13T08:41:29.141410+0000","last_unstale":"2024-09-13T08:41:31.474024+0000","last_undegraded":"2024-09-13T08:41:31.474024+0000","last_fullsized":"2024-09-13T08:41:31.474024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:36.990282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475780+0000","last_change":"2024-09-13T08:41:30.170258+0000","last_active":"2024-09-13T08:41:31.475780+0000","last_peered":"2024-09-13T08:41:31.475780+0000","last_clean":"2024-09-13T08:41:31.475780+0000","last_became_active":"2024-09-13T08:41:30.169900+0000","last_became_peered":"2024-09-13T08:41:30.169900+0000","last_unstale":"2024-09-13T08:41:31.475780+0000","last_undegraded":"2024-09-13T08:41:31.475780+0000","last_fullsized":"2024-09-13T08:41:31.475780+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.577924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150429+0000","last_change":"2024-09-13T08:41:29.152478+0000","last_active":"2024-09-13T08:41:31.150429+0000","last_peered":"2024-09-13T08:41:31.150429+0000","last_clean":"2024-09-13T08:41:31.150429+0000","last_became_active":"2024-09-13T08:41:29.152174+0000","last_became_peered":"2024-09-13T08:41:29.152174+0000","last_unstale":"2024-09-13T08:41:31.150429+0000","last_undegraded":"2024-09-13T08:41:31.150429+0000","last_fullsized":"2024-09-13T08:41:31.150429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:27.809813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475948+0000","last_change":"2024-09-13T08:41:30.157129+0000","last_active":"2024-09-13T08:41:31.475948+0000","last_peered":"2024-09-13T08:41:31.475948+0000","last_clean":"2024-09-13T08:41:31.475948+0000","last_became_active":"2024-09-13T08:41:30.156648+0000","last_became_peered":"2024-09-13T08:41:30.156648+0000","last_unstale":"2024-09-13T08:41:31.475948+0000","last_undegraded":"2024-09-13T08:41:31.475948+0000","last_fullsized":"2024-09-13T08:41:31.475948+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:28.126509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475814+0000","last_change":"2024-09-13T08:41:29.156147+0000","last_active":"2024-09-13T08:41:31.475814+0000","last_peered":"2024-09-13T08:41:31.475814+0000","last_clean":"2024-09-13T08:41:31.475814+0000","last_became_active":"2024-09-13T08:41:29.155857+0000","last_became_peered":"2024-09-13T08:41:29.155857+0000","last_unstale":"2024-09-13T08:41:31.475814+0000","last_undegraded":"2024-09-13T08:41:31.475814+0000","last_fullsized":"2024-09-13T08:41:31.475814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:27.046693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152642+0000","last_change":"2024-09-13T08:41:30.160794+0000","last_active":"2024-09-13T08:41:31.152642+0000","last_peered":"2024-09-13T08:41:31.152642+0000","last_clean":"2024-09-13T08:41:31.152642+0000","last_became_active":"2024-09-13T08:41:30.160339+0000","last_became_peered":"2024-09-13T08:41:30.160339+0000","last_unstale":"2024-09-13T08:41:31.152642+0000","last_undegraded":"2024-09-13T08:41:31.152642+0000","last_fullsized":"2024-09-13T08:41:31.152642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:56.686232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474046+0000","last_change":"2024-09-13T08:41:29.154023+0000","last_active":"2024-09-13T08:41:31.474046+0000","last_peered":"2024-09-13T08:41:31.474046+0000","last_clean":"2024-09-13T08:41:31.474046+0000","last_became_active":"2024-09-13T08:41:29.152087+0000","last_became_peered":"2024-09-13T08:41:29.152087+0000","last_unstale":"2024-09-13T08:41:31.474046+0000","last_undegraded":"2024-09-13T08:41:31.474046+0000","last_fullsized":"2024-09-13T08:41:31.474046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:38:02.653402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152683+0000","last_change":"2024-09-13T08:41:30.170644+0000","last_active":"2024-09-13T08:41:31.152683+0000","last_peered":"2024-09-13T08:41:31.152683+0000","last_clean":"2024-09-13T08:41:31.152683+0000","last_became_active":"2024-09-13T08:41:30.170350+0000","last_became_peered":"2024-09-13T08:41:30.170350+0000","last_unstale":"2024-09-13T08:41:31.152683+0000","last_undegraded":"2024-09-13T08:41:31.152683+0000","last_fullsized":"2024-09-13T08:41:31.152683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:58.357807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474990+0000","last_change":"2024-09-13T08:41:29.143199+0000","last_active":"2024-09-13T08:41:31.474990+0000","last_peered":"2024-09-13T08:41:31.474990+0000","last_clean":"2024-09-13T08:41:31.474990+0000","last_became_active":"2024-09-13T08:41:29.142930+0000","last_became_peered":"2024-09-13T08:41:29.142930+0000","last_unstale":"2024-09-13T08:41:31.474990+0000","last_undegraded":"2024-09-13T08:41:31.474990+0000","last_fullsized":"2024-09-13T08:41:31.474990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:36.019466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474439+0000","last_change":"2024-09-13T08:41:30.168382+0000","last_active":"2024-09-13T08:41:31.474439+0000","last_peered":"2024-09-13T08:41:31.474439+0000","last_clean":"2024-09-13T08:41:31.474439+0000","last_became_active":"2024-09-13T08:41:30.167748+0000","last_became_peered":"2024-09-13T08:41:30.167748+0000","last_unstale":"2024-09-13T08:41:31.474439+0000","last_undegraded":"2024-09-13T08:41:31.474439+0000","last_fullsized":"2024-09-13T08:41:31.474439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:43.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475680+0000","last_change":"2024-09-13T08:41:29.148961+0000","last_active":"2024-09-13T08:41:31.475680+0000","last_peered":"2024-09-13T08:41:31.475680+0000","last_clean":"2024-09-13T08:41:31.475680+0000","last_became_active":"2024-09-13T08:41:29.146591+0000","last_became_peered":"2024-09-13T08:41:29.146591+0000","last_unstale":"2024-09-13T08:41:31.475680+0000","last_undegraded":"2024-09-13T08:41:31.475680+0000","last_fullsized":"2024-09-13T08:41:31.475680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:25.694758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474398+0000","last_change":"2024-09-13T08:41:30.175697+0000","last_active":"2024-09-13T08:41:31.474398+0000","last_peered":"2024-09-13T08:41:31.474398+0000","last_clean":"2024-09-13T08:41:31.474398+0000","last_became_active":"2024-09-13T08:41:30.174133+0000","last_became_peered":"2024-09-13T08:41:30.174133+0000","last_unstale":"2024-09-13T08:41:31.474398+0000","last_undegraded":"2024-09-13T08:41:31.474398+0000","last_fullsized":"2024-09-13T08:41:31.474398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:45.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474547+0000","last_change":"2024-09-13T08:41:29.154197+0000","last_active":"2024-09-13T08:41:31.474547+0000","last_peered":"2024-09-13T08:41:31.474547+0000","last_clean":"2024-09-13T08:41:31.474547+0000","last_became_active":"2024-09-13T08:41:29.152859+0000","last_became_peered":"2024-09-13T08:41:29.152859+0000","last_unstale":"2024-09-13T08:41:31.474547+0000","last_undegraded":"2024-09-13T08:41:31.474547+0000","last_fullsized":"2024-09-13T08:41:31.474547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:35.228437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474394+0000","last_change":"2024-09-13T08:41:30.158406+0000","last_active":"2024-09-13T08:41:31.474394+0000","last_peered":"2024-09-13T08:41:31.474394+0000","last_clean":"2024-09-13T08:41:31.474394+0000","last_became_active":"2024-09-13T08:41:30.157975+0000","last_became_peered":"2024-09-13T08:41:30.157975+0000","last_unstale":"2024-09-13T08:41:31.474394+0000","last_undegraded":"2024-09-13T08:41:31.474394+0000","last_fullsized":"2024-09-13T08:41:31.474394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:54.914413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150950+0000","last_change":"2024-09-13T08:41:29.154649+0000","last_active":"2024-09-13T08:41:31.150950+0000","last_peered":"2024-09-13T08:41:31.150950+0000","last_clean":"2024-09-13T08:41:31.150950+0000","last_became_active":"2024-09-13T08:41:29.154327+0000","last_became_peered":"2024-09-13T08:41:29.154327+0000","last_unstale":"2024-09-13T08:41:31.150950+0000","last_undegraded":"2024-09-13T08:41:31.150950+0000","last_fullsized":"2024-09-13T08:41:31.150950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:18.313976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152456+0000","last_change":"2024-09-13T08:41:30.173642+0000","last_active":"2024-09-13T08:41:31.152456+0000","last_peered":"2024-09-13T08:41:31.152456+0000","last_clean":"2024-09-13T08:41:31.152456+0000","last_became_active":"2024-09-13T08:41:30.173362+0000","last_became_peered":"2024-09-13T08:41:30.173362+0000","last_unstale":"2024-09-13T08:41:31.152456+0000","last_undegraded":"2024-09-13T08:41:31.152456+0000","last_fullsized":"2024-09-13T08:41:31.152456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.753257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150731+0000","last_change":"2024-09-13T08:41:29.150523+0000","last_active":"2024-09-13T08:41:31.150731+0000","last_peered":"2024-09-13T08:41:31.150731+0000","last_clean":"2024-09-13T08:41:31.150731+0000","last_became_active":"2024-09-13T08:41:29.149999+0000","last_became_peered":"2024-09-13T08:41:29.149999+0000","last_unstale":"2024-09-13T08:41:31.150731+0000","last_undegraded":"2024-09-13T08:41:31.150731+0000","last_fullsized":"2024-09-13T08:41:31.150731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:32.616347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475559+0000","last_change":"2024-09-13T08:41:30.172825+0000","last_active":"2024-09-13T08:41:31.475559+0000","last_peered":"2024-09-13T08:41:31.475559+0000","last_clean":"2024-09-13T08:41:31.475559+0000","last_became_active":"2024-09-13T08:41:30.172459+0000","last_became_peered":"2024-09-13T08:41:30.172459+0000","last_unstale":"2024-09-13T08:41:31.475559+0000","last_undegraded":"2024-09-13T08:41:31.475559+0000","last_fullsized":"2024-09-13T08:41:31.475559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:08.614713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150663+0000","last_change":"2024-09-13T08:41:29.156767+0000","last_active":"2024-09-13T08:41:31.150663+0000","last_peered":"2024-09-13T08:41:31.150663+0000","last_clean":"2024-09-13T08:41:31.150663+0000","last_became_active":"2024-09-13T08:41:29.156127+0000","last_became_peered":"2024-09-13T08:41:29.156127+0000","last_unstale":"2024-09-13T08:41:31.150663+0000","last_undegraded":"2024-09-13T08:41:31.150663+0000","last_fullsized":"2024-09-13T08:41:31.150663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:26.199094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474754+0000","last_change":"2024-09-13T08:41:30.176974+0000","last_active":"2024-09-13T08:41:31.474754+0000","last_peered":"2024-09-13T08:41:31.474754+0000","last_clean":"2024-09-13T08:41:31.474754+0000","last_became_active":"2024-09-13T08:41:30.176715+0000","last_became_peered":"2024-09-13T08:41:30.176715+0000","last_unstale":"2024-09-13T08:41:31.474754+0000","last_undegraded":"2024-09-13T08:41:31.474754+0000","last_fullsized":"2024-09-13T08:41:31.474754+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474730+0000","last_change":"2024-09-13T08:41:29.148076+0000","last_active":"2024-09-13T08:41:31.474730+0000","last_peered":"2024-09-13T08:41:31.474730+0000","last_clean":"2024-09-13T08:41:31.474730+0000","last_became_active":"2024-09-13T08:41:29.147012+0000","last_became_peered":"2024-09-13T08:41:29.147012+0000","last_unstale":"2024-09-13T08:41:31.474730+0000","last_undegraded":"2024-09-13T08:41:31.474730+0000","last_fullsized":"2024-09-13T08:41:31.474730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:18.467245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.173815+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.173512+0000","last_became_peered":"2024-09-13T08:41:30.173512+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:17.533940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475003+0000","last_change":"2024-09-13T08:41:29.148851+0000","last_active":"2024-09-13T08:41:31.475003+0000","last_peered":"2024-09-13T08:41:31.475003+0000","last_clean":"2024-09-13T08:41:31.475003+0000","last_became_active":"2024-09-13T08:41:29.148567+0000","last_became_peered":"2024-09-13T08:41:29.148567+0000","last_unstale":"2024-09-13T08:41:31.475003+0000","last_undegraded":"2024-09-13T08:41:31.475003+0000","last_fullsized":"2024-09-13T08:41:31.475003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.901791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151894+0000","last_change":"2024-09-13T08:41:30.164444+0000","last_active":"2024-09-13T08:41:31.151894+0000","last_peered":"2024-09-13T08:41:31.151894+0000","last_clean":"2024-09-13T08:41:31.151894+0000","last_became_active":"2024-09-13T08:41:30.164177+0000","last_became_peered":"2024-09-13T08:41:30.164177+0000","last_unstale":"2024-09-13T08:41:31.151894+0000","last_undegraded":"2024-09-13T08:41:31.151894+0000","last_fullsized":"2024-09-13T08:41:31.151894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:10.013440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474879+0000","last_change":"2024-09-13T08:41:29.147550+0000","last_active":"2024-09-13T08:41:31.474879+0000","last_peered":"2024-09-13T08:41:31.474879+0000","last_clean":"2024-09-13T08:41:31.474879+0000","last_became_active":"2024-09-13T08:41:29.147183+0000","last_became_peered":"2024-09-13T08:41:29.147183+0000","last_unstale":"2024-09-13T08:41:31.474879+0000","last_undegraded":"2024-09-13T08:41:31.474879+0000","last_fullsized":"2024-09-13T08:41:31.474879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:09.961905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150069+0000","last_change":"2024-09-13T08:41:30.188192+0000","last_active":"2024-09-13T08:41:31.150069+0000","last_peered":"2024-09-13T08:41:31.150069+0000","last_clean":"2024-09-13T08:41:31.150069+0000","last_became_active":"2024-09-13T08:41:30.187402+0000","last_became_peered":"2024-09-13T08:41:30.187402+0000","last_unstale":"2024-09-13T08:41:31.150069+0000","last_undegraded":"2024-09-13T08:41:31.150069+0000","last_fullsized":"2024-09-13T08:41:31.150069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:23.226310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152027+0000","last_change":"2024-09-13T08:41:29.144692+0000","last_active":"2024-09-13T08:41:31.152027+0000","last_peered":"2024-09-13T08:41:31.152027+0000","last_clean":"2024-09-13T08:41:31.152027+0000","last_became_active":"2024-09-13T08:41:29.144396+0000","last_became_peered":"2024-09-13T08:41:29.144396+0000","last_unstale":"2024-09-13T08:41:31.152027+0000","last_undegraded":"2024-09-13T08:41:31.152027+0000","last_fullsized":"2024-09-13T08:41:31.152027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:05.544795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474559+0000","last_change":"2024-09-13T08:41:30.166665+0000","last_active":"2024-09-13T08:41:31.474559+0000","last_peered":"2024-09-13T08:41:31.474559+0000","last_clean":"2024-09-13T08:41:31.474559+0000","last_became_active":"2024-09-13T08:41:30.166364+0000","last_became_peered":"2024-09-13T08:41:30.166364+0000","last_unstale":"2024-09-13T08:41:31.474559+0000","last_undegraded":"2024-09-13T08:41:31.474559+0000","last_fullsized":"2024-09-13T08:41:31.474559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:37.021694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474651+0000","last_change":"2024-09-13T08:41:29.156770+0000","last_active":"2024-09-13T08:41:31.474651+0000","last_peered":"2024-09-13T08:41:31.474651+0000","last_clean":"2024-09-13T08:41:31.474651+0000","last_became_active":"2024-09-13T08:41:29.156466+0000","last_became_peered":"2024-09-13T08:41:29.156466+0000","last_unstale":"2024-09-13T08:41:31.474651+0000","last_undegraded":"2024-09-13T08:41:31.474651+0000","last_fullsized":"2024-09-13T08:41:31.474651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:04.189295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474295+0000","last_change":"2024-09-13T08:41:30.170922+0000","last_active":"2024-09-13T08:41:31.474295+0000","last_peered":"2024-09-13T08:41:31.474295+0000","last_clean":"2024-09-13T08:41:31.474295+0000","last_became_active":"2024-09-13T08:41:30.170663+0000","last_became_peered":"2024-09-13T08:41:30.170663+0000","last_unstale":"2024-09-13T08:41:31.474295+0000","last_undegraded":"2024-09-13T08:41:31.474295+0000","last_fullsized":"2024-09-13T08:41:31.474295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:07:26.254945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474197+0000","last_change":"2024-09-13T08:41:29.149340+0000","last_active":"2024-09-13T08:41:31.474197+0000","last_peered":"2024-09-13T08:41:31.474197+0000","last_clean":"2024-09-13T08:41:31.474197+0000","last_became_active":"2024-09-13T08:41:29.149060+0000","last_became_peered":"2024-09-13T08:41:29.149060+0000","last_unstale":"2024-09-13T08:41:31.474197+0000","last_undegraded":"2024-09-13T08:41:31.474197+0000","last_fullsized":"2024-09-13T08:41:31.474197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:44.441597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151653+0000","last_change":"2024-09-13T08:41:30.177150+0000","last_active":"2024-09-13T08:41:31.151653+0000","last_peered":"2024-09-13T08:41:31.151653+0000","last_clean":"2024-09-13T08:41:31.151653+0000","last_became_active":"2024-09-13T08:41:30.176850+0000","last_became_peered":"2024-09-13T08:41:30.176850+0000","last_unstale":"2024-09-13T08:41:31.151653+0000","last_undegraded":"2024-09-13T08:41:31.151653+0000","last_fullsized":"2024-09-13T08:41:31.151653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:44.657225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475011+0000","last_change":"2024-09-13T08:41:29.144964+0000","last_active":"2024-09-13T08:41:31.475011+0000","last_peered":"2024-09-13T08:41:31.475011+0000","last_clean":"2024-09-13T08:41:31.475011+0000","last_became_active":"2024-09-13T08:41:29.144673+0000","last_became_peered":"2024-09-13T08:41:29.144673+0000","last_unstale":"2024-09-13T08:41:31.475011+0000","last_undegraded":"2024-09-13T08:41:31.475011+0000","last_fullsized":"2024-09-13T08:41:31.475011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:12.107299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":207,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474058+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:41:31.474058+0000","last_peered":"2024-09-13T08:41:31.474058+0000","last_clean":"2024-09-13T08:41:31.474058+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:41:31.474058+0000","last_undegraded":"2024-09-13T08:41:31.474058+0000","last_fullsized":"2024-09-13T08:41:31.474058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":388196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":388230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509498,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":977042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509498,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":386710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509498,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":387166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509499,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1511424,"data_stored":976952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":921600,"data_stored":386654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":929792,"data_stored":387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:41:36.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:41:36.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:41:39.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:41:39.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 -- 172.21.15.31:0/725180027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 msgr2=0x7f08e00cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 --2- 172.21.15.31:0/725180027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00cf850 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f08e00cc8c0 tx=0x7f08d402a560 comp rx=0 tx=0).stop 2024-09-13T08:41:39.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 -- 172.21.15.31:0/725180027 shutdown_connections 2024-09-13T08:41:39.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 --2- 172.21.15.31:0/725180027 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e015f9e0 0x7f08e015fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 --2- 172.21.15.31:0/725180027 >> v2:172.21.15.45:3300/0 conn(0x7f08e00ce3c0 0x7f08e00ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 --2- 172.21.15.31:0/725180027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00cf850 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 -- 172.21.15.31:0/725180027 >> 172.21.15.31:0/725180027 conn(0x7f08e00cc910 msgr2=0x7f08e00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 -- 172.21.15.31:0/725180027 shutdown_connections 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.347+0000 7f08e8600640 1 -- 172.21.15.31:0/725180027 wait complete. 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.349+0000 7f08e8600640 1 Processor -- start 2024-09-13T08:41:39.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.349+0000 7f08e8600640 1 -- start start 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e00ce3c0 0x7f08e01f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e015f9e0 0x7f08e00caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08e00cb9d0 con 0x7f08e015bd70 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f08e00cb850 con 0x7f08e015f9e0 2024-09-13T08:41:39.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e8600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f08e00cbc50 con 0x7f08e00ce3c0 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59726/0 (socket says 172.21.15.31:59726) 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 -- 172.21.15.31:0/2566527660 learned_addr learned my addr 172.21.15.31:0/2566527660 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5c00640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e00ce3c0 0x7f08e01f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e6600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e015f9e0 0x7f08e00caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 -- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e00ce3c0 msgr2=0x7f08e01f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e00ce3c0 0x7f08e01f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 -- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e015f9e0 msgr2=0x7f08e00caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e015f9e0 0x7f08e00caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.350+0000 7f08e5200640 1 -- 172.21.15.31:0/2566527660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e01f4650 con 0x7f08e015bd70 2024-09-13T08:41:39.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08e5200640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00d750 tx=0x7f08cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:39.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc017070 con 0x7f08e015bd70 2024-09-13T08:41:39.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08cc009b20 con 0x7f08e015bd70 2024-09-13T08:41:39.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc010460 con 0x7f08e015bd70 2024-09-13T08:41:39.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e01f4930 con 0x7f08e015bd70 2024-09-13T08:41:39.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.351+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e01f4ef0 con 0x7f08e015bd70 2024-09-13T08:41:39.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.352+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e00ccf40 con 0x7f08e015bd70 2024-09-13T08:41:39.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.354+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f08cc01a070 con 0x7f08e015bd70 2024-09-13T08:41:39.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.354+0000 7f08dd600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 0x7f08c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.355+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f08cc09b120 con 0x7f08e015bd70 2024-09-13T08:41:39.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.355+0000 7f08e5c00640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 0x7f08c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.356+0000 7f08e5c00640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 0x7f08c007b0a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f08d4033000 tx=0x7f08d4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:39.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.359+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f08cc062ae0 con 0x7f08e015bd70 2024-09-13T08:41:39.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.550+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08e00008d0 con 0x7f08c0078be0 2024-09-13T08:41:39.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.555+0000 7f08dd600640 1 -- 172.21.15.31:0/2566527660 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7f08e00008d0 con 0x7f08c0078be0 2024-09-13T08:41:39.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:41:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 msgr2=0x7f08c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 0x7f08c007b0a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f08d4033000 tx=0x7f08d4004730 comp rx=0 tx=0).stop 2024-09-13T08:41:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 msgr2=0x7f08e00ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00d750 tx=0x7f08cc00dc10 comp rx=0 tx=0).stop 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 shutdown_connections 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e015f9e0 0x7f08e00caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e015bd70 0x7f08e00ca4c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f08c0078be0 0x7f08c007b0a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 --2- 172.21.15.31:0/2566527660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f08e00ce3c0 0x7f08e01f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 >> 172.21.15.31:0/2566527660 conn(0x7f08e00cc910 msgr2=0x7f08e00cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 shutdown_connections 2024-09-13T08:41:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.559+0000 7f08e8600640 1 -- 172.21.15.31:0/2566527660 wait complete. 2024-09-13T08:41:39.577 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-13T08:41:39.104616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262072,"kb_used_data":8392,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712648,"statfs":{"total":773094113280,"available":772825751552,"internally_reserved":0,"allocated":8593408,"data_stored":4278228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.956360"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150712+0000","last_change":"2024-09-13T08:41:30.177953+0000","last_active":"2024-09-13T08:41:31.150712+0000","last_peered":"2024-09-13T08:41:31.150712+0000","last_clean":"2024-09-13T08:41:31.150712+0000","last_became_active":"2024-09-13T08:41:30.177705+0000","last_became_peered":"2024-09-13T08:41:30.177705+0000","last_unstale":"2024-09-13T08:41:31.150712+0000","last_undegraded":"2024-09-13T08:41:31.150712+0000","last_fullsized":"2024-09-13T08:41:31.150712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:51.644500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.171555+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.171277+0000","last_became_peered":"2024-09-13T08:41:30.171277+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152712+0000","last_change":"2024-09-13T08:41:30.184527+0000","last_active":"2024-09-13T08:41:31.152712+0000","last_peered":"2024-09-13T08:41:31.152712+0000","last_clean":"2024-09-13T08:41:31.152712+0000","last_became_active":"2024-09-13T08:41:30.184178+0000","last_became_peered":"2024-09-13T08:41:30.184178+0000","last_unstale":"2024-09-13T08:41:31.152712+0000","last_undegraded":"2024-09-13T08:41:31.152712+0000","last_fullsized":"2024-09-13T08:41:31.152712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:11.304643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152599+0000","last_change":"2024-09-13T08:41:30.177541+0000","last_active":"2024-09-13T08:41:31.152599+0000","last_peered":"2024-09-13T08:41:31.152599+0000","last_clean":"2024-09-13T08:41:31.152599+0000","last_became_active":"2024-09-13T08:41:30.177252+0000","last_became_peered":"2024-09-13T08:41:30.177252+0000","last_unstale":"2024-09-13T08:41:31.152599+0000","last_undegraded":"2024-09-13T08:41:31.152599+0000","last_fullsized":"2024-09-13T08:41:31.152599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:05.041863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152229+0000","last_change":"2024-09-13T08:41:30.176493+0000","last_active":"2024-09-13T08:41:31.152229+0000","last_peered":"2024-09-13T08:41:31.152229+0000","last_clean":"2024-09-13T08:41:31.152229+0000","last_became_active":"2024-09-13T08:41:30.176063+0000","last_became_peered":"2024-09-13T08:41:30.176063+0000","last_unstale":"2024-09-13T08:41:31.152229+0000","last_undegraded":"2024-09-13T08:41:31.152229+0000","last_fullsized":"2024-09-13T08:41:31.152229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:10.675747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475149+0000","last_change":"2024-09-13T08:41:30.180783+0000","last_active":"2024-09-13T08:41:31.475149+0000","last_peered":"2024-09-13T08:41:31.475149+0000","last_clean":"2024-09-13T08:41:31.475149+0000","last_became_active":"2024-09-13T08:41:30.180485+0000","last_became_peered":"2024-09-13T08:41:30.180485+0000","last_unstale":"2024-09-13T08:41:31.475149+0000","last_undegraded":"2024-09-13T08:41:31.475149+0000","last_fullsized":"2024-09-13T08:41:31.475149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474644+0000","last_change":"2024-09-13T08:41:30.178989+0000","last_active":"2024-09-13T08:41:31.474644+0000","last_peered":"2024-09-13T08:41:31.474644+0000","last_clean":"2024-09-13T08:41:31.474644+0000","last_became_active":"2024-09-13T08:41:30.178708+0000","last_became_peered":"2024-09-13T08:41:30.178708+0000","last_unstale":"2024-09-13T08:41:31.474644+0000","last_undegraded":"2024-09-13T08:41:31.474644+0000","last_fullsized":"2024-09-13T08:41:31.474644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:02.877430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149574+0000","last_change":"2024-09-13T08:41:30.176643+0000","last_active":"2024-09-13T08:41:31.149574+0000","last_peered":"2024-09-13T08:41:31.149574+0000","last_clean":"2024-09-13T08:41:31.149574+0000","last_became_active":"2024-09-13T08:41:30.176347+0000","last_became_peered":"2024-09-13T08:41:30.176347+0000","last_unstale":"2024-09-13T08:41:31.149574+0000","last_undegraded":"2024-09-13T08:41:31.149574+0000","last_fullsized":"2024-09-13T08:41:31.149574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:45.146060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152963+0000","last_change":"2024-09-13T08:41:30.156159+0000","last_active":"2024-09-13T08:41:31.152963+0000","last_peered":"2024-09-13T08:41:31.152963+0000","last_clean":"2024-09-13T08:41:31.152963+0000","last_became_active":"2024-09-13T08:41:30.155807+0000","last_became_peered":"2024-09-13T08:41:30.155807+0000","last_unstale":"2024-09-13T08:41:31.152963+0000","last_undegraded":"2024-09-13T08:41:31.152963+0000","last_fullsized":"2024-09-13T08:41:31.152963+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:11.356687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475634+0000","last_change":"2024-09-13T08:41:30.181209+0000","last_active":"2024-09-13T08:41:31.475634+0000","last_peered":"2024-09-13T08:41:31.475634+0000","last_clean":"2024-09-13T08:41:31.475634+0000","last_became_active":"2024-09-13T08:41:30.179580+0000","last_became_peered":"2024-09-13T08:41:30.179580+0000","last_unstale":"2024-09-13T08:41:31.475634+0000","last_undegraded":"2024-09-13T08:41:31.475634+0000","last_fullsized":"2024-09-13T08:41:31.475634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:36.230665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150715+0000","last_change":"2024-09-13T08:41:30.175255+0000","last_active":"2024-09-13T08:41:31.150715+0000","last_peered":"2024-09-13T08:41:31.150715+0000","last_clean":"2024-09-13T08:41:31.150715+0000","last_became_active":"2024-09-13T08:41:30.174389+0000","last_became_peered":"2024-09-13T08:41:30.174389+0000","last_unstale":"2024-09-13T08:41:31.150715+0000","last_undegraded":"2024-09-13T08:41:31.150715+0000","last_fullsized":"2024-09-13T08:41:31.150715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:57:33.926281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475051+0000","last_change":"2024-09-13T08:41:30.167545+0000","last_active":"2024-09-13T08:41:31.475051+0000","last_peered":"2024-09-13T08:41:31.475051+0000","last_clean":"2024-09-13T08:41:31.475051+0000","last_became_active":"2024-09-13T08:41:30.167268+0000","last_became_peered":"2024-09-13T08:41:30.167268+0000","last_unstale":"2024-09-13T08:41:31.475051+0000","last_undegraded":"2024-09-13T08:41:31.475051+0000","last_fullsized":"2024-09-13T08:41:31.475051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:01.162295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152039+0000","last_change":"2024-09-13T08:41:30.177514+0000","last_active":"2024-09-13T08:41:31.152039+0000","last_peered":"2024-09-13T08:41:31.152039+0000","last_clean":"2024-09-13T08:41:31.152039+0000","last_became_active":"2024-09-13T08:41:30.177234+0000","last_became_peered":"2024-09-13T08:41:30.177234+0000","last_unstale":"2024-09-13T08:41:31.152039+0000","last_undegraded":"2024-09-13T08:41:31.152039+0000","last_fullsized":"2024-09-13T08:41:31.152039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:04:59.795415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474710+0000","last_change":"2024-09-13T08:41:30.172201+0000","last_active":"2024-09-13T08:41:31.474710+0000","last_peered":"2024-09-13T08:41:31.474710+0000","last_clean":"2024-09-13T08:41:31.474710+0000","last_became_active":"2024-09-13T08:41:30.171269+0000","last_became_peered":"2024-09-13T08:41:30.171269+0000","last_unstale":"2024-09-13T08:41:31.474710+0000","last_undegraded":"2024-09-13T08:41:31.474710+0000","last_fullsized":"2024-09-13T08:41:31.474710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.336999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474006+0000","last_change":"2024-09-13T08:41:30.169784+0000","last_active":"2024-09-13T08:41:31.474006+0000","last_peered":"2024-09-13T08:41:31.474006+0000","last_clean":"2024-09-13T08:41:31.474006+0000","last_became_active":"2024-09-13T08:41:30.169508+0000","last_became_peered":"2024-09-13T08:41:30.169508+0000","last_unstale":"2024-09-13T08:41:31.474006+0000","last_undegraded":"2024-09-13T08:41:31.474006+0000","last_fullsized":"2024-09-13T08:41:31.474006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:32.762281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149585+0000","last_change":"2024-09-13T08:41:30.168753+0000","last_active":"2024-09-13T08:41:31.149585+0000","last_peered":"2024-09-13T08:41:31.149585+0000","last_clean":"2024-09-13T08:41:31.149585+0000","last_became_active":"2024-09-13T08:41:30.168428+0000","last_became_peered":"2024-09-13T08:41:30.168428+0000","last_unstale":"2024-09-13T08:41:31.149585+0000","last_undegraded":"2024-09-13T08:41:31.149585+0000","last_fullsized":"2024-09-13T08:41:31.149585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:28.857492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152183+0000","last_change":"2024-09-13T08:41:30.174176+0000","last_active":"2024-09-13T08:41:31.152183+0000","last_peered":"2024-09-13T08:41:31.152183+0000","last_clean":"2024-09-13T08:41:31.152183+0000","last_became_active":"2024-09-13T08:41:30.173898+0000","last_became_peered":"2024-09-13T08:41:30.173898+0000","last_unstale":"2024-09-13T08:41:31.152183+0000","last_undegraded":"2024-09-13T08:41:31.152183+0000","last_fullsized":"2024-09-13T08:41:31.152183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:43:41.193530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152346+0000","last_change":"2024-09-13T08:41:30.179501+0000","last_active":"2024-09-13T08:41:31.152346+0000","last_peered":"2024-09-13T08:41:31.152346+0000","last_clean":"2024-09-13T08:41:31.152346+0000","last_became_active":"2024-09-13T08:41:30.179215+0000","last_became_peered":"2024-09-13T08:41:30.179215+0000","last_unstale":"2024-09-13T08:41:31.152346+0000","last_undegraded":"2024-09-13T08:41:31.152346+0000","last_fullsized":"2024-09-13T08:41:31.152346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:41:33.182476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150478+0000","last_change":"2024-09-13T08:41:30.176585+0000","last_active":"2024-09-13T08:41:31.150478+0000","last_peered":"2024-09-13T08:41:31.150478+0000","last_clean":"2024-09-13T08:41:31.150478+0000","last_became_active":"2024-09-13T08:41:30.173677+0000","last_became_peered":"2024-09-13T08:41:30.173677+0000","last_unstale":"2024-09-13T08:41:31.150478+0000","last_undegraded":"2024-09-13T08:41:31.150478+0000","last_fullsized":"2024-09-13T08:41:31.150478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:18.526089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152370+0000","last_change":"2024-09-13T08:41:30.179532+0000","last_active":"2024-09-13T08:41:31.152370+0000","last_peered":"2024-09-13T08:41:31.152370+0000","last_clean":"2024-09-13T08:41:31.152370+0000","last_became_active":"2024-09-13T08:41:30.179182+0000","last_became_peered":"2024-09-13T08:41:30.179182+0000","last_unstale":"2024-09-13T08:41:31.152370+0000","last_undegraded":"2024-09-13T08:41:31.152370+0000","last_fullsized":"2024-09-13T08:41:31.152370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:03.650279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475052+0000","last_change":"2024-09-13T08:41:30.155465+0000","last_active":"2024-09-13T08:41:31.475052+0000","last_peered":"2024-09-13T08:41:31.475052+0000","last_clean":"2024-09-13T08:41:31.475052+0000","last_became_active":"2024-09-13T08:41:30.155056+0000","last_became_peered":"2024-09-13T08:41:30.155056+0000","last_unstale":"2024-09-13T08:41:31.475052+0000","last_undegraded":"2024-09-13T08:41:31.475052+0000","last_fullsized":"2024-09-13T08:41:31.475052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:26.842118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475329+0000","last_change":"2024-09-13T08:41:30.178147+0000","last_active":"2024-09-13T08:41:31.475329+0000","last_peered":"2024-09-13T08:41:31.475329+0000","last_clean":"2024-09-13T08:41:31.475329+0000","last_became_active":"2024-09-13T08:41:30.175910+0000","last_became_peered":"2024-09-13T08:41:30.175910+0000","last_unstale":"2024-09-13T08:41:31.475329+0000","last_undegraded":"2024-09-13T08:41:31.475329+0000","last_fullsized":"2024-09-13T08:41:31.475329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:19.316204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474864+0000","last_change":"2024-09-13T08:41:30.190277+0000","last_active":"2024-09-13T08:41:31.474864+0000","last_peered":"2024-09-13T08:41:31.474864+0000","last_clean":"2024-09-13T08:41:31.474864+0000","last_became_active":"2024-09-13T08:41:30.189911+0000","last_became_peered":"2024-09-13T08:41:30.189911+0000","last_unstale":"2024-09-13T08:41:31.474864+0000","last_undegraded":"2024-09-13T08:41:31.474864+0000","last_fullsized":"2024-09-13T08:41:31.474864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:46.494807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149710+0000","last_change":"2024-09-13T08:41:30.179319+0000","last_active":"2024-09-13T08:41:31.149710+0000","last_peered":"2024-09-13T08:41:31.149710+0000","last_clean":"2024-09-13T08:41:31.149710+0000","last_became_active":"2024-09-13T08:41:30.179050+0000","last_became_peered":"2024-09-13T08:41:30.179050+0000","last_unstale":"2024-09-13T08:41:31.149710+0000","last_undegraded":"2024-09-13T08:41:31.149710+0000","last_fullsized":"2024-09-13T08:41:31.149710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:01.953440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475362+0000","last_change":"2024-09-13T08:41:30.190025+0000","last_active":"2024-09-13T08:41:31.475362+0000","last_peered":"2024-09-13T08:41:31.475362+0000","last_clean":"2024-09-13T08:41:31.475362+0000","last_became_active":"2024-09-13T08:41:30.189682+0000","last_became_peered":"2024-09-13T08:41:30.189682+0000","last_unstale":"2024-09-13T08:41:31.475362+0000","last_undegraded":"2024-09-13T08:41:31.475362+0000","last_fullsized":"2024-09-13T08:41:31.475362+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:08.385273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150264+0000","last_change":"2024-09-13T08:41:30.190500+0000","last_active":"2024-09-13T08:41:31.150264+0000","last_peered":"2024-09-13T08:41:31.150264+0000","last_clean":"2024-09-13T08:41:31.150264+0000","last_became_active":"2024-09-13T08:41:30.190211+0000","last_became_peered":"2024-09-13T08:41:30.190211+0000","last_unstale":"2024-09-13T08:41:31.150264+0000","last_undegraded":"2024-09-13T08:41:31.150264+0000","last_fullsized":"2024-09-13T08:41:31.150264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:55:48.809149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151986+0000","last_change":"2024-09-13T08:41:30.168525+0000","last_active":"2024-09-13T08:41:31.151986+0000","last_peered":"2024-09-13T08:41:31.151986+0000","last_clean":"2024-09-13T08:41:31.151986+0000","last_became_active":"2024-09-13T08:41:30.167750+0000","last_became_peered":"2024-09-13T08:41:30.167750+0000","last_unstale":"2024-09-13T08:41:31.151986+0000","last_undegraded":"2024-09-13T08:41:31.151986+0000","last_fullsized":"2024-09-13T08:41:31.151986+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:37.422258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474553+0000","last_change":"2024-09-13T08:41:30.165606+0000","last_active":"2024-09-13T08:41:31.474553+0000","last_peered":"2024-09-13T08:41:31.474553+0000","last_clean":"2024-09-13T08:41:31.474553+0000","last_became_active":"2024-09-13T08:41:30.165277+0000","last_became_peered":"2024-09-13T08:41:30.165277+0000","last_unstale":"2024-09-13T08:41:31.474553+0000","last_undegraded":"2024-09-13T08:41:31.474553+0000","last_fullsized":"2024-09-13T08:41:31.474553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152266+0000","last_change":"2024-09-13T08:41:30.170008+0000","last_active":"2024-09-13T08:41:31.152266+0000","last_peered":"2024-09-13T08:41:31.152266+0000","last_clean":"2024-09-13T08:41:31.152266+0000","last_became_active":"2024-09-13T08:41:30.169516+0000","last_became_peered":"2024-09-13T08:41:30.169516+0000","last_unstale":"2024-09-13T08:41:31.152266+0000","last_undegraded":"2024-09-13T08:41:31.152266+0000","last_fullsized":"2024-09-13T08:41:31.152266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:12:20.327460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149957+0000","last_change":"2024-09-13T08:41:30.171602+0000","last_active":"2024-09-13T08:41:31.149957+0000","last_peered":"2024-09-13T08:41:31.149957+0000","last_clean":"2024-09-13T08:41:31.149957+0000","last_became_active":"2024-09-13T08:41:30.170882+0000","last_became_peered":"2024-09-13T08:41:30.170882+0000","last_unstale":"2024-09-13T08:41:31.149957+0000","last_undegraded":"2024-09-13T08:41:31.149957+0000","last_fullsized":"2024-09-13T08:41:31.149957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:17.491430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474487+0000","last_change":"2024-09-13T08:41:30.165505+0000","last_active":"2024-09-13T08:41:31.474487+0000","last_peered":"2024-09-13T08:41:31.474487+0000","last_clean":"2024-09-13T08:41:31.474487+0000","last_became_active":"2024-09-13T08:41:30.165205+0000","last_became_peered":"2024-09-13T08:41:30.165205+0000","last_unstale":"2024-09-13T08:41:31.474487+0000","last_undegraded":"2024-09-13T08:41:31.474487+0000","last_fullsized":"2024-09-13T08:41:31.474487+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:53.060236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474225+0000","last_change":"2024-09-13T08:41:30.166147+0000","last_active":"2024-09-13T08:41:31.474225+0000","last_peered":"2024-09-13T08:41:31.474225+0000","last_clean":"2024-09-13T08:41:31.474225+0000","last_became_active":"2024-09-13T08:41:30.165114+0000","last_became_peered":"2024-09-13T08:41:30.165114+0000","last_unstale":"2024-09-13T08:41:31.474225+0000","last_undegraded":"2024-09-13T08:41:31.474225+0000","last_fullsized":"2024-09-13T08:41:31.474225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:51:53.742788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150526+0000","last_change":"2024-09-13T08:41:30.184240+0000","last_active":"2024-09-13T08:41:31.150526+0000","last_peered":"2024-09-13T08:41:31.150526+0000","last_clean":"2024-09-13T08:41:31.150526+0000","last_became_active":"2024-09-13T08:41:30.183893+0000","last_became_peered":"2024-09-13T08:41:30.183893+0000","last_unstale":"2024-09-13T08:41:31.150526+0000","last_undegraded":"2024-09-13T08:41:31.150526+0000","last_fullsized":"2024-09-13T08:41:31.150526+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:51.357207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"75'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474599+0000","last_change":"2024-09-13T08:41:29.151268+0000","last_active":"2024-09-13T08:41:31.474599+0000","last_peered":"2024-09-13T08:41:31.474599+0000","last_clean":"2024-09-13T08:41:31.474599+0000","last_became_active":"2024-09-13T08:41:29.150960+0000","last_became_peered":"2024-09-13T08:41:29.150960+0000","last_unstale":"2024-09-13T08:41:31.474599+0000","last_undegraded":"2024-09-13T08:41:31.474599+0000","last_fullsized":"2024-09-13T08:41:31.474599+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:42.453175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152927+0000","last_change":"2024-09-13T08:41:30.164561+0000","last_active":"2024-09-13T08:41:31.152927+0000","last_peered":"2024-09-13T08:41:31.152927+0000","last_clean":"2024-09-13T08:41:31.152927+0000","last_became_active":"2024-09-13T08:41:30.164290+0000","last_became_peered":"2024-09-13T08:41:30.164290+0000","last_unstale":"2024-09-13T08:41:31.152927+0000","last_undegraded":"2024-09-13T08:41:31.152927+0000","last_fullsized":"2024-09-13T08:41:31.152927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:33.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152403+0000","last_change":"2024-09-13T08:41:29.144190+0000","last_active":"2024-09-13T08:41:31.152403+0000","last_peered":"2024-09-13T08:41:31.152403+0000","last_clean":"2024-09-13T08:41:31.152403+0000","last_became_active":"2024-09-13T08:41:29.143622+0000","last_became_peered":"2024-09-13T08:41:29.143622+0000","last_unstale":"2024-09-13T08:41:31.152403+0000","last_undegraded":"2024-09-13T08:41:31.152403+0000","last_fullsized":"2024-09-13T08:41:31.152403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:37.575695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474834+0000","last_change":"2024-09-13T08:41:30.176252+0000","last_active":"2024-09-13T08:41:31.474834+0000","last_peered":"2024-09-13T08:41:31.474834+0000","last_clean":"2024-09-13T08:41:31.474834+0000","last_became_active":"2024-09-13T08:41:30.174505+0000","last_became_peered":"2024-09-13T08:41:30.174505+0000","last_unstale":"2024-09-13T08:41:31.474834+0000","last_undegraded":"2024-09-13T08:41:31.474834+0000","last_fullsized":"2024-09-13T08:41:31.474834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150540+0000","last_change":"2024-09-13T08:41:29.153502+0000","last_active":"2024-09-13T08:41:31.150540+0000","last_peered":"2024-09-13T08:41:31.150540+0000","last_clean":"2024-09-13T08:41:31.150540+0000","last_became_active":"2024-09-13T08:41:29.153256+0000","last_became_peered":"2024-09-13T08:41:29.153256+0000","last_unstale":"2024-09-13T08:41:31.150540+0000","last_undegraded":"2024-09-13T08:41:31.150540+0000","last_fullsized":"2024-09-13T08:41:31.150540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:11.750149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475366+0000","last_change":"2024-09-13T08:41:30.180280+0000","last_active":"2024-09-13T08:41:31.475366+0000","last_peered":"2024-09-13T08:41:31.475366+0000","last_clean":"2024-09-13T08:41:31.475366+0000","last_became_active":"2024-09-13T08:41:30.179979+0000","last_became_peered":"2024-09-13T08:41:30.179979+0000","last_unstale":"2024-09-13T08:41:31.475366+0000","last_undegraded":"2024-09-13T08:41:31.475366+0000","last_fullsized":"2024-09-13T08:41:31.475366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:39.299088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474892+0000","last_change":"2024-09-13T08:41:29.142380+0000","last_active":"2024-09-13T08:41:31.474892+0000","last_peered":"2024-09-13T08:41:31.474892+0000","last_clean":"2024-09-13T08:41:31.474892+0000","last_became_active":"2024-09-13T08:41:29.142093+0000","last_became_peered":"2024-09-13T08:41:29.142093+0000","last_unstale":"2024-09-13T08:41:31.474892+0000","last_undegraded":"2024-09-13T08:41:31.474892+0000","last_fullsized":"2024-09-13T08:41:31.474892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:05.945048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.149958+0000","last_change":"2024-09-13T08:41:30.175452+0000","last_active":"2024-09-13T08:41:31.149958+0000","last_peered":"2024-09-13T08:41:31.149958+0000","last_clean":"2024-09-13T08:41:31.149958+0000","last_became_active":"2024-09-13T08:41:30.175101+0000","last_became_peered":"2024-09-13T08:41:30.175101+0000","last_unstale":"2024-09-13T08:41:31.149958+0000","last_undegraded":"2024-09-13T08:41:31.149958+0000","last_fullsized":"2024-09-13T08:41:31.149958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:35.678193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474813+0000","last_change":"2024-09-13T08:41:29.152256+0000","last_active":"2024-09-13T08:41:31.474813+0000","last_peered":"2024-09-13T08:41:31.474813+0000","last_clean":"2024-09-13T08:41:31.474813+0000","last_became_active":"2024-09-13T08:41:29.151880+0000","last_became_peered":"2024-09-13T08:41:29.151880+0000","last_unstale":"2024-09-13T08:41:31.474813+0000","last_undegraded":"2024-09-13T08:41:31.474813+0000","last_fullsized":"2024-09-13T08:41:31.474813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:45.557782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474596+0000","last_change":"2024-09-13T08:41:30.179754+0000","last_active":"2024-09-13T08:41:31.474596+0000","last_peered":"2024-09-13T08:41:31.474596+0000","last_clean":"2024-09-13T08:41:31.474596+0000","last_became_active":"2024-09-13T08:41:30.176002+0000","last_became_peered":"2024-09-13T08:41:30.176002+0000","last_unstale":"2024-09-13T08:41:31.474596+0000","last_undegraded":"2024-09-13T08:41:31.474596+0000","last_fullsized":"2024-09-13T08:41:31.474596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:21:32.139399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474629+0000","last_change":"2024-09-13T08:41:29.144056+0000","last_active":"2024-09-13T08:41:31.474629+0000","last_peered":"2024-09-13T08:41:31.474629+0000","last_clean":"2024-09-13T08:41:31.474629+0000","last_became_active":"2024-09-13T08:41:29.143787+0000","last_became_peered":"2024-09-13T08:41:29.143787+0000","last_unstale":"2024-09-13T08:41:31.474629+0000","last_undegraded":"2024-09-13T08:41:31.474629+0000","last_fullsized":"2024-09-13T08:41:31.474629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:07.698273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474317+0000","last_change":"2024-09-13T08:41:30.176412+0000","last_active":"2024-09-13T08:41:31.474317+0000","last_peered":"2024-09-13T08:41:31.474317+0000","last_clean":"2024-09-13T08:41:31.474317+0000","last_became_active":"2024-09-13T08:41:30.176124+0000","last_became_peered":"2024-09-13T08:41:30.176124+0000","last_unstale":"2024-09-13T08:41:31.474317+0000","last_undegraded":"2024-09-13T08:41:31.474317+0000","last_fullsized":"2024-09-13T08:41:31.474317+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:19:31.333677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474680+0000","last_change":"2024-09-13T08:41:29.142404+0000","last_active":"2024-09-13T08:41:31.474680+0000","last_peered":"2024-09-13T08:41:31.474680+0000","last_clean":"2024-09-13T08:41:31.474680+0000","last_became_active":"2024-09-13T08:41:29.142078+0000","last_became_peered":"2024-09-13T08:41:29.142078+0000","last_unstale":"2024-09-13T08:41:31.474680+0000","last_undegraded":"2024-09-13T08:41:31.474680+0000","last_fullsized":"2024-09-13T08:41:31.474680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:42.389929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474073+0000","last_change":"2024-09-13T08:41:30.171632+0000","last_active":"2024-09-13T08:41:31.474073+0000","last_peered":"2024-09-13T08:41:31.474073+0000","last_clean":"2024-09-13T08:41:31.474073+0000","last_became_active":"2024-09-13T08:41:30.171351+0000","last_became_peered":"2024-09-13T08:41:30.171351+0000","last_unstale":"2024-09-13T08:41:31.474073+0000","last_undegraded":"2024-09-13T08:41:31.474073+0000","last_fullsized":"2024-09-13T08:41:31.474073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:36.601619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.473982+0000","last_change":"2024-09-13T08:41:29.141838+0000","last_active":"2024-09-13T08:41:31.473982+0000","last_peered":"2024-09-13T08:41:31.473982+0000","last_clean":"2024-09-13T08:41:31.473982+0000","last_became_active":"2024-09-13T08:41:29.141458+0000","last_became_peered":"2024-09-13T08:41:29.141458+0000","last_unstale":"2024-09-13T08:41:31.473982+0000","last_undegraded":"2024-09-13T08:41:31.473982+0000","last_fullsized":"2024-09-13T08:41:31.473982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:57.962170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475749+0000","last_change":"2024-09-13T08:41:30.186535+0000","last_active":"2024-09-13T08:41:31.475749+0000","last_peered":"2024-09-13T08:41:31.475749+0000","last_clean":"2024-09-13T08:41:31.475749+0000","last_became_active":"2024-09-13T08:41:30.186257+0000","last_became_peered":"2024-09-13T08:41:30.186257+0000","last_unstale":"2024-09-13T08:41:31.475749+0000","last_undegraded":"2024-09-13T08:41:31.475749+0000","last_fullsized":"2024-09-13T08:41:31.475749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:05.191523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150332+0000","last_change":"2024-09-13T08:41:29.149880+0000","last_active":"2024-09-13T08:41:31.150332+0000","last_peered":"2024-09-13T08:41:31.150332+0000","last_clean":"2024-09-13T08:41:31.150332+0000","last_became_active":"2024-09-13T08:41:29.149556+0000","last_became_peered":"2024-09-13T08:41:29.149556+0000","last_unstale":"2024-09-13T08:41:31.150332+0000","last_undegraded":"2024-09-13T08:41:31.150332+0000","last_fullsized":"2024-09-13T08:41:31.150332+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:39.387524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475671+0000","last_change":"2024-09-13T08:41:30.171210+0000","last_active":"2024-09-13T08:41:31.475671+0000","last_peered":"2024-09-13T08:41:31.475671+0000","last_clean":"2024-09-13T08:41:31.475671+0000","last_became_active":"2024-09-13T08:41:30.170494+0000","last_became_peered":"2024-09-13T08:41:30.170494+0000","last_unstale":"2024-09-13T08:41:31.475671+0000","last_undegraded":"2024-09-13T08:41:31.475671+0000","last_fullsized":"2024-09-13T08:41:31.475671+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475717+0000","last_change":"2024-09-13T08:41:29.154606+0000","last_active":"2024-09-13T08:41:31.475717+0000","last_peered":"2024-09-13T08:41:31.475717+0000","last_clean":"2024-09-13T08:41:31.475717+0000","last_became_active":"2024-09-13T08:41:29.154326+0000","last_became_peered":"2024-09-13T08:41:29.154326+0000","last_unstale":"2024-09-13T08:41:31.475717+0000","last_undegraded":"2024-09-13T08:41:31.475717+0000","last_fullsized":"2024-09-13T08:41:31.475717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:52.695133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150201+0000","last_change":"2024-09-13T08:41:30.191552+0000","last_active":"2024-09-13T08:41:31.150201+0000","last_peered":"2024-09-13T08:41:31.150201+0000","last_clean":"2024-09-13T08:41:31.150201+0000","last_became_active":"2024-09-13T08:41:30.191295+0000","last_became_peered":"2024-09-13T08:41:30.191295+0000","last_unstale":"2024-09-13T08:41:31.150201+0000","last_undegraded":"2024-09-13T08:41:31.150201+0000","last_fullsized":"2024-09-13T08:41:31.150201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:34.894273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474613+0000","last_change":"2024-09-13T08:41:29.147972+0000","last_active":"2024-09-13T08:41:31.474613+0000","last_peered":"2024-09-13T08:41:31.474613+0000","last_clean":"2024-09-13T08:41:31.474613+0000","last_became_active":"2024-09-13T08:41:29.147657+0000","last_became_peered":"2024-09-13T08:41:29.147657+0000","last_unstale":"2024-09-13T08:41:31.474613+0000","last_undegraded":"2024-09-13T08:41:31.474613+0000","last_fullsized":"2024-09-13T08:41:31.474613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:37.087395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152158+0000","last_change":"2024-09-13T08:41:30.168763+0000","last_active":"2024-09-13T08:41:31.152158+0000","last_peered":"2024-09-13T08:41:31.152158+0000","last_clean":"2024-09-13T08:41:31.152158+0000","last_became_active":"2024-09-13T08:41:30.168493+0000","last_became_peered":"2024-09-13T08:41:30.168493+0000","last_unstale":"2024-09-13T08:41:31.152158+0000","last_undegraded":"2024-09-13T08:41:31.152158+0000","last_fullsized":"2024-09-13T08:41:31.152158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:27.265527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"75'6","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474699+0000","last_change":"2024-09-13T08:41:29.142862+0000","last_active":"2024-09-13T08:41:31.474699+0000","last_peered":"2024-09-13T08:41:31.474699+0000","last_clean":"2024-09-13T08:41:31.474699+0000","last_became_active":"2024-09-13T08:41:29.140928+0000","last_became_peered":"2024-09-13T08:41:29.140928+0000","last_unstale":"2024-09-13T08:41:31.474699+0000","last_undegraded":"2024-09-13T08:41:31.474699+0000","last_fullsized":"2024-09-13T08:41:31.474699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:29:54.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475544+0000","last_change":"2024-09-13T08:41:30.166884+0000","last_active":"2024-09-13T08:41:31.475544+0000","last_peered":"2024-09-13T08:41:31.475544+0000","last_clean":"2024-09-13T08:41:31.475544+0000","last_became_active":"2024-09-13T08:41:30.166578+0000","last_became_peered":"2024-09-13T08:41:30.166578+0000","last_unstale":"2024-09-13T08:41:31.475544+0000","last_undegraded":"2024-09-13T08:41:31.475544+0000","last_fullsized":"2024-09-13T08:41:31.475544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:06.547024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474519+0000","last_change":"2024-09-13T08:41:29.144444+0000","last_active":"2024-09-13T08:41:31.474519+0000","last_peered":"2024-09-13T08:41:31.474519+0000","last_clean":"2024-09-13T08:41:31.474519+0000","last_became_active":"2024-09-13T08:41:29.144236+0000","last_became_peered":"2024-09-13T08:41:29.144236+0000","last_unstale":"2024-09-13T08:41:31.474519+0000","last_undegraded":"2024-09-13T08:41:31.474519+0000","last_fullsized":"2024-09-13T08:41:31.474519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:31.977262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475053+0000","last_change":"2024-09-13T08:41:30.178971+0000","last_active":"2024-09-13T08:41:31.475053+0000","last_peered":"2024-09-13T08:41:31.475053+0000","last_clean":"2024-09-13T08:41:31.475053+0000","last_became_active":"2024-09-13T08:41:30.178662+0000","last_became_peered":"2024-09-13T08:41:30.178662+0000","last_unstale":"2024-09-13T08:41:31.475053+0000","last_undegraded":"2024-09-13T08:41:31.475053+0000","last_fullsized":"2024-09-13T08:41:31.475053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.425812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474733+0000","last_change":"2024-09-13T08:41:29.148940+0000","last_active":"2024-09-13T08:41:31.474733+0000","last_peered":"2024-09-13T08:41:31.474733+0000","last_clean":"2024-09-13T08:41:31.474733+0000","last_became_active":"2024-09-13T08:41:29.148677+0000","last_became_peered":"2024-09-13T08:41:29.148677+0000","last_unstale":"2024-09-13T08:41:31.474733+0000","last_undegraded":"2024-09-13T08:41:31.474733+0000","last_fullsized":"2024-09-13T08:41:31.474733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:36:49.700010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151295+0000","last_change":"2024-09-13T08:41:30.176836+0000","last_active":"2024-09-13T08:41:31.151295+0000","last_peered":"2024-09-13T08:41:31.151295+0000","last_clean":"2024-09-13T08:41:31.151295+0000","last_became_active":"2024-09-13T08:41:30.176561+0000","last_became_peered":"2024-09-13T08:41:30.176561+0000","last_unstale":"2024-09-13T08:41:31.151295+0000","last_undegraded":"2024-09-13T08:41:31.151295+0000","last_fullsized":"2024-09-13T08:41:31.151295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:27.201322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474722+0000","last_change":"2024-09-13T08:41:29.137115+0000","last_active":"2024-09-13T08:41:31.474722+0000","last_peered":"2024-09-13T08:41:31.474722+0000","last_clean":"2024-09-13T08:41:31.474722+0000","last_became_active":"2024-09-13T08:41:29.136800+0000","last_became_peered":"2024-09-13T08:41:29.136800+0000","last_unstale":"2024-09-13T08:41:31.474722+0000","last_undegraded":"2024-09-13T08:41:31.474722+0000","last_fullsized":"2024-09-13T08:41:31.474722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:04:53.975223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151142+0000","last_change":"2024-09-13T08:41:30.164645+0000","last_active":"2024-09-13T08:41:31.151142+0000","last_peered":"2024-09-13T08:41:31.151142+0000","last_clean":"2024-09-13T08:41:31.151142+0000","last_became_active":"2024-09-13T08:41:30.164362+0000","last_became_peered":"2024-09-13T08:41:30.164362+0000","last_unstale":"2024-09-13T08:41:31.151142+0000","last_undegraded":"2024-09-13T08:41:31.151142+0000","last_fullsized":"2024-09-13T08:41:31.151142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:39.424459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"75'8","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474024+0000","last_change":"2024-09-13T08:41:29.141724+0000","last_active":"2024-09-13T08:41:31.474024+0000","last_peered":"2024-09-13T08:41:31.474024+0000","last_clean":"2024-09-13T08:41:31.474024+0000","last_became_active":"2024-09-13T08:41:29.141410+0000","last_became_peered":"2024-09-13T08:41:29.141410+0000","last_unstale":"2024-09-13T08:41:31.474024+0000","last_undegraded":"2024-09-13T08:41:31.474024+0000","last_fullsized":"2024-09-13T08:41:31.474024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:36.990282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475780+0000","last_change":"2024-09-13T08:41:30.170258+0000","last_active":"2024-09-13T08:41:31.475780+0000","last_peered":"2024-09-13T08:41:31.475780+0000","last_clean":"2024-09-13T08:41:31.475780+0000","last_became_active":"2024-09-13T08:41:30.169900+0000","last_became_peered":"2024-09-13T08:41:30.169900+0000","last_unstale":"2024-09-13T08:41:31.475780+0000","last_undegraded":"2024-09-13T08:41:31.475780+0000","last_fullsized":"2024-09-13T08:41:31.475780+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:38.577924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150429+0000","last_change":"2024-09-13T08:41:29.152478+0000","last_active":"2024-09-13T08:41:31.150429+0000","last_peered":"2024-09-13T08:41:31.150429+0000","last_clean":"2024-09-13T08:41:31.150429+0000","last_became_active":"2024-09-13T08:41:29.152174+0000","last_became_peered":"2024-09-13T08:41:29.152174+0000","last_unstale":"2024-09-13T08:41:31.150429+0000","last_undegraded":"2024-09-13T08:41:31.150429+0000","last_fullsized":"2024-09-13T08:41:31.150429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:27.809813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475948+0000","last_change":"2024-09-13T08:41:30.157129+0000","last_active":"2024-09-13T08:41:31.475948+0000","last_peered":"2024-09-13T08:41:31.475948+0000","last_clean":"2024-09-13T08:41:31.475948+0000","last_became_active":"2024-09-13T08:41:30.156648+0000","last_became_peered":"2024-09-13T08:41:30.156648+0000","last_unstale":"2024-09-13T08:41:31.475948+0000","last_undegraded":"2024-09-13T08:41:31.475948+0000","last_fullsized":"2024-09-13T08:41:31.475948+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:28.126509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475814+0000","last_change":"2024-09-13T08:41:29.156147+0000","last_active":"2024-09-13T08:41:31.475814+0000","last_peered":"2024-09-13T08:41:31.475814+0000","last_clean":"2024-09-13T08:41:31.475814+0000","last_became_active":"2024-09-13T08:41:29.155857+0000","last_became_peered":"2024-09-13T08:41:29.155857+0000","last_unstale":"2024-09-13T08:41:31.475814+0000","last_undegraded":"2024-09-13T08:41:31.475814+0000","last_fullsized":"2024-09-13T08:41:31.475814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:27.046693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152642+0000","last_change":"2024-09-13T08:41:30.160794+0000","last_active":"2024-09-13T08:41:31.152642+0000","last_peered":"2024-09-13T08:41:31.152642+0000","last_clean":"2024-09-13T08:41:31.152642+0000","last_became_active":"2024-09-13T08:41:30.160339+0000","last_became_peered":"2024-09-13T08:41:30.160339+0000","last_unstale":"2024-09-13T08:41:31.152642+0000","last_undegraded":"2024-09-13T08:41:31.152642+0000","last_fullsized":"2024-09-13T08:41:31.152642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:56.686232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474046+0000","last_change":"2024-09-13T08:41:29.154023+0000","last_active":"2024-09-13T08:41:31.474046+0000","last_peered":"2024-09-13T08:41:31.474046+0000","last_clean":"2024-09-13T08:41:31.474046+0000","last_became_active":"2024-09-13T08:41:29.152087+0000","last_became_peered":"2024-09-13T08:41:29.152087+0000","last_unstale":"2024-09-13T08:41:31.474046+0000","last_undegraded":"2024-09-13T08:41:31.474046+0000","last_fullsized":"2024-09-13T08:41:31.474046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:38:02.653402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152683+0000","last_change":"2024-09-13T08:41:30.170644+0000","last_active":"2024-09-13T08:41:31.152683+0000","last_peered":"2024-09-13T08:41:31.152683+0000","last_clean":"2024-09-13T08:41:31.152683+0000","last_became_active":"2024-09-13T08:41:30.170350+0000","last_became_peered":"2024-09-13T08:41:30.170350+0000","last_unstale":"2024-09-13T08:41:31.152683+0000","last_undegraded":"2024-09-13T08:41:31.152683+0000","last_fullsized":"2024-09-13T08:41:31.152683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:58.357807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474990+0000","last_change":"2024-09-13T08:41:29.143199+0000","last_active":"2024-09-13T08:41:31.474990+0000","last_peered":"2024-09-13T08:41:31.474990+0000","last_clean":"2024-09-13T08:41:31.474990+0000","last_became_active":"2024-09-13T08:41:29.142930+0000","last_became_peered":"2024-09-13T08:41:29.142930+0000","last_unstale":"2024-09-13T08:41:31.474990+0000","last_undegraded":"2024-09-13T08:41:31.474990+0000","last_fullsized":"2024-09-13T08:41:31.474990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:36.019466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474439+0000","last_change":"2024-09-13T08:41:30.168382+0000","last_active":"2024-09-13T08:41:31.474439+0000","last_peered":"2024-09-13T08:41:31.474439+0000","last_clean":"2024-09-13T08:41:31.474439+0000","last_became_active":"2024-09-13T08:41:30.167748+0000","last_became_peered":"2024-09-13T08:41:30.167748+0000","last_unstale":"2024-09-13T08:41:31.474439+0000","last_undegraded":"2024-09-13T08:41:31.474439+0000","last_fullsized":"2024-09-13T08:41:31.474439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:43.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475680+0000","last_change":"2024-09-13T08:41:29.148961+0000","last_active":"2024-09-13T08:41:31.475680+0000","last_peered":"2024-09-13T08:41:31.475680+0000","last_clean":"2024-09-13T08:41:31.475680+0000","last_became_active":"2024-09-13T08:41:29.146591+0000","last_became_peered":"2024-09-13T08:41:29.146591+0000","last_unstale":"2024-09-13T08:41:31.475680+0000","last_undegraded":"2024-09-13T08:41:31.475680+0000","last_fullsized":"2024-09-13T08:41:31.475680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:25.694758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474398+0000","last_change":"2024-09-13T08:41:30.175697+0000","last_active":"2024-09-13T08:41:31.474398+0000","last_peered":"2024-09-13T08:41:31.474398+0000","last_clean":"2024-09-13T08:41:31.474398+0000","last_became_active":"2024-09-13T08:41:30.174133+0000","last_became_peered":"2024-09-13T08:41:30.174133+0000","last_unstale":"2024-09-13T08:41:31.474398+0000","last_undegraded":"2024-09-13T08:41:31.474398+0000","last_fullsized":"2024-09-13T08:41:31.474398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:45.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474547+0000","last_change":"2024-09-13T08:41:29.154197+0000","last_active":"2024-09-13T08:41:31.474547+0000","last_peered":"2024-09-13T08:41:31.474547+0000","last_clean":"2024-09-13T08:41:31.474547+0000","last_became_active":"2024-09-13T08:41:29.152859+0000","last_became_peered":"2024-09-13T08:41:29.152859+0000","last_unstale":"2024-09-13T08:41:31.474547+0000","last_undegraded":"2024-09-13T08:41:31.474547+0000","last_fullsized":"2024-09-13T08:41:31.474547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:35.228437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474394+0000","last_change":"2024-09-13T08:41:30.158406+0000","last_active":"2024-09-13T08:41:31.474394+0000","last_peered":"2024-09-13T08:41:31.474394+0000","last_clean":"2024-09-13T08:41:31.474394+0000","last_became_active":"2024-09-13T08:41:30.157975+0000","last_became_peered":"2024-09-13T08:41:30.157975+0000","last_unstale":"2024-09-13T08:41:31.474394+0000","last_undegraded":"2024-09-13T08:41:31.474394+0000","last_fullsized":"2024-09-13T08:41:31.474394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:54.914413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150950+0000","last_change":"2024-09-13T08:41:29.154649+0000","last_active":"2024-09-13T08:41:31.150950+0000","last_peered":"2024-09-13T08:41:31.150950+0000","last_clean":"2024-09-13T08:41:31.150950+0000","last_became_active":"2024-09-13T08:41:29.154327+0000","last_became_peered":"2024-09-13T08:41:29.154327+0000","last_unstale":"2024-09-13T08:41:31.150950+0000","last_undegraded":"2024-09-13T08:41:31.150950+0000","last_fullsized":"2024-09-13T08:41:31.150950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:18.313976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152456+0000","last_change":"2024-09-13T08:41:30.173642+0000","last_active":"2024-09-13T08:41:31.152456+0000","last_peered":"2024-09-13T08:41:31.152456+0000","last_clean":"2024-09-13T08:41:31.152456+0000","last_became_active":"2024-09-13T08:41:30.173362+0000","last_became_peered":"2024-09-13T08:41:30.173362+0000","last_unstale":"2024-09-13T08:41:31.152456+0000","last_undegraded":"2024-09-13T08:41:31.152456+0000","last_fullsized":"2024-09-13T08:41:31.152456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.753257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150731+0000","last_change":"2024-09-13T08:41:29.150523+0000","last_active":"2024-09-13T08:41:31.150731+0000","last_peered":"2024-09-13T08:41:31.150731+0000","last_clean":"2024-09-13T08:41:31.150731+0000","last_became_active":"2024-09-13T08:41:29.149999+0000","last_became_peered":"2024-09-13T08:41:29.149999+0000","last_unstale":"2024-09-13T08:41:31.150731+0000","last_undegraded":"2024-09-13T08:41:31.150731+0000","last_fullsized":"2024-09-13T08:41:31.150731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:32.616347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475559+0000","last_change":"2024-09-13T08:41:30.172825+0000","last_active":"2024-09-13T08:41:31.475559+0000","last_peered":"2024-09-13T08:41:31.475559+0000","last_clean":"2024-09-13T08:41:31.475559+0000","last_became_active":"2024-09-13T08:41:30.172459+0000","last_became_peered":"2024-09-13T08:41:30.172459+0000","last_unstale":"2024-09-13T08:41:31.475559+0000","last_undegraded":"2024-09-13T08:41:31.475559+0000","last_fullsized":"2024-09-13T08:41:31.475559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:08.614713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150663+0000","last_change":"2024-09-13T08:41:29.156767+0000","last_active":"2024-09-13T08:41:31.150663+0000","last_peered":"2024-09-13T08:41:31.150663+0000","last_clean":"2024-09-13T08:41:31.150663+0000","last_became_active":"2024-09-13T08:41:29.156127+0000","last_became_peered":"2024-09-13T08:41:29.156127+0000","last_unstale":"2024-09-13T08:41:31.150663+0000","last_undegraded":"2024-09-13T08:41:31.150663+0000","last_fullsized":"2024-09-13T08:41:31.150663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:42:26.199094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474754+0000","last_change":"2024-09-13T08:41:30.176974+0000","last_active":"2024-09-13T08:41:31.474754+0000","last_peered":"2024-09-13T08:41:31.474754+0000","last_clean":"2024-09-13T08:41:31.474754+0000","last_became_active":"2024-09-13T08:41:30.176715+0000","last_became_peered":"2024-09-13T08:41:30.176715+0000","last_unstale":"2024-09-13T08:41:31.474754+0000","last_undegraded":"2024-09-13T08:41:31.474754+0000","last_fullsized":"2024-09-13T08:41:31.474754+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:10.652745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474730+0000","last_change":"2024-09-13T08:41:29.148076+0000","last_active":"2024-09-13T08:41:31.474730+0000","last_peered":"2024-09-13T08:41:31.474730+0000","last_clean":"2024-09-13T08:41:31.474730+0000","last_became_active":"2024-09-13T08:41:29.147012+0000","last_became_peered":"2024-09-13T08:41:29.147012+0000","last_unstale":"2024-09-13T08:41:31.474730+0000","last_undegraded":"2024-09-13T08:41:31.474730+0000","last_fullsized":"2024-09-13T08:41:31.474730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:21:18.467245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475576+0000","last_change":"2024-09-13T08:41:30.173815+0000","last_active":"2024-09-13T08:41:31.475576+0000","last_peered":"2024-09-13T08:41:31.475576+0000","last_clean":"2024-09-13T08:41:31.475576+0000","last_became_active":"2024-09-13T08:41:30.173512+0000","last_became_peered":"2024-09-13T08:41:30.173512+0000","last_unstale":"2024-09-13T08:41:31.475576+0000","last_undegraded":"2024-09-13T08:41:31.475576+0000","last_fullsized":"2024-09-13T08:41:31.475576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:17.533940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475003+0000","last_change":"2024-09-13T08:41:29.148851+0000","last_active":"2024-09-13T08:41:31.475003+0000","last_peered":"2024-09-13T08:41:31.475003+0000","last_clean":"2024-09-13T08:41:31.475003+0000","last_became_active":"2024-09-13T08:41:29.148567+0000","last_became_peered":"2024-09-13T08:41:29.148567+0000","last_unstale":"2024-09-13T08:41:31.475003+0000","last_undegraded":"2024-09-13T08:41:31.475003+0000","last_fullsized":"2024-09-13T08:41:31.475003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.901791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151894+0000","last_change":"2024-09-13T08:41:30.164444+0000","last_active":"2024-09-13T08:41:31.151894+0000","last_peered":"2024-09-13T08:41:31.151894+0000","last_clean":"2024-09-13T08:41:31.151894+0000","last_became_active":"2024-09-13T08:41:30.164177+0000","last_became_peered":"2024-09-13T08:41:30.164177+0000","last_unstale":"2024-09-13T08:41:31.151894+0000","last_undegraded":"2024-09-13T08:41:31.151894+0000","last_fullsized":"2024-09-13T08:41:31.151894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:10.013440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474879+0000","last_change":"2024-09-13T08:41:29.147550+0000","last_active":"2024-09-13T08:41:31.474879+0000","last_peered":"2024-09-13T08:41:31.474879+0000","last_clean":"2024-09-13T08:41:31.474879+0000","last_became_active":"2024-09-13T08:41:29.147183+0000","last_became_peered":"2024-09-13T08:41:29.147183+0000","last_unstale":"2024-09-13T08:41:31.474879+0000","last_undegraded":"2024-09-13T08:41:31.474879+0000","last_fullsized":"2024-09-13T08:41:31.474879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:09.961905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.150069+0000","last_change":"2024-09-13T08:41:30.188192+0000","last_active":"2024-09-13T08:41:31.150069+0000","last_peered":"2024-09-13T08:41:31.150069+0000","last_clean":"2024-09-13T08:41:31.150069+0000","last_became_active":"2024-09-13T08:41:30.187402+0000","last_became_peered":"2024-09-13T08:41:30.187402+0000","last_unstale":"2024-09-13T08:41:31.150069+0000","last_undegraded":"2024-09-13T08:41:31.150069+0000","last_fullsized":"2024-09-13T08:41:31.150069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:23.226310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.152027+0000","last_change":"2024-09-13T08:41:29.144692+0000","last_active":"2024-09-13T08:41:31.152027+0000","last_peered":"2024-09-13T08:41:31.152027+0000","last_clean":"2024-09-13T08:41:31.152027+0000","last_became_active":"2024-09-13T08:41:29.144396+0000","last_became_peered":"2024-09-13T08:41:29.144396+0000","last_unstale":"2024-09-13T08:41:31.152027+0000","last_undegraded":"2024-09-13T08:41:31.152027+0000","last_fullsized":"2024-09-13T08:41:31.152027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:05.544795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474559+0000","last_change":"2024-09-13T08:41:30.166665+0000","last_active":"2024-09-13T08:41:31.474559+0000","last_peered":"2024-09-13T08:41:31.474559+0000","last_clean":"2024-09-13T08:41:31.474559+0000","last_became_active":"2024-09-13T08:41:30.166364+0000","last_became_peered":"2024-09-13T08:41:30.166364+0000","last_unstale":"2024-09-13T08:41:31.474559+0000","last_undegraded":"2024-09-13T08:41:31.474559+0000","last_fullsized":"2024-09-13T08:41:31.474559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:37.021694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474651+0000","last_change":"2024-09-13T08:41:29.156770+0000","last_active":"2024-09-13T08:41:31.474651+0000","last_peered":"2024-09-13T08:41:31.474651+0000","last_clean":"2024-09-13T08:41:31.474651+0000","last_became_active":"2024-09-13T08:41:29.156466+0000","last_became_peered":"2024-09-13T08:41:29.156466+0000","last_unstale":"2024-09-13T08:41:31.474651+0000","last_undegraded":"2024-09-13T08:41:31.474651+0000","last_fullsized":"2024-09-13T08:41:31.474651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:04.189295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474295+0000","last_change":"2024-09-13T08:41:30.170922+0000","last_active":"2024-09-13T08:41:31.474295+0000","last_peered":"2024-09-13T08:41:31.474295+0000","last_clean":"2024-09-13T08:41:31.474295+0000","last_became_active":"2024-09-13T08:41:30.170663+0000","last_became_peered":"2024-09-13T08:41:30.170663+0000","last_unstale":"2024-09-13T08:41:31.474295+0000","last_undegraded":"2024-09-13T08:41:31.474295+0000","last_fullsized":"2024-09-13T08:41:31.474295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:07:26.254945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474197+0000","last_change":"2024-09-13T08:41:29.149340+0000","last_active":"2024-09-13T08:41:31.474197+0000","last_peered":"2024-09-13T08:41:31.474197+0000","last_clean":"2024-09-13T08:41:31.474197+0000","last_became_active":"2024-09-13T08:41:29.149060+0000","last_became_peered":"2024-09-13T08:41:29.149060+0000","last_unstale":"2024-09-13T08:41:31.474197+0000","last_undegraded":"2024-09-13T08:41:31.474197+0000","last_fullsized":"2024-09-13T08:41:31.474197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:58:44.441597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.151653+0000","last_change":"2024-09-13T08:41:30.177150+0000","last_active":"2024-09-13T08:41:31.151653+0000","last_peered":"2024-09-13T08:41:31.151653+0000","last_clean":"2024-09-13T08:41:31.151653+0000","last_became_active":"2024-09-13T08:41:30.176850+0000","last_became_peered":"2024-09-13T08:41:30.176850+0000","last_unstale":"2024-09-13T08:41:31.151653+0000","last_undegraded":"2024-09-13T08:41:31.151653+0000","last_fullsized":"2024-09-13T08:41:31.151653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:29.119254+0000","last_clean_scrub_stamp":"2024-09-13T08:41:29.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:44.657225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.475011+0000","last_change":"2024-09-13T08:41:29.144964+0000","last_active":"2024-09-13T08:41:31.475011+0000","last_peered":"2024-09-13T08:41:31.475011+0000","last_clean":"2024-09-13T08:41:31.475011+0000","last_became_active":"2024-09-13T08:41:29.144673+0000","last_became_peered":"2024-09-13T08:41:29.144673+0000","last_unstale":"2024-09-13T08:41:31.475011+0000","last_undegraded":"2024-09-13T08:41:31.475011+0000","last_fullsized":"2024-09-13T08:41:31.475011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:41:28.117026+0000","last_clean_scrub_stamp":"2024-09-13T08:41:28.117026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:02:12.107299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":207,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T08:41:31.474058+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:41:31.474058+0000","last_peered":"2024-09-13T08:41:31.474058+0000","last_clean":"2024-09-13T08:41:31.474058+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:41:31.474058+0000","last_undegraded":"2024-09-13T08:41:31.474058+0000","last_fullsized":"2024-09-13T08:41:31.474058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":388196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509499,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":388230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509499,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":977042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509499,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":386710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509499,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":387166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1511424,"data_stored":976952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":921600,"data_stored":386654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542202,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":929792,"data_stored":387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:41:39.581 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:41:39.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:41:39.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 -- 172.21.15.31:0/4276470370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 msgr2=0x7f4b1406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 --2- 172.21.15.31:0/4276470370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b1406fa60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009830 tx=0x7f4b0802a480 comp rx=0 tx=0).stop 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 -- 172.21.15.31:0/4276470370 shutdown_connections 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 --2- 172.21.15.31:0/4276470370 >> v2:172.21.15.45:3300/0 conn(0x7f4b140ccaf0 0x7f4b1406e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 --2- 172.21.15.31:0/4276470370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b1406fa60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 -- 172.21.15.31:0/4276470370 >> 172.21.15.31:0/4276470370 conn(0x7f4b140b1520 msgr2=0x7f4b140b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 -- 172.21.15.31:0/4276470370 shutdown_connections 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.785+0000 7f4b1b000640 1 -- 172.21.15.31:0/4276470370 wait complete. 2024-09-13T08:41:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.787+0000 7f4b1b000640 1 Processor -- start 2024-09-13T08:41:39.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.787+0000 7f4b1b000640 1 -- start start 2024-09-13T08:41:39.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b140b5300 0x7f4b140c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b140a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1406ed20 con 0x7f4b140ccaf0 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1406eba0 con 0x7f4b140b5300 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1b000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1406eea0 con 0x7f4b1415bb70 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b19200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b19200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59728/0 (socket says 172.21.15.31:59728) 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b19200640 1 -- 172.21.15.31:0/1008682560 learned_addr learned my addr 172.21.15.31:0/1008682560 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b19c00640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b140b5300 0x7f4b140c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.788+0000 7f4b1a600640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b140a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 -- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 msgr2=0x7f4b140a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b140a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 -- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b140b5300 msgr2=0x7f4b140c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:39.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b140b5300 0x7f4b140c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:39.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 -- 172.21.15.31:0/1008682560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b140c6aa0 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b19200640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000c420 tx=0x7f4b0000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00017070 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b000053c0 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b140c6d40 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00003850 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.789+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b140c7300 con 0x7f4b140ccaf0 2024-09-13T08:41:39.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.790+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1406f350 con 0x7f4b140ccaf0 2024-09-13T08:41:39.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.792+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f4b0001a070 con 0x7f4b140ccaf0 2024-09-13T08:41:39.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.793+0000 7f4b11600640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 0x7f4af407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:39.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.793+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f4b0009a2e0 con 0x7f4b140ccaf0 2024-09-13T08:41:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.793+0000 7f4b19c00640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 0x7f4af407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.794+0000 7f4b19c00640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 0x7f4af407b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4b08034000 tx=0x7f4b08004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:39.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:39.797+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4b00061d20 con 0x7f4b140ccaf0 2024-09-13T08:41:40.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.065+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4b14069700 con 0x7f4b140ccaf0 2024-09-13T08:41:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.068+0000 7f4b11600640 1 -- 172.21.15.31:0/1008682560 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f4b00066750 con 0x7f4b140ccaf0 2024-09-13T08:41:40.069 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:41:40.069 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:41:40.069 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:41:40.069 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:41:40.070 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:41:40.070 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 msgr2=0x7f4af407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 0x7f4af407b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4b08034000 tx=0x7f4b08004730 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 msgr2=0x7f4b140a9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000c420 tx=0x7f4b0000c8e0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 shutdown_connections 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f4af4078be0 0x7f4af407b0a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b1415bb70 0x7f4b140a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b140ccaf0 0x7f4b140a9480 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 --2- 172.21.15.31:0/1008682560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b140b5300 0x7f4b140c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 >> 172.21.15.31:0/1008682560 conn(0x7f4b140b1520 msgr2=0x7f4b140cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 shutdown_connections 2024-09-13T08:41:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.071+0000 7f4b1b000640 1 -- 172.21.15.31:0/1008682560 wait complete. 2024-09-13T08:41:40.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 -- 172.21.15.31:0/594447898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 msgr2=0x7ff1880cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 --2- 172.21.15.31:0/594447898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff1880cf850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff17c005fd0 tx=0x7ff17c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 -- 172.21.15.31:0/594447898 shutdown_connections 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 --2- 172.21.15.31:0/594447898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815fe20 0x7ff1880cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 --2- 172.21.15.31:0/594447898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff1880cf850 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 -- 172.21.15.31:0/594447898 >> 172.21.15.31:0/594447898 conn(0x7ff1880cc910 msgr2=0x7ff1880cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 -- 172.21.15.31:0/594447898 shutdown_connections 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.250+0000 7ff18f600640 1 -- 172.21.15.31:0/594447898 wait complete. 2024-09-13T08:41:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.252+0000 7ff18f600640 1 Processor -- start 2024-09-13T08:41:40.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.252+0000 7ff18f600640 1 -- start start 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff18815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815f9e0 0x7ff1881f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880cec50 con 0x7ff18815fe20 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880cead0 con 0x7ff18815bd70 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18f600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff1880cb850 con 0x7ff18815f9e0 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59744/0 (socket says 172.21.15.31:59744) 2024-09-13T08:41:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18d600640 1 -- 172.21.15.31:0/4215120240 learned_addr learned my addr 172.21.15.31:0/4215120240 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff18cc00640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff18815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.253+0000 7ff187e00640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815f9e0 0x7ff1881f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 -- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815f9e0 msgr2=0x7ff1881f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815f9e0 0x7ff1881f80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 -- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 msgr2=0x7ff18815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff18815e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 -- 172.21.15.31:0/4215120240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1880ae4a0 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18d600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff18000ec20 tx=0x7ff18000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180003b70 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff180003e00 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1880ae780 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180004280 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.254+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18815eb20 con 0x7ff18815fe20 2024-09-13T08:41:40.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.255+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18809ce20 con 0x7ff18815fe20 2024-09-13T08:41:40.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.256+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7ff180004420 con 0x7ff18815fe20 2024-09-13T08:41:40.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.257+0000 7ff184c00640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 0x7ff16807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:40.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.257+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7ff18009be50 con 0x7ff18815fe20 2024-09-13T08:41:40.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.258+0000 7ff18cc00640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 0x7ff16807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:40.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.259+0000 7ff18cc00640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 0x7ff16807b070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff17c033000 tx=0x7ff17c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:40.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.261+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff180063810 con 0x7ff18815fe20 2024-09-13T08:41:40.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.494+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1880ca660 con 0x7ff18815fe20 2024-09-13T08:41:40.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.495+0000 7ff184c00640 1 -- 172.21.15.31:0/4215120240 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5222 (secure 0 0 0) 0x7ff180068240 con 0x7ff18815fe20 2024-09-13T08:41:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:40.496 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":47,"btime":"2024-09-13T08:41:33:169589+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:40.496 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 47 2024-09-13T08:41:40.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 msgr2=0x7ff16807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 0x7ff16807b070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff17c033000 tx=0x7ff17c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 msgr2=0x7ff1881f85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff18000ec20 tx=0x7ff18000c6a0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 shutdown_connections 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff18815fe20 0x7ff1881f85e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ff168078bb0 0x7ff16807b070 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff18815f9e0 0x7ff1881f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 --2- 172.21.15.31:0/4215120240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18815bd70 0x7ff18815e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 >> 172.21.15.31:0/4215120240 conn(0x7ff1880cc910 msgr2=0x7ff1880cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 shutdown_connections 2024-09-13T08:41:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:40.497+0000 7ff18f600640 1 -- 172.21.15.31:0/4215120240 wait complete. 2024-09-13T08:41:40.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:41:29.577486+0000', 'modified': '2024-09-13T08:41:32.607487+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6162}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6162': {'gid': 6162, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6835/1098295543', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6833', 'nonce': 1098295543}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 1098295543}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6162, 'qdb_cluster': [6162]} 2024-09-13T08:41:40.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:41:40.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 -- 172.21.15.31:0/1596558132 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec1419d0 msgr2=0x7f3dec141db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 --2- 172.21.15.31:0/1596558132 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec1419d0 0x7f3dec141db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc005fd0 tx=0x7f3ddc02a540 comp rx=0 tx=0).stop 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 -- 172.21.15.31:0/1596558132 shutdown_connections 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 --2- 172.21.15.31:0/1596558132 >> v2:172.21.15.45:3300/0 conn(0x7f3dec0d05c0 0x7f3dec06cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 --2- 172.21.15.31:0/1596558132 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec1419d0 0x7f3dec141db0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.081+0000 7f3df0c00640 1 -- 172.21.15.31:0/1596558132 >> 172.21.15.31:0/1596558132 conn(0x7f3dec0d0f70 msgr2=0x7f3dec0d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.082+0000 7f3df0c00640 1 -- 172.21.15.31:0/1596558132 shutdown_connections 2024-09-13T08:41:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.082+0000 7f3df0c00640 1 -- 172.21.15.31:0/1596558132 wait complete. 2024-09-13T08:41:41.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.083+0000 7f3df0c00640 1 Processor -- start 2024-09-13T08:41:41.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.083+0000 7f3df0c00640 1 -- start start 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 0x7f3dec1f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dec1419d0 0x7f3dec0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec0d0ba0 con 0x7f3dec0d05c0 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec0d0a20 con 0x7f3dec1419d0 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3df0c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec071610 con 0x7f3dec06d040 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.084+0000 7f3de9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59754/0 (socket says 172.21.15.31:59754) 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 -- 172.21.15.31:0/4150038984 learned_addr learned my addr 172.21.15.31:0/4150038984 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9e00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 0x7f3dec1f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3dea800640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dec1419d0 0x7f3dec0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 -- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 msgr2=0x7f3dec1f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 0x7f3dec1f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 -- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dec1419d0 msgr2=0x7f3dec0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dec1419d0 0x7f3dec0c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 -- 172.21.15.31:0/4150038984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec1f5670 con 0x7f3dec0d05c0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9e00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 0x7f3dec1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de9400640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9f0 tx=0x7f3dd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de2000640 1 -- 172.21.15.31:0/4150038984 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800cd80 con 0x7f3dec0d05c0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3de2000640 1 -- 172.21.15.31:0/4150038984 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd8010430 con 0x7f3dec0d05c0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.085+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dec15bae0 con 0x7f3dec0d05c0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.086+0000 7f3de2000640 1 -- 172.21.15.31:0/4150038984 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8010840 con 0x7f3dec0d05c0 2024-09-13T08:41:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.086+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dec15c0a0 con 0x7f3dec0d05c0 2024-09-13T08:41:41.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.088+0000 7f3de2000640 1 -- 172.21.15.31:0/4150038984 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f3dd80109e0 con 0x7f3dec0d05c0 2024-09-13T08:41:41.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.089+0000 7f3de2000640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 0x7f3dcc07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.089+0000 7f3df3a88740 1 Processor -- start 2024-09-13T08:41:41.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.089+0000 7f3de2000640 1 -- 172.21.15.31:0/4150038984 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f3dd809bab0 con 0x7f3dec0d05c0 2024-09-13T08:41:41.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.089+0000 7f3df3a88740 1 -- start start 2024-09-13T08:41:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3de9e00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 0x7f3dcc07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 0x562274bbd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562274afc430 con 0x562274b79fa0 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562274aec5d0 con 0x562274bbe8b0 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.090+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x562274adba00 con 0x562274b79a20 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9e00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 0x7f3dcc07b070 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc033000 tx=0x7f3ddc004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3dea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 0x562274bbd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3dea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59764/0 (socket says 172.21.15.31:59764) 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3dea800640 1 -- 172.21.15.31:0/3725686557 learned_addr learned my addr 172.21.15.31:0/3725686557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9e00640 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 -- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 msgr2=0x562274bbec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 -- 172.21.15.31:0/3725686557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 msgr2=0x562274bbd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 -- 172.21.15.31:0/3725686557 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562274aa4520 con 0x562274b79a20 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.091+0000 7f3de9400640 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 0x562274bbd390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8099aa0 tx=0x7f3dd8062e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c329869eafaed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3dd6600640 1 -- 172.21.15.31:0/3725686557 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80680f0 con 0x562274b79a20 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3dd6600640 1 -- 172.21.15.31:0/3725686557 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd806cfa0 con 0x562274b79a20 2024-09-13T08:41:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3dd6600640 1 -- 172.21.15.31:0/3725686557 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8062ad0 con 0x562274b79a20 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 -- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 msgr2=0x562274bbd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 0x562274bbd390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8099aa0 tx=0x7f3dd8062e70 comp rx=0 tx=0).stop 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 -- 172.21.15.31:0/3725686557 shutdown_connections 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b79a20 0x562274bbd390 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 --2- 172.21.15.31:0/3725686557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.092+0000 7f3df3a88740 1 -- 172.21.15.31:0/3725686557 >> 172.21.15.31:0/3725686557 conn(0x562274b74ff0 msgr2=0x562274b78a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.093+0000 7f3df3a88740 1 -- 172.21.15.31:0/3725686557 shutdown_connections 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.093+0000 7f3df3a88740 1 -- 172.21.15.31:0/3725686557 wait complete. 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.093+0000 7f3df3a88740 1 -- start start 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.093+0000 7f3df3a88740 1 Processor -- start 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274bbb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562274a82330 con 0x562274b79fa0 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562274adf810 con 0x562274bbe8b0 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x56227484c620 con 0x562274b910e0 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.094+0000 7f3df3a88740 10 client.0 _ll_register_callbacks cb 0x562274a87420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59774/0 (socket says 172.21.15.31:59774) 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 -- 172.21.15.31:0/2694762240 learned_addr learned my addr 172.21.15.31:0/2694762240 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9400640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3dea800640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274bbb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 -- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 msgr2=0x562274bbb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274bbb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 -- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 msgr2=0x562274bbb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 -- 172.21.15.31:0/2694762240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56227483d850 con 0x562274b79fa0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0096c0 tx=0x7f3ddc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f0da1ce53d9f7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.095+0000 7f3dcaa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc04d070 con 0x562274b79fa0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f3ddc03b870 con 0x562274b79fa0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd806cfa0 con 0x562274b79fa0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3ddc058050 con 0x562274b79fa0 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 10 client.6300 fetch_fsmap learned FSMap version 47 2024-09-13T08:41:41.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) -- 0x5622748b46b0 con 0x562274b79fa0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd806cfa0 con 0x562274b79fa0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 47) ==== 3502+0+0 (secure 0 0 0) 0x7f3ddc05b020 con 0x562274b79fa0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 10 client.6300 fetch_fsmap finished waiting for FSMap version 47 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 10 client.6300 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 20 client.6300 populate_metadata read hostname 'smithi031' 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274b8de10 0x562274ba1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 4 client.6300 mds_command: new command op to 6162 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.096+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 --> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562274aee520 con 0x562274b8de10 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.097+0000 7f3de9400640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274b8de10 0x562274ba1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.097+0000 7f3de9400640 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274b8de10 0x562274ba1de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83c93c16122186 server_cookie=4ec562754671eddd in_seq=0 out_seq=0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.097+0000 7f3dcaa00640 10 client.6300 ms_handle_connect on v2:172.21.15.31:6833/1098295543 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.098+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2694762240 <== mds.0 v2:172.21.15.31:6833/1098295543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562274aee520 con 0x562274b8de10 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.098+0000 7f3dcaa00640 10 client.6300 handle_command_reply: tid=0 2024-09-13T08:41:41.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.098+0000 7f3df3a88740 1 client.6300 shutdown 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 msgr2=0x562274bcd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0096c0 tx=0x7f3ddc009260 comp rx=0 tx=0).stop 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 shutdown_connections 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274b8de10 0x562274ba1de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274bbb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274bbb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2694762240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bcd820 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 >> 172.21.15.31:0/2694762240 conn(0x562274b79a20 msgr2=0x562274bb73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3dcaa00640 0 client.6300 ms_handle_reset on v2:172.21.15.31:6833/1098295543 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 shutdown_connections 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 1 -- 172.21.15.31:0/2694762240 wait complete. 2024-09-13T08:41:41.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.099+0000 7f3df3a88740 20 client.6300 trim_cache size 0 max 16384 2024-09-13T08:41:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.118+0000 7f3df3a88740 1 Processor -- start 2024-09-13T08:41:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.118+0000 7f3df3a88740 1 -- start start 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 0x562274b9eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec2001c0 con 0x562274b910e0 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec0b5b30 con 0x562274b79fa0 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec0b6180 con 0x562274bbe8b0 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3de9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3de9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59782/0 (socket says 172.21.15.31:59782) 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.119+0000 7f3de9400640 1 -- 172.21.15.31:0/2754257112 learned_addr learned my addr 172.21.15.31:0/2754257112 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 0x562274b9eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 -- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274bbe8b0 msgr2=0x562274b73b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 -- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 msgr2=0x562274b9eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 0x562274b9eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 -- 172.21.15.31:0/2754257112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562274bbb770 con 0x562274b910e0 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 0x562274b9eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3de9400640 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c6a0 tx=0x7f3dd801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75da97546ca1bd0f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3dd5c00640 1 -- 172.21.15.31:0/2754257112 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8082070 con 0x562274b910e0 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3dd5c00640 1 -- 172.21.15.31:0/2754257112 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd8062870 con 0x562274b910e0 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.120+0000 7f3dd5c00640 1 -- 172.21.15.31:0/2754257112 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8085b20 con 0x562274b910e0 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- 172.21.15.31:0/2754257112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 msgr2=0x562274bb73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c6a0 tx=0x7f3dd801a040 comp rx=0 tx=0).stop 2024-09-13T08:41:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- 172.21.15.31:0/2754257112 shutdown_connections 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b910e0 0x562274bb73a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2754257112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274b79fa0 0x562274b9eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- 172.21.15.31:0/2754257112 >> 172.21.15.31:0/2754257112 conn(0x562274b79a20 msgr2=0x562274bb67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- 172.21.15.31:0/2754257112 shutdown_connections 2024-09-13T08:41:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- 172.21.15.31:0/2754257112 wait complete. 2024-09-13T08:41:41.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.121+0000 7f3df3a88740 1 -- start start 2024-09-13T08:41:41.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.122+0000 7f3df3a88740 1 Processor -- start 2024-09-13T08:41:41.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274c24d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562274ba2690 con 0x562274b79fa0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562274b8dc90 con 0x562274bbe8b0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x562274ba2ce0 con 0x562274b910e0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3df3a88740 10 client.0 _ll_register_callbacks cb 0x7f3ddc004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59794/0 (socket says 172.21.15.31:59794) 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 -- 172.21.15.31:0/2167908522 learned_addr learned my addr 172.21.15.31:0/2167908522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9400640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274c24d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3dea800640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 -- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 msgr2=0x562274c24d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274c24d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 -- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 msgr2=0x562274b73b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3de9e00640 1 -- 172.21.15.31:0/2167908522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562274b9fb30 con 0x562274b79fa0 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.123+0000 7f3dea800640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3de9e00640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc03b880 tx=0x7f3ddc03b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd60f7a25cc28beb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc061060 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f3ddc052850 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd8062870 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3ddc067050 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 10 client.6306 fetch_fsmap learned FSMap version 47 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) -- 0x562274c25270 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd8062870 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 47) ==== 3502+0+0 (secure 0 0 0) 0x7f3ddc06a020 con 0x562274b79fa0 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 10 client.6306 fetch_fsmap finished waiting for FSMap version 47 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 10 client.6306 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.124+0000 7f3df3a88740 20 client.6306 populate_metadata read hostname 'smithi031' 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274bcc390 0x562274b71e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3df3a88740 4 client.6306 mds_command: new command op to 6162 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 --> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] -- command(tid 0: {"prefix": "status"}) -- 0x562274ae9fd0 con 0x562274bcc390 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3de9400640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274bcc390 0x562274b71e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3dcaa00640 10 client.6306 ms_handle_connect on v2:172.21.15.31:6833/1098295543 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3de9400640 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274bcc390 0x562274b71e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da0f2378c791ca5 server_cookie=d26a53b03b92ead8 in_seq=0 out_seq=0 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3dcaa00640 1 -- 172.21.15.31:0/2167908522 <== mds.0 v2:172.21.15.31:6833/1098295543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x562274ae9fd0 con 0x562274bcc390 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3dcaa00640 10 client.6306 handle_command_reply: tid=0 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.125+0000 7f3df3a88740 1 client.6306 shutdown 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 msgr2=0x562274bbb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc03b880 tx=0x7f3ddc03b8b0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 shutdown_connections 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:6833/1098295543,v1:172.21.15.31:6835/1098295543] conn(0x562274bcc390 0x562274b71e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562274bbe8b0 0x562274b73b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x562274b910e0 0x562274c24d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3dcaa00640 0 client.6306 ms_handle_reset on v2:172.21.15.31:6833/1098295543 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 --2- 172.21.15.31:0/2167908522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562274b79fa0 0x562274bbb070 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 >> 172.21.15.31:0/2167908522 conn(0x562274b79a20 msgr2=0x562274bb67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 shutdown_connections 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 1 -- 172.21.15.31:0/2167908522 wait complete. 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.126+0000 7f3df3a88740 20 client.6306 trim_cache size 0 max 16384 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:41:41.134 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 6162, 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 10.979494451000001, 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 47, 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 76, 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 76, 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 16.237985848000001 2024-09-13T08:41:41.135 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 msgr2=0x7f3dcc07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 0x7f3dcc07b070 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc033000 tx=0x7f3ddc004730 comp rx=0 tx=0).stop 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 msgr2=0x7f3dec0c6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9f0 tx=0x7f3dd800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 shutdown_connections 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dec1419d0 0x7f3dec0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dec0d05c0 0x7f3dec0c6790 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3dcc078bb0 0x7f3dcc07b070 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 --2- 172.21.15.31:0/4150038984 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3dec06d040 0x7f3dec1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 >> 172.21.15.31:0/4150038984 conn(0x7f3dec0d0f70 msgr2=0x7f3dec0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 shutdown_connections 2024-09-13T08:41:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.127+0000 7f3df0c00640 1 -- 172.21.15.31:0/4150038984 wait complete. 2024-09-13T08:41:41.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 -- 172.21.15.31:0/2417239349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815fe20 msgr2=0x7fc2380cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 --2- 172.21.15.31:0/2417239349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815fe20 0x7fc2380cbc50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc230005fd0 tx=0x7fc23002aaa0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 -- 172.21.15.31:0/2417239349 shutdown_connections 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 --2- 172.21.15.31:0/2417239349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815fe20 0x7fc2380cbc50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 --2- 172.21.15.31:0/2417239349 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc23815bd70 0x7fc2380cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 -- 172.21.15.31:0/2417239349 >> 172.21.15.31:0/2417239349 conn(0x7fc2380cc910 msgr2=0x7fc2380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.303+0000 7fc23ca00640 1 -- 172.21.15.31:0/2417239349 shutdown_connections 2024-09-13T08:41:41.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.304+0000 7fc23ca00640 1 -- 172.21.15.31:0/2417239349 wait complete. 2024-09-13T08:41:41.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.305+0000 7fc23ca00640 1 Processor -- start 2024-09-13T08:41:41.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.305+0000 7fc23ca00640 1 -- start start 2024-09-13T08:41:41.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc23815f9e0 0x7fc238151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc23815fe20 0x7fc238153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2380cec50 con 0x7fc23815bd70 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2380cead0 con 0x7fc23815f9e0 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc23ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc2380cb850 con 0x7fc23815fe20 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc235e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc235e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59826/0 (socket says 172.21.15.31:59826) 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.306+0000 7fc235e00640 1 -- 172.21.15.31:0/1315219328 learned_addr learned my addr 172.21.15.31:0/1315219328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 -- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc23815fe20 msgr2=0x7fc238153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc23815fe20 0x7fc238153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 -- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc23815f9e0 msgr2=0x7fc238151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc23815f9e0 0x7fc238151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 -- 172.21.15.31:0/1315219328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2380c51e0 con 0x7fc23815bd70 2024-09-13T08:41:41.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc235e00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc22800d950 tx=0x7fc22800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228017070 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc228009cc0 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2280106c0 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.307+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2380c54c0 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.308+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2381524f0 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.308+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2380cf850 con 0x7fc23815bd70 2024-09-13T08:41:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.310+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fc228010860 con 0x7fc23815bd70 2024-09-13T08:41:41.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.311+0000 7fc22ea00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 0x7fc21407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:41.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.311+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fc22809b320 con 0x7fc23815bd70 2024-09-13T08:41:41.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.311+0000 7fc22d400640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 0x7fc21407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:41.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.312+0000 7fc22d400640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 0x7fc21407b0a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc220000f60 tx=0x7fc2200079b0 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:41.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.315+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc228063400 con 0x7fc23815bd70 2024-09-13T08:41:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.547+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc2380b5df0 con 0x7fc23815bd70 2024-09-13T08:41:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.547+0000 7fc22ea00640 1 -- 172.21.15.31:0/1315219328 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) ==== 53+0+73 (secure 0 0 0) 0x7fc228062d00 con 0x7fc23815bd70 2024-09-13T08:41:41.549 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 msgr2=0x7fc21407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 0x7fc21407b0a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc220000f60 tx=0x7fc2200079b0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 msgr2=0x7fc238154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc22800d950 tx=0x7fc22800de10 comp rx=0 tx=0).stop 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 shutdown_connections 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fc214078be0 0x7fc21407b0a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc23815fe20 0x7fc238153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc23815f9e0 0x7fc238151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 --2- 172.21.15.31:0/1315219328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23815bd70 0x7fc238154610 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 >> 172.21.15.31:0/1315219328 conn(0x7fc2380cc910 msgr2=0x7fc2380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 shutdown_connections 2024-09-13T08:41:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41.550+0000 7fc23ca00640 1 -- 172.21.15.31:0/1315219328 wait complete. 2024-09-13T08:41:41.566 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:41:41.567 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:41:41.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:41.567 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42430sec preferred_lft 42430sec 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:41:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:41.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:41.607 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:41:41.697 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:41:41.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:41.700 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:41:41.736 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:41:41.736 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:41:41.736 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:41:41.737 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:41:41.737 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:41:41.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:41:41.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:41:41.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:41.935 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:41:41.970 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:41:41.970 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:41:41.971 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:41:41.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:41.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:41:42.068 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:41:42.068 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:42.103 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:41:42.103 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:41:42.103 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:42.178 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:41:42.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:42.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:42.308 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:42.308 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:41:42.336 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:42.336 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:41:42.336 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:41:42.336 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:41:42.336 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:41:42.338 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:41:42.339 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:41:42.339 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:41:42.339 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:41:42.339 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:41:42.342 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:41:42.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:42.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:42.398 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:41:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:41:42.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:41:42.495 INFO:teuthology.orchestra.run.smithi031.stdout:client6318 2024-09-13T08:41:42.495 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:42.496 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:42.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:41:42.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 -- 172.21.15.31:0/1676761090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07740b1910 msgr2=0x7f07740b1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 --2- 172.21.15.31:0/1676761090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07740b1910 0x7f07740b1d60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0774070b80 tx=0x7f076c02aac0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 -- 172.21.15.31:0/1676761090 shutdown_connections 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 --2- 172.21.15.31:0/1676761090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07740b1910 0x7f07740b1d60 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 --2- 172.21.15.31:0/1676761090 >> v2:172.21.15.45:3300/0 conn(0x7f07740ca5c0 0x7f07740cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 -- 172.21.15.31:0/1676761090 >> 172.21.15.31:0/1676761090 conn(0x7f07740cc910 msgr2=0x7f07740cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 -- 172.21.15.31:0/1676761090 shutdown_connections 2024-09-13T08:41:42.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.730+0000 7f077be00640 1 -- 172.21.15.31:0/1676761090 wait complete. 2024-09-13T08:41:42.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.732+0000 7f077be00640 1 Processor -- start 2024-09-13T08:41:42.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.732+0000 7f077be00640 1 -- start start 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 0x7f07740b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07740ca5c0 0x7f077415b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0774073b10 con 0x7f077415dfc0 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0774073990 con 0x7f07740b1910 2024-09-13T08:41:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f077be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0774073c90 con 0x7f07740ca5c0 2024-09-13T08:41:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0779e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0779400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 0x7f07740b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0778a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07740ca5c0 0x7f077415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0779400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 0x7f07740b58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:49014/0 (socket says 172.21.15.31:49014) 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0779e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59846/0 (socket says 172.21.15.31:59846) 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.733+0000 7f0779400640 1 -- 172.21.15.31:0/1832012315 learned_addr learned my addr 172.21.15.31:0/1832012315 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 -- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07740ca5c0 msgr2=0x7f077415b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07740ca5c0 0x7f077415b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 -- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 msgr2=0x7f07740b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 0x7f07740b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 -- 172.21.15.31:0/1832012315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07740c8ad0 con 0x7f077415dfc0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0779e00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f076c032000 tx=0x7f076c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c044070 con 0x7f077415dfc0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f076c004250 con 0x7f077415dfc0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c004660 con 0x7f077415dfc0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.734+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07740c8db0 con 0x7f077415dfc0 2024-09-13T08:41:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.735+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07741f8910 con 0x7f077415dfc0 2024-09-13T08:41:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.736+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07740b1d60 con 0x7f077415dfc0 2024-09-13T08:41:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.737+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f076c02c860 con 0x7f077415dfc0 2024-09-13T08:41:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.738+0000 7f0771600640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 0x7f075407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:42.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.738+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f076c0bb390 con 0x7f077415dfc0 2024-09-13T08:41:42.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.738+0000 7f0779400640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 0x7f075407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:42.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.739+0000 7f0779400640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 0x7f075407b070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f07740b3fc0 tx=0x7f0768009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:42.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.742+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f076c032030 con 0x7f077415dfc0 2024-09-13T08:41:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.976+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f077406ac50 con 0x7f077415dfc0 2024-09-13T08:41:42.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.977+0000 7f0771600640 1 -- 172.21.15.31:0/1832012315 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) ==== 53+0+73 (secure 0 0 0) 0x7f076c082de0 con 0x7f077415dfc0 2024-09-13T08:41:42.978 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:41:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 msgr2=0x7f075407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 0x7f075407b070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f07740b3fc0 tx=0x7f0768009040 comp rx=0 tx=0).stop 2024-09-13T08:41:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 msgr2=0x7f07740c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f076c032000 tx=0x7f076c003f10 comp rx=0 tx=0).stop 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 shutdown_connections 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f077415dfc0 0x7f07740c8280 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f0754078bb0 0x7f075407b070 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07740ca5c0 0x7f077415b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 --2- 172.21.15.31:0/1832012315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07740b1910 0x7f07740b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 >> 172.21.15.31:0/1832012315 conn(0x7f07740cc910 msgr2=0x7f07740b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 shutdown_connections 2024-09-13T08:41:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:42.979+0000 7f077be00640 1 -- 172.21.15.31:0/1832012315 wait complete. 2024-09-13T08:41:42.996 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:41:42.997 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:41:42.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:42.997 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:41:43.038 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:41:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:41:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:41:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42433sec preferred_lft 42433sec 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:41:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:41:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.041 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:41:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:41:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:41:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:41:43.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.041 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:41:43.137 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:41:43.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.141 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:41:43.178 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:41:43.178 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:41:43.178 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:41:43.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:41:43.178 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:41:43.179 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:41:43.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:41:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:41:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:41:43.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.377 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:41:43.412 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:41:43.413 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:41:43.413 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:41:43.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:41:43.510 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:41:43.510 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:43.545 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:41:43.546 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:41:43.546 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:43.620 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:41:43.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:43.748 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:43.748 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:41:43.775 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:41:43.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:41:43.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:41:43.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:41:43.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:41:43.778 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:41:43.778 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:41:43.778 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:41:43.778 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:41:43.778 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:41:43.781 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:41:43.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:43.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:43.837 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:41:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:41:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:41:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:41:43.927 INFO:teuthology.orchestra.run.smithi045.stdout:client5630 2024-09-13T08:41:43.927 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:43.927 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:43.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 -- 172.21.15.31:0/2473434506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 msgr2=0x7fae6c0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 --2- 172.21.15.31:0/2473434506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c0cd780 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fae54005fd0 tx=0x7fae5402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 -- 172.21.15.31:0/2473434506 shutdown_connections 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 --2- 172.21.15.31:0/2473434506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c0cd780 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 --2- 172.21.15.31:0/2473434506 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 0x7fae6c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 -- 172.21.15.31:0/2473434506 >> 172.21.15.31:0/2473434506 conn(0x7fae6c0cfbb0 msgr2=0x7fae6c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 -- 172.21.15.31:0/2473434506 shutdown_connections 2024-09-13T08:41:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.137+0000 7fae72800640 1 -- 172.21.15.31:0/2473434506 wait complete. 2024-09-13T08:41:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.139+0000 7fae72800640 1 Processor -- start 2024-09-13T08:41:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.139+0000 7fae72800640 1 -- start start 2024-09-13T08:41:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6c0745f0 0x7fae6c159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 0x7fae6c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c0ce130 con 0x7fae6c0741b0 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c0cdfb0 con 0x7fae6c0745f0 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae72800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fae6c0cb0f0 con 0x7fae6c15acb0 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae6b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae6b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59864/0 (socket says 172.21.15.31:59864) 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae6b400640 1 -- 172.21.15.31:0/3470374434 learned_addr learned my addr 172.21.15.31:0/3470374434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:44.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae62a00640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6c0745f0 0x7fae6c159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.140+0000 7fae6be00640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 0x7fae6c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 -- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 msgr2=0x7fae6c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 0x7fae6c159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 -- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6c0745f0 msgr2=0x7fae6c159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6c0745f0 0x7fae6c159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 -- 172.21.15.31:0/3470374434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6c0af310 con 0x7fae6c0741b0 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae6b400640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00d750 tx=0x7fae5c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c017070 con 0x7fae6c0741b0 2024-09-13T08:41:44.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae5c004310 con 0x7fae6c0741b0 2024-09-13T08:41:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c00de60 con 0x7fae6c0741b0 2024-09-13T08:41:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6c1f28e0 con 0x7fae6c0741b0 2024-09-13T08:41:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.141+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6c0c9e50 con 0x7fae6c0741b0 2024-09-13T08:41:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.143+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6c15e470 con 0x7fae6c0741b0 2024-09-13T08:41:44.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.144+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fae5c009c30 con 0x7fae6c0741b0 2024-09-13T08:41:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.145+0000 7fae63e00640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 0x7fae4807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.146+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fae5c09b300 con 0x7fae6c0741b0 2024-09-13T08:41:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.146+0000 7fae62a00640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 0x7fae4807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:44.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.146+0000 7fae62a00640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 0x7fae4807b070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fae54033000 tx=0x7fae54004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:44.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.150+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fae6c15e470 con 0x7fae6c0741b0 2024-09-13T08:41:44.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:44.388+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fae6c0cdae0 con 0x7fae6c0741b0 2024-09-13T08:41:45.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.214+0000 7fae63e00640 1 -- 172.21.15.31:0/3470374434 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v48) ==== 136+0+0 (secure 0 0 0) 0x7fae6c0cdae0 con 0x7fae6c0741b0 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 msgr2=0x7fae4807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 0x7fae4807b070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fae54033000 tx=0x7fae54004730 comp rx=0 tx=0).stop 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 msgr2=0x7fae6c1f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00d750 tx=0x7fae5c00dc10 comp rx=0 tx=0).stop 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 shutdown_connections 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fae48078bb0 0x7fae4807b070 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae6c15acb0 0x7fae6c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6c0745f0 0x7fae6c159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 --2- 172.21.15.31:0/3470374434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae6c0741b0 0x7fae6c1f6930 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 >> 172.21.15.31:0/3470374434 conn(0x7fae6c0cfbb0 msgr2=0x7fae6c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 shutdown_connections 2024-09-13T08:41:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.217+0000 7fae72800640 1 -- 172.21.15.31:0/3470374434 wait complete. 2024-09-13T08:41:45.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:41:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393be00640 1 -- 172.21.15.31:0/1223932376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 msgr2=0x7f39200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393be00640 1 --2- 172.21.15.31:0/1223932376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f39200051c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3934001e10 tx=0x7f3920005830 comp rx=0 tx=0).stop 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393b400640 1 -- 172.21.15.31:0/1223932376 reap_dead start 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393be00640 1 -- 172.21.15.31:0/1223932376 shutdown_connections 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393be00640 1 -- 172.21.15.31:0/1223932376 >> 172.21.15.31:0/1223932376 conn(0x7f393c0b2000 msgr2=0x7f393c0b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.402+0000 7f393be00640 1 -- 172.21.15.31:0/1223932376 shutdown_connections 2024-09-13T08:41:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.403+0000 7f393be00640 1 -- 172.21.15.31:0/1223932376 wait complete. 2024-09-13T08:41:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.404+0000 7f393be00640 1 Processor -- start 2024-09-13T08:41:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.404+0000 7f393be00640 1 -- start start 2024-09-13T08:41:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393c09a290 0x7f393c09d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393c09de70 0x7f393c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f393c06d970 con 0x7f3920004d40 2024-09-13T08:41:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f393c06d7f0 con 0x7f393c09de70 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f393c15d480 con 0x7f393c09a290 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.405+0000 7f393aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55602/0 (socket says 172.21.15.31:55602) 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 -- 172.21.15.31:0/1619223 learned_addr learned my addr 172.21.15.31:0/1619223 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393b400640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393c09de70 0x7f393c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f3932000640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393c09a290 0x7f393c09d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 -- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393c09a290 msgr2=0x7f393c09d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393c09a290 0x7f393c09d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 -- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393c09de70 msgr2=0x7f393c0b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393c09de70 0x7f393c0b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 -- 172.21.15.31:0/1619223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393c1f85c0 con 0x7f3920004d40 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f393aa00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f392c0066b0 tx=0x7f392c003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c00f780 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f392c016020 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.406+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c00fbc0 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.407+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393c09a770 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.407+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393c1f8b40 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.408+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393c0cc140 con 0x7f3920004d40 2024-09-13T08:41:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.409+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f392c011430 con 0x7f3920004d40 2024-09-13T08:41:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.410+0000 7f3933400640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 0x7f391807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.410+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f392c09b7a0 con 0x7f3920004d40 2024-09-13T08:41:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.410+0000 7f3932000640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 0x7f391807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.411+0000 7f3932000640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 0x7f391807b070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f393c069730 tx=0x7f3924005f90 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:45.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.414+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f392c020030 con 0x7f3920004d40 2024-09-13T08:41:45.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.633+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f393c09bdb0 con 0x7f3918078bb0 2024-09-13T08:41:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.636+0000 7f3933400640 1 -- 172.21.15.31:0/1619223 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f3924012b40 con 0x7f3918078bb0 2024-09-13T08:41:45.636 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:41:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:41:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:45.637 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.637+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 msgr2=0x7f391807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 0x7f391807b070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f393c069730 tx=0x7f3924005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 msgr2=0x7f393c0cfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f392c0066b0 tx=0x7f392c003a80 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 shutdown_connections 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f393c09de70 0x7f393c0b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3918078bb0 0x7f391807b070 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f393c09a290 0x7f393c09d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 --2- 172.21.15.31:0/1619223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920004d40 0x7f393c0cfc60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 >> 172.21.15.31:0/1619223 conn(0x7f393c0b2000 msgr2=0x7f393c070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 shutdown_connections 2024-09-13T08:41:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.638+0000 7f393be00640 1 -- 172.21.15.31:0/1619223 wait complete. 2024-09-13T08:41:45.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:41:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 -- 172.21.15.31:0/3115356738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d2710 msgr2=0x7f01c80b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 --2- 172.21.15.31:0/3115356738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d2710 0x7f01c80b3460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f01c806cfe0 tx=0x7f01c002a8f0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 -- 172.21.15.31:0/3115356738 shutdown_connections 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 --2- 172.21.15.31:0/3115356738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d2710 0x7f01c80b3460 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 --2- 172.21.15.31:0/3115356738 >> v2:172.21.15.45:3300/0 conn(0x7f01c80d1d90 0x7f01c80d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 --2- 172.21.15.31:0/3115356738 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 0x7f01c80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 -- 172.21.15.31:0/3115356738 >> 172.21.15.31:0/3115356738 conn(0x7f01c80d3e90 msgr2=0x7f01c80d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 -- 172.21.15.31:0/3115356738 shutdown_connections 2024-09-13T08:41:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.818+0000 7f01cca00640 1 -- 172.21.15.31:0/3115356738 wait complete. 2024-09-13T08:41:45.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.820+0000 7f01cca00640 1 Processor -- start 2024-09-13T08:41:45.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.820+0000 7f01cca00640 1 -- start start 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01c80d2710 0x7f01c815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 0x7f01c80cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01c80b3b20 con 0x7f01c80d1d90 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f01c80b39a0 con 0x7f01c80d2710 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01cca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f01c80b3ca0 con 0x7f01c8161470 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55632/0 (socket says 172.21.15.31:55632) 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 -- 172.21.15.31:0/3129526139 learned_addr learned my addr 172.21.15.31:0/3129526139 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c7200640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 0x7f01c80cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01bfe00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01c80d2710 0x7f01c815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 -- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 msgr2=0x7f01c80cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 0x7f01c80cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 -- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01c80d2710 msgr2=0x7f01c815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01c80d2710 0x7f01c815bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.821+0000 7f01c6800640 1 -- 172.21.15.31:0/3129526139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c815c460 con 0x7f01c80d1d90 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01c6800640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f01b800d750 tx=0x7f01b800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8017070 con 0x7f01c80d1d90 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01b8004310 con 0x7f01c80d1d90 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b800de60 con 0x7f01c80d1d90 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c815c740 con 0x7f01c80d1d90 2024-09-13T08:41:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.822+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c815cd00 con 0x7f01c80d1d90 2024-09-13T08:41:45.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.823+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c815d130 con 0x7f01c80d1d90 2024-09-13T08:41:45.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.824+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f01b8009c30 con 0x7f01c80d1d90 2024-09-13T08:41:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.825+0000 7f01bea00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 0x7f01a407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.825+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f01b809b320 con 0x7f01c80d1d90 2024-09-13T08:41:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.825+0000 7f01bfe00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 0x7f01a407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.826+0000 7f01bfe00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 0x7f01a407b070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01c80ca4b0 tx=0x7f01b0005c70 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:45.829+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f01b8020020 con 0x7f01c80d1d90 2024-09-13T08:41:46.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.025+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f01c80b66c0 con 0x7f01c80d1d90 2024-09-13T08:41:46.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.025+0000 7f01bea00640 1 -- 172.21.15.31:0/3129526139 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f01b80634c0 con 0x7f01c80d1d90 2024-09-13T08:41:46.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 msgr2=0x7f01a407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 0x7f01a407b070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01c80ca4b0 tx=0x7f01b0005c70 comp rx=0 tx=0).stop 2024-09-13T08:41:46.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 msgr2=0x7f01c80d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f01b800d750 tx=0x7f01b800dc10 comp rx=0 tx=0).stop 2024-09-13T08:41:46.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 shutdown_connections 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f01a4078bb0 0x7f01a407b070 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f01c8161470 0x7f01c80cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01c80d2710 0x7f01c815bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 --2- 172.21.15.31:0/3129526139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c80d1d90 0x7f01c80d09d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 >> 172.21.15.31:0/3129526139 conn(0x7f01c80d3e90 msgr2=0x7f01c806e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 shutdown_connections 2024-09-13T08:41:46.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.028+0000 7f01cca00640 1 -- 172.21.15.31:0/3129526139 wait complete. 2024-09-13T08:41:46.045 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:46.045 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:46.085 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:46.085 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:46.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:41:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 -- 172.21.15.31:0/1054269498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5415be30 msgr2=0x7fef540cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 --2- 172.21.15.31:0/1054269498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5415be30 0x7fef540cdb80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fef44009830 tx=0x7fef4402a4a0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 -- 172.21.15.31:0/1054269498 shutdown_connections 2024-09-13T08:41:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 --2- 172.21.15.31:0/1054269498 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 0x7fef540ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 --2- 172.21.15.31:0/1054269498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5415be30 0x7fef540cdb80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 -- 172.21.15.31:0/1054269498 >> 172.21.15.31:0/1054269498 conn(0x7fef540b18f0 msgr2=0x7fef540b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 -- 172.21.15.31:0/1054269498 shutdown_connections 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.362+0000 7fef59200640 1 -- 172.21.15.31:0/1054269498 wait complete. 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.364+0000 7fef59200640 1 Processor -- start 2024-09-13T08:41:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.364+0000 7fef59200640 1 -- start start 2024-09-13T08:41:46.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef540ce600 0x7fef541f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 0x7fef541f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef540733e0 con 0x7fef5415be30 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef54160160 con 0x7fef540ce600 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef59200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fef541602e0 con 0x7fef540ced20 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55636/0 (socket says 172.21.15.31:55636) 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 -- 172.21.15.31:0/2588738487 learned_addr learned my addr 172.21.15.31:0/2588738487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef51e00640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef540ce600 0x7fef541f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef49400640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 0x7fef541f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 -- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 msgr2=0x7fef541f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 0x7fef541f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 -- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef540ce600 msgr2=0x7fef541f7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef540ce600 0x7fef541f7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.365+0000 7fef52800640 1 -- 172.21.15.31:0/2588738487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef541f3700 con 0x7fef5415be30 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef52800640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00cc40 tx=0x7fef4c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c016020 con 0x7fef5415be30 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef541f39e0 con 0x7fef5415be30 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef4c00ce50 con 0x7fef5415be30 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c012680 con 0x7fef5415be30 2024-09-13T08:41:46.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.366+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef54159570 con 0x7fef5415be30 2024-09-13T08:41:46.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.367+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef54160800 con 0x7fef5415be30 2024-09-13T08:41:46.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.368+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fef4c019020 con 0x7fef5415be30 2024-09-13T08:41:46.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.369+0000 7fef4aa00640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 0x7fef3007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.369+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fef4c09b840 con 0x7fef5415be30 2024-09-13T08:41:46.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.370+0000 7fef51e00640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 0x7fef3007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.370+0000 7fef51e00640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 0x7fef3007b070 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef44034000 tx=0x7fef44004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:46.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.373+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fef4c020020 con 0x7fef5415be30 2024-09-13T08:41:46.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.581+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fef54159740 con 0x7fef5415be30 2024-09-13T08:41:46.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.582+0000 7fef4aa00640 1 -- 172.21.15.31:0/2588738487 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) ==== 80+0+0 (secure 0 0 0) 0x7fef4c0638d0 con 0x7fef5415be30 2024-09-13T08:41:46.583 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.584+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 msgr2=0x7fef3007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.584+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 0x7fef3007b070 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef44034000 tx=0x7fef44004730 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.584+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 msgr2=0x7fef54145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.584+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00cc40 tx=0x7fef4c00a430 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.584+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 shutdown_connections 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef5415be30 0x7fef54145430 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fef30078bb0 0x7fef3007b070 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef540ced20 0x7fef541f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 --2- 172.21.15.31:0/2588738487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef540ce600 0x7fef541f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 >> 172.21.15.31:0/2588738487 conn(0x7fef540b18f0 msgr2=0x7fef540aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 shutdown_connections 2024-09-13T08:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.585+0000 7fef59200640 1 -- 172.21.15.31:0/2588738487 wait complete. 2024-09-13T08:41:46.602 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:41:46.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:46.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:41:46.665 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:41:46.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:46.665 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:41:46.702 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:46.702 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:46.742 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:46.742 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:46.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:41:46.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 -- 172.21.15.31:0/1386226916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc15e2e0 msgr2=0x7f3fcc15e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/1386226916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc15e6c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0c7570 tx=0x7f3fbc02a460 comp rx=0 tx=0).stop 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 -- 172.21.15.31:0/1386226916 shutdown_connections 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/1386226916 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0d0130 0x7f3fcc0cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/1386226916 >> v2:172.21.15.45:3300/0 conn(0x7f3fcc0ce0e0 0x7f3fcc0cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/1386226916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc15e6c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 -- 172.21.15.31:0/1386226916 >> 172.21.15.31:0/1386226916 conn(0x7f3fcc0b2000 msgr2=0x7f3fcc0b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 -- 172.21.15.31:0/1386226916 shutdown_connections 2024-09-13T08:41:46.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.982+0000 7f3fcbe00640 1 -- 172.21.15.31:0/1386226916 wait complete. 2024-09-13T08:41:46.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.984+0000 7f3fcbe00640 1 Processor -- start 2024-09-13T08:41:46.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.984+0000 7f3fcbe00640 1 -- start start 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0ce0e0 0x7f3fcc156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc0741b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc068e30 con 0x7f3fcc0d0130 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0afbd0 con 0x7f3fcc15e2e0 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcbe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc0cb660 con 0x7f3fcc0ce0e0 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55660/0 (socket says 172.21.15.31:55660) 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fc3e00640 1 -- 172.21.15.31:0/3719269706 learned_addr learned my addr 172.21.15.31:0/3719269706 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcb400640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc0741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.985+0000 7f3fcaa00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0ce0e0 0x7f3fcc156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 -- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0ce0e0 msgr2=0x7f3fcc156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0ce0e0 0x7f3fcc156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 -- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fcc15e2e0 msgr2=0x7f3fcc0741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc0741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 -- 172.21.15.31:0/3719269706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc0c6b00 con 0x7f3fcc0d0130 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc3e00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400d750 tx=0x7f3fb400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4017070 con 0x7f3fcc0d0130 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fb4009d20 con 0x7f3fcc0d0130 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400de60 con 0x7f3fcc0d0130 2024-09-13T08:41:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc0c6de0 con 0x7f3fcc0d0130 2024-09-13T08:41:46.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.986+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc156f70 con 0x7f3fcc0d0130 2024-09-13T08:41:46.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.987+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fcc15e6c0 con 0x7f3fcc0d0130 2024-09-13T08:41:46.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.989+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f3fb401a070 con 0x7f3fcc0d0130 2024-09-13T08:41:46.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.989+0000 7f3fc2a00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 0x7f3fa807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:46.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.990+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f3fb409b110 con 0x7f3fcc0d0130 2024-09-13T08:41:46.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.990+0000 7f3fcaa00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 0x7f3fa807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:46.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.991+0000 7f3fcaa00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 0x7f3fa807b0a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc033000 tx=0x7f3fbc004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:46.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:46.994+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3fb4062ad0 con 0x7f3fcc0d0130 2024-09-13T08:41:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.198+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3fcc15db00 con 0x7f3fcc0d0130 2024-09-13T08:41:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.199+0000 7f3fc2a00640 1 -- 172.21.15.31:0/3719269706 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) ==== 80+0+0 (secure 0 0 0) 0x7f3fb4067500 con 0x7f3fcc0d0130 2024-09-13T08:41:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 msgr2=0x7f3fa807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 0x7f3fa807b0a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc033000 tx=0x7f3fbc004730 comp rx=0 tx=0).stop 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 msgr2=0x7f3fcc156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400d750 tx=0x7f3fb400dc10 comp rx=0 tx=0).stop 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 shutdown_connections 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fcc15e2e0 0x7f3fcc0741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc0d0130 0x7f3fcc156300 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f3fa8078be0 0x7f3fa807b0a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 --2- 172.21.15.31:0/3719269706 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fcc0ce0e0 0x7f3fcc156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 >> 172.21.15.31:0/3719269706 conn(0x7f3fcc0b2000 msgr2=0x7f3fcc0704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 shutdown_connections 2024-09-13T08:41:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.201+0000 7f3fcbe00640 1 -- 172.21.15.31:0/3719269706 wait complete. 2024-09-13T08:41:47.218 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:41:47.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:47.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:41:47.287 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:41:47.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:41:47.288 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:41:47.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 -- 172.21.15.31:0/3263108925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 msgr2=0x7f75f4162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 --2- 172.21.15.31:0/3263108925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4162060 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f75e4005fd0 tx=0x7f75e402a660 comp rx=0 tx=0).stop 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 -- 172.21.15.31:0/3263108925 shutdown_connections 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 --2- 172.21.15.31:0/3263108925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 0x7f75f41625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 --2- 172.21.15.31:0/3263108925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4162060 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 -- 172.21.15.31:0/3263108925 >> 172.21.15.31:0/3263108925 conn(0x7f75f409d1c0 msgr2=0x7f75f40d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 -- 172.21.15.31:0/3263108925 shutdown_connections 2024-09-13T08:41:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.485+0000 7f75f3200640 1 -- 172.21.15.31:0/3263108925 wait complete. 2024-09-13T08:41:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.487+0000 7f75f3200640 1 Processor -- start 2024-09-13T08:41:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.487+0000 7f75f3200640 1 -- start start 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.487+0000 7f75f3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 0x7f75f40b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40d2620 con 0x7f75f40d44f0 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40d24a0 con 0x7f75f40d20c0 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f75f40d27a0 con 0x7f75f40cf2d0 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55666/0 (socket says 172.21.15.31:55666) 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 -- 172.21.15.31:0/2931954159 learned_addr learned my addr 172.21.15.31:0/2931954159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f1e00640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 0x7f75f40b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f1400640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 -- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 msgr2=0x7f75f40b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 0x7f75f40b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 -- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 msgr2=0x7f75f4152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.488+0000 7f75f2800640 1 -- 172.21.15.31:0/2931954159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41f46d0 con 0x7f75f40d44f0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75f2800640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c6b0 tx=0x7f75ec00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec003b70 con 0x7f75f40d44f0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75ec003d30 con 0x7f75f40d44f0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec005330 con 0x7f75f40d44f0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f41f49b0 con 0x7f75f40d44f0 2024-09-13T08:41:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.489+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41f4f70 con 0x7f75f40d44f0 2024-09-13T08:41:47.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.491+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f40b57b0 con 0x7f75f40d44f0 2024-09-13T08:41:47.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.492+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f75ec0054d0 con 0x7f75f40d44f0 2024-09-13T08:41:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.492+0000 7f75ea000640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 0x7f75d407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.493+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f75ec09ac70 con 0x7f75f40d44f0 2024-09-13T08:41:47.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.493+0000 7f75f1e00640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 0x7f75d407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:47.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.494+0000 7f75f1e00640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 0x7f75d407b070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f75e4033000 tx=0x7f75e4004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.497+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f75ec062670 con 0x7f75f40d44f0 2024-09-13T08:41:47.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.734+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f408df70 con 0x7f75f40d44f0 2024-09-13T08:41:47.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.735+0000 7f75ea000640 1 -- 172.21.15.31:0/2931954159 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7f75ec0670a0 con 0x7f75f40d44f0 2024-09-13T08:41:47.736 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:47.736 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:47.736 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 msgr2=0x7f75d407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 0x7f75d407b070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f75e4033000 tx=0x7f75e4004730 comp rx=0 tx=0).stop 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 msgr2=0x7f75f40d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c6b0 tx=0x7f75ec00cb70 comp rx=0 tx=0).stop 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 shutdown_connections 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75f40d44f0 0x7f75f40d0860 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75f40d20c0 0x7f75f4152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f75d4078bb0 0x7f75d407b070 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 --2- 172.21.15.31:0/2931954159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75f40cf2d0 0x7f75f40b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:47.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 >> 172.21.15.31:0/2931954159 conn(0x7f75f409d1c0 msgr2=0x7f75f41621f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:47.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 shutdown_connections 2024-09-13T08:41:47.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:47.737+0000 7f75f3200640 1 -- 172.21.15.31:0/2931954159 wait complete. 2024-09-13T08:41:47.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:41:47.931 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-09-13T08:41:47.932 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-13T08:41:47.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol52730 2024-09-13T08:41:48.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 -- 172.21.15.31:0/3701005155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 msgr2=0x7f5e440cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/3701005155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e440cb740 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009830 tx=0x7f5e2c02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 -- 172.21.15.31:0/3701005155 shutdown_connections 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/3701005155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e440cb740 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 -- 172.21.15.31:0/3701005155 >> 172.21.15.31:0/3701005155 conn(0x7f5e440cfbb0 msgr2=0x7f5e440cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 -- 172.21.15.31:0/3701005155 shutdown_connections 2024-09-13T08:41:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.096+0000 7f5e4bc00640 1 -- 172.21.15.31:0/3701005155 wait complete. 2024-09-13T08:41:48.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.098+0000 7f5e4bc00640 1 Processor -- start 2024-09-13T08:41:48.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.098+0000 7f5e4bc00640 1 -- start start 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 0x7f5e440c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e4415bfc0 0x7f5e440c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4406e740 con 0x7f5e44072560 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4406e5c0 con 0x7f5e4415bfc0 2024-09-13T08:41:48.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e4bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5e440b30f0 con 0x7f5e440cbc80 2024-09-13T08:41:48.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e49200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e49200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55672/0 (socket says 172.21.15.31:55672) 2024-09-13T08:41:48.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.099+0000 7f5e49200640 1 -- 172.21.15.31:0/1169114395 learned_addr learned my addr 172.21.15.31:0/1169114395 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e3be00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 0x7f5e440c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49c00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e4415bfc0 0x7f5e440c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 -- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 msgr2=0x7f5e440c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 0x7f5e440c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 -- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e4415bfc0 msgr2=0x7f5e440c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e4415bfc0 0x7f5e440c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 -- 172.21.15.31:0/1169114395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e440c6610 con 0x7f5e44072560 2024-09-13T08:41:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e3be00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 0x7f5e440c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e49200640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400c430 tx=0x7f5e3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e34017070 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e340053c0 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e441f8240 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.100+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e34003850 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.101+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e441f8800 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.102+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e440cb740 con 0x7f5e44072560 2024-09-13T08:41:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.103+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7f5e3401a070 con 0x7f5e44072560 2024-09-13T08:41:48.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.104+0000 7f5e41600640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 0x7f5e2007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.104+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7f5e3409a360 con 0x7f5e44072560 2024-09-13T08:41:48.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.104+0000 7f5e3be00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 0x7f5e2007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.105+0000 7f5e3be00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 0x7f5e2007b070 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c034000 tx=0x7f5e2c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.108+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5e34062470 con 0x7f5e44072560 2024-09-13T08:41:48.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.331+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 --> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol52730", "target": ["mon-mgr", ""]}) -- 0x7f5e4415b350 con 0x7f5e20078bb0 2024-09-13T08:41:48.340 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:48.340+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:48.340 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:48.340+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:48.340 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:48.340+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:48.340 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:48.340+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:48.340 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-09-13T08:41:48.340+0000 7fe267c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:48.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.379+0000 7f5e41600640 1 -- 172.21.15.31:0/1169114395 <== mgr.4112 v2:172.21.15.45:6800/4176094944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5e4415b350 con 0x7f5e20078bb0 2024-09-13T08:41:48.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 msgr2=0x7f5e2007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 0x7f5e2007b070 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c034000 tx=0x7f5e2c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:48.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 msgr2=0x7f5e44149170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400c430 tx=0x7f5e3400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 shutdown_connections 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e4415bfc0 0x7f5e440c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7f5e20078bb0 0x7f5e2007b070 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e440cbc80 0x7f5e440c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 --2- 172.21.15.31:0/1169114395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e44072560 0x7f5e44149170 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 >> 172.21.15.31:0/1169114395 conn(0x7f5e440cfbb0 msgr2=0x7f5e44152fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 shutdown_connections 2024-09-13T08:41:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.381+0000 7f5e4bc00640 1 -- 172.21.15.31:0/1169114395 wait complete. 2024-09-13T08:41:48.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:48.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb13be00640 1 -- 172.21.15.31:0/453582455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13402b720 con 0x7fb120004d40 2024-09-13T08:41:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 -- 172.21.15.31:0/453582455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb120004d40 msgr2=0x7fb1200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 --2- 172.21.15.31:0/453582455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb120004d40 0x7fb1200051c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c670 tx=0x7fb120005830 comp rx=0 tx=0).stop 2024-09-13T08:41:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb141600640 1 -- 172.21.15.31:0/453582455 reap_dead start 2024-09-13T08:41:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 -- 172.21.15.31:0/453582455 shutdown_connections 2024-09-13T08:41:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 -- 172.21.15.31:0/453582455 >> 172.21.15.31:0/453582455 conn(0x7fb13c0cc910 msgr2=0x7fb13c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 -- 172.21.15.31:0/453582455 shutdown_connections 2024-09-13T08:41:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.559+0000 7fb143600640 1 -- 172.21.15.31:0/453582455 wait complete. 2024-09-13T08:41:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.561+0000 7fb143600640 1 Processor -- start 2024-09-13T08:41:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.561+0000 7fb143600640 1 -- start start 2024-09-13T08:41:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb120004d40 0x7fb13c0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb13c15a950 0x7fb13c151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1600f0 con 0x7fb13c15afb0 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c0cb850 con 0x7fb120004d40 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb143600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb13c0b15a0 con 0x7fb13c15a950 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb141600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb141600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55682/0 (socket says 172.21.15.31:55682) 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb141600640 1 -- 172.21.15.31:0/3056825049 learned_addr learned my addr 172.21.15.31:0/3056825049 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb140c00640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb120004d40 0x7fb13c0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.562+0000 7fb133e00640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb13c15a950 0x7fb13c151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 -- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb13c15a950 msgr2=0x7fb13c151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb13c15a950 0x7fb13c151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 -- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb120004d40 msgr2=0x7fb13c0b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb120004d40 0x7fb13c0b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 -- 172.21.15.31:0/3056825049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c1f5b10 con 0x7fb13c15afb0 2024-09-13T08:41:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb141600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c670 tx=0x7fb134003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134039070 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb13402ba10 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13402be40 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c1f5dd0 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.563+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1f6390 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.564+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c0cc440 con 0x7fb13c15afb0 2024-09-13T08:41:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.566+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fb13403c020 con 0x7fb13c15afb0 2024-09-13T08:41:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.566+0000 7fb139600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 0x7fb11807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.567+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fb1340baa30 con 0x7fb13c15afb0 2024-09-13T08:41:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.567+0000 7fb140c00640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 0x7fb11807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.568+0000 7fb140c00640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 0x7fb11807b070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb12c006fd0 tx=0x7fb12c006b10 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:48.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.571+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb134082ca0 con 0x7fb13c15afb0 2024-09-13T08:41:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.805+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb13c15fdc0 con 0x7fb13c15afb0 2024-09-13T08:41:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.805+0000 7fb139600640 1 -- 172.21.15.31:0/3056825049 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7fb1340825a0 con 0x7fb13c15afb0 2024-09-13T08:41:48.806 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:48.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 msgr2=0x7fb11807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 0x7fb11807b070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb12c006fd0 tx=0x7fb12c006b10 comp rx=0 tx=0).stop 2024-09-13T08:41:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 msgr2=0x7fb13c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c670 tx=0x7fb134003ba0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 shutdown_connections 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb13c15afb0 0x7fb13c15b430 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fb118078bb0 0x7fb11807b070 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb13c15a950 0x7fb13c151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 --2- 172.21.15.31:0/3056825049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb120004d40 0x7fb13c0b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 >> 172.21.15.31:0/3056825049 conn(0x7fb13c0cc910 msgr2=0x7fb13c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 shutdown_connections 2024-09-13T08:41:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:48.808+0000 7fb143600640 1 -- 172.21.15.31:0/3056825049 wait complete. 2024-09-13T08:41:48.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:41:49.004 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":5675,"entity":{"name":{"type":"client","num":5675},"addr":{"type":"any","addr":"172.21.15.45:0","nonce":1917184858}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":0.64798926800000001,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.967822784821635,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5675 172.21.15.45:0/1917184858","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"x","hostname":"smithi045","pid":"35542","root":"/"}}] 2024-09-13T08:41:49.005 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi045", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35542", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.45:0", "nonce": 1917184858, "type": "any" }, "name": { "num": 5675, "type": "client" } }, "id": 5675, "inst": "client.5675 172.21.15.45:0/1917184858", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.967822784821635 }, "state": "open", "uptime": 0.647989268 } ] 2024-09-13T08:41:49.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.165+0000 7fbbd3600640 1 -- 172.21.15.31:0/89291874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15bd70 msgr2=0x7fbbcc0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.165+0000 7fbbd3600640 1 --2- 172.21.15.31:0/89291874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc0cf850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009830 tx=0x7fbbbc02a540 comp rx=0 tx=0).stop 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.166+0000 7fbbd3600640 1 -- 172.21.15.31:0/89291874 shutdown_connections 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.166+0000 7fbbd3600640 1 --2- 172.21.15.31:0/89291874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc0cf850 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.166+0000 7fbbd3600640 1 -- 172.21.15.31:0/89291874 >> 172.21.15.31:0/89291874 conn(0x7fbbcc0cc910 msgr2=0x7fbbcc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.166+0000 7fbbd3600640 1 -- 172.21.15.31:0/89291874 shutdown_connections 2024-09-13T08:41:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.166+0000 7fbbd3600640 1 -- 172.21.15.31:0/89291874 wait complete. 2024-09-13T08:41:49.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.167+0000 7fbbd3600640 1 Processor -- start 2024-09-13T08:41:49.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.167+0000 7fbbd3600640 1 -- start start 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15f9e0 0x7fbbcc0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbbcc15fe20 0x7fbbcc0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cec50 con 0x7fbbcc15bd70 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cead0 con 0x7fbbcc15f9e0 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.168+0000 7fbbd3600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cb850 con 0x7fbbcc15fe20 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55690/0 (socket says 172.21.15.31:55690) 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 -- 172.21.15.31:0/1384469398 learned_addr learned my addr 172.21.15.31:0/1384469398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:49.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbc3e00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15f9e0 0x7fbbcc0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd1600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbbcc15fe20 0x7fbbcc0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 -- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbbcc15fe20 msgr2=0x7fbbcc0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbbcc15fe20 0x7fbbcc0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 -- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15f9e0 msgr2=0x7fbbcc0c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15f9e0 0x7fbbcc0c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 -- 172.21.15.31:0/1384469398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc0c5510 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.169+0000 7fbbd0c00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009950 tx=0x7fbbbc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.170+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc005c90 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.170+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbbc005020 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.170+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc0c57f0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.170+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc005450 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.170+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc0c5db0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.171+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc0cf850 con 0x7fbbcc15bd70 2024-09-13T08:41:49.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.172+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 101593+0+0 (secure 0 0 0) 0x7fbbbc039040 con 0x7fbbcc15bd70 2024-09-13T08:41:49.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.174+0000 7fbbc9600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 0x7fbba807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:49.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.174+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7fbbbc0bbfe0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.174+0000 7fbbc3e00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 0x7fbba807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:49.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.175+0000 7fbbc3e00640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 0x7fbba807b0a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc051080 tx=0x7fbbb4009040 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:49.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.178+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbbbc040030 con 0x7fbbcc15bd70 2024-09-13T08:41:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.235+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 12) ==== 101644+0+0 (secure 0 0 0) 0x7fbbbc07fca0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.425+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fbbcc0b5df0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.433+0000 7fbbc9600640 1 -- 172.21.15.31:0/1384469398 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v12) ==== 81+0+530109 (secure 0 0 0) 0x7fbbbc0893a0 con 0x7fbbcc15bd70 2024-09-13T08:41:49.441 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 12, 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 4112, 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4176094944 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4176094944 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.45:6801/4176094944", 2024-09-13T08:41:49.443 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2024-09-13T08:36:49.024454+0000", 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540719139924082687, 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 4118, 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540719139924082687, 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.444 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.446 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.448 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.449 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.452 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-09-13T08:41:49.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.455 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.461 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-13T08:41:49.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-13T08:41:49.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-09-13T08:41:49.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-09-13T08:41:49.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.470 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.473 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-09-13T08:41:49.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-09-13T08:41:49.476 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-09-13T08:41:49.477 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.480 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-09-13T08:41:49.483 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-13T08:41:49.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.486 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-13T08:41:49.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-13T08:41:49.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-13T08:41:49.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-09-13T08:41:49.490 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-09-13T08:41:49.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.495 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.496 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-09-13T08:41:49.497 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.497 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.497 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.499 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:49.500 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.502 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.503 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-13T08:41:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-13T08:41:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.508 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nginx": { 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nginx", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nginx container image", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-09-13T08:41:49.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-09-13T08:41:49.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.512 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_oauth2_proxy": { 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_oauth2_proxy", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "oauth2-proxy container image", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba_metrics": { 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba_metrics", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-13T08:41:49.520 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-13T08:41:49.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.524 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.526 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-09-13T08:41:49.528 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-09-13T08:41:49.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.534 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.535 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.539 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-13T08:41:49.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.542 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.544 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.545 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.545 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.545 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.547 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-13T08:41:49.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-13T08:41:49.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.547 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.547 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-13T08:41:49.548 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-13T08:41:49.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-09-13T08:41:49.550 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-09-13T08:41:49.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-09-13T08:41:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.555 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.555 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.555 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.556 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-13T08:41:49.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.558 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.558 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.558 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.558 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.558 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-09-13T08:41:49.559 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.560 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-09-13T08:41:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-09-13T08:41:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-09-13T08:41:49.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-09-13T08:41:49.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.563 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.563 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-09-13T08:41:49.563 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-13T08:41:49.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.565 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-09-13T08:41:49.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-09-13T08:41:49.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-13T08:41:49.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.569 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.569 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-13T08:41:49.569 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-09-13T08:41:49.570 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-13T08:41:49.572 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.572 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.572 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.574 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.574 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-13T08:41:49.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.575 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.576 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-09-13T08:41:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-09-13T08:41:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_keepalive_count_max": { 2024-09-13T08:41:49.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_keepalive_count_max", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_keepalive_interval": { 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_keepalive_interval", 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-13T08:41:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-09-13T08:41:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-13T08:41:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-09-13T08:41:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-09-13T08:41:49.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.621 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.624 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-13T08:41:49.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-13T08:41:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-13T08:41:49.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-13T08:41:49.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-13T08:41:49.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-09-13T08:41:49.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-09-13T08:41:49.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.636 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.649 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-13T08:41:49.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-09-13T08:41:49.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.653 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.654 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.664 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:49.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-09-13T08:41:49.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-09-13T08:41:49.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.672 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.678 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-09-13T08:41:49.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.685 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.689 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.690 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.693 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.694 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.696 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-13T08:41:49.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-09-13T08:41:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-09-13T08:41:49.719 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-13T08:41:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.734 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.736 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.739 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.740 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.743 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-09-13T08:41:49.746 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-13T08:41:49.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.748 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.760 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.768 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.769 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.771 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.774 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.775 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.784 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-09-13T08:41:49.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-09-13T08:41:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-09-13T08:41:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-09-13T08:41:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-09-13T08:41:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-09-13T08:41:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-09-13T08:41:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.842 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.845 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.858 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.859 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max_requests": { 2024-09-13T08:41:49.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_requests", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.861 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:49.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.865 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.867 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.868 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "secondary_zone_period_retry_limit": { 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.873 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.876 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-09-13T08:41:49.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.884 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-09-13T08:41:49.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-09-13T08:41:49.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-09-13T08:41:49.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-09-13T08:41:49.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-09-13T08:41:49.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-09-13T08:41:49.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smb", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "internal_store_backend": { 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "internal_store_backend", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.905 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "update_orchestration": { 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "update_orchestration", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-09-13T08:41:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-09-13T08:41:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-09-13T08:41:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-09-13T08:41:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-09-13T08:41:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-09-13T08:41:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.955 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.964 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.965 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.972 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.986 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.989 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: "restful" 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-09-13T08:41:49.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:49.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:49.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.000 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-09-13T08:41:50.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.008 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-09-13T08:41:50.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.021 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.023 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-09-13T08:41:50.038 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-09-13T08:41:50.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-13T08:41:50.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-13T08:41:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-13T08:41:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-13T08:41:50.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-09-13T08:41:50.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nginx": { 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nginx", 2024-09-13T08:41:50.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nginx container image", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-09-13T08:41:50.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-09-13T08:41:50.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_oauth2_proxy": { 2024-09-13T08:41:50.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_oauth2_proxy", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "oauth2-proxy container image", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-09-13T08:41:50.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-09-13T08:41:50.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.060 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba_metrics": { 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba_metrics", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-13T08:41:50.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-13T08:41:50.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.068 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.079 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-09-13T08:41:50.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-09-13T08:41:50.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-09-13T08:41:50.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.088 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.089 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.090 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-09-13T08:41:50.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_keepalive_count_max": { 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_keepalive_count_max", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_keepalive_interval": { 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_keepalive_interval", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-13T08:41:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-09-13T08:41:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-13T08:41:50.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.104 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.107 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.108 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-09-13T08:41:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.111 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-13T08:41:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-13T08:41:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-13T08:41:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-13T08:41:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-13T08:41:50.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-13T08:41:50.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-13T08:41:50.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-09-13T08:41:50.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-09-13T08:41:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.157 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.161 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-09-13T08:41:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:50.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.171 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.173 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.174 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-09-13T08:41:50.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.180 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.181 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.184 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.194 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-09-13T08:41:50.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-09-13T08:41:50.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:50.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-09-13T08:41:50.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.207 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-09-13T08:41:50.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.214 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-09-13T08:41:50.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.219 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.220 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.226 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.228 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.230 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.232 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.234 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.237 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-09-13T08:41:50.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.242 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.248 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.254 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.266 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.276 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.279 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.280 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.283 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.284 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-09-13T08:41:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.299 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.304 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.305 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.309 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.310 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-13T08:41:50.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-09-13T08:41:50.318 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-09-13T08:41:50.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.320 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.321 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-09-13T08:41:50.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-09-13T08:41:50.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.333 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.335 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max_requests": { 2024-09-13T08:41:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_requests", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-09-13T08:41:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "secondary_zone_period_retry_limit": { 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.351 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-09-13T08:41:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-09-13T08:41:50.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-09-13T08:41:50.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-09-13T08:41:50.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-09-13T08:41:50.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-09-13T08:41:50.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smb", 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "internal_store_backend": { 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "internal_store_backend", 2024-09-13T08:41:50.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.377 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.378 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "update_orchestration": { 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "update_orchestration", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.385 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.386 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.387 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.388 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-09-13T08:41:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-13T08:41:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-09-13T08:41:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-09-13T08:41:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-09-13T08:41:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-09-13T08:41:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "services": {}, 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-09-13T08:41:50.469 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-09-13T08:41:50.470 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2024-09-13T08:41:50.471 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "squid": [ 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-09-13T08:41:50.472 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 0, 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 224785773 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.474 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 462607798 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2115686593 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 583368432 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.476 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1597311998 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 887477527 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1259704583 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2456757066 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2626498261 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2712511176 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-13T08:41:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:0", 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1917184858 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:41:50.481 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:41:50.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 msgr2=0x7fbba807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 0x7fbba807b0a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc051080 tx=0x7fbbb4009040 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 msgr2=0x7fbbcc15b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009950 tx=0x7fbbbc003c10 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 shutdown_connections 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7fbba8078be0 0x7fbba807b0a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbbcc15fe20 0x7fbbcc0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc15f9e0 0x7fbbcc0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 --2- 172.21.15.31:0/1384469398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbcc15bd70 0x7fbbcc15b330 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 >> 172.21.15.31:0/1384469398 conn(0x7fbbcc0cc910 msgr2=0x7fbbcc0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 shutdown_connections 2024-09-13T08:41:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:49.438+0000 7fbbd3600640 1 -- 172.21.15.31:0/1384469398 wait complete. 2024-09-13T08:41:50.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr fail x 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 -- 172.21.15.31:0/2111484270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 msgr2=0x7ffa8c0cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 --2- 172.21.15.31:0/2111484270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c0cdb80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c09e090 tx=0x7ffa7c02a560 comp rx=0 tx=0).stop 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 -- 172.21.15.31:0/2111484270 shutdown_connections 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 --2- 172.21.15.31:0/2111484270 >> v2:172.21.15.45:3300/0 conn(0x7ffa8c0ced20 0x7ffa8c0ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 --2- 172.21.15.31:0/2111484270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c0cdb80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.659+0000 7ffa90a00640 1 -- 172.21.15.31:0/2111484270 >> 172.21.15.31:0/2111484270 conn(0x7ffa8c0b18f0 msgr2=0x7ffa8c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.660+0000 7ffa90a00640 1 -- 172.21.15.31:0/2111484270 shutdown_connections 2024-09-13T08:41:50.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.660+0000 7ffa90a00640 1 -- 172.21.15.31:0/2111484270 wait complete. 2024-09-13T08:41:50.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.661+0000 7ffa90a00640 1 Processor -- start 2024-09-13T08:41:50.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.661+0000 7ffa90a00640 1 -- start start 2024-09-13T08:41:50.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa8c0ce600 0x7ffa8c1f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:50.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa8c0ced20 0x7ffa8c09c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:50.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c0749f0 con 0x7ffa8c15be30 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c074870 con 0x7ffa8c0ced20 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa90a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffa8c160160 con 0x7ffa8c0ce600 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.662+0000 7ffa8a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55708/0 (socket says 172.21.15.31:55708) 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 -- 172.21.15.31:0/1881793898 learned_addr learned my addr 172.21.15.31:0/1881793898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa81400640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa8c0ced20 0x7ffa8c09c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:50.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa89e00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa8c0ce600 0x7ffa8c1f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 -- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa8c0ce600 msgr2=0x7ffa8c1f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa8c0ce600 0x7ffa8c1f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 -- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa8c0ced20 msgr2=0x7ffa8c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa8c0ced20 0x7ffa8c09c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 -- 172.21.15.31:0/1881793898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8c1f8450 con 0x7ffa8c15be30 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.663+0000 7ffa8a800640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400c780 tx=0x7ffa8400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:50.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.664+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84017070 con 0x7ffa8c15be30 2024-09-13T08:41:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.664+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa84003c60 con 0x7ffa8c15be30 2024-09-13T08:41:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.664+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840052d0 con 0x7ffa8c15be30 2024-09-13T08:41:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.664+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8c1f8730 con 0x7ffa8c15be30 2024-09-13T08:41:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.664+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8c1f8cf0 con 0x7ffa8c15be30 2024-09-13T08:41:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.665+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa8c1605c0 con 0x7ffa8c15be30 2024-09-13T08:41:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.666+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 101644+0+0 (secure 0 0 0) 0x7ffa8401a070 con 0x7ffa8c15be30 2024-09-13T08:41:50.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.668+0000 7ffa82a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 0x7ffa6807b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:50.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.668+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5709+0+0 (secure 0 0 0) 0x7ffa8409a630 con 0x7ffa8c15be30 2024-09-13T08:41:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.668+0000 7ffa89e00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 0x7ffa6807b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.669+0000 7ffa89e00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 0x7ffa6807b100 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c033000 tx=0x7ffa7c004730 comp rx=0 tx=0).ready entity=mgr.4112 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:50.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.672+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffa84061fb0 con 0x7ffa8c15be30 2024-09-13T08:41:50.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.922+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr fail", "who": "x"} v 0) -- 0x7ffa8c0b25f0 con 0x7ffa8c15be30 2024-09-13T08:41:50.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.929+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr fail", "who": "x"}]=0 v13) ==== 68+0+0 (secure 0 0 0) 0x7ffa840669e0 con 0x7ffa8c15be30 2024-09-13T08:41:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.932+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 13) ==== 50665+0+0 (secure 0 0 0) 0x7ffa8405e220 con 0x7ffa8c15be30 2024-09-13T08:41:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.932+0000 7ffa82a00640 1 -- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 msgr2=0x7ffa6807b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.932+0000 7ffa82a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 0x7ffa6807b100 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c033000 tx=0x7ffa7c004730 comp rx=0 tx=0).stop 2024-09-13T08:41:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 msgr2=0x7ffa8c1f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400c780 tx=0x7ffa8400cb80 comp rx=0 tx=0).stop 2024-09-13T08:41:50.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 shutdown_connections 2024-09-13T08:41:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa8c15be30 0x7ffa8c1f79e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa8c0ced20 0x7ffa8c09c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:6800/4176094944,v1:172.21.15.45:6801/4176094944] conn(0x7ffa68078c40 0x7ffa6807b100 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 --2- 172.21.15.31:0/1881793898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa8c0ce600 0x7ffa8c1f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:50.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.934+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 >> 172.21.15.31:0/1881793898 conn(0x7ffa8c0b18f0 msgr2=0x7ffa8c0cb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:50.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.935+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 shutdown_connections 2024-09-13T08:41:50.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:50.935+0000 7ffa90a00640 1 -- 172.21.15.31:0/1881793898 wait complete. 2024-09-13T08:41:50.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:51.061 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.060+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.062 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.060+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.062 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.060+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.062 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.060+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.062 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.060+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.062 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.061+0000 7fe171600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.063 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.061+0000 7fe171600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.063 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.061+0000 7fe171600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.063 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.061+0000 7fe171600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.063 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:41:51.061+0000 7fe171600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:41:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 -- 172.21.15.31:0/845275560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015f9e0 msgr2=0x7fad2015fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 --2- 172.21.15.31:0/845275560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015f9e0 0x7fad2015fe20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fad2009e090 tx=0x7fad0802a600 comp rx=0 tx=0).stop 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 -- 172.21.15.31:0/845275560 shutdown_connections 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 --2- 172.21.15.31:0/845275560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015f9e0 0x7fad2015fe20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 --2- 172.21.15.31:0/845275560 >> v2:172.21.15.45:3300/0 conn(0x7fad2015bd70 0x7fad200cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.165+0000 7fad26200640 1 -- 172.21.15.31:0/845275560 >> 172.21.15.31:0/845275560 conn(0x7fad200cc910 msgr2=0x7fad200cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.166+0000 7fad26200640 1 -- 172.21.15.31:0/845275560 shutdown_connections 2024-09-13T08:41:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.166+0000 7fad26200640 1 -- 172.21.15.31:0/845275560 wait complete. 2024-09-13T08:41:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.167+0000 7fad26200640 1 Processor -- start 2024-09-13T08:41:51.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.167+0000 7fad26200640 1 -- start start 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 0x7fad2014d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 0x7fad201f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad2015f9e0 0x7fad201f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad200cb9d0 con 0x7fad2015bd70 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fad200cb850 con 0x7fad200ce3c0 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.168+0000 7fad26200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fad200cc360 con 0x7fad2015f9e0 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 0x7fad201f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad1f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 0x7fad2014d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:51.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad1f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 0x7fad2014d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:48260/0 (socket says 172.21.15.31:48260) 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad1f400640 1 -- 172.21.15.31:0/2487471451 learned_addr learned my addr 172.21.15.31:0/2487471451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad1fe00640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad2015f9e0 0x7fad201f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 -- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad2015f9e0 msgr2=0x7fad201f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad2015f9e0 0x7fad201f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 -- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 msgr2=0x7fad2014d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 0x7fad2014d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.169+0000 7fad16a00640 1 -- 172.21.15.31:0/2487471451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2009aa10 con 0x7fad2015bd70 2024-09-13T08:41:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad16a00640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 0x7fad201f5c60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fad200b26b0 tx=0x7fad080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0803e030 con 0x7fad2015bd70 2024-09-13T08:41:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad08003d90 con 0x7fad2015bd70 2024-09-13T08:41:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08035040 con 0x7fad2015bd70 2024-09-13T08:41:51.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2009acf0 con 0x7fad2015bd70 2024-09-13T08:41:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.170+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad200c66e0 con 0x7fad2015bd70 2024-09-13T08:41:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.171+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad200cf850 con 0x7fad2015bd70 2024-09-13T08:41:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.171+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 50665+0+0 (secure 0 0 0) 0x7fad08041030 con 0x7fad2015bd70 2024-09-13T08:41:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.172+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6355+0+0 (secure 0 0 0) 0x7fad08073e40 con 0x7fad2015bd70 2024-09-13T08:41:51.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.177+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fad0802ae20 con 0x7fad2015bd70 2024-09-13T08:41:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.408+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad200728c0 con 0x7fad2015bd70 2024-09-13T08:41:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.409+0000 7fad17e00640 1 -- 172.21.15.31:0/2487471451 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7fad0804cd90 con 0x7fad2015bd70 2024-09-13T08:41:51.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:51.410 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.411+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 msgr2=0x7fad201f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.411+0000 7fad26200640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 0x7fad201f5c60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fad200b26b0 tx=0x7fad080038a0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 shutdown_connections 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad2015f9e0 0x7fad201f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad2015bd70 0x7fad201f5c60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 --2- 172.21.15.31:0/2487471451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad200ce3c0 0x7fad2014d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 >> 172.21.15.31:0/2487471451 conn(0x7fad200cc910 msgr2=0x7fad2015e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 shutdown_connections 2024-09-13T08:41:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:51.412+0000 7fad26200640 1 -- 172.21.15.31:0/2487471451 wait complete. 2024-09-13T08:41:51.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-13T08:41:51.608 INFO:teuthology.orchestra.run.smithi031.stdout:[{"id":6160,"entity":{"name":{"type":"client","num":6160},"addr":{"type":"any","addr":"172.21.15.31:0","nonce":2988258439}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.528991239,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9975754957913647,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6160 172.21.15.31:0/2988258439","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"y","hostname":"smithi031","pid":"35924","root":"/"}},{"id":5711,"entity":{"name":{"type":"client","num":5711},"addr":{"type":"any","addr":"172.21.15.31:0","nonce":1339354638}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.52999122200000004,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9975778034445044,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5711 172.21.15.31:0/1339354638","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"d454760ef8a93a9dd7f4e4dedfb085faac216e7e","ceph_version":"ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)","entity_id":"y","hostname":"smithi031","pid":"35924","root":"/"}}] 2024-09-13T08:41:51.608 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi031", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35924", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.31:0", "nonce": 2988258439, "type": "any" }, "name": { "num": 6160, "type": "client" } }, "id": 6160, "inst": "client.6160 172.21.15.31:0/2988258439", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9975754957913647 }, "state": "open", "uptime": 0.528991239 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "d454760ef8a93a9dd7f4e4dedfb085faac216e7e", "ceph_version": "ceph version 19.3.0-4881-gd454760e (d454760ef8a93a9dd7f4e4dedfb085faac216e7e) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi031", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35924", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.31:0", "nonce": 1339354638, "type": "any" }, "name": { "num": 5711, "type": "client" } }, "id": 5711, "inst": "client.5711 172.21.15.31:0/1339354638", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9975778034445044 }, "state": "open", "uptime": 0.529991222 } ] 2024-09-13T08:41:51.609 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T08:41:51.609 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:41:51.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:51.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:41:51.780 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:41:51.781 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:51.781 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:51.781 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:51.816 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:41:51.816 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:51.816 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:51.816 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:51.852 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:41:51.852 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:51.852 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:41:51.852 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:51.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:41:51.892 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:41:51.892 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:41:51.892 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:41:51.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:41:51.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 -- 172.21.15.31:0/1534945290 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f392415dfc0 msgr2=0x7f39240cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 --2- 172.21.15.31:0/1534945290 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f392415dfc0 0x7f39240cc300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3914005fd0 tx=0x7f391402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 -- 172.21.15.31:0/1534945290 shutdown_connections 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 --2- 172.21.15.31:0/1534945290 >> v2:172.21.15.45:3300/0 conn(0x7f39240ca5c0 0x7f39240ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 --2- 172.21.15.31:0/1534945290 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f392415dfc0 0x7f39240cc300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 -- 172.21.15.31:0/1534945290 >> 172.21.15.31:0/1534945290 conn(0x7f39240cc910 msgr2=0x7f39240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 -- 172.21.15.31:0/1534945290 shutdown_connections 2024-09-13T08:41:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.099+0000 7f3929e00640 1 -- 172.21.15.31:0/1534945290 wait complete. 2024-09-13T08:41:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.100+0000 7f3929e00640 1 Processor -- start 2024-09-13T08:41:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 -- start start 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39240ca5c0 0x7f39240c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392415dfc0 0x7f39240b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3924074010 con 0x7f392415dfc0 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3924073e90 con 0x7f39240b1910 2024-09-13T08:41:52.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.101+0000 7f3929e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39241f89d0 con 0x7f39240ca5c0 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:48280/0 (socket says 172.21.15.31:48280) 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 -- 172.21.15.31:0/2341917182 learned_addr learned my addr 172.21.15.31:0/2341917182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f391a000640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39240ca5c0 0x7f39240c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3923400640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392415dfc0 0x7f39240b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 -- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39240ca5c0 msgr2=0x7f39240c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39240ca5c0 0x7f39240c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 -- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392415dfc0 msgr2=0x7f39240b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392415dfc0 0x7f39240b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.102+0000 7f3922a00640 1 -- 172.21.15.31:0/2341917182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39241f8c20 con 0x7f39240b1910 2024-09-13T08:41:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f3922a00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3914034040 tx=0x7f3914005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391403e030 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39140039f0 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914035040 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39241f8f00 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.103+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39241f94c0 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.104+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39241f98a0 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.104+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7f3914041030 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.105+0000 7f391be00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 0x7f3900040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.105+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f3914073ed0 con 0x7f39240b1910 2024-09-13T08:41:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.105+0000 7f391a000640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 0x7f3900040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.106+0000 7f391a000640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 0x7f3900040bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f390c0095f0 tx=0x7f390c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.110+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f39241f98a0 con 0x7f39240b1910 2024-09-13T08:41:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.348+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39240b20d0 con 0x7f39240b1910 2024-09-13T08:41:52.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.350+0000 7f391be00640 1 -- 172.21.15.31:0/2341917182 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7f39140303a0 con 0x7f39240b1910 2024-09-13T08:41:52.351 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:52.351 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:52.351 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 msgr2=0x7f3900040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 0x7f3900040bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f390c0095f0 tx=0x7f390c005f90 comp rx=0 tx=0).stop 2024-09-13T08:41:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 msgr2=0x7f39241f65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3914034040 tx=0x7f3914005e80 comp rx=0 tx=0).stop 2024-09-13T08:41:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 shutdown_connections 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f390003e710 0x7f3900040bd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392415dfc0 0x7f39240b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39240ca5c0 0x7f39240c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 --2- 172.21.15.31:0/2341917182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39240b1910 0x7f39241f65a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 >> 172.21.15.31:0/2341917182 conn(0x7f39240cc910 msgr2=0x7f39240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 shutdown_connections 2024-09-13T08:41:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.352+0000 7f3929e00640 1 -- 172.21.15.31:0/2341917182 wait complete. 2024-09-13T08:41:52.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:52.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 -- 172.21.15.31:0/2509572669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 msgr2=0x7f8e980cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/2509572669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980cf280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009830 tx=0x7f8e8c02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 -- 172.21.15.31:0/2509572669 shutdown_connections 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/2509572669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980cf280 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 -- 172.21.15.31:0/2509572669 >> 172.21.15.31:0/2509572669 conn(0x7f8e980cc910 msgr2=0x7f8e980cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 -- 172.21.15.31:0/2509572669 shutdown_connections 2024-09-13T08:41:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.529+0000 7f8e9fe00640 1 -- 172.21.15.31:0/2509572669 wait complete. 2024-09-13T08:41:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.531+0000 7f8e9fe00640 1 Processor -- start 2024-09-13T08:41:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.531+0000 7f8e9fe00640 1 -- start start 2024-09-13T08:41:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e9815af90 0x7f8e9815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98073f20 con 0x7f8e980cad60 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8e980cbab0 con 0x7f8e980ceea0 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8e980740a0 con 0x7f8e9815af90 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55838/0 (socket says 172.21.15.31:55838) 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9d400640 1 -- 172.21.15.31:0/3383449087 learned_addr learned my addr 172.21.15.31:0/3383449087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9ca00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.532+0000 7f8e9de00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e9815af90 0x7f8e9815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 -- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e9815af90 msgr2=0x7f8e9815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e9815af90 0x7f8e9815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 -- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 msgr2=0x7f8e980b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 -- 172.21.15.31:0/3383449087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e981f5d00 con 0x7f8e980cad60 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9d400640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009950 tx=0x7f8e8c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c040070 con 0x7f8e980cad60 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e8c005310 con 0x7f8e980cad60 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c005c90 con 0x7f8e980cad60 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e980b51a0 con 0x7f8e980cad60 2024-09-13T08:41:52.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.533+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e981f5220 con 0x7f8e980cad60 2024-09-13T08:41:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.534+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7f8e8c039040 con 0x7f8e980cad60 2024-09-13T08:41:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.534+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e980cf280 con 0x7f8e980cad60 2024-09-13T08:41:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.535+0000 7f8e95600640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 0x7f8e78040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.535+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f8e8c074c40 con 0x7f8e980cad60 2024-09-13T08:41:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.535+0000 7f8e9ca00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 0x7f8e78040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.536+0000 7f8e9ca00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 0x7f8e78040bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e980c2c00 tx=0x7f8e84009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.540+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8e8c04b2a0 con 0x7f8e980cad60 2024-09-13T08:41:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.770+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e980b5e10 con 0x7f8e980cad60 2024-09-13T08:41:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.771+0000 7f8e95600640 1 -- 172.21.15.31:0/3383449087 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7f8e8c078070 con 0x7f8e980cad60 2024-09-13T08:41:52.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:52.772 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:52.772 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 msgr2=0x7f8e78040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 0x7f8e78040bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e980c2c00 tx=0x7f8e84009040 comp rx=0 tx=0).stop 2024-09-13T08:41:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 msgr2=0x7f8e981f8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009950 tx=0x7f8e8c032c10 comp rx=0 tx=0).stop 2024-09-13T08:41:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 shutdown_connections 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8e7803e710 0x7f8e78040bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e9815af90 0x7f8e9815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e980ceea0 0x7f8e980b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 --2- 172.21.15.31:0/3383449087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980cad60 0x7f8e981f8ee0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 >> 172.21.15.31:0/3383449087 conn(0x7f8e980cc910 msgr2=0x7f8e9815d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 shutdown_connections 2024-09-13T08:41:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.773+0000 7f8e9fe00640 1 -- 172.21.15.31:0/3383449087 wait complete. 2024-09-13T08:41:52.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.950+0000 7f2c13200640 1 -- 172.21.15.31:0/774040379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 msgr2=0x7f2c140b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.950+0000 7f2c13200640 1 --2- 172.21.15.31:0/774040379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140b3210 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2c00005fd0 tx=0x7f2c0002a4b0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 -- 172.21.15.31:0/774040379 shutdown_connections 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 --2- 172.21.15.31:0/774040379 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c14160180 0x7f2c140ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 --2- 172.21.15.31:0/774040379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140b3210 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 -- 172.21.15.31:0/774040379 >> 172.21.15.31:0/774040379 conn(0x7f2c140d1c70 msgr2=0x7f2c140d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 -- 172.21.15.31:0/774040379 shutdown_connections 2024-09-13T08:41:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.951+0000 7f2c13200640 1 -- 172.21.15.31:0/774040379 wait complete. 2024-09-13T08:41:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.952+0000 7f2c13200640 1 Processor -- start 2024-09-13T08:41:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.952+0000 7f2c13200640 1 -- start start 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c1415f150 0x7f2c140c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c14160180 0x7f2c140cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c140aed80 con 0x7f2c1415f530 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2c14160820 con 0x7f2c14160180 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.953+0000 7f2c13200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2c140aef00 con 0x7f2c1415f150 2024-09-13T08:41:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55852/0 (socket says 172.21.15.31:55852) 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 -- 172.21.15.31:0/3028397353 learned_addr learned my addr 172.21.15.31:0/3028397353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11e00640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c1415f150 0x7f2c140c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c12800640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c14160180 0x7f2c140cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 -- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c1415f150 msgr2=0x7f2c140c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c1415f150 0x7f2c140c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 -- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c14160180 msgr2=0x7f2c140cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c14160180 0x7f2c140cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 -- 172.21.15.31:0/3028397353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1406f330 con 0x7f2c1415f530 2024-09-13T08:41:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.954+0000 7f2c11400640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2c000099f0 tx=0x7f2c00003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.955+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0003f070 con 0x7f2c1415f530 2024-09-13T08:41:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.955+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c000047a0 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.955+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00035030 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.955+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1406f8b0 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.955+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1406ed70 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.956+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c140ae840 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.956+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7f2c00042070 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.956+0000 7f2c0a000640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 0x7f2bf4040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.957+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f2c00074cc0 con 0x7f2c1415f530 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.957+0000 7f2c11e00640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 0x7f2bf4040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.957+0000 7f2c11e00640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 0x7f2bf4040bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2c040017c0 tx=0x7f2c04009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:52.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:52.961+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2c0003acd0 con 0x7f2c1415f530 2024-09-13T08:41:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.166+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c14074940 con 0x7f2c1415f530 2024-09-13T08:41:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.167+0000 7f2c0a000640 1 -- 172.21.15.31:0/3028397353 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+17573 (secure 0 0 0) 0x7f2c0003b460 con 0x7f2c1415f530 2024-09-13T08:41:53.168 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:53.168 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:51.926419+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:41:27.848143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T08:41:28.532134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"10.16","mappings":[{"from":2,"to":7}]},{"pgid":"11.2e","mappings":[{"from":1,"to":2}]},{"pgid":"11.3e","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/583368432":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6801/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2626498261":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1597311998":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2712511176":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2115686593":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/462607798":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1917184858":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6800/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/224785773":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2456757066":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/887477527":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1259704583":"2024-09-14T08:41:50.923556+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 msgr2=0x7f2bf4040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 0x7f2bf4040bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2c040017c0 tx=0x7f2c04009040 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 msgr2=0x7f2c140cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2c000099f0 tx=0x7f2c00003910 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 shutdown_connections 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bf403e710 0x7f2bf4040bd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c14160180 0x7f2c140cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c1415f530 0x7f2c140cc750 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 --2- 172.21.15.31:0/3028397353 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c1415f150 0x7f2c140c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 >> 172.21.15.31:0/3028397353 conn(0x7f2c140d1c70 msgr2=0x7f2c140b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:53.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 shutdown_connections 2024-09-13T08:41:53.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.170+0000 7f2c13200640 1 -- 172.21.15.31:0/3028397353 wait complete. 2024-09-13T08:41:53.187 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:41:53.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 -- 172.21.15.31:0/981917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40b52d0 msgr2=0x7f3ee406f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 --2- 172.21.15.31:0/981917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40b52d0 0x7f3ee406f9b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40ce2e0 tx=0x7f3ed002a530 comp rx=0 tx=0).stop 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 -- 172.21.15.31:0/981917101 shutdown_connections 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 --2- 172.21.15.31:0/981917101 >> v2:172.21.15.45:3300/0 conn(0x7f3ee40ccb00 0x7f3ee4152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 --2- 172.21.15.31:0/981917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40b52d0 0x7f3ee406f9b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 -- 172.21.15.31:0/981917101 >> 172.21.15.31:0/981917101 conn(0x7f3ee406cb00 msgr2=0x7f3ee406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 -- 172.21.15.31:0/981917101 shutdown_connections 2024-09-13T08:41:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.348+0000 7f3ee9800640 1 -- 172.21.15.31:0/981917101 wait complete. 2024-09-13T08:41:53.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.349+0000 7f3ee9800640 1 Processor -- start 2024-09-13T08:41:53.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.349+0000 7f3ee9800640 1 -- start start 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 0x7f3ee40cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ee40cc720 0x7f3ee41f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee4152bc0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee40703b0 con 0x7f3ee40b52d0 2024-09-13T08:41:53.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.350+0000 7f3ee9800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee4152d40 con 0x7f3ee40cc720 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 0x7f3ee40cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55900/0 (socket says 172.21.15.31:55900) 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 -- 172.21.15.31:0/4206291983 learned_addr learned my addr 172.21.15.31:0/4206291983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee3400640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ee40cc720 0x7f3ee41f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 -- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ee40cc720 msgr2=0x7f3ee41f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ee40cc720 0x7f3ee41f8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 -- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 msgr2=0x7f3ee40cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 0x7f3ee40cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 -- 172.21.15.31:0/4206291983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee41f8da0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee3e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 0x7f3ee40cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:41:53.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.351+0000 7f3ee8e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00ea40 tx=0x7f3edc00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.352+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00ce40 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.352+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3edc004c10 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.352+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee40ca620 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.352+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc005020 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.352+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee40cabe0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.353+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee406f9b0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.353+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7f3edc0051c0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.353+0000 7f3edbe00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 0x7f3ec4040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.354+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f3edc0548b0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.354+0000 7f3ee3e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 0x7f3ec4040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.354+0000 7f3ee3e00640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 0x7f3ec4040bd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4007f20 tx=0x7f3ed4007eb0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.358+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3ee41f8da0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.596+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ee406cf80 con 0x7f3ee40ccb00 2024-09-13T08:41:53.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.597+0000 7f3edbe00640 1 -- 172.21.15.31:0/4206291983 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7f3edc00f3c0 con 0x7f3ee40ccb00 2024-09-13T08:41:53.599 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:53.599 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:53.599 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 msgr2=0x7f3ec4040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 0x7f3ec4040bd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4007f20 tx=0x7f3ed4007eb0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 msgr2=0x7f3ee40b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00ea40 tx=0x7f3edc00ef00 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 shutdown_connections 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3ec403e710 0x7f3ec4040bd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ee40ccb00 0x7f3ee40b0640 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ee40cc720 0x7f3ee41f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 --2- 172.21.15.31:0/4206291983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ee40b52d0 0x7f3ee40cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.599+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 >> 172.21.15.31:0/4206291983 conn(0x7f3ee406cb00 msgr2=0x7f3ee40b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:53.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.600+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 shutdown_connections 2024-09-13T08:41:53.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.600+0000 7f3ee9800640 1 -- 172.21.15.31:0/4206291983 wait complete. 2024-09-13T08:41:53.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:41:53.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 -- 172.21.15.31:0/973555097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0cc720 msgr2=0x7f65ec06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 --2- 172.21.15.31:0/973555097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0cc720 0x7f65ec06f9b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009830 tx=0x7f65dc02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 -- 172.21.15.31:0/973555097 shutdown_connections 2024-09-13T08:41:53.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 --2- 172.21.15.31:0/973555097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0cc720 0x7f65ec06f9b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 -- 172.21.15.31:0/973555097 >> 172.21.15.31:0/973555097 conn(0x7f65ec06cb00 msgr2=0x7f65ec06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:53.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 -- 172.21.15.31:0/973555097 shutdown_connections 2024-09-13T08:41:53.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.776+0000 7f65ebe00640 1 -- 172.21.15.31:0/973555097 wait complete. 2024-09-13T08:41:53.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.777+0000 7f65ebe00640 1 Processor -- start 2024-09-13T08:41:53.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 -- start start 2024-09-13T08:41:53.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 0x7f65ec14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 0x7f65ec155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec070320 con 0x7f65ec0ccb00 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec0701a0 con 0x7f65ec0b52d0 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.778+0000 7f65ebe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65ec0704a0 con 0x7f65ec0cc720 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 0x7f65ec14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 0x7f65ec155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65ea000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 0x7f65ec14f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:48356/0 (socket says 172.21.15.31:48356) 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65ea000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:46796/0 (socket says 172.21.15.31:46796) 2024-09-13T08:41:53.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eaa00640 1 -- 172.21.15.31:0/3026281201 learned_addr learned my addr 172.21.15.31:0/3026281201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:53.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 -- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 msgr2=0x7f65ec154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 -- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 msgr2=0x7f65ec14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 0x7f65ec14f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65eb400640 1 -- 172.21.15.31:0/3026281201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec1537b0 con 0x7f65ec0ccb00 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.779+0000 7f65ea000640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65eb400640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 0x7f65ec155170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f65e400cb10 tx=0x7f65e400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4016020 con 0x7f65ec0ccb00 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65e400ad60 con 0x7f65ec0ccb00 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4012680 con 0x7f65ec0ccb00 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ec153a90 con 0x7f65ec0ccb00 2024-09-13T08:41:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.780+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ec154050 con 0x7f65ec0ccb00 2024-09-13T08:41:53.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.781+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ec0c3270 con 0x7f65ec0ccb00 2024-09-13T08:41:53.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.781+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7f65e4019020 con 0x7f65ec0ccb00 2024-09-13T08:41:53.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.782+0000 7f65e2a00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 0x7f65cc040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:53.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.782+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f65e4053f80 con 0x7f65ec0ccb00 2024-09-13T08:41:53.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.782+0000 7f65eaa00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 0x7f65cc040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:53.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.783+0000 7f65eaa00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 0x7f65cc040bd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0095f0 tx=0x7f65dc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:53.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:53.787+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f65e4020020 con 0x7f65ec0ccb00 2024-09-13T08:41:54.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.021+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65ec06e8f0 con 0x7f65ec0ccb00 2024-09-13T08:41:54.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.021+0000 7f65e2a00640 1 -- 172.21.15.31:0/3026281201 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5222 (secure 0 0 0) 0x7f65e401bd90 con 0x7f65ec0ccb00 2024-09-13T08:41:54.024 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:54.024 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"btime":"2024-09-13T08:41:45:208616+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:41:29.577486+0000","modified":"2024-09-13T08:41:32.607487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6162},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6162":{"gid":6162,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1098295543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1098295543},{"type":"v1","addr":"172.21.15.31:6835","nonce":1098295543}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6162,"qdb_cluster":[6162]},"id":5}]} 2024-09-13T08:41:54.024 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 48 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 msgr2=0x7f65cc040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 0x7f65cc040bd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0095f0 tx=0x7f65dc004730 comp rx=0 tx=0).stop 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 msgr2=0x7f65ec155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 0x7f65ec155170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f65e400cb10 tx=0x7f65e400a060 comp rx=0 tx=0).stop 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 shutdown_connections 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65cc03e710 0x7f65cc040bd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec0ccb00 0x7f65ec155170 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65ec0cc720 0x7f65ec154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 --2- 172.21.15.31:0/3026281201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65ec0b52d0 0x7f65ec14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 >> 172.21.15.31:0/3026281201 conn(0x7f65ec06cb00 msgr2=0x7f65ec15d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 shutdown_connections 2024-09-13T08:41:54.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.024+0000 7f65ebe00640 1 -- 172.21.15.31:0/3026281201 wait complete. 2024-09-13T08:41:54.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:54.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 -- 172.21.15.31:0/3393648133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 msgr2=0x7fa34c06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 --2- 172.21.15.31:0/3393648133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c06f9b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009830 tx=0x7fa33c02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 -- 172.21.15.31:0/3393648133 shutdown_connections 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 --2- 172.21.15.31:0/3393648133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c06f9b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 -- 172.21.15.31:0/3393648133 >> 172.21.15.31:0/3393648133 conn(0x7fa34c06cb00 msgr2=0x7fa34c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 -- 172.21.15.31:0/3393648133 shutdown_connections 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.203+0000 7fa352600640 1 -- 172.21.15.31:0/3393648133 wait complete. 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.205+0000 7fa352600640 1 Processor -- start 2024-09-13T08:41:54.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.205+0000 7fa352600640 1 -- start start 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c0c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 0x7fa34c0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c070320 con 0x7fa34c0b52d0 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c0701a0 con 0x7fa34c0cc720 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa352600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa34c0704a0 con 0x7fa34c0ccb00 2024-09-13T08:41:54.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa351200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa351200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55926/0 (socket says 172.21.15.31:55926) 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa351200640 1 -- 172.21.15.31:0/946669489 learned_addr learned my addr 172.21.15.31:0/946669489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa343e00640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c0c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.206+0000 7fa351c00640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 0x7fa34c0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 -- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 msgr2=0x7fa34c0c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 0x7fa34c0c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 -- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 msgr2=0x7fa34c0c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c0c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 -- 172.21.15.31:0/946669489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c1443f0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351c00640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 0x7fa34c0c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa351200640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa33c000c00 tx=0x7fa33c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c040070 con 0x7fa34c0b52d0 2024-09-13T08:41:54.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa33c0038e0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c003d30 con 0x7fa34c0b52d0 2024-09-13T08:41:54.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34c141810 con 0x7fa34c0b52d0 2024-09-13T08:41:54.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.207+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa34c141dd0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.208+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34c1530a0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.208+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 14) ==== 51000+0+0 (secure 0 0 0) 0x7fa33c039040 con 0x7fa34c0b52d0 2024-09-13T08:41:54.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.209+0000 7fa349600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 0x7fa328040bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.209+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7fa33c075150 con 0x7fa34c0b52d0 2024-09-13T08:41:54.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.209+0000 7fa343e00640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 0x7fa328040bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.210+0000 7fa343e00640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 0x7fa328040bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa34c15be40 tx=0x7fa334009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:54.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.214+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa33c030a00 con 0x7fa34c0b52d0 2024-09-13T08:41:54.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.252+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 15) ==== 51056+0+0 (secure 0 0 0) 0x7fa33c03c420 con 0x7fa34c0b52d0 2024-09-13T08:41:54.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.415+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa34c141fa0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.417+0000 7fa349600640 1 -- 172.21.15.31:0/946669489 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+17573 (secure 0 0 0) 0x7fa33c04aca0 con 0x7fa34c0b52d0 2024-09-13T08:41:54.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:54.419 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":78,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:51.926419+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:41:27.848143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T08:41:28.532134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"10.16","mappings":[{"from":2,"to":7}]},{"pgid":"11.2e","mappings":[{"from":1,"to":2}]},{"pgid":"11.3e","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/583368432":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6801/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2626498261":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1597311998":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2712511176":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2115686593":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/462607798":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1917184858":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6800/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/224785773":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2456757066":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/887477527":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1259704583":"2024-09-14T08:41:50.923556+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:54.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 msgr2=0x7fa328040bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 0x7fa328040bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa34c15be40 tx=0x7fa334009040 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 msgr2=0x7fa34c15b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa33c000c00 tx=0x7fa33c032c10 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 shutdown_connections 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa32803e710 0x7fa328040bd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa34c0ccb00 0x7fa34c0c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa34c0cc720 0x7fa34c0c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 --2- 172.21.15.31:0/946669489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa34c0b52d0 0x7fa34c15b0b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 >> 172.21.15.31:0/946669489 conn(0x7fa34c06cb00 msgr2=0x7fa34c15e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:54.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.419+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 shutdown_connections 2024-09-13T08:41:54.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.420+0000 7fa352600640 1 -- 172.21.15.31:0/946669489 wait complete. 2024-09-13T08:41:54.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:41:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 -- 172.21.15.31:0/2522234547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 msgr2=0x7f0f580cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/2522234547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f580cdb80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009830 tx=0x7f0f4802a510 comp rx=0 tx=0).stop 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 -- 172.21.15.31:0/2522234547 shutdown_connections 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/2522234547 >> v2:172.21.15.45:3300/0 conn(0x7f0f58073860 0x7f0f58073d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/2522234547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f580cdb80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 -- 172.21.15.31:0/2522234547 >> 172.21.15.31:0/2522234547 conn(0x7f0f580b18f0 msgr2=0x7f0f580b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 -- 172.21.15.31:0/2522234547 shutdown_connections 2024-09-13T08:41:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.596+0000 7f0f5ea00640 1 -- 172.21.15.31:0/2522234547 wait complete. 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.597+0000 7f0f5ea00640 1 Processor -- start 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 -- start start 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f580ced20 0x7f0f58159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f581f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0f580733e0 con 0x7f0f58073860 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f580ce9e0 con 0x7f0f580ced20 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.598+0000 7f0f5ea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0f58074250 con 0x7f0f5815be30 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55936/0 (socket says 172.21.15.31:55936) 2024-09-13T08:41:54.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 -- 172.21.15.31:0/532879651 learned_addr learned my addr 172.21.15.31:0/532879651 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f5ca00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f581f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f4f400640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f580ced20 0x7f0f58159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 -- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 msgr2=0x7f0f581f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f581f7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 -- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f580ced20 msgr2=0x7f0f58159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f580ced20 0x7f0f58159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 -- 172.21.15.31:0/532879651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5815a790 con 0x7f0f58073860 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.599+0000 7f0f57e00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0f4802ab00 tx=0x7f0f48033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.600+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48005c90 con 0x7f0f58073860 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.600+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f48031e50 con 0x7f0f58073860 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.600+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f5815aa70 con 0x7f0f58073860 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.600+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48005260 con 0x7f0f58073860 2024-09-13T08:41:54.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.600+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f5815b030 con 0x7f0f58073860 2024-09-13T08:41:54.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.601+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f58074740 con 0x7f0f58073860 2024-09-13T08:41:54.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.601+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 51056+0+0 (secure 0 0 0) 0x7f0f48005400 con 0x7f0f58073860 2024-09-13T08:41:54.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.602+0000 7f0f54c00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 0x7f0f34040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:54.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.602+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6355+0+0 (secure 0 0 0) 0x7f0f48074ba0 con 0x7f0f58073860 2024-09-13T08:41:54.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.602+0000 7f0f4f400640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 0x7f0f34040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:54.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.602+0000 7f0f4f400640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 0x7f0f34040c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f40007fd0 tx=0x7f0f40001410 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:54.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.606+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0f4803f030 con 0x7f0f58073860 2024-09-13T08:41:54.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.837+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0f580b25f0 con 0x7f0f58073860 2024-09-13T08:41:54.840 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:41:54.838+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:41:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.840+0000 7f0f54c00640 1 -- 172.21.15.31:0/532879651 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) ==== 161+0+0 (secure 0 0 0) 0x7f0f48033e10 con 0x7f0f58073860 2024-09-13T08:41:54.842 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:41:54.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 msgr2=0x7f0f34040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 0x7f0f34040c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f40007fd0 tx=0x7f0f40001410 comp rx=0 tx=0).stop 2024-09-13T08:41:54.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 msgr2=0x7f0f5815ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0f4802ab00 tx=0x7f0f48033c20 comp rx=0 tx=0).stop 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 shutdown_connections 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0f3403e7a0 0x7f0f34040c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0f5815be30 0x7f0f581f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f580ced20 0x7f0f58159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 --2- 172.21.15.31:0/532879651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f58073860 0x7f0f5815ef00 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.843+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 >> 172.21.15.31:0/532879651 conn(0x7f0f580b18f0 msgr2=0x7f0f580cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.844+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 shutdown_connections 2024-09-13T08:41:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:54.844+0000 7f0f5ea00640 1 -- 172.21.15.31:0/532879651 wait complete. 2024-09-13T08:41:54.856 INFO:tasks.ceph.mds.c.smithi031.stderr: -8477> 2024-09-13T08:41:36.123+0000 7fd149e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:41:54.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:41:54.912 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:41:55.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 -- 172.21.15.31:0/3406739181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 msgr2=0x7f79ac0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:55.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 --2- 172.21.15.31:0/3406739181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac0d0690 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f79a0005fd0 tx=0x7f79a002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 -- 172.21.15.31:0/3406739181 shutdown_connections 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 --2- 172.21.15.31:0/3406739181 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 0x7f79ac0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 --2- 172.21.15.31:0/3406739181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac0d0690 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 -- 172.21.15.31:0/3406739181 >> 172.21.15.31:0/3406739181 conn(0x7f79ac0cfb70 msgr2=0x7f79ac0cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 -- 172.21.15.31:0/3406739181 shutdown_connections 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.028+0000 7f79b4400640 1 -- 172.21.15.31:0/3406739181 wait complete. 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.030+0000 7f79b4400640 1 Processor -- start 2024-09-13T08:41:55.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.030+0000 7f79b4400640 1 -- start start 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 0x7f79ac152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79ac0cb570 0x7f79ac0c5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac06ff20 con 0x7f79ac15c040 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac06fda0 con 0x7f79ac0cb570 2024-09-13T08:41:55.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b4400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f79ac0700a0 con 0x7f79ac072560 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39798/0 (socket says 172.21.15.31:39798) 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 -- 172.21.15.31:0/868149180 learned_addr learned my addr 172.21.15.31:0/868149180 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79ac0cb570 0x7f79ac0c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b1a00640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 0x7f79ac152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 -- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 msgr2=0x7f79ac152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 0x7f79ac152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 -- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79ac0cb570 msgr2=0x7f79ac0c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79ac0cb570 0x7f79ac0c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.031+0000 7f79b2400640 1 -- 172.21.15.31:0/868149180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac0c5900 con 0x7f79ac15c040 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79b2400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c770 tx=0x7f79a400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4017070 con 0x7f79ac15c040 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79a4005070 con 0x7f79ac15c040 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a40054c0 con 0x7f79ac15c040 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac0c5be0 con 0x7f79ac15c040 2024-09-13T08:41:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.032+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ac0c61a0 con 0x7f79ac15c040 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.033+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ac0cbd10 con 0x7f79ac15c040 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.034+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 51056+0+0 (secure 0 0 0) 0x7f79a401a070 con 0x7f79ac15c040 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.034+0000 7f79a9600640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 0x7f798c040c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.034+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6441+0+0 (secure 0 0 0) 0x7f79a4053300 con 0x7f79ac15c040 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.034+0000 7f79b1a00640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 0x7f798c040c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:55.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.035+0000 7f79b1a00640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 0x7f798c040c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79a0033000 tx=0x7f79a0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:55.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.039+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f79a400ab00 con 0x7f79ac15c040 2024-09-13T08:41:55.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.271+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f79ac0af5a0 con 0x7f79ac15c040 2024-09-13T08:41:55.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.845+0000 7f79a9600640 1 -- 172.21.15.31:0/868149180 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) ==== 104+0+0 (secure 0 0 0) 0x7f79a4004cb0 con 0x7f79ac15c040 2024-09-13T08:41:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 msgr2=0x7f798c040c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 0x7f798c040c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79a0033000 tx=0x7f79a0004730 comp rx=0 tx=0).stop 2024-09-13T08:41:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 msgr2=0x7f79ac15e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c770 tx=0x7f79a400cc30 comp rx=0 tx=0).stop 2024-09-13T08:41:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 shutdown_connections 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f798c03e7d0 0x7f798c040c90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79ac15c040 0x7f79ac15e4a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79ac0cb570 0x7f79ac0c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 --2- 172.21.15.31:0/868149180 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f79ac072560 0x7f79ac152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 >> 172.21.15.31:0/868149180 conn(0x7f79ac0cfb70 msgr2=0x7f79ac0c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 shutdown_connections 2024-09-13T08:41:55.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:55.849+0000 7f79b4400640 1 -- 172.21.15.31:0/868149180 wait complete. 2024-09-13T08:41:55.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:41:56.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 -- 172.21.15.31:0/2583131710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cd0004800 msgr2=0x7f5cd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:56.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 --2- 172.21.15.31:0/2583131710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cd0004800 0x7f5cd0004c80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009b80 tx=0x7f5cd0005830 comp rx=0 tx=0).stop 2024-09-13T08:41:56.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cede00640 1 -- 172.21.15.31:0/2583131710 reap_dead start 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 -- 172.21.15.31:0/2583131710 shutdown_connections 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 -- 172.21.15.31:0/2583131710 >> 172.21.15.31:0/2583131710 conn(0x7f5ce80cfbb0 msgr2=0x7f5ce80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 -- 172.21.15.31:0/2583131710 shutdown_connections 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.037+0000 7f5cefe00640 1 -- 172.21.15.31:0/2583131710 wait complete. 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.038+0000 7f5cefe00640 1 Processor -- start 2024-09-13T08:41:56.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.039+0000 7f5cefe00640 1 -- start start 2024-09-13T08:41:56.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.039+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd0004800 0x7f5ce80c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:56.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.039+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:56.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ce81451a0 0x7f5ce81e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:56.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5cefe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce80b3270 con 0x7f5ce8157120 2024-09-13T08:41:56.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5cefe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce80b30f0 con 0x7f5cd0004800 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5cefe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce80b33f0 con 0x7f5ce81451a0 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39826/0 (socket says 172.21.15.31:39826) 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 -- 172.21.15.31:0/3688345858 learned_addr learned my addr 172.21.15.31:0/3688345858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5cede00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ce81451a0 0x7f5ce81e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ced400640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd0004800 0x7f5ce80c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 -- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ce81451a0 msgr2=0x7f5ce81e8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ce81451a0 0x7f5ce81e8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 -- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd0004800 msgr2=0x7f5ce80c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd0004800 0x7f5ce80c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.040+0000 7f5ceca00640 1 -- 172.21.15.31:0/3688345858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce80c6c80 con 0x7f5ce8157120 2024-09-13T08:41:56.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5ceca00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009ca0 tx=0x7f5cd4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4038040 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cd402b7e0 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce80c6f20 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4004280 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.041+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce80c74e0 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.042+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 51056+0+0 (secure 0 0 0) 0x7f5cd4045020 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.042+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce8072560 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.043+0000 7f5ce5600640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 0x7f5cc8040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.043+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6441+0+0 (secure 0 0 0) 0x7f5cd4074270 con 0x7f5ce8157120 2024-09-13T08:41:56.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.043+0000 7f5ced400640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 0x7f5cc8040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:56.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.044+0000 7f5ced400640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 0x7f5cc8040c60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005fd0 tx=0x7f5cdc009500 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:56.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.048+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5cd4033760 con 0x7f5ce8157120 2024-09-13T08:41:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.245+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5ce8148050 con 0x7f5ce8157120 2024-09-13T08:41:56.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.264+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f5cd4033ae0 con 0x7f5ce8157120 2024-09-13T08:41:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.847+0000 7f5ce5600640 1 -- 172.21.15.31:0/3688345858 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v80) ==== 181+0+0 (secure 0 0 0) 0x7f5cd406b030 con 0x7f5ce8157120 2024-09-13T08:41:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:41:56.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.852+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 msgr2=0x7f5cc8040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:56.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.852+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 0x7f5cc8040c60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005fd0 tx=0x7f5cdc009500 comp rx=0 tx=0).stop 2024-09-13T08:41:56.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.852+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 msgr2=0x7f5ce8147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:56.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.852+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009ca0 tx=0x7f5cd4033670 comp rx=0 tx=0).stop 2024-09-13T08:41:56.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 shutdown_connections 2024-09-13T08:41:56.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5cc803e7a0 0x7f5cc8040c60 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005fd0 tx=0x7f5cdc009500 comp rx=0 tx=0).stop 2024-09-13T08:41:56.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5ce81451a0 0x7f5ce81e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce8157120 0x7f5ce8147b10 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 --2- 172.21.15.31:0/3688345858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd0004800 0x7f5ce80c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 >> 172.21.15.31:0/3688345858 conn(0x7f5ce80cfbb0 msgr2=0x7f5ce815eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 shutdown_connections 2024-09-13T08:41:56.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:56.853+0000 7f5cefe00640 1 -- 172.21.15.31:0/3688345858 wait complete. 2024-09-13T08:41:56.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:41:57.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 -- 172.21.15.31:0/157146421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f764c0040b0 msgr2=0x7f764c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:57.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 --2- 172.21.15.31:0/157146421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f764c0040b0 0x7f764c004800 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7654001eb0 tx=0x7f764c005830 comp rx=0 tx=0).stop 2024-09-13T08:41:57.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766a800640 1 -- 172.21.15.31:0/157146421 reap_dead start 2024-09-13T08:41:57.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 -- 172.21.15.31:0/157146421 shutdown_connections 2024-09-13T08:41:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 -- 172.21.15.31:0/157146421 >> 172.21.15.31:0/157146421 conn(0x7f76640b1520 msgr2=0x7f76640b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 -- 172.21.15.31:0/157146421 shutdown_connections 2024-09-13T08:41:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.045+0000 7f766b200640 1 -- 172.21.15.31:0/157146421 wait complete. 2024-09-13T08:41:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.047+0000 7f766b200640 1 Processor -- start 2024-09-13T08:41:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.047+0000 7f766b200640 1 -- start start 2024-09-13T08:41:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f764c0040b0 0x7f766415cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76640c7d60 0x7f76640ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76640cc850 con 0x7f76640c78e0 2024-09-13T08:41:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7664073de0 con 0x7f764c0040b0 2024-09-13T08:41:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f76640cc9d0 con 0x7f76640c7d60 2024-09-13T08:41:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f7669400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f7669400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39842/0 (socket says 172.21.15.31:39842) 2024-09-13T08:41:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f7669400640 1 -- 172.21.15.31:0/3206250703 learned_addr learned my addr 172.21.15.31:0/3206250703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f7669e00640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f764c0040b0 0x7f766415cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f766a800640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76640c7d60 0x7f76640ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.048+0000 7f7669400640 1 -- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76640c7d60 msgr2=0x7f76640ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f7669400640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76640c7d60 0x7f76640ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f7669400640 1 -- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f764c0040b0 msgr2=0x7f766415cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f7669400640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f764c0040b0 0x7f766415cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f7669400640 1 -- 172.21.15.31:0/3206250703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76640c8280 con 0x7f76640c78e0 2024-09-13T08:41:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f7669400640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f765402b2d0 tx=0x7f7654006830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654039070 con 0x7f76640c78e0 2024-09-13T08:41:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76540035f0 con 0x7f76640c78e0 2024-09-13T08:41:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766415c1c0 con 0x7f76640c78e0 2024-09-13T08:41:57.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654003a20 con 0x7f76640c78e0 2024-09-13T08:41:57.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.049+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766415c780 con 0x7f76640c78e0 2024-09-13T08:41:57.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.051+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766406fc80 con 0x7f76640c78e0 2024-09-13T08:41:57.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.052+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f7654045020 con 0x7f76640c78e0 2024-09-13T08:41:57.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.053+0000 7f765e000640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 0x7f764807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:57.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.053+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5974+0+0 (secure 0 0 0) 0x7f76540ba860 con 0x7f76640c78e0 2024-09-13T08:41:57.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.053+0000 7f7669e00640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 0x7f764807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:57.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.054+0000 7f7669e00640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 0x7f764807ae30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f766415f3d0 tx=0x7f765800b260 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:57.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.057+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f766415bfe0 con 0x7f76640c78e0 2024-09-13T08:41:57.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.254+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f76640b3e00 con 0x7f76640c78e0 2024-09-13T08:41:57.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.853+0000 7f765e000640 1 -- 172.21.15.31:0/3206250703 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v81) ==== 169+0+0 (secure 0 0 0) 0x7f76540824b0 con 0x7f76640c78e0 2024-09-13T08:41:57.855 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:41:57.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.857+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 msgr2=0x7f764807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:57.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.857+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 0x7f764807ae30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f766415f3d0 tx=0x7f765800b260 comp rx=0 tx=0).stop 2024-09-13T08:41:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.857+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 msgr2=0x7f76641f4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.857+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f765402b2d0 tx=0x7f7654006830 comp rx=0 tx=0).stop 2024-09-13T08:41:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 shutdown_connections 2024-09-13T08:41:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7648078970 0x7f764807ae30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76640c7d60 0x7f76640ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76640c78e0 0x7f76641f4a80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 --2- 172.21.15.31:0/3206250703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f764c0040b0 0x7f766415cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:57.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 >> 172.21.15.31:0/3206250703 conn(0x7f76640b1520 msgr2=0x7f76640b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:57.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 shutdown_connections 2024-09-13T08:41:57.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:57.858+0000 7f766b200640 1 -- 172.21.15.31:0/3206250703 wait complete. 2024-09-13T08:41:57.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.041+0000 7f1b4aa00640 1 -- 172.21.15.31:0/1640872126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 msgr2=0x7f1b440cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.041+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/1640872126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440cee40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1b440ceda0 tx=0x7f1b3802a4b0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 -- 172.21.15.31:0/1640872126 shutdown_connections 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/1640872126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b4406e670 0x7f1b440cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/1640872126 >> v2:172.21.15.45:3300/0 conn(0x7f1b440ce4b0 0x7f1b4415f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/1640872126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440cee40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 -- 172.21.15.31:0/1640872126 >> 172.21.15.31:0/1640872126 conn(0x7f1b440cc7a0 msgr2=0x7f1b440ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:58.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 -- 172.21.15.31:0/1640872126 shutdown_connections 2024-09-13T08:41:58.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.042+0000 7f1b4aa00640 1 -- 172.21.15.31:0/1640872126 wait complete. 2024-09-13T08:41:58.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.043+0000 7f1b4aa00640 1 Processor -- start 2024-09-13T08:41:58.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.043+0000 7f1b4aa00640 1 -- start start 2024-09-13T08:41:58.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 0x7f1b441f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 0x7f1b440cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b44160170 con 0x7f1b4415bb70 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4415fff0 con 0x7f1b4406e670 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.044+0000 7f1b4aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4406fec0 con 0x7f1b440ce4b0 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39862/0 (socket says 172.21.15.31:39862) 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 -- 172.21.15.31:0/4157245988 learned_addr learned my addr 172.21.15.31:0/4157245988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b48c00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 0x7f1b440cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b49600640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 0x7f1b441f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 -- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 msgr2=0x7f1b440cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 0x7f1b440cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 -- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 msgr2=0x7f1b441f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 0x7f1b441f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 -- 172.21.15.31:0/4157245988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b441f5200 con 0x7f1b4415bb70 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b49600640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 0x7f1b441f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b48c00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 0x7f1b440cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.045+0000 7f1b4a000640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00eb10 tx=0x7f1b3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.046+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c004b20 con 0x7f1b4415bb70 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.046+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b3c004ce0 con 0x7f1b4415bb70 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.046+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b441f54e0 con 0x7f1b4415bb70 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.046+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c005280 con 0x7f1b4415bb70 2024-09-13T08:41:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.046+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b440cb530 con 0x7f1b4415bb70 2024-09-13T08:41:58.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.047+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b4415f820 con 0x7f1b4415bb70 2024-09-13T08:41:58.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.048+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f1b3c005420 con 0x7f1b4415bb70 2024-09-13T08:41:58.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.049+0000 7f1b41600640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 0x7f1b2407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.049+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f1b3c09b680 con 0x7f1b4415bb70 2024-09-13T08:41:58.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.049+0000 7f1b49600640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 0x7f1b2407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.050+0000 7f1b49600640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 0x7f1b2407ae30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b38034000 tx=0x7f1b38004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:58.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.054+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1b3c063c30 con 0x7f1b4415bb70 2024-09-13T08:41:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.265+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f1b440cf140 con 0x7f1b4415bb70 2024-09-13T08:41:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.269+0000 7f1b41600640 1 -- 172.21.15.31:0/4157245988 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f1b3c063530 con 0x7f1b4415bb70 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 msgr2=0x7f1b2407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 0x7f1b2407ae30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b38034000 tx=0x7f1b38004730 comp rx=0 tx=0).stop 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 msgr2=0x7f1b440caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00eb10 tx=0x7f1b3c00c2d0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 shutdown_connections 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b24078970 0x7f1b2407ae30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b4415bb70 0x7f1b440caff0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b440ce4b0 0x7f1b440cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 --2- 172.21.15.31:0/4157245988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4406e670 0x7f1b441f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 >> 172.21.15.31:0/4157245988 conn(0x7f1b440cc7a0 msgr2=0x7f1b44159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 shutdown_connections 2024-09-13T08:41:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.272+0000 7f1b4aa00640 1 -- 172.21.15.31:0/4157245988 wait complete. 2024-09-13T08:41:58.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.449+0000 7f1b89400640 1 -- 172.21.15.31:0/1352766426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b8415dfa0 msgr2=0x7f1b840d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.449+0000 7f1b89400640 1 --2- 172.21.15.31:0/1352766426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b8415dfa0 0x7f1b840d2500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009830 tx=0x7f1b7402a510 comp rx=0 tx=0).stop 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 -- 172.21.15.31:0/1352766426 shutdown_connections 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 --2- 172.21.15.31:0/1352766426 >> v2:172.21.15.45:3300/0 conn(0x7f1b840d42e0 0x7f1b840b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 --2- 172.21.15.31:0/1352766426 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b8415dfa0 0x7f1b840d2500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 -- 172.21.15.31:0/1352766426 >> 172.21.15.31:0/1352766426 conn(0x7f1b8408d8b0 msgr2=0x7f1b8409d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 -- 172.21.15.31:0/1352766426 shutdown_connections 2024-09-13T08:41:58.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.450+0000 7f1b89400640 1 -- 172.21.15.31:0/1352766426 wait complete. 2024-09-13T08:41:58.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.451+0000 7f1b89400640 1 Processor -- start 2024-09-13T08:41:58.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.451+0000 7f1b89400640 1 -- start start 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 0x7f1b8415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b840d42e0 0x7f1b84161930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b840b1900 con 0x7f1b8415dfa0 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1b840b1780 con 0x7f1b840d42e0 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.452+0000 7f1b89400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1b840b1a80 con 0x7f1b840b47f0 2024-09-13T08:41:58.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39874/0 (socket says 172.21.15.31:39874) 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 -- 172.21.15.31:0/3567277279 learned_addr learned my addr 172.21.15.31:0/3567277279 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b83e00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 0x7f1b8415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b7be00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b840d42e0 0x7f1b84161930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 -- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 msgr2=0x7f1b8415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 0x7f1b8415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 -- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b840d42e0 msgr2=0x7f1b84161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b840d42e0 0x7f1b84161930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 -- 172.21.15.31:0/3567277279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b840cb1d0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b83e00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 0x7f1b8415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.453+0000 7f1b88a00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00eb10 tx=0x7f1b7c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.454+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c004b20 con 0x7f1b8415dfa0 2024-09-13T08:41:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.454+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b7c004ce0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.454+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c005280 con 0x7f1b8415dfa0 2024-09-13T08:41:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.454+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b840cb4b0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.454+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b840cba70 con 0x7f1b8415dfa0 2024-09-13T08:41:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.455+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b840b1f30 con 0x7f1b8415dfa0 2024-09-13T08:41:58.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.456+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f1b7c005420 con 0x7f1b8415dfa0 2024-09-13T08:41:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.457+0000 7f1b80c00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 0x7f1b6007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.457+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f1b7c09b8a0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.457+0000 7f1b83e00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 0x7f1b6007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:58.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.458+0000 7f1b83e00640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 0x7f1b6007ae30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b74034000 tx=0x7f1b74004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.461+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1b7c063db0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.729+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f1b840cbc40 con 0x7f1b8415dfa0 2024-09-13T08:41:58.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.855+0000 7f1b80c00640 1 -- 172.21.15.31:0/3567277279 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v265) ==== 131+0+0 (secure 0 0 0) 0x7f1b7c0636b0 con 0x7f1b8415dfa0 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 msgr2=0x7f1b6007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 0x7f1b6007ae30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b74034000 tx=0x7f1b74004730 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 msgr2=0x7f1b84161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00eb10 tx=0x7f1b7c00c2d0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 shutdown_connections 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b60078970 0x7f1b6007ae30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b8415dfa0 0x7f1b84161e70 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b840d42e0 0x7f1b84161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 --2- 172.21.15.31:0/3567277279 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b840b47f0 0x7f1b8415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:58.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 >> 172.21.15.31:0/3567277279 conn(0x7f1b8408d8b0 msgr2=0x7f1b840d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:58.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 shutdown_connections 2024-09-13T08:41:58.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:58.858+0000 7f1b89400640 1 -- 172.21.15.31:0/3567277279 wait complete. 2024-09-13T08:41:58.875 INFO:tasks.cephfs_test_runner:test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-13T08:41:58.876 INFO:tasks.cephfs_test_runner:Starting test: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) Test the creating vols, subvols subvolgroups fails when their names uses 2024-09-13T08:41:58.876 INFO:tasks.cephfs_test_runner:test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:41:58.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 -- 172.21.15.31:0/2699667299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 msgr2=0x7fd058152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 --2- 172.21.15.31:0/2699667299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd058152e50 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd050005fd0 tx=0x7fd05002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 -- 172.21.15.31:0/2699667299 shutdown_connections 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 --2- 172.21.15.31:0/2699667299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd058152e50 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 -- 172.21.15.31:0/2699667299 >> 172.21.15.31:0/2699667299 conn(0x7fd0580cc910 msgr2=0x7fd0580cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 -- 172.21.15.31:0/2699667299 shutdown_connections 2024-09-13T08:41:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.035+0000 7fd05e400640 1 -- 172.21.15.31:0/2699667299 wait complete. 2024-09-13T08:41:59.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.037+0000 7fd05e400640 1 Processor -- start 2024-09-13T08:41:59.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.037+0000 7fd05e400640 1 -- start start 2024-09-13T08:41:59.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 0x7fd058156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 0x7fd0580b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0580b1ca0 con 0x7fd0580746d0 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0580b1b20 con 0x7fd058152a70 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd05e400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd0580b1e20 con 0x7fd0580cf980 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd057400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd057400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39892/0 (socket says 172.21.15.31:39892) 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd057400640 1 -- 172.21.15.31:0/4275753443 learned_addr learned my addr 172.21.15.31:0/4275753443 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd057e00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 0x7fd0580b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.038+0000 7fd04ea00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 0x7fd058156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 -- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 msgr2=0x7fd058156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 0x7fd058156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 -- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 msgr2=0x7fd0580b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 0x7fd0580b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 -- 172.21.15.31:0/4275753443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0580b3870 con 0x7fd0580746d0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd04ea00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 0x7fd058156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057e00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 0x7fd0580b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd057400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c450 tx=0x7fd04800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048016020 con 0x7fd0580746d0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd048012920 con 0x7fd0580746d0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048012d70 con 0x7fd0580746d0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0580b3b50 con 0x7fd0580746d0 2024-09-13T08:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.039+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0580b5800 con 0x7fd0580746d0 2024-09-13T08:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.041+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0580b2280 con 0x7fd0580746d0 2024-09-13T08:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.042+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fd048019020 con 0x7fd0580746d0 2024-09-13T08:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.043+0000 7fd04fe00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 0x7fd03407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.043+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7fd04809aa50 con 0x7fd0580746d0 2024-09-13T08:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.043+0000 7fd04ea00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 0x7fd03407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.044+0000 7fd04ea00640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 0x7fd03407ae60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0580ad550 tx=0x7fd040009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.047+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd048020020 con 0x7fd0580746d0 2024-09-13T08:41:59.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.241+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0580ad130 con 0x7fd0580746d0 2024-09-13T08:41:59.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.242+0000 7fd04fe00640 1 -- 172.21.15.31:0/4275753443 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+13793 (secure 0 0 0) 0x7fd048062f70 con 0x7fd0580746d0 2024-09-13T08:41:59.244 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:41:59.244 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:41:57.849657+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6833/1098295543":"2024-09-14T08:41:54.839808+0000","172.21.15.45:0/1259704583":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/887477527":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2456757066":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/462607798":"2024-09-14T08:41:50.923556+0000","172.21.15.31:6835/1098295543":"2024-09-14T08:41:54.839808+0000","172.21.15.45:0/1917184858":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2115686593":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/1597311998":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2626498261":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6801/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/224785773":"2024-09-14T08:41:50.923556+0000","172.21.15.45:6800/4176094944":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/2712511176":"2024-09-14T08:41:50.923556+0000","172.21.15.45:0/583368432":"2024-09-14T08:41:50.923556+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 msgr2=0x7fd03407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 0x7fd03407ae60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0580ad550 tx=0x7fd040009040 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 msgr2=0x7fd0581524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c450 tx=0x7fd04800c910 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 shutdown_connections 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0340789a0 0x7fd03407ae60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd058152a70 0x7fd0580b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd0580cf980 0x7fd058156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 --2- 172.21.15.31:0/4275753443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0580746d0 0x7fd0581524e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.244+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 >> 172.21.15.31:0/4275753443 conn(0x7fd0580cc910 msgr2=0x7fd0580cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.245+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 shutdown_connections 2024-09-13T08:41:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.245+0000 7fd05e400640 1 -- 172.21.15.31:0/4275753443 wait complete. 2024-09-13T08:41:59.263 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:41:59.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 -- 172.21.15.31:0/4219194680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0258003a30 msgr2=0x7f0258003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 --2- 172.21.15.31:0/4219194680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0258003a30 0x7f0258003e50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0264009b50 tx=0x7f0258005830 comp rx=0 tx=0).stop 2024-09-13T08:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f0273e00640 1 -- 172.21.15.31:0/4219194680 reap_dead start 2024-09-13T08:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 -- 172.21.15.31:0/4219194680 shutdown_connections 2024-09-13T08:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 -- 172.21.15.31:0/4219194680 >> 172.21.15.31:0/4219194680 conn(0x7f02740cfbb0 msgr2=0x7f02740cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 -- 172.21.15.31:0/4219194680 shutdown_connections 2024-09-13T08:41:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.426+0000 7f027a400640 1 -- 172.21.15.31:0/4219194680 wait complete. 2024-09-13T08:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.428+0000 7f027a400640 1 Processor -- start 2024-09-13T08:41:59.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.428+0000 7f027a400640 1 -- start start 2024-09-13T08:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0258003a30 0x7f02741f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0274150050 0x7f0274152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02740cbfd0 con 0x7f02741e8850 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02740cbe50 con 0x7f0274150050 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f027a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0274072560 con 0x7f0258003a30 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f0273e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f0273e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39906/0 (socket says 172.21.15.31:39906) 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f0273e00640 1 -- 172.21.15.31:0/1159915067 learned_addr learned my addr 172.21.15.31:0/1159915067 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f026aa00640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0274150050 0x7f0274152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.429+0000 7f0273400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0258003a30 0x7f02741f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 -- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0258003a30 msgr2=0x7f02741f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0258003a30 0x7f02741f5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 -- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0274150050 msgr2=0x7f0274152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0274150050 0x7f0274152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 -- 172.21.15.31:0/1159915067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0274158440 con 0x7f02741e8850 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f0273e00640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f02740cc600 tx=0x7f026c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c015070 con 0x7f02741e8850 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f026c00cb10 con 0x7f02741e8850 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c011460 con 0x7f02741e8850 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0274151690 con 0x7f02741e8850 2024-09-13T08:41:59.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.430+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0274151c50 con 0x7f02741e8850 2024-09-13T08:41:59.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.431+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02741e9210 con 0x7f02741e8850 2024-09-13T08:41:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.432+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f026c018070 con 0x7f02741e8850 2024-09-13T08:41:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.433+0000 7f026be00640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 0x7f025007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.433+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f026c09afe0 con 0x7f02741e8850 2024-09-13T08:41:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.434+0000 7f0273400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 0x7f025007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.434+0000 7f0273400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 0x7f025007ae30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0264035000 tx=0x7f026402d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.437+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f026c062ea0 con 0x7f02741e8850 2024-09-13T08:41:59.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.632+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02740cb740 con 0x7f02741e8850 2024-09-13T08:41:59.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.632+0000 7f026be00640 1 -- 172.21.15.31:0/1159915067 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f026c0678d0 con 0x7f02741e8850 2024-09-13T08:41:59.634 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:41:59.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 msgr2=0x7f025007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 0x7f025007ae30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0264035000 tx=0x7f026402d040 comp rx=0 tx=0).stop 2024-09-13T08:41:59.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 msgr2=0x7f02741e8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f02740cc600 tx=0x7f026c00ed10 comp rx=0 tx=0).stop 2024-09-13T08:41:59.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 shutdown_connections 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0250078970 0x7f025007ae30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02741e8850 0x7f02741e8cd0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0274150050 0x7f0274152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 --2- 172.21.15.31:0/1159915067 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0258003a30 0x7f02741f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 >> 172.21.15.31:0/1159915067 conn(0x7f02740cfbb0 msgr2=0x7f027415ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 shutdown_connections 2024-09-13T08:41:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.635+0000 7f027a400640 1 -- 172.21.15.31:0/1159915067 wait complete. 2024-09-13T08:41:59.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-13T08:41:59.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 -- 172.21.15.31:0/1266135866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 msgr2=0x7f69f40cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 --2- 172.21.15.31:0/1266135866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f40cc300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009830 tx=0x7f69e402a4a0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 -- 172.21.15.31:0/1266135866 shutdown_connections 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 --2- 172.21.15.31:0/1266135866 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40ca5c0 0x7f69f40ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 --2- 172.21.15.31:0/1266135866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f40cc300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 -- 172.21.15.31:0/1266135866 >> 172.21.15.31:0/1266135866 conn(0x7f69f40cc910 msgr2=0x7f69f40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 -- 172.21.15.31:0/1266135866 shutdown_connections 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.812+0000 7f69fa600640 1 -- 172.21.15.31:0/1266135866 wait complete. 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.814+0000 7f69fa600640 1 Processor -- start 2024-09-13T08:41:59.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.814+0000 7f69fa600640 1 -- start start 2024-09-13T08:41:59.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40b1910 0x7f69f4153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f4155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4074010 con 0x7f69f40ca5c0 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4073e90 con 0x7f69f415dfc0 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69fa600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f69f4155b30 con 0x7f69f40b1910 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39916/0 (socket says 172.21.15.31:39916) 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 -- 172.21.15.31:0/2991751623 learned_addr learned my addr 172.21.15.31:0/2991751623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69f3400640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40b1910 0x7f69f4153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69f3e00640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f4155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 -- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40b1910 msgr2=0x7f69f4153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40b1910 0x7f69f4153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 -- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 msgr2=0x7f69f4155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f4155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69ebe00640 1 -- 172.21.15.31:0/2991751623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f40ca2a0 con 0x7f69f40ca5c0 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.815+0000 7f69f3e00640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f4155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69ebe00640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00e9f0 tx=0x7f69dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc00cd80 con 0x7f69f40ca5c0 2024-09-13T08:41:59.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69dc010430 con 0x7f69f40ca5c0 2024-09-13T08:41:59.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f40c9a60 con 0x7f69f40ca5c0 2024-09-13T08:41:59.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc010840 con 0x7f69f40ca5c0 2024-09-13T08:41:59.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.816+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f40c9fc0 con 0x7f69f40ca5c0 2024-09-13T08:41:59.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.817+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f40b2110 con 0x7f69f40ca5c0 2024-09-13T08:41:59.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.818+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f69dc0109e0 con 0x7f69f40ca5c0 2024-09-13T08:41:59.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.819+0000 7f69eb400640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 0x7f69d007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:41:59.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.819+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f69dc09b700 con 0x7f69f40ca5c0 2024-09-13T08:41:59.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.819+0000 7f69f3400640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 0x7f69d007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:41:59.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.820+0000 7f69f3400640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 0x7f69d007ae30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f69e4034000 tx=0x7f69e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:41:59.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:41:59.824+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f69f40c9a60 con 0x7f69f40ca5c0 2024-09-13T08:42:00.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.106+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f69f40af5d0 con 0x7f69f40ca5c0 2024-09-13T08:42:00.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.863+0000 7f69eb400640 1 -- 172.21.15.31:0/2991751623 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v267) ==== 149+0+0 (secure 0 0 0) 0x7f69dc063bc0 con 0x7f69f40ca5c0 2024-09-13T08:42:00.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 msgr2=0x7f69d007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:00.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 0x7f69d007ae30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f69e4034000 tx=0x7f69e4004730 comp rx=0 tx=0).stop 2024-09-13T08:42:00.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 msgr2=0x7f69f40c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00e9f0 tx=0x7f69dc00eeb0 comp rx=0 tx=0).stop 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 shutdown_connections 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69d0078970 0x7f69d007ae30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f415dfc0 0x7f69f4155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69f40ca5c0 0x7f69f40c6f60 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 --2- 172.21.15.31:0/2991751623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69f40b1910 0x7f69f4153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.867+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 >> 172.21.15.31:0/2991751623 conn(0x7f69f40cc910 msgr2=0x7f69f415d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.868+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 shutdown_connections 2024-09-13T08:42:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:00.868+0000 7f69fa600640 1 -- 172.21.15.31:0/2991751623 wait complete. 2024-09-13T08:42:00.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:42:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 -- 172.21.15.31:0/134889009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940cf850 msgr2=0x7f91940ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 --2- 172.21.15.31:0/134889009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940cf850 0x7f91940ce3c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f917c009830 tx=0x7f917c02a510 comp rx=0 tx=0).stop 2024-09-13T08:42:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 -- 172.21.15.31:0/134889009 shutdown_connections 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 --2- 172.21.15.31:0/134889009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940ce900 0x7f91940cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 --2- 172.21.15.31:0/134889009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940cf850 0x7f91940ce3c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 -- 172.21.15.31:0/134889009 >> 172.21.15.31:0/134889009 conn(0x7f91940cc910 msgr2=0x7f91940cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 -- 172.21.15.31:0/134889009 shutdown_connections 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.054+0000 7f9199c00640 1 -- 172.21.15.31:0/134889009 wait complete. 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.056+0000 7f9199c00640 1 Processor -- start 2024-09-13T08:42:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.056+0000 7f9199c00640 1 -- start start 2024-09-13T08:42:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940cf850 0x7f919415c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 0x7f919415c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91941600f0 con 0x7f91940ce900 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91940cb850 con 0x7f919415bd70 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9199c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9194148980 con 0x7f91940cf850 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39934/0 (socket says 172.21.15.31:39934) 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 -- 172.21.15.31:0/752496343 learned_addr learned my addr 172.21.15.31:0/752496343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f918a000640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940cf850 0x7f919415c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9193400640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 0x7f919415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 -- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940cf850 msgr2=0x7f919415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940cf850 0x7f919415c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 -- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 msgr2=0x7f919415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 0x7f919415c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9192a00640 1 -- 172.21.15.31:0/752496343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919415e730 con 0x7f91940ce900 2024-09-13T08:42:01.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.057+0000 7f9193400640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 0x7f919415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:01.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f9192a00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f918400c3a0 tx=0x7f918400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:01.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184011790 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9184017070 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184011cc0 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919415ea10 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.058+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91941f4ce0 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.059+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91940ce3c0 con 0x7f91940ce900 2024-09-13T08:42:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.060+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f918401c460 con 0x7f91940ce900 2024-09-13T08:42:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.062+0000 7f918b400640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 0x7f917007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.062+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f918409a910 con 0x7f91940ce900 2024-09-13T08:42:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.062+0000 7f918a000640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 0x7f917007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.063+0000 7f918a000640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 0x7f917007ae30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f917c034000 tx=0x7f917c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:01.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.066+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9184062d90 con 0x7f91940ce900 2024-09-13T08:42:01.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.263+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f9194074980 con 0x7f91940ce900 2024-09-13T08:42:01.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.263+0000 7f918b400640 1 -- 172.21.15.31:0/752496343 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7f9184062690 con 0x7f91940ce900 2024-09-13T08:42:01.265 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:42:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:42:01.267 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:42:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 msgr2=0x7f917007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 0x7f917007ae30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f917c034000 tx=0x7f917c004730 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 msgr2=0x7f91941f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f918400c3a0 tx=0x7f918400c860 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 shutdown_connections 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9170078970 0x7f917007ae30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f919415bd70 0x7f919415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91940cf850 0x7f919415c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 --2- 172.21.15.31:0/752496343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91940ce900 0x7f91941f47a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 >> 172.21.15.31:0/752496343 conn(0x7f91940cc910 msgr2=0x7f9194157450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 shutdown_connections 2024-09-13T08:42:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.266+0000 7f9199c00640 1 -- 172.21.15.31:0/752496343 wait complete. 2024-09-13T08:42:01.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:42:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 -- 172.21.15.31:0/348812990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d00cbe50 msgr2=0x7f31d00b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 --2- 172.21.15.31:0/348812990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d00cbe50 0x7f31d00b29e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f31c8005fd0 tx=0x7f31c802a940 comp rx=0 tx=0).stop 2024-09-13T08:42:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 -- 172.21.15.31:0/348812990 shutdown_connections 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 --2- 172.21.15.31:0/348812990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d00cbe50 0x7f31d00b29e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 -- 172.21.15.31:0/348812990 >> 172.21.15.31:0/348812990 conn(0x7f31d00cfbb0 msgr2=0x7f31d00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 -- 172.21.15.31:0/348812990 shutdown_connections 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.447+0000 7f31d6200640 1 -- 172.21.15.31:0/348812990 wait complete. 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.449+0000 7f31d6200640 1 Processor -- start 2024-09-13T08:42:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.449+0000 7f31d6200640 1 -- start start 2024-09-13T08:42:01.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d0072560 0x7f31d015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31d00cbe50 0x7f31d01f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00b2f20 con 0x7f31d015bfc0 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00d0a00 con 0x7f31d0072560 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31d6200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f31d00b30a0 con 0x7f31d00cbe50 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39958/0 (socket says 172.21.15.31:39958) 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31cfe00640 1 -- 172.21.15.31:0/1123158376 learned_addr learned my addr 172.21.15.31:0/1123158376 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31c6a00640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31d00cbe50 0x7f31d01f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.450+0000 7f31cf400640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d0072560 0x7f31d015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 -- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31d00cbe50 msgr2=0x7f31d01f8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31d00cbe50 0x7f31d01f8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 -- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d0072560 msgr2=0x7f31d015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d0072560 0x7f31d015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 -- 172.21.15.31:0/1123158376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d015f4c0 con 0x7f31d015bfc0 2024-09-13T08:42:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31cfe00640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009cd0 tx=0x7f31c8031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8004240 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31c80043e0 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d00c6350 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8004830 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.451+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d015e250 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.452+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d00b29e0 con 0x7f31d015bfc0 2024-09-13T08:42:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.454+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f31c804c440 con 0x7f31d015bfc0 2024-09-13T08:42:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.455+0000 7f31c7e00640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 0x7f31ac07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.455+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7f31c80bc410 con 0x7f31d015bfc0 2024-09-13T08:42:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.455+0000 7f31cf400640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 0x7f31ac07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.455+0000 7f31cf400640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 0x7f31ac07ae30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f31c0001720 tx=0x7f31c0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:01.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.459+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f31c80842d0 con 0x7f31d015bfc0 2024-09-13T08:42:01.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.654+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f31d0153780 con 0x7f31d015bfc0 2024-09-13T08:42:01.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.658+0000 7f31c7e00640 1 -- 172.21.15.31:0/1123158376 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f31c8088d00 con 0x7f31d015bfc0 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.660+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 msgr2=0x7f31ac07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.660+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 0x7f31ac07ae30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f31c0001720 tx=0x7f31c0009040 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.660+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 msgr2=0x7f31d00c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.660+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009cd0 tx=0x7f31c8031e20 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 shutdown_connections 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f31ac078970 0x7f31ac07ae30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31d015bfc0 0x7f31d00c8ed0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f31d00cbe50 0x7f31d01f8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 --2- 172.21.15.31:0/1123158376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d0072560 0x7f31d015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 >> 172.21.15.31:0/1123158376 conn(0x7f31d00cfbb0 msgr2=0x7f31d00ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:01.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 shutdown_connections 2024-09-13T08:42:01.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:01.661+0000 7f31d6200640 1 -- 172.21.15.31:0/1123158376 wait complete. 2024-09-13T08:42:01.678 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:42:01.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:01.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:01.863 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:42:01.864 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:01.864 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:01.900 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:42:01.900 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:01.900 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:01.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:42:01.939 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:01.939 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:01.976 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:42:01.976 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:01.977 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:02.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:42:02.016 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:42:02.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 -- 172.21.15.31:0/647378693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34072850 msgr2=0x7fca34073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 --2- 172.21.15.31:0/647378693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34072850 0x7fca34073fc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fca20009830 tx=0x7fca2002a560 comp rx=0 tx=0).stop 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 -- 172.21.15.31:0/647378693 shutdown_connections 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 --2- 172.21.15.31:0/647378693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34072850 0x7fca34073fc0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 --2- 172.21.15.31:0/647378693 >> v2:172.21.15.45:3300/0 conn(0x7fca34152f10 0x7fca340cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 -- 172.21.15.31:0/647378693 >> 172.21.15.31:0/647378693 conn(0x7fca340cc910 msgr2=0x7fca340cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 -- 172.21.15.31:0/647378693 shutdown_connections 2024-09-13T08:42:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.175+0000 7fca3b400640 1 -- 172.21.15.31:0/647378693 wait complete. 2024-09-13T08:42:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.177+0000 7fca3b400640 1 Processor -- start 2024-09-13T08:42:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.177+0000 7fca3b400640 1 -- start start 2024-09-13T08:42:02.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca34072850 0x7fca3409d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34074500 0x7fca341f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca340ca4e0 con 0x7fca34152f10 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca340cc6b0 con 0x7fca34074500 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca3b400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fca340ca660 con 0x7fca34072850 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca39400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca39400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39978/0 (socket says 172.21.15.31:39978) 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca39400640 1 -- 172.21.15.31:0/227668928 learned_addr learned my addr 172.21.15.31:0/227668928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca33e00640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34074500 0x7fca341f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.178+0000 7fca38a00640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca34072850 0x7fca3409d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 -- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca34072850 msgr2=0x7fca3409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca34072850 0x7fca3409d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 -- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34074500 msgr2=0x7fca341f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34074500 0x7fca341f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 -- 172.21.15.31:0/227668928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca341f5500 con 0x7fca34152f10 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca39400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c7a0 tx=0x7fca2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c017070 con 0x7fca34152f10 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca2c005070 con 0x7fca34152f10 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0054c0 con 0x7fca34152f10 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.179+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca341561a0 con 0x7fca34152f10 2024-09-13T08:42:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.180+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca34153f60 con 0x7fca34152f10 2024-09-13T08:42:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.180+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca340b2250 con 0x7fca34152f10 2024-09-13T08:42:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.182+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fca2c01a070 con 0x7fca34152f10 2024-09-13T08:42:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.183+0000 7fca30c00640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 0x7fca1407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.183+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5502+0+0 (secure 0 0 0) 0x7fca2c09a170 con 0x7fca34152f10 2024-09-13T08:42:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.183+0000 7fca38a00640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 0x7fca1407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.184+0000 7fca38a00640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 0x7fca1407ae30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca28009b30 tx=0x7fca28005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.187+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fca341f5500 con 0x7fca34152f10 2024-09-13T08:42:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.418+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca340b5d40 con 0x7fca34152f10 2024-09-13T08:42:02.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.418+0000 7fca30c00640 1 -- 172.21.15.31:0/227668928 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+3905 (secure 0 0 0) 0x7fca2c061fa0 con 0x7fca34152f10 2024-09-13T08:42:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:02.421 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"btime":"2024-09-13T08:41:55:842358+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1515084768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6835","nonce":1515084768},{"type":"v1","addr":"172.21.15.45:6837","nonce":1515084768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5914,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6836/1345142907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1345142907},{"type":"v1","addr":"172.21.15.45:6836","nonce":1345142907}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/722247618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":722247618},{"type":"v1","addr":"172.21.15.31:6834","nonce":722247618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6498,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/461881370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":461881370},{"type":"v1","addr":"172.21.15.31:6835","nonce":461881370}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[]} 2024-09-13T08:42:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 50 2024-09-13T08:42:02.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.420+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 msgr2=0x7fca1407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.420+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 0x7fca1407ae30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca28009b30 tx=0x7fca28005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.420+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 msgr2=0x7fca340b0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.420+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c7a0 tx=0x7fca2c00cc60 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.420+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 shutdown_connections 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca14078970 0x7fca1407ae30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca34152f10 0x7fca340b0920 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34074500 0x7fca341f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 --2- 172.21.15.31:0/227668928 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca34072850 0x7fca3409d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 >> 172.21.15.31:0/227668928 conn(0x7fca340cc910 msgr2=0x7fca3415ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 shutdown_connections 2024-09-13T08:42:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:02.421+0000 7fca3b400640 1 -- 172.21.15.31:0/227668928 wait complete. 2024-09-13T08:42:02.438 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:42:02.438 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:42:02.438 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:42:02.438 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:02.439 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:42:02.439 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:42:02.439 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:42:02.439 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:02.439 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:42:02.439 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:42:02.439 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:42:02.440 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:02.440 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:42:02.440 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:42:02.440 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:42:02.440 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:02.455 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:42:02.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:42:02.456 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:42:02.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:42:02.458 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:42:02.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:42:02.499 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:42:02.499 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:02.501 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:42:02.501 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:02.503 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:42:02.503 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:02.627 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:42:02.627 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:42:02.637 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:42:04.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-13T08:42:06.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:42:06.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:42:08.457 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:42:08.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:42:08.462 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:42:08.463 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:08.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:42:08.598 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:42:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 -- 172.21.15.31:0/367844778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 msgr2=0x7f29180cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 --2- 172.21.15.31:0/367844778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180cdb80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f291809e090 tx=0x7f290c02a540 comp rx=0 tx=0).stop 2024-09-13T08:42:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 -- 172.21.15.31:0/367844778 shutdown_connections 2024-09-13T08:42:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 --2- 172.21.15.31:0/367844778 >> v2:172.21.15.45:3300/0 conn(0x7f29180ced20 0x7f29180ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:08.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 --2- 172.21.15.31:0/367844778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180cdb80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:08.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 -- 172.21.15.31:0/367844778 >> 172.21.15.31:0/367844778 conn(0x7f29180b18f0 msgr2=0x7f29180b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:08.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 -- 172.21.15.31:0/367844778 shutdown_connections 2024-09-13T08:42:08.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.634+0000 7f291fc00640 1 -- 172.21.15.31:0/367844778 wait complete. 2024-09-13T08:42:08.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.636+0000 7f291fc00640 1 Processor -- start 2024-09-13T08:42:08.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.636+0000 7f291fc00640 1 -- start start 2024-09-13T08:42:08.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29180ce600 0x7f291814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:08.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:08.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:08.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29180749f0 con 0x7f291815be30 2024-09-13T08:42:08.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2918074870 con 0x7f29180ce600 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291fc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2918160160 con 0x7f29180ced20 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f2917e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f291d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29180ce600 0x7f291814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f2917e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:54528/0 (socket says 172.21.15.31:54528) 2024-09-13T08:42:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.637+0000 7f2917e00640 1 -- 172.21.15.31:0/2668989641 learned_addr learned my addr 172.21.15.31:0/2668989641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:08.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 -- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29180ce600 msgr2=0x7f291814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:08.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29180ce600 0x7f291814c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:08.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 -- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 msgr2=0x7f29180aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 -- 172.21.15.31:0/2668989641 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291814cca0 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f291dc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.638+0000 7f2917e00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.639+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904017070 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.639+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2904009d20 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.639+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29040106c0 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.639+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2918159200 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.639+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29180cbdf0 con 0x7f29180ced20 2024-09-13T08:42:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.640+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29181605c0 con 0x7f29180ced20 2024-09-13T08:42:08.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.641+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f2904010860 con 0x7f29180ced20 2024-09-13T08:42:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.642+0000 7f2914c00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 0x7f28f807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.642+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 5502+0+0 (secure 0 0 0) 0x7f290409aeb0 con 0x7f29180ced20 2024-09-13T08:42:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.642+0000 7f291d200640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 0x7f28f807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.643+0000 7f291d200640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 0x7f28f807ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f290c034000 tx=0x7f290c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.647+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f291814cca0 con 0x7f29180ced20 2024-09-13T08:42:08.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:08.857+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f29180b25f0 con 0x7f29180ced20 2024-09-13T08:42:09.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.673+0000 7f2914c00640 1 -- 172.21.15.31:0/2668989641 <== mon.2 v2:172.21.15.45:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v86) ==== 97+0+0 (secure 0 0 0) 0x7f2904062d70 con 0x7f29180ced20 2024-09-13T08:42:09.675 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:42:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 msgr2=0x7f28f807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 0x7f28f807ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f290c034000 tx=0x7f290c004730 comp rx=0 tx=0).stop 2024-09-13T08:42:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 msgr2=0x7f2918158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400de10 comp rx=0 tx=0).stop 2024-09-13T08:42:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 shutdown_connections 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28f8078970 0x7f28f807ae30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f291815be30 0x7f29180aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29180ced20 0x7f2918158cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 --2- 172.21.15.31:0/2668989641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29180ce600 0x7f291814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 >> 172.21.15.31:0/2668989641 conn(0x7f29180b18f0 msgr2=0x7f29180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 shutdown_connections 2024-09-13T08:42:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.676+0000 7f291fc00640 1 -- 172.21.15.31:0/2668989641 wait complete. 2024-09-13T08:42:09.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:42:09.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.858+0000 7f3ce8800640 1 -- 172.21.15.31:0/2460143940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cc8005210 msgr2=0x7f3cc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.858+0000 7f3ce8800640 1 --2- 172.21.15.31:0/2460143940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cc8005210 0x7f3cc800b9c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009ed0 tx=0x7f3cc8005830 comp rx=0 tx=0).stop 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.858+0000 7f3ce8800640 1 -- 172.21.15.31:0/2460143940 shutdown_connections 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.858+0000 7f3ce8800640 1 --2- 172.21.15.31:0/2460143940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cc8005210 0x7f3cc800b9c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.858+0000 7f3ce8800640 1 -- 172.21.15.31:0/2460143940 >> 172.21.15.31:0/2460143940 conn(0x7f3ce00cc910 msgr2=0x7f3ce00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.859+0000 7f3ce8800640 1 -- 172.21.15.31:0/2460143940 shutdown_connections 2024-09-13T08:42:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.859+0000 7f3ce8800640 1 -- 172.21.15.31:0/2460143940 wait complete. 2024-09-13T08:42:09.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.860+0000 7f3ce8800640 1 Processor -- start 2024-09-13T08:42:09.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.860+0000 7f3ce8800640 1 -- start start 2024-09-13T08:42:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 0x7f3ce009d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ce00c49f0 0x7f3ce00c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce015c150 con 0x7f3ce009aa10 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00cb850 con 0x7f3cc8005210 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce8800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00c6370 con 0x7f3ce00c49f0 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.861+0000 7f3ce5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 0x7f3ce009d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55662/0 (socket says 172.21.15.31:55662) 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 -- 172.21.15.31:0/1828333191 learned_addr learned my addr 172.21.15.31:0/1828333191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 0x7f3ce009d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:52890/0 (socket says 172.21.15.31:52890) 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce6800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ce00c49f0 0x7f3ce00c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 -- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ce00c49f0 msgr2=0x7f3ce00c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:09.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ce00c49f0 0x7f3ce00c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 -- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 msgr2=0x7f3ce009d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 0x7f3ce009d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 -- 172.21.15.31:0/1828333191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00c5760 con 0x7f3ce009aa10 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3ce5400640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0003f00 tx=0x7f3cd0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0011640 con 0x7f3ce009aa10 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cd0016020 con 0x7f3ce009aa10 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.862+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0011a80 con 0x7f3ce009aa10 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.863+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce00c5a40 con 0x7f3ce009aa10 2024-09-13T08:42:09.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.863+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce00c6000 con 0x7f3ce009aa10 2024-09-13T08:42:09.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.864+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00cc220 con 0x7f3ce009aa10 2024-09-13T08:42:09.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.865+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f3cd0011c20 con 0x7f3ce009aa10 2024-09-13T08:42:09.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.866+0000 7f3cda000640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 0x7f3cc407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:09.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.866+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 4857+0+0 (secure 0 0 0) 0x7f3cd009b3a0 con 0x7f3ce009aa10 2024-09-13T08:42:09.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.866+0000 7f3ce5e00640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 0x7f3cc407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:09.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.867+0000 7f3ce5e00640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 0x7f3cc407ae60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001d80 tx=0x7f3cd40067a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:09.870+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3cd0020020 con 0x7f3ce009aa10 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.135+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3ce00ae9a0 con 0x7f3ce009aa10 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.136+0000 7f3cda000640 1 -- 172.21.15.31:0/1828333191 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7f3ce00cc5a0 con 0x7f3ce009aa10 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.138 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:42:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.144 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.146 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:42:10.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:42:10.149 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:42:10.151 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.138+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 msgr2=0x7f3cc407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.138+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 0x7f3cc407ae60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001d80 tx=0x7f3cd40067a0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.138+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 msgr2=0x7f3ce009ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.138+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0003f00 tx=0x7f3cd0001630 comp rx=0 tx=0).stop 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 shutdown_connections 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3cc40789a0 0x7f3cc407ae60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ce00c49f0 0x7f3ce00c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce009aa10 0x7f3ce009ae90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 --2- 172.21.15.31:0/1828333191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc8005210 0x7f3ce009d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 >> 172.21.15.31:0/1828333191 conn(0x7f3ce00cc910 msgr2=0x7f3ce00ca4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 shutdown_connections 2024-09-13T08:42:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.139+0000 7f3ce8800640 1 -- 172.21.15.31:0/1828333191 wait complete. 2024-09-13T08:42:10.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:42:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 -- 172.21.15.31:0/2642978849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 msgr2=0x7f33840cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 --2- 172.21.15.31:0/2642978849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f33840cc060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f33840cfe60 tx=0x7f337c02a920 comp rx=0 tx=0).stop 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 -- 172.21.15.31:0/2642978849 shutdown_connections 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 --2- 172.21.15.31:0/2642978849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f33840cc060 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 -- 172.21.15.31:0/2642978849 >> 172.21.15.31:0/2642978849 conn(0x7f33840cc910 msgr2=0x7f33840cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 -- 172.21.15.31:0/2642978849 shutdown_connections 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.316+0000 7f3389600640 1 -- 172.21.15.31:0/2642978849 wait complete. 2024-09-13T08:42:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.318+0000 7f3389600640 1 Processor -- start 2024-09-13T08:42:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.318+0000 7f3389600640 1 -- start start 2024-09-13T08:42:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f338415e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33840cf850 0x7f3384157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33841600f0 con 0x7f338415bd70 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33840cead0 con 0x7f33840cbc50 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3389600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f338415e700 con 0x7f33840cf850 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3383400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3383400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55670/0 (socket says 172.21.15.31:55670) 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3383400640 1 -- 172.21.15.31:0/948424407 learned_addr learned my addr 172.21.15.31:0/948424407 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3382a00640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f338415e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.319+0000 7f3382000640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33840cf850 0x7f3384157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 -- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33840cf850 msgr2=0x7f3384157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33840cf850 0x7f3384157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 -- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 msgr2=0x7f338415e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f338415e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 -- 172.21.15.31:0/948424407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338415e9d0 con 0x7f338415bd70 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3383400640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f337c009ca0 tx=0x7f337c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c004240 con 0x7f338415bd70 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f337c0043e0 con 0x7f338415bd70 2024-09-13T08:42:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338415ecb0 con 0x7f338415bd70 2024-09-13T08:42:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c004810 con 0x7f338415bd70 2024-09-13T08:42:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.320+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33841f8270 con 0x7f338415bd70 2024-09-13T08:42:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.322+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338406ac10 con 0x7f338415bd70 2024-09-13T08:42:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.323+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f337c03b450 con 0x7f338415bd70 2024-09-13T08:42:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.323+0000 7f337aa00640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 0x7f336407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.324+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 4857+0+0 (secure 0 0 0) 0x7f337c0bc2a0 con 0x7f338415bd70 2024-09-13T08:42:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.324+0000 7f3382a00640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 0x7f336407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.324+0000 7f3382a00640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 0x7f336407ae30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33740012f0 tx=0x7f3374009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.328+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f337c02d070 con 0x7f338415bd70 2024-09-13T08:42:10.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.527+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f33841eaf10 con 0x7f338415bd70 2024-09-13T08:42:10.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.527+0000 7f337aa00640 1 -- 172.21.15.31:0/948424407 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) ==== 84+0+65 (secure 0 0 0) 0x7f337c084350 con 0x7f338415bd70 2024-09-13T08:42:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:10.529 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:42:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:42:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:42:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:10.530 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 msgr2=0x7f336407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 0x7f336407ae30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33740012f0 tx=0x7f3374009040 comp rx=0 tx=0).stop 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 msgr2=0x7f33841f8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f337c009ca0 tx=0x7f337c003c20 comp rx=0 tx=0).stop 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 shutdown_connections 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3364078970 0x7f336407ae30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338415bd70 0x7f33841f8a70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33840cf850 0x7f3384157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 --2- 172.21.15.31:0/948424407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33840cbc50 0x7f338415e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 >> 172.21.15.31:0/948424407 conn(0x7f33840cc910 msgr2=0x7f33840ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 shutdown_connections 2024-09-13T08:42:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.530+0000 7f3389600640 1 -- 172.21.15.31:0/948424407 wait complete. 2024-09-13T08:42:10.548 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:42:10.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:42:10.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 -- 172.21.15.31:0/3069794278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a815bda0 msgr2=0x7f60a80cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 --2- 172.21.15.31:0/3069794278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a815bda0 0x7f60a80cc6a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f60a80d1ae0 tx=0x7f609802a560 comp rx=0 tx=0).stop 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 -- 172.21.15.31:0/3069794278 shutdown_connections 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 --2- 172.21.15.31:0/3069794278 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a80ce2d0 0x7f60a80ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 --2- 172.21.15.31:0/3069794278 >> v2:172.21.15.45:3300/0 conn(0x7f60a80ccbe0 0x7f60a80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 --2- 172.21.15.31:0/3069794278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a815bda0 0x7f60a80cc6a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 -- 172.21.15.31:0/3069794278 >> 172.21.15.31:0/3069794278 conn(0x7f60a80d1b90 msgr2=0x7f60a80d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 -- 172.21.15.31:0/3069794278 shutdown_connections 2024-09-13T08:42:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.712+0000 7f60ae200640 1 -- 172.21.15.31:0/3069794278 wait complete. 2024-09-13T08:42:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.714+0000 7f60ae200640 1 Processor -- start 2024-09-13T08:42:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.714+0000 7f60ae200640 1 -- start start 2024-09-13T08:42:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 0x7f60a81f7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 0x7f60a81f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f60a80cee20 con 0x7f60a80ccbe0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60a80ceca0 con 0x7f60a80ce2d0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ae200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f60a80cefa0 con 0x7f60a815bda0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ace00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ace00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55692/0 (socket says 172.21.15.31:55692) 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ace00640 1 -- 172.21.15.31:0/2553359903 learned_addr learned my addr 172.21.15.31:0/2553359903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f60ad800640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 0x7f60a81f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.715+0000 7f609fe00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 0x7f60a81f7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 -- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 msgr2=0x7f60a81f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 0x7f60a81f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 -- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 msgr2=0x7f60a81f7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 0x7f60a81f7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 -- 172.21.15.31:0/2553359903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a815c2f0 con 0x7f60a80ccbe0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ad800640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 0x7f60a81f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f609fe00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 0x7f60a81f7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ace00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6098034040 tx=0x7f60980039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609803e030 con 0x7f60a80ccbe0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60980047a0 con 0x7f60a80ccbe0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a806fb00 con 0x7f60a80ccbe0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.716+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098035040 con 0x7f60a80ccbe0 2024-09-13T08:42:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.717+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a815c870 con 0x7f60a80ccbe0 2024-09-13T08:42:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.717+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60a80ce760 con 0x7f60a80ccbe0 2024-09-13T08:42:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.719+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f6098041030 con 0x7f60a80ccbe0 2024-09-13T08:42:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.720+0000 7f60a5600640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 0x7f608407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.720+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 4857+0+0 (secure 0 0 0) 0x7f60980baf40 con 0x7f60a80ccbe0 2024-09-13T08:42:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.720+0000 7f609fe00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 0x7f608407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.720+0000 7f609fe00640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 0x7f608407ae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60a80cea20 tx=0x7f6090005f10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.724+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f60980836d0 con 0x7f60a80ccbe0 2024-09-13T08:42:10.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-13T08:42:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:10.924+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f60a815c640 con 0x7f60a80ccbe0 2024-09-13T08:42:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.679+0000 7f60a5600640 1 -- 172.21.15.31:0/2553359903 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v87) ==== 138+0+0 (secure 0 0 0) 0x7f6098082fd0 con 0x7f60a80ccbe0 2024-09-13T08:42:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:42:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.682+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 msgr2=0x7f608407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.682+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 0x7f608407ae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60a80cea20 tx=0x7f6090005f10 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.682+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 msgr2=0x7f60a8159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.682+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6098034040 tx=0x7f60980039e0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 shutdown_connections 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6084078970 0x7f608407ae30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f60a815bda0 0x7f60a81f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60a80ce2d0 0x7f60a81f7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 --2- 172.21.15.31:0/2553359903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60a80ccbe0 0x7f60a8159010 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 >> 172.21.15.31:0/2553359903 conn(0x7f60a80d1b90 msgr2=0x7f60a815a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 shutdown_connections 2024-09-13T08:42:11.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.683+0000 7f60ae200640 1 -- 172.21.15.31:0/2553359903 wait complete. 2024-09-13T08:42:11.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:42:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 -- 172.21.15.31:0/1960637810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cdcc0 msgr2=0x7fb6e415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/1960637810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e415e820 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc005fd0 tx=0x7fb6dc02aa90 comp rx=0 tx=0).stop 2024-09-13T08:42:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 -- 172.21.15.31:0/1960637810 shutdown_connections 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/1960637810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e415e820 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/1960637810 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cf4d0 0x7fb6e40cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 -- 172.21.15.31:0/1960637810 >> 172.21.15.31:0/1960637810 conn(0x7fb6e40cfbb0 msgr2=0x7fb6e40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 -- 172.21.15.31:0/1960637810 shutdown_connections 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.878+0000 7fb6e8e00640 1 -- 172.21.15.31:0/1960637810 wait complete. 2024-09-13T08:42:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.879+0000 7fb6e8e00640 1 Processor -- start 2024-09-13T08:42:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.880+0000 7fb6e8e00640 1 -- start start 2024-09-13T08:42:11.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e40c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:11.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e415acb0 0x7fb6e40b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e415eee0 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e415ed60 con 0x7fb6e40cdcc0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e8e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e415f060 con 0x7fb6e415acb0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55710/0 (socket says 172.21.15.31:55710) 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 -- 172.21.15.31:0/3910902297 learned_addr learned my addr 172.21.15.31:0/3910902297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e2800640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e415acb0 0x7fb6e40b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6e1e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e40c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:11.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 -- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e415acb0 msgr2=0x7fb6e40b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e415acb0 0x7fb6e40b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 -- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cdcc0 msgr2=0x7fb6e40c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e40c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.881+0000 7fb6d9400640 1 -- 172.21.15.31:0/3910902297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e41f8c60 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6d9400640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00d770 tx=0x7fb6cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc017070 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6cc009d20 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e41f4bc0 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc004280 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.882+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e40c9c00 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.883+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e415f580 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.885+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fb6cc0035f0 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.886+0000 7fb6daa00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 0x7fb6c007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:11.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.886+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5227+0+0 (secure 0 0 0) 0x7fb6cc09ae70 con 0x7fb6e40cf4d0 2024-09-13T08:42:11.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.886+0000 7fb6e1e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 0x7fb6c007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:11.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.887+0000 7fb6e1e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 0x7fb6c007ae30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4001720 tx=0x7fb6d4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:11.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:11.890+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb6e41f4bc0 con 0x7fb6e40cf4d0 2024-09-13T08:42:12.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.086+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb6e40aea20 con 0x7fb6e40cf4d0 2024-09-13T08:42:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.682+0000 7fb6daa00640 1 -- 172.21.15.31:0/3910902297 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) ==== 170+0+0 (secure 0 0 0) 0x7fb6cc062f30 con 0x7fb6e40cf4d0 2024-09-13T08:42:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:42:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.685+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 msgr2=0x7fb6c007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.685+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 0x7fb6c007ae30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4001720 tx=0x7fb6d4009040 comp rx=0 tx=0).stop 2024-09-13T08:42:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.685+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 msgr2=0x7fb6e41510f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.685+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00d770 tx=0x7fb6cc00dc30 comp rx=0 tx=0).stop 2024-09-13T08:42:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 shutdown_connections 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078970 0x7fb6c007ae30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e415acb0 0x7fb6e40b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cf4d0 0x7fb6e41510f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 --2- 172.21.15.31:0/3910902297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cdcc0 0x7fb6e40c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 >> 172.21.15.31:0/3910902297 conn(0x7fb6e40cfbb0 msgr2=0x7fb6e40b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 shutdown_connections 2024-09-13T08:42:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.686+0000 7fb6e8e00640 1 -- 172.21.15.31:0/3910902297 wait complete. 2024-09-13T08:42:12.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:42:12.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 -- 172.21.15.31:0/105378327 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 msgr2=0x7fa1600b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 --2- 172.21.15.31:0/105378327 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b1480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa158005fd0 tx=0x7fa15802a780 comp rx=0 tx=0).stop 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 -- 172.21.15.31:0/105378327 shutdown_connections 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 --2- 172.21.15.31:0/105378327 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b1480 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 --2- 172.21.15.31:0/105378327 >> v2:172.21.15.45:3300/0 conn(0x7fa160161470 0x7fa1600b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 -- 172.21.15.31:0/105378327 >> 172.21.15.31:0/105378327 conn(0x7fa1600d3e90 msgr2=0x7fa1600d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 -- 172.21.15.31:0/105378327 shutdown_connections 2024-09-13T08:42:12.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.883+0000 7fa15f200640 1 -- 172.21.15.31:0/105378327 wait complete. 2024-09-13T08:42:12.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.885+0000 7fa15f200640 1 Processor -- start 2024-09-13T08:42:12.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.885+0000 7fa15f200640 1 -- start start 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 0x7fa1600a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1600b1bd0 con 0x7fa160161470 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1600b5290 con 0x7fa1600d1d90 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa1600b1d50 con 0x7fa1600d28e0 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55720/0 (socket says 172.21.15.31:55720) 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15e800640 1 -- 172.21.15.31:0/4229690772 learned_addr learned my addr 172.21.15.31:0/4229690772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15de00640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 0x7fa1600a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.886+0000 7fa15d400640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 -- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 msgr2=0x7fa1600b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 -- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 msgr2=0x7fa1600cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 -- 172.21.15.31:0/4229690772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa160150cb0 con 0x7fa1600d1d90 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15e800640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:42:12.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa15de00640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 0x7fa1600a1520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa15000ea20 tx=0x7fa15000eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15000ccc0 con 0x7fa1600d1d90 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa15000ce60 con 0x7fa1600d1d90 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.887+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150004280 con 0x7fa1600d1d90 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.888+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1600b8680 con 0x7fa1600d1d90 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.888+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16015f310 con 0x7fa1600d1d90 2024-09-13T08:42:12.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.889+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1600b1480 con 0x7fa1600d1d90 2024-09-13T08:42:12.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.890+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fa1500047a0 con 0x7fa1600d1d90 2024-09-13T08:42:12.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.891+0000 7fa156000640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 0x7fa14007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:12.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.891+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5609+0+0 (secure 0 0 0) 0x7fa15009b8b0 con 0x7fa1600d1d90 2024-09-13T08:42:12.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.891+0000 7fa15d400640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 0x7fa14007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:12.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.892+0000 7fa15d400640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 0x7fa14007ae30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1600caaa0 tx=0x7fa14c007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:12.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:12.895+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa1500636f0 con 0x7fa1600d1d90 2024-09-13T08:42:13.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.128+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa1600b8ba0 con 0x7fa1600d1d90 2024-09-13T08:42:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.699+0000 7fa156000640 1 -- 172.21.15.31:0/4229690772 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v54) ==== 174+0+0 (secure 0 0 0) 0x7fa150068120 con 0x7fa1600d1d90 2024-09-13T08:42:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 12 and data pool 13 2024-09-13T08:42:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.707+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 msgr2=0x7fa14007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.707+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 0x7fa14007ae30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1600caaa0 tx=0x7fa14c007e90 comp rx=0 tx=0).stop 2024-09-13T08:42:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.707+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 msgr2=0x7fa1600a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.707+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 0x7fa1600a1520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa15000ea20 tx=0x7fa15000eee0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 shutdown_connections 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa140078970 0x7fa14007ae30 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1600caaa0 tx=0x7fa14c007e90 comp rx=0 tx=0).stop 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa160161470 0x7fa1600cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1600d28e0 0x7fa1600b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 --2- 172.21.15.31:0/4229690772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1600d1d90 0x7fa1600a1520 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.708+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 >> 172.21.15.31:0/4229690772 conn(0x7fa1600d3e90 msgr2=0x7fa16006e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:13.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.710+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 shutdown_connections 2024-09-13T08:42:13.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.710+0000 7fa15f200640 1 -- 172.21.15.31:0/4229690772 wait complete. 2024-09-13T08:42:13.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 -- 172.21.15.31:0/3027020660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 msgr2=0x7fafc4161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 --2- 172.21.15.31:0/3027020660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc4161b90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fafb4005fd0 tx=0x7fafb402a940 comp rx=0 tx=0).stop 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 -- 172.21.15.31:0/3027020660 shutdown_connections 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 --2- 172.21.15.31:0/3027020660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc4161b90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 -- 172.21.15.31:0/3027020660 >> 172.21.15.31:0/3027020660 conn(0x7fafc406cb00 msgr2=0x7fafc406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 -- 172.21.15.31:0/3027020660 shutdown_connections 2024-09-13T08:42:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.909+0000 7fafc3200640 1 -- 172.21.15.31:0/3027020660 wait complete. 2024-09-13T08:42:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.911+0000 7fafc3200640 1 Processor -- start 2024-09-13T08:42:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.911+0000 7fafc3200640 1 -- start start 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc40c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 0x7fafc414d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40ce5d0 con 0x7fafc40cf2f0 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40ce450 con 0x7fafc40caa00 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fafc40ce750 con 0x7fafc40cc3e0 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55730/0 (socket says 172.21.15.31:55730) 2024-09-13T08:42:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 -- 172.21.15.31:0/3246668050 learned_addr learned my addr 172.21.15.31:0/3246668050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc1e00640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc40c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc1400640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 0x7fafc414d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 -- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 msgr2=0x7fafc414d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 0x7fafc414d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 -- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 msgr2=0x7fafc40c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc40c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc2800640 1 -- 172.21.15.31:0/3246668050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc40b5910 con 0x7fafc40cf2f0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc1e00640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc40c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.912+0000 7fafc1400640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 0x7fafc414d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafc2800640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00c400 tx=0x7fafbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011790 con 0x7fafc40cf2f0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafbc017070 con 0x7fafc40cf2f0 2024-09-13T08:42:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc40b5630 con 0x7fafc40cf2f0 2024-09-13T08:42:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011d00 con 0x7fafc40cf2f0 2024-09-13T08:42:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.913+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc40c5c90 con 0x7fafc40cf2f0 2024-09-13T08:42:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.914+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc40cec00 con 0x7fafc40cf2f0 2024-09-13T08:42:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.916+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fafbc01c460 con 0x7fafc40cf2f0 2024-09-13T08:42:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.916+0000 7fafba000640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 0x7fafa407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.917+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5709+0+0 (secure 0 0 0) 0x7fafbc09aa30 con 0x7fafc40cf2f0 2024-09-13T08:42:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.917+0000 7fafc1e00640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 0x7fafa407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.917+0000 7fafc1e00640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 0x7fafa407ae30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafb4034000 tx=0x7fafb4004ab0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:13.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:13.921+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fafbc062de0 con 0x7fafc40cf2f0 2024-09-13T08:42:14.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.115+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafc40b5340 con 0x7fafc40cf2f0 2024-09-13T08:42:14.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.116+0000 7fafba000640 1 -- 172.21.15.31:0/3246668050 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16697 (secure 0 0 0) 0x7fafbc0626e0 con 0x7fafc40cf2f0 2024-09-13T08:42:14.118 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:14.118 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:13.130980+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:10.935693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:12.097514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:14.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 msgr2=0x7fafa407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 0x7fafa407ae30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafb4034000 tx=0x7fafb4004ab0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 msgr2=0x7fafc41f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00c400 tx=0x7fafbc00c8c0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 shutdown_connections 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fafa4078970 0x7fafa407ae30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc40cf2f0 0x7fafc41f7ab0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafc40cc3e0 0x7fafc414d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 --2- 172.21.15.31:0/3246668050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc40caa00 0x7fafc40c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 >> 172.21.15.31:0/3246668050 conn(0x7fafc406cb00 msgr2=0x7fafc415fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 shutdown_connections 2024-09-13T08:42:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.118+0000 7fafc3200640 1 -- 172.21.15.31:0/3246668050 wait complete. 2024-09-13T08:42:14.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.296+0000 7fbecbe00640 1 -- 172.21.15.31:0/3561362350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbeb0003c10 msgr2=0x7fbeb0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.296+0000 7fbecbe00640 1 --2- 172.21.15.31:0/3561362350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbeb0003c10 0x7fbeb0004010 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009b80 tx=0x7fbeb0005830 comp rx=0 tx=0).stop 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.296+0000 7fbecb400640 1 -- 172.21.15.31:0/3561362350 reap_dead start 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.296+0000 7fbecbe00640 1 -- 172.21.15.31:0/3561362350 shutdown_connections 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.296+0000 7fbecbe00640 1 -- 172.21.15.31:0/3561362350 >> 172.21.15.31:0/3561362350 conn(0x7fbecc0d1c70 msgr2=0x7fbecc0d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.297+0000 7fbecbe00640 1 -- 172.21.15.31:0/3561362350 shutdown_connections 2024-09-13T08:42:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.297+0000 7fbecbe00640 1 -- 172.21.15.31:0/3561362350 wait complete. 2024-09-13T08:42:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.298+0000 7fbecbe00640 1 Processor -- start 2024-09-13T08:42:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.298+0000 7fbecbe00640 1 -- start start 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbeb0003c10 0x7fbecc0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbecc1f5da0 0x7fbecc15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0b3210 con 0x7fbecc1f66d0 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc160840 con 0x7fbecc1f5da0 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbecbe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc0b3390 con 0x7fbeb0003c10 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.299+0000 7fbec2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55752/0 (socket says 172.21.15.31:55752) 2024-09-13T08:42:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 -- 172.21.15.31:0/1413462417 learned_addr learned my addr 172.21.15.31:0/1413462417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbecb400640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbecc1f5da0 0x7fbecc15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbecaa00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbeb0003c10 0x7fbecc0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 -- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbeb0003c10 msgr2=0x7fbecc0c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbeb0003c10 0x7fbecc0c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 -- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbecc1f5da0 msgr2=0x7fbecc15dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbecc1f5da0 0x7fbecc15dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 -- 172.21.15.31:0/1413462417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc0c8ee0 con 0x7fbecc1f66d0 2024-09-13T08:42:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec2000640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006dc0 tx=0x7fbeb40049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.300+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4015020 con 0x7fbecc1f66d0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.301+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbeb40118b0 con 0x7fbecc1f66d0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.301+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4011d00 con 0x7fbecc1f66d0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.301+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc0c91c0 con 0x7fbecc1f66d0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.301+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc0c9780 con 0x7fbecc1f66d0 2024-09-13T08:42:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.302+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbecc15fa70 con 0x7fbecc1f66d0 2024-09-13T08:42:14.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.303+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fbeb4018020 con 0x7fbecc1f66d0 2024-09-13T08:42:14.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.304+0000 7fbec3e00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 0x7fbea807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.304+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5709+0+0 (secure 0 0 0) 0x7fbeb409b5a0 con 0x7fbecc1f66d0 2024-09-13T08:42:14.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.304+0000 7fbecaa00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 0x7fbea807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.305+0000 7fbecaa00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 0x7fbea807ae30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbebc035000 tx=0x7fbebc02d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:14.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.309+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbeb401f020 con 0x7fbecc1f66d0 2024-09-13T08:42:14.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.505+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbecc074940 con 0x7fbecc1f66d0 2024-09-13T08:42:14.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.506+0000 7fbec3e00640 1 -- 172.21.15.31:0/1413462417 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16697 (secure 0 0 0) 0x7fbecc0b1ce0 con 0x7fbecc1f66d0 2024-09-13T08:42:14.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:14.508 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:13.130980+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:10.935693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:12.097514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 msgr2=0x7fbea807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 0x7fbea807ae30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbebc035000 tx=0x7fbebc02d040 comp rx=0 tx=0).stop 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 msgr2=0x7fbecc0cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006dc0 tx=0x7fbeb40049d0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 shutdown_connections 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbea8078970 0x7fbea807ae30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbecc1f5da0 0x7fbecc15dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbecc1f66d0 0x7fbecc0cb4f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 --2- 172.21.15.31:0/1413462417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbeb0003c10 0x7fbecc0c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 >> 172.21.15.31:0/1413462417 conn(0x7fbecc0d1c70 msgr2=0x7fbecc071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:14.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 shutdown_connections 2024-09-13T08:42:14.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.509+0000 7fbecbe00640 1 -- 172.21.15.31:0/1413462417 wait complete. 2024-09-13T08:42:14.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.689+0000 7fabf1a00640 1 -- 172.21.15.31:0/3062176772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15bfc0 msgr2=0x7fabec072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.689+0000 7fabf1a00640 1 --2- 172.21.15.31:0/3062176772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15bfc0 0x7fabec072560 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009860 tx=0x7fabdc02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.690+0000 7fabf1a00640 1 -- 172.21.15.31:0/3062176772 shutdown_connections 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.690+0000 7fabf1a00640 1 --2- 172.21.15.31:0/3062176772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec15bfc0 0x7fabec072560 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.690+0000 7fabf1a00640 1 -- 172.21.15.31:0/3062176772 >> 172.21.15.31:0/3062176772 conn(0x7fabec0cfbb0 msgr2=0x7fabec0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.690+0000 7fabf1a00640 1 -- 172.21.15.31:0/3062176772 shutdown_connections 2024-09-13T08:42:14.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.690+0000 7fabf1a00640 1 -- 172.21.15.31:0/3062176772 wait complete. 2024-09-13T08:42:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.691+0000 7fabf1a00640 1 Processor -- start 2024-09-13T08:42:14.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.691+0000 7fabf1a00640 1 -- start start 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec06fa60 0x7fabec150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 0x7fabec15e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabec06e760 con 0x7fabec15bfc0 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fabec0cc060 con 0x7fabec0c9ad0 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.692+0000 7fabf1a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fabec0b30f0 con 0x7fabec06fa60 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55762/0 (socket says 172.21.15.31:55762) 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 -- 172.21.15.31:0/340141731 learned_addr learned my addr 172.21.15.31:0/340141731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabeaa00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec06fa60 0x7fabec150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabeb400640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 0x7fabec15e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 -- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec06fa60 msgr2=0x7fabec150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec06fa60 0x7fabec150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 -- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 msgr2=0x7fabec15e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 0x7fabec15e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 -- 172.21.15.31:0/340141731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec15d370 con 0x7fabec15bfc0 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabeb400640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 0x7fabec15e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:14.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.693+0000 7fabe2000640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fabd400c430 tx=0x7fabd400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.694+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4017070 con 0x7fabec15bfc0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.694+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabd40035f0 con 0x7fabec15bfc0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.694+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec15d650 con 0x7fabec15bfc0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.694+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4003a40 con 0x7fabec15bfc0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.694+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec15dc10 con 0x7fabec15bfc0 2024-09-13T08:42:14.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.695+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec072560 con 0x7fabec15bfc0 2024-09-13T08:42:14.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.696+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fabd401a070 con 0x7fabec15bfc0 2024-09-13T08:42:14.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.698+0000 7fabe3400640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 0x7fabc807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:14.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.698+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5709+0+0 (secure 0 0 0) 0x7fabd4099e90 con 0x7fabec15bfc0 2024-09-13T08:42:14.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.698+0000 7fabeaa00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 0x7fabc807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:14.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.699+0000 7fabeaa00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 0x7fabc807ae30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fabdc034000 tx=0x7fabdc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.702+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fabd40622d0 con 0x7fabec15bfc0 2024-09-13T08:42:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:14.943+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fabec0cb310 con 0x7fabec15bfc0 2024-09-13T08:42:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.721+0000 7fabe3400640 1 -- 172.21.15.31:0/340141731 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) ==== 118+0+0 (secure 0 0 0) 0x7fabd409b0b0 con 0x7fabec15bfc0 2024-09-13T08:42:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 msgr2=0x7fabc807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 0x7fabc807ae30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fabdc034000 tx=0x7fabdc004730 comp rx=0 tx=0).stop 2024-09-13T08:42:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 msgr2=0x7fabec0c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fabd400c430 tx=0x7fabd400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 shutdown_connections 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fabc8078970 0x7fabc807ae30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabec0c9ad0 0x7fabec15e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec15bfc0 0x7fabec0c9590 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 --2- 172.21.15.31:0/340141731 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fabec06fa60 0x7fabec150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 >> 172.21.15.31:0/340141731 conn(0x7fabec0cfbb0 msgr2=0x7fabec159520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 shutdown_connections 2024-09-13T08:42:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.724+0000 7fabf1a00640 1 -- 172.21.15.31:0/340141731 wait complete. 2024-09-13T08:42:15.744 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:42:15.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.913+0000 7fafb7400640 1 -- 172.21.15.31:0/2881210579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf94004800 msgr2=0x7faf94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.913+0000 7fafb7400640 1 --2- 172.21.15.31:0/2881210579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf94004800 0x7faf94004c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fafb00cf4e0 tx=0x7faf94005830 comp rx=0 tx=0).stop 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.913+0000 7fafb5400640 1 -- 172.21.15.31:0/2881210579 reap_dead start 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.914+0000 7fafb7400640 1 -- 172.21.15.31:0/2881210579 shutdown_connections 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.914+0000 7fafb7400640 1 -- 172.21.15.31:0/2881210579 >> 172.21.15.31:0/2881210579 conn(0x7fafb00d1dd0 msgr2=0x7fafb00d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.914+0000 7fafb7400640 1 -- 172.21.15.31:0/2881210579 shutdown_connections 2024-09-13T08:42:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.914+0000 7fafb7400640 1 -- 172.21.15.31:0/2881210579 wait complete. 2024-09-13T08:42:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.915+0000 7fafb7400640 1 Processor -- start 2024-09-13T08:42:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.915+0000 7fafb7400640 1 -- start start 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf94004800 0x7fafb01f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafb01f5a00 0x7fafb015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafb00b5c60 con 0x7fafb0155d10 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafb00d2bf0 con 0x7faf94004800 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.916+0000 7fafb7400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fafb00b5de0 con 0x7fafb01f5a00 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57732/0 (socket says 172.21.15.31:57732) 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 -- 172.21.15.31:0/1647867464 learned_addr learned my addr 172.21.15.31:0/1647867464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafb4a00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf94004800 0x7fafb01f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafb5400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafb01f5a00 0x7fafb015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 -- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafb01f5a00 msgr2=0x7fafb015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafb01f5a00 0x7fafb015dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 -- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf94004800 msgr2=0x7fafb01f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf94004800 0x7fafb01f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 -- 172.21.15.31:0/1647867464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb00c9730 con 0x7fafb0155d10 2024-09-13T08:42:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.917+0000 7fafa7e00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7faf98009c10 tx=0x7faf98033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.918+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98038040 con 0x7fafb0155d10 2024-09-13T08:42:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.918+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf98004020 con 0x7fafb0155d10 2024-09-13T08:42:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.918+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98004470 con 0x7fafb0155d10 2024-09-13T08:42:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.918+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb00c9a10 con 0x7fafb0155d10 2024-09-13T08:42:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.918+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb00c9fd0 con 0x7fafb0155d10 2024-09-13T08:42:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.919+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb00b74f0 con 0x7fafb0155d10 2024-09-13T08:42:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.920+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7faf98045020 con 0x7fafb0155d10 2024-09-13T08:42:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.921+0000 7fafad600640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 0x7faf8c07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.922+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7faf980bb1d0 con 0x7fafb0155d10 2024-09-13T08:42:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.922+0000 7fafb4a00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 0x7faf8c07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.922+0000 7fafb4a00640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 0x7faf8c07ae60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafa0001290 tx=0x7fafa0006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:15.925+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7faf98083060 con 0x7fafb0155d10 2024-09-13T08:42:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.160+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fafb015a990 con 0x7fafb0155d10 2024-09-13T08:42:16.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.728+0000 7fafad600640 1 -- 172.21.15.31:0/1647867464 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) ==== 122+0+0 (secure 0 0 0) 0x7faf98087a90 con 0x7fafb0155d10 2024-09-13T08:42:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 msgr2=0x7faf8c07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 0x7faf8c07ae60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafa0001290 tx=0x7fafa0006040 comp rx=0 tx=0).stop 2024-09-13T08:42:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 msgr2=0x7fafb01f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7faf98009c10 tx=0x7faf98033670 comp rx=0 tx=0).stop 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 shutdown_connections 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faf8c0789a0 0x7faf8c07ae60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fafb01f5a00 0x7fafb015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb0155d10 0x7fafb01f54c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 --2- 172.21.15.31:0/1647867464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf94004800 0x7fafb01f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 >> 172.21.15.31:0/1647867464 conn(0x7fafb00d1dd0 msgr2=0x7fafb00717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 shutdown_connections 2024-09-13T08:42:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.731+0000 7fafb7400640 1 -- 172.21.15.31:0/1647867464 wait complete. 2024-09-13T08:42:16.749 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:42:16.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:16.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 -- 172.21.15.31:0/1608342489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc0ce3c0 msgr2=0x7fcfbc0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:16.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 --2- 172.21.15.31:0/1608342489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc0ce3c0 0x7fcfbc0ce850 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc09e090 tx=0x7fcfb402aa30 comp rx=0 tx=0).stop 2024-09-13T08:42:16.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 -- 172.21.15.31:0/1608342489 shutdown_connections 2024-09-13T08:42:16.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 --2- 172.21.15.31:0/1608342489 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc15f9e0 0x7fcfbc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 --2- 172.21.15.31:0/1608342489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc0ce3c0 0x7fcfbc0ce850 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 --2- 172.21.15.31:0/1608342489 >> v2:172.21.15.45:3300/0 conn(0x7fcfbc15bd70 0x7fcfbc0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 -- 172.21.15.31:0/1608342489 >> 172.21.15.31:0/1608342489 conn(0x7fcfbc0cc910 msgr2=0x7fcfbc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 -- 172.21.15.31:0/1608342489 shutdown_connections 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.911+0000 7fcfc1200640 1 -- 172.21.15.31:0/1608342489 wait complete. 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.912+0000 7fcfc1200640 1 Processor -- start 2024-09-13T08:42:16.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.912+0000 7fcfc1200640 1 -- start start 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc0ce3c0 0x7fcfbc09c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 0x7fcfbc0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc0cb9d0 con 0x7fcfbc15bd70 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc0cb850 con 0x7fcfbc15f9e0 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.913+0000 7fcfc1200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc0cbc50 con 0x7fcfbc0ce3c0 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57752/0 (socket says 172.21.15.31:57752) 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 -- 172.21.15.31:0/3866241052 learned_addr learned my addr 172.21.15.31:0/3866241052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9e00640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc0ce3c0 0x7fcfbc09c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfba800640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 0x7fcfbc0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 -- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc0ce3c0 msgr2=0x7fcfbc09c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc0ce3c0 0x7fcfbc09c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 -- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 msgr2=0x7fcfbc0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 0x7fcfbc0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 -- 172.21.15.31:0/3866241052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc0b2d60 con 0x7fcfbc15bd70 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfba800640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 0x7fcfbc0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.914+0000 7fcfb9400640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800d750 tx=0x7fcfa800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.915+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8017070 con 0x7fcfbc15bd70 2024-09-13T08:42:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.915+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa8009d20 con 0x7fcfbc15bd70 2024-09-13T08:42:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.915+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800de60 con 0x7fcfbc15bd70 2024-09-13T08:42:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.915+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc0b3040 con 0x7fcfbc15bd70 2024-09-13T08:42:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.915+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc0b3600 con 0x7fcfbc15bd70 2024-09-13T08:42:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.916+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfbc0ccd90 con 0x7fcfbc15bd70 2024-09-13T08:42:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.917+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fcfa8004280 con 0x7fcfbc15bd70 2024-09-13T08:42:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.918+0000 7fcfb2000640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 0x7fcf9c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.918+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fcfa809ab00 con 0x7fcfbc15bd70 2024-09-13T08:42:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.918+0000 7fcfb9e00640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 0x7fcf9c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.919+0000 7fcfb9e00640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 0x7fcf9c07ae30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009bd0 tx=0x7fcfac005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:16.922+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcfa80628f0 con 0x7fcfbc15bd70 2024-09-13T08:42:17.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.155+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfbc0cf820 con 0x7fcfbc15bd70 2024-09-13T08:42:17.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.156+0000 7fcfb2000640 1 -- 172.21.15.31:0/3866241052 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5225 (secure 0 0 0) 0x7fcfa8067320 con 0x7fcfbc15bd70 2024-09-13T08:42:17.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:17.157 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":57,"btime":"2024-09-13T08:42:16:723057+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3778871942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3778871942},{"type":"v1","addr":"172.21.15.45:6835","nonce":3778871942}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:17.157 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 57 2024-09-13T08:42:17.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 msgr2=0x7fcf9c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 0x7fcf9c07ae30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009bd0 tx=0x7fcfac005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:17.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 msgr2=0x7fcfbc09d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800d750 tx=0x7fcfa800dc10 comp rx=0 tx=0).stop 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 shutdown_connections 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcf9c078970 0x7fcf9c07ae30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfbc15f9e0 0x7fcfbc0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfbc15bd70 0x7fcfbc09d090 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 --2- 172.21.15.31:0/3866241052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcfbc0ce3c0 0x7fcfbc09c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 >> 172.21.15.31:0/3866241052 conn(0x7fcfbc0cc910 msgr2=0x7fcfbc0ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 shutdown_connections 2024-09-13T08:42:17.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.158+0000 7fcfc1200640 1 -- 172.21.15.31:0/3866241052 wait complete. 2024-09-13T08:42:17.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.334+0000 7f8a70e00640 1 -- 172.21.15.31:0/2309074398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15f9e0 msgr2=0x7f8a6c15fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.334+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2309074398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15f9e0 0x7f8a6c15fe20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009830 tx=0x7f8a5402a520 comp rx=0 tx=0).stop 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 -- 172.21.15.31:0/2309074398 shutdown_connections 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2309074398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15f9e0 0x7f8a6c15fe20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2309074398 >> v2:172.21.15.45:3300/0 conn(0x7f8a6c15bd70 0x7f8a6c0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 -- 172.21.15.31:0/2309074398 >> 172.21.15.31:0/2309074398 conn(0x7f8a6c0cc910 msgr2=0x7f8a6c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 -- 172.21.15.31:0/2309074398 shutdown_connections 2024-09-13T08:42:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.335+0000 7f8a70e00640 1 -- 172.21.15.31:0/2309074398 wait complete. 2024-09-13T08:42:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.336+0000 7f8a70e00640 1 Processor -- start 2024-09-13T08:42:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.336+0000 7f8a70e00640 1 -- start start 2024-09-13T08:42:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a6c0ce3c0 0x7f8a6c09cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15bd70 0x7f8a6c0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0cb9d0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0cb850 con 0x7f8a6c0ce3c0 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.337+0000 7f8a70e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c0cc360 con 0x7f8a6c15bd70 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57764/0 (socket says 172.21.15.31:57764) 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 -- 172.21.15.31:0/2494160197 learned_addr learned my addr 172.21.15.31:0/2494160197 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a61400640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15bd70 0x7f8a6c0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a69e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a6c0ce3c0 0x7f8a6c09cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 -- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15bd70 msgr2=0x7f8a6c0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15bd70 0x7f8a6c0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 -- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a6c0ce3c0 msgr2=0x7f8a6c09cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a6c0ce3c0 0x7f8a6c09cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 -- 172.21.15.31:0/2494160197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c09aab0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.338+0000 7f8a6a800640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c7a0 tx=0x7f8a6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.339+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64003b70 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.339+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a64003d10 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.339+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c09adf0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.339+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400aab0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.339+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c09b630 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.340+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6c0cf850 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.341+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f8a6400ac50 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.342+0000 7f8a62a00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 0x7f8a4807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.342+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f8a6409a9c0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.342+0000 7f8a69e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 0x7f8a4807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.343+0000 7f8a69e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 0x7f8a4807ae30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009530 tx=0x7f8a5c009260 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:17.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.346+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8a6c09adf0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.539+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a6c0b5df0 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.540+0000 7f8a62a00640 1 -- 172.21.15.31:0/2494160197 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16684 (secure 0 0 0) 0x7f8a64062720 con 0x7f8a6c15f9e0 2024-09-13T08:42:17.541 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:17.542 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:14.710157+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:10.935693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:12.097514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 msgr2=0x7f8a4807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 0x7f8a4807ae30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009530 tx=0x7f8a5c009260 comp rx=0 tx=0).stop 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 msgr2=0x7f8a6c09bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c7a0 tx=0x7f8a6400cc60 comp rx=0 tx=0).stop 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 shutdown_connections 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a48078970 0x7f8a4807ae30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6c15f9e0 0x7f8a6c09bb30 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a6c15bd70 0x7f8a6c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 --2- 172.21.15.31:0/2494160197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a6c0ce3c0 0x7f8a6c09cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 >> 172.21.15.31:0/2494160197 conn(0x7f8a6c0cc910 msgr2=0x7f8a6c0ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 shutdown_connections 2024-09-13T08:42:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.543+0000 7f8a70e00640 1 -- 172.21.15.31:0/2494160197 wait complete. 2024-09-13T08:42:17.560 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:42:17.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:42:17.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 -- 172.21.15.31:0/2088740398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c15bd70 msgr2=0x7efd0c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 --2- 172.21.15.31:0/2088740398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c15bd70 0x7efd0c0cf850 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0c6f10 tx=0x7efcfc02a560 comp rx=0 tx=0).stop 2024-09-13T08:42:17.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 -- 172.21.15.31:0/2088740398 shutdown_connections 2024-09-13T08:42:17.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 --2- 172.21.15.31:0/2088740398 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c0ce3c0 0x7efd0c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 --2- 172.21.15.31:0/2088740398 >> v2:172.21.15.45:3300/0 conn(0x7efd0c15f9e0 0x7efd0c15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 --2- 172.21.15.31:0/2088740398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c15bd70 0x7efd0c0cf850 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 -- 172.21.15.31:0/2088740398 >> 172.21.15.31:0/2088740398 conn(0x7efd0c0cc910 msgr2=0x7efd0c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 -- 172.21.15.31:0/2088740398 shutdown_connections 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.721+0000 7efd10a00640 1 -- 172.21.15.31:0/2088740398 wait complete. 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.723+0000 7efd10a00640 1 Processor -- start 2024-09-13T08:42:17.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.723+0000 7efd10a00640 1 -- start start 2024-09-13T08:42:17.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c15bd70 0x7efd0c14bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c15f9e0 0x7efd0c14e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0cb9d0 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0cb850 con 0x7efd0c15f9e0 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd10a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c0cbc50 con 0x7efd0c15bd70 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd09e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd09e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57778/0 (socket says 172.21.15.31:57778) 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd09e00640 1 -- 172.21.15.31:0/2364193510 learned_addr learned my addr 172.21.15.31:0/2364193510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.724+0000 7efd01400640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c15bd70 0x7efd0c14bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd0a800640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c15f9e0 0x7efd0c14e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 -- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c15bd70 msgr2=0x7efd0c14bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c15bd70 0x7efd0c14bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 -- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c15f9e0 msgr2=0x7efd0c14e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c15f9e0 0x7efd0c14e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 -- 172.21.15.31:0/2364193510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c14ebf0 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd09e00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009850 tx=0x7efcfc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc03f070 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcfc0047a0 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc035030 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.725+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0c14eed0 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.726+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0c0c5c50 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.727+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0c0cf850 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.728+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7efcfc042070 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.729+0000 7efd02a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 0x7efce807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:17.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.729+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7efcfc0bb9e0 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.729+0000 7efd01400640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 0x7efce807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:17.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.730+0000 7efd01400640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 0x7efce807ae30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0b38f0 tx=0x7efcf400b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:17.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.733+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efcfc083740 con 0x7efd0c0ce3c0 2024-09-13T08:42:17.881 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:42:17.880+0000 7f0456000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:42:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.926+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd0c0ae8d0 con 0x7efce8078970 2024-09-13T08:42:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.931+0000 7efd02a00640 1 -- 172.21.15.31:0/2364193510 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252079 (secure 0 0 0) 0x7efd0c0ae8d0 con 0x7efce8078970 2024-09-13T08:42:17.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:17.937 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.935+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 msgr2=0x7efce807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.935+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 0x7efce807ae30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0b38f0 tx=0x7efcf400b2e0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.935+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 msgr2=0x7efd0c0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.935+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009850 tx=0x7efcfc003910 comp rx=0 tx=0).stop 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 shutdown_connections 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078970 0x7efce807ae30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c15f9e0 0x7efd0c14e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c15bd70 0x7efd0c14bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 --2- 172.21.15.31:0/2364193510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c0ce3c0 0x7efd0c0b33b0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:17.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 >> 172.21.15.31:0/2364193510 conn(0x7efd0c0cc910 msgr2=0x7efd0c15b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:17.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 shutdown_connections 2024-09-13T08:42:17.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:17.936+0000 7efd10a00640 1 -- 172.21.15.31:0/2364193510 wait complete. 2024-09-13T08:42:17.953 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-13T08:42:16.964319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263516,"kb_used_data":9800,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711204,"statfs":{"total":773094113280,"available":772824272896,"internally_reserved":0,"allocated":10035200,"data_stored":4973036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.279769"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868063+0000","last_change":"2024-09-13T08:42:13.727717+0000","last_active":"2024-09-13T08:42:14.868063+0000","last_peered":"2024-09-13T08:42:14.868063+0000","last_clean":"2024-09-13T08:42:14.868063+0000","last_became_active":"2024-09-13T08:42:13.727422+0000","last_became_peered":"2024-09-13T08:42:13.727422+0000","last_unstale":"2024-09-13T08:42:14.868063+0000","last_undegraded":"2024-09-13T08:42:14.868063+0000","last_fullsized":"2024-09-13T08:42:14.868063+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.890339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721178+0000","last_change":"2024-09-13T08:42:13.717934+0000","last_active":"2024-09-13T08:42:14.721178+0000","last_peered":"2024-09-13T08:42:14.721178+0000","last_clean":"2024-09-13T08:42:14.721178+0000","last_became_active":"2024-09-13T08:42:13.717666+0000","last_became_peered":"2024-09-13T08:42:13.717666+0000","last_unstale":"2024-09-13T08:42:14.721178+0000","last_undegraded":"2024-09-13T08:42:14.721178+0000","last_fullsized":"2024-09-13T08:42:14.721178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:03.318733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927875+0000","last_change":"2024-09-13T08:42:13.717033+0000","last_active":"2024-09-13T08:42:14.927875+0000","last_peered":"2024-09-13T08:42:14.927875+0000","last_clean":"2024-09-13T08:42:14.927875+0000","last_became_active":"2024-09-13T08:42:13.716769+0000","last_became_peered":"2024-09-13T08:42:13.716769+0000","last_unstale":"2024-09-13T08:42:14.927875+0000","last_undegraded":"2024-09-13T08:42:14.927875+0000","last_fullsized":"2024-09-13T08:42:14.927875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:13.927727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928750+0000","last_change":"2024-09-13T08:42:13.723529+0000","last_active":"2024-09-13T08:42:14.928750+0000","last_peered":"2024-09-13T08:42:14.928750+0000","last_clean":"2024-09-13T08:42:14.928750+0000","last_became_active":"2024-09-13T08:42:13.722988+0000","last_became_peered":"2024-09-13T08:42:13.722988+0000","last_unstale":"2024-09-13T08:42:14.928750+0000","last_undegraded":"2024-09-13T08:42:14.928750+0000","last_fullsized":"2024-09-13T08:42:14.928750+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:45.201238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926478+0000","last_change":"2024-09-13T08:42:13.714158+0000","last_active":"2024-09-13T08:42:14.926478+0000","last_peered":"2024-09-13T08:42:14.926478+0000","last_clean":"2024-09-13T08:42:14.926478+0000","last_became_active":"2024-09-13T08:42:13.713830+0000","last_became_peered":"2024-09-13T08:42:13.713830+0000","last_unstale":"2024-09-13T08:42:14.926478+0000","last_undegraded":"2024-09-13T08:42:14.926478+0000","last_fullsized":"2024-09-13T08:42:14.926478+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:36.860483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927608+0000","last_change":"2024-09-13T08:42:13.719723+0000","last_active":"2024-09-13T08:42:14.927608+0000","last_peered":"2024-09-13T08:42:14.927608+0000","last_clean":"2024-09-13T08:42:14.927608+0000","last_became_active":"2024-09-13T08:42:13.719445+0000","last_became_peered":"2024-09-13T08:42:13.719445+0000","last_unstale":"2024-09-13T08:42:14.927608+0000","last_undegraded":"2024-09-13T08:42:14.927608+0000","last_fullsized":"2024-09-13T08:42:14.927608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:25:04.934648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720763+0000","last_change":"2024-09-13T08:42:13.709204+0000","last_active":"2024-09-13T08:42:14.720763+0000","last_peered":"2024-09-13T08:42:14.720763+0000","last_clean":"2024-09-13T08:42:14.720763+0000","last_became_active":"2024-09-13T08:42:13.708846+0000","last_became_peered":"2024-09-13T08:42:13.708846+0000","last_unstale":"2024-09-13T08:42:14.720763+0000","last_undegraded":"2024-09-13T08:42:14.720763+0000","last_fullsized":"2024-09-13T08:42:14.720763+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:21.173905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718712+0000","last_change":"2024-09-13T08:42:13.718001+0000","last_active":"2024-09-13T08:42:14.718712+0000","last_peered":"2024-09-13T08:42:14.718712+0000","last_clean":"2024-09-13T08:42:14.718712+0000","last_became_active":"2024-09-13T08:42:13.715806+0000","last_became_peered":"2024-09-13T08:42:13.715806+0000","last_unstale":"2024-09-13T08:42:14.718712+0000","last_undegraded":"2024-09-13T08:42:14.718712+0000","last_fullsized":"2024-09-13T08:42:14.718712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:00.134424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868184+0000","last_change":"2024-09-13T08:42:13.734450+0000","last_active":"2024-09-13T08:42:14.868184+0000","last_peered":"2024-09-13T08:42:14.868184+0000","last_clean":"2024-09-13T08:42:14.868184+0000","last_became_active":"2024-09-13T08:42:13.734154+0000","last_became_peered":"2024-09-13T08:42:13.734154+0000","last_unstale":"2024-09-13T08:42:14.868184+0000","last_undegraded":"2024-09-13T08:42:14.868184+0000","last_fullsized":"2024-09-13T08:42:14.868184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:47.892780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927937+0000","last_change":"2024-09-13T08:42:13.706263+0000","last_active":"2024-09-13T08:42:14.927937+0000","last_peered":"2024-09-13T08:42:14.927937+0000","last_clean":"2024-09-13T08:42:14.927937+0000","last_became_active":"2024-09-13T08:42:13.705967+0000","last_became_peered":"2024-09-13T08:42:13.705967+0000","last_unstale":"2024-09-13T08:42:14.927937+0000","last_undegraded":"2024-09-13T08:42:14.927937+0000","last_fullsized":"2024-09-13T08:42:14.927937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:09.450278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927598+0000","last_change":"2024-09-13T08:42:13.726096+0000","last_active":"2024-09-13T08:42:14.927598+0000","last_peered":"2024-09-13T08:42:14.927598+0000","last_clean":"2024-09-13T08:42:14.927598+0000","last_became_active":"2024-09-13T08:42:13.725858+0000","last_became_peered":"2024-09-13T08:42:13.725858+0000","last_unstale":"2024-09-13T08:42:14.927598+0000","last_undegraded":"2024-09-13T08:42:14.927598+0000","last_fullsized":"2024-09-13T08:42:14.927598+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:54.237813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718893+0000","last_change":"2024-09-13T08:42:13.728350+0000","last_active":"2024-09-13T08:42:14.718893+0000","last_peered":"2024-09-13T08:42:14.718893+0000","last_clean":"2024-09-13T08:42:14.718893+0000","last_became_active":"2024-09-13T08:42:13.728128+0000","last_became_peered":"2024-09-13T08:42:13.728128+0000","last_unstale":"2024-09-13T08:42:14.718893+0000","last_undegraded":"2024-09-13T08:42:14.718893+0000","last_fullsized":"2024-09-13T08:42:14.718893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:32.371215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926889+0000","last_change":"2024-09-13T08:42:13.733438+0000","last_active":"2024-09-13T08:42:14.926889+0000","last_peered":"2024-09-13T08:42:14.926889+0000","last_clean":"2024-09-13T08:42:14.926889+0000","last_became_active":"2024-09-13T08:42:13.733182+0000","last_became_peered":"2024-09-13T08:42:13.733182+0000","last_unstale":"2024-09-13T08:42:14.926889+0000","last_undegraded":"2024-09-13T08:42:14.926889+0000","last_fullsized":"2024-09-13T08:42:14.926889+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:53:05.488068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720747+0000","last_change":"2024-09-13T08:42:13.720292+0000","last_active":"2024-09-13T08:42:14.720747+0000","last_peered":"2024-09-13T08:42:14.720747+0000","last_clean":"2024-09-13T08:42:14.720747+0000","last_became_active":"2024-09-13T08:42:13.719871+0000","last_became_peered":"2024-09-13T08:42:13.719871+0000","last_unstale":"2024-09-13T08:42:14.720747+0000","last_undegraded":"2024-09-13T08:42:14.720747+0000","last_fullsized":"2024-09-13T08:42:14.720747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:42.823706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720240+0000","last_change":"2024-09-13T08:42:13.714961+0000","last_active":"2024-09-13T08:42:14.720240+0000","last_peered":"2024-09-13T08:42:14.720240+0000","last_clean":"2024-09-13T08:42:14.720240+0000","last_became_active":"2024-09-13T08:42:13.714657+0000","last_became_peered":"2024-09-13T08:42:13.714657+0000","last_unstale":"2024-09-13T08:42:14.720240+0000","last_undegraded":"2024-09-13T08:42:14.720240+0000","last_fullsized":"2024-09-13T08:42:14.720240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:25.609683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927254+0000","last_change":"2024-09-13T08:42:13.724074+0000","last_active":"2024-09-13T08:42:14.927254+0000","last_peered":"2024-09-13T08:42:14.927254+0000","last_clean":"2024-09-13T08:42:14.927254+0000","last_became_active":"2024-09-13T08:42:13.723771+0000","last_became_peered":"2024-09-13T08:42:13.723771+0000","last_unstale":"2024-09-13T08:42:14.927254+0000","last_undegraded":"2024-09-13T08:42:14.927254+0000","last_fullsized":"2024-09-13T08:42:14.927254+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:01.053496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719339+0000","last_change":"2024-09-13T08:42:13.728598+0000","last_active":"2024-09-13T08:42:14.719339+0000","last_peered":"2024-09-13T08:42:14.719339+0000","last_clean":"2024-09-13T08:42:14.719339+0000","last_became_active":"2024-09-13T08:42:13.728266+0000","last_became_peered":"2024-09-13T08:42:13.728266+0000","last_unstale":"2024-09-13T08:42:14.719339+0000","last_undegraded":"2024-09-13T08:42:14.719339+0000","last_fullsized":"2024-09-13T08:42:14.719339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:58:17.488347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927235+0000","last_change":"2024-09-13T08:42:13.738004+0000","last_active":"2024-09-13T08:42:14.927235+0000","last_peered":"2024-09-13T08:42:14.927235+0000","last_clean":"2024-09-13T08:42:14.927235+0000","last_became_active":"2024-09-13T08:42:13.737811+0000","last_became_peered":"2024-09-13T08:42:13.737811+0000","last_unstale":"2024-09-13T08:42:14.927235+0000","last_undegraded":"2024-09-13T08:42:14.927235+0000","last_fullsized":"2024-09-13T08:42:14.927235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:28.219291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720959+0000","last_change":"2024-09-13T08:42:13.717770+0000","last_active":"2024-09-13T08:42:14.720959+0000","last_peered":"2024-09-13T08:42:14.720959+0000","last_clean":"2024-09-13T08:42:14.720959+0000","last_became_active":"2024-09-13T08:42:13.717461+0000","last_became_peered":"2024-09-13T08:42:13.717461+0000","last_unstale":"2024-09-13T08:42:14.720959+0000","last_undegraded":"2024-09-13T08:42:14.720959+0000","last_fullsized":"2024-09-13T08:42:14.720959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:35.251021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721513+0000","last_change":"2024-09-13T08:42:13.718309+0000","last_active":"2024-09-13T08:42:14.721513+0000","last_peered":"2024-09-13T08:42:14.721513+0000","last_clean":"2024-09-13T08:42:14.721513+0000","last_became_active":"2024-09-13T08:42:13.718004+0000","last_became_peered":"2024-09-13T08:42:13.718004+0000","last_unstale":"2024-09-13T08:42:14.721513+0000","last_undegraded":"2024-09-13T08:42:14.721513+0000","last_fullsized":"2024-09-13T08:42:14.721513+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.890339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928650+0000","last_change":"2024-09-13T08:42:13.726325+0000","last_active":"2024-09-13T08:42:14.928650+0000","last_peered":"2024-09-13T08:42:14.928650+0000","last_clean":"2024-09-13T08:42:14.928650+0000","last_became_active":"2024-09-13T08:42:13.723647+0000","last_became_peered":"2024-09-13T08:42:13.723647+0000","last_unstale":"2024-09-13T08:42:14.928650+0000","last_undegraded":"2024-09-13T08:42:14.928650+0000","last_fullsized":"2024-09-13T08:42:14.928650+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:36:42.725742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720962+0000","last_change":"2024-09-13T08:42:13.706853+0000","last_active":"2024-09-13T08:42:14.720962+0000","last_peered":"2024-09-13T08:42:14.720962+0000","last_clean":"2024-09-13T08:42:14.720962+0000","last_became_active":"2024-09-13T08:42:13.705968+0000","last_became_peered":"2024-09-13T08:42:13.705968+0000","last_unstale":"2024-09-13T08:42:14.720962+0000","last_undegraded":"2024-09-13T08:42:14.720962+0000","last_fullsized":"2024-09-13T08:42:14.720962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:19.266805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718710+0000","last_change":"2024-09-13T08:42:13.717922+0000","last_active":"2024-09-13T08:42:14.718710+0000","last_peered":"2024-09-13T08:42:14.718710+0000","last_clean":"2024-09-13T08:42:14.718710+0000","last_became_active":"2024-09-13T08:42:13.717663+0000","last_became_peered":"2024-09-13T08:42:13.717663+0000","last_unstale":"2024-09-13T08:42:14.718710+0000","last_undegraded":"2024-09-13T08:42:14.718710+0000","last_fullsized":"2024-09-13T08:42:14.718710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:06.306557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927267+0000","last_change":"2024-09-13T08:42:13.720025+0000","last_active":"2024-09-13T08:42:14.927267+0000","last_peered":"2024-09-13T08:42:14.927267+0000","last_clean":"2024-09-13T08:42:14.927267+0000","last_became_active":"2024-09-13T08:42:13.714339+0000","last_became_peered":"2024-09-13T08:42:13.714339+0000","last_unstale":"2024-09-13T08:42:14.927267+0000","last_undegraded":"2024-09-13T08:42:14.927267+0000","last_fullsized":"2024-09-13T08:42:14.927267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:53.575506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927687+0000","last_change":"2024-09-13T08:42:13.734026+0000","last_active":"2024-09-13T08:42:14.927687+0000","last_peered":"2024-09-13T08:42:14.927687+0000","last_clean":"2024-09-13T08:42:14.927687+0000","last_became_active":"2024-09-13T08:42:13.733254+0000","last_became_peered":"2024-09-13T08:42:13.733254+0000","last_unstale":"2024-09-13T08:42:14.927687+0000","last_undegraded":"2024-09-13T08:42:14.927687+0000","last_fullsized":"2024-09-13T08:42:14.927687+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:47.212345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926722+0000","last_change":"2024-09-13T08:42:13.721968+0000","last_active":"2024-09-13T08:42:14.926722+0000","last_peered":"2024-09-13T08:42:14.926722+0000","last_clean":"2024-09-13T08:42:14.926722+0000","last_became_active":"2024-09-13T08:42:13.721320+0000","last_became_peered":"2024-09-13T08:42:13.721320+0000","last_unstale":"2024-09-13T08:42:14.926722+0000","last_undegraded":"2024-09-13T08:42:14.926722+0000","last_fullsized":"2024-09-13T08:42:14.926722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:25.783453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926902+0000","last_change":"2024-09-13T08:42:13.718719+0000","last_active":"2024-09-13T08:42:14.926902+0000","last_peered":"2024-09-13T08:42:14.926902+0000","last_clean":"2024-09-13T08:42:14.926902+0000","last_became_active":"2024-09-13T08:42:13.718460+0000","last_became_peered":"2024-09-13T08:42:13.718460+0000","last_unstale":"2024-09-13T08:42:14.926902+0000","last_undegraded":"2024-09-13T08:42:14.926902+0000","last_fullsized":"2024-09-13T08:42:14.926902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:39.280717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867966+0000","last_change":"2024-09-13T08:42:13.734182+0000","last_active":"2024-09-13T08:42:14.867966+0000","last_peered":"2024-09-13T08:42:14.867966+0000","last_clean":"2024-09-13T08:42:14.867966+0000","last_became_active":"2024-09-13T08:42:13.733912+0000","last_became_peered":"2024-09-13T08:42:13.733912+0000","last_unstale":"2024-09-13T08:42:14.867966+0000","last_undegraded":"2024-09-13T08:42:14.867966+0000","last_fullsized":"2024-09-13T08:42:14.867966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:24.588964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928784+0000","last_change":"2024-09-13T08:42:13.728566+0000","last_active":"2024-09-13T08:42:14.928784+0000","last_peered":"2024-09-13T08:42:14.928784+0000","last_clean":"2024-09-13T08:42:14.928784+0000","last_became_active":"2024-09-13T08:42:13.728200+0000","last_became_peered":"2024-09-13T08:42:13.728200+0000","last_unstale":"2024-09-13T08:42:14.928784+0000","last_undegraded":"2024-09-13T08:42:14.928784+0000","last_fullsized":"2024-09-13T08:42:14.928784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:39.765773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720907+0000","last_change":"2024-09-13T08:42:13.718642+0000","last_active":"2024-09-13T08:42:14.720907+0000","last_peered":"2024-09-13T08:42:14.720907+0000","last_clean":"2024-09-13T08:42:14.720907+0000","last_became_active":"2024-09-13T08:42:13.717548+0000","last_became_peered":"2024-09-13T08:42:13.717548+0000","last_unstale":"2024-09-13T08:42:14.720907+0000","last_undegraded":"2024-09-13T08:42:14.720907+0000","last_fullsized":"2024-09-13T08:42:14.720907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:01.754683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927447+0000","last_change":"2024-09-13T08:42:13.717314+0000","last_active":"2024-09-13T08:42:14.927447+0000","last_peered":"2024-09-13T08:42:14.927447+0000","last_clean":"2024-09-13T08:42:14.927447+0000","last_became_active":"2024-09-13T08:42:13.715807+0000","last_became_peered":"2024-09-13T08:42:13.715807+0000","last_unstale":"2024-09-13T08:42:14.927447+0000","last_undegraded":"2024-09-13T08:42:14.927447+0000","last_fullsized":"2024-09-13T08:42:14.927447+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:14.609541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927259+0000","last_change":"2024-09-13T08:42:13.713984+0000","last_active":"2024-09-13T08:42:14.927259+0000","last_peered":"2024-09-13T08:42:14.927259+0000","last_clean":"2024-09-13T08:42:14.927259+0000","last_became_active":"2024-09-13T08:42:13.713707+0000","last_became_peered":"2024-09-13T08:42:13.713707+0000","last_unstale":"2024-09-13T08:42:14.927259+0000","last_undegraded":"2024-09-13T08:42:14.927259+0000","last_fullsized":"2024-09-13T08:42:14.927259+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:06.306557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719460+0000","last_change":"2024-09-13T08:42:13.738304+0000","last_active":"2024-09-13T08:42:14.719460+0000","last_peered":"2024-09-13T08:42:14.719460+0000","last_clean":"2024-09-13T08:42:14.719460+0000","last_became_active":"2024-09-13T08:42:13.736841+0000","last_became_peered":"2024-09-13T08:42:13.736841+0000","last_unstale":"2024-09-13T08:42:14.719460+0000","last_undegraded":"2024-09-13T08:42:14.719460+0000","last_fullsized":"2024-09-13T08:42:14.719460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:10.798955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"89'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926641+0000","last_change":"2024-09-13T08:42:12.710331+0000","last_active":"2024-09-13T08:42:14.926641+0000","last_peered":"2024-09-13T08:42:14.926641+0000","last_clean":"2024-09-13T08:42:14.926641+0000","last_became_active":"2024-09-13T08:42:12.707336+0000","last_became_peered":"2024-09-13T08:42:12.707336+0000","last_unstale":"2024-09-13T08:42:14.926641+0000","last_undegraded":"2024-09-13T08:42:14.926641+0000","last_fullsized":"2024-09-13T08:42:14.926641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:05.856547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868293+0000","last_change":"2024-09-13T08:42:13.733527+0000","last_active":"2024-09-13T08:42:14.868293+0000","last_peered":"2024-09-13T08:42:14.868293+0000","last_clean":"2024-09-13T08:42:14.868293+0000","last_became_active":"2024-09-13T08:42:13.733241+0000","last_became_peered":"2024-09-13T08:42:13.733241+0000","last_unstale":"2024-09-13T08:42:14.868293+0000","last_undegraded":"2024-09-13T08:42:14.868293+0000","last_fullsized":"2024-09-13T08:42:14.868293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:43:10.763388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927843+0000","last_change":"2024-09-13T08:42:12.715005+0000","last_active":"2024-09-13T08:42:14.927843+0000","last_peered":"2024-09-13T08:42:14.927843+0000","last_clean":"2024-09-13T08:42:14.927843+0000","last_became_active":"2024-09-13T08:42:12.714756+0000","last_became_peered":"2024-09-13T08:42:12.714756+0000","last_unstale":"2024-09-13T08:42:14.927843+0000","last_undegraded":"2024-09-13T08:42:14.927843+0000","last_fullsized":"2024-09-13T08:42:14.927843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:17.407632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719029+0000","last_change":"2024-09-13T08:42:13.735466+0000","last_active":"2024-09-13T08:42:14.719029+0000","last_peered":"2024-09-13T08:42:14.719029+0000","last_clean":"2024-09-13T08:42:14.719029+0000","last_became_active":"2024-09-13T08:42:13.735009+0000","last_became_peered":"2024-09-13T08:42:13.735009+0000","last_unstale":"2024-09-13T08:42:14.719029+0000","last_undegraded":"2024-09-13T08:42:14.719029+0000","last_fullsized":"2024-09-13T08:42:14.719029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:34.919273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927600+0000","last_change":"2024-09-13T08:42:12.713812+0000","last_active":"2024-09-13T08:42:14.927600+0000","last_peered":"2024-09-13T08:42:14.927600+0000","last_clean":"2024-09-13T08:42:14.927600+0000","last_became_active":"2024-09-13T08:42:12.713386+0000","last_became_peered":"2024-09-13T08:42:12.713386+0000","last_unstale":"2024-09-13T08:42:14.927600+0000","last_undegraded":"2024-09-13T08:42:14.927600+0000","last_fullsized":"2024-09-13T08:42:14.927600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:14:10.679987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868301+0000","last_change":"2024-09-13T08:42:13.728918+0000","last_active":"2024-09-13T08:42:14.868301+0000","last_peered":"2024-09-13T08:42:14.868301+0000","last_clean":"2024-09-13T08:42:14.868301+0000","last_became_active":"2024-09-13T08:42:13.727479+0000","last_became_peered":"2024-09-13T08:42:13.727479+0000","last_unstale":"2024-09-13T08:42:14.868301+0000","last_undegraded":"2024-09-13T08:42:14.868301+0000","last_fullsized":"2024-09-13T08:42:14.868301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:15.394252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928661+0000","last_change":"2024-09-13T08:42:12.715846+0000","last_active":"2024-09-13T08:42:14.928661+0000","last_peered":"2024-09-13T08:42:14.928661+0000","last_clean":"2024-09-13T08:42:14.928661+0000","last_became_active":"2024-09-13T08:42:12.715294+0000","last_became_peered":"2024-09-13T08:42:12.715294+0000","last_unstale":"2024-09-13T08:42:14.928661+0000","last_undegraded":"2024-09-13T08:42:14.928661+0000","last_fullsized":"2024-09-13T08:42:14.928661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:53:05.142788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928590+0000","last_change":"2024-09-13T08:42:13.729047+0000","last_active":"2024-09-13T08:42:14.928590+0000","last_peered":"2024-09-13T08:42:14.928590+0000","last_clean":"2024-09-13T08:42:14.928590+0000","last_became_active":"2024-09-13T08:42:13.728208+0000","last_became_peered":"2024-09-13T08:42:13.728208+0000","last_unstale":"2024-09-13T08:42:14.928590+0000","last_undegraded":"2024-09-13T08:42:14.928590+0000","last_fullsized":"2024-09-13T08:42:14.928590+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718567+0000","last_change":"2024-09-13T08:42:12.714445+0000","last_active":"2024-09-13T08:42:14.718567+0000","last_peered":"2024-09-13T08:42:14.718567+0000","last_clean":"2024-09-13T08:42:14.718567+0000","last_became_active":"2024-09-13T08:42:12.714140+0000","last_became_peered":"2024-09-13T08:42:12.714140+0000","last_unstale":"2024-09-13T08:42:14.718567+0000","last_undegraded":"2024-09-13T08:42:14.718567+0000","last_fullsized":"2024-09-13T08:42:14.718567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:29.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927635+0000","last_change":"2024-09-13T08:42:13.713892+0000","last_active":"2024-09-13T08:42:14.927635+0000","last_peered":"2024-09-13T08:42:14.927635+0000","last_clean":"2024-09-13T08:42:14.927635+0000","last_became_active":"2024-09-13T08:42:13.713564+0000","last_became_peered":"2024-09-13T08:42:13.713564+0000","last_unstale":"2024-09-13T08:42:14.927635+0000","last_undegraded":"2024-09-13T08:42:14.927635+0000","last_fullsized":"2024-09-13T08:42:14.927635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:57:30.530634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927303+0000","last_change":"2024-09-13T08:42:12.710419+0000","last_active":"2024-09-13T08:42:14.927303+0000","last_peered":"2024-09-13T08:42:14.927303+0000","last_clean":"2024-09-13T08:42:14.927303+0000","last_became_active":"2024-09-13T08:42:12.710147+0000","last_became_peered":"2024-09-13T08:42:12.710147+0000","last_unstale":"2024-09-13T08:42:14.927303+0000","last_undegraded":"2024-09-13T08:42:14.927303+0000","last_fullsized":"2024-09-13T08:42:14.927303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:07.477664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868148+0000","last_change":"2024-09-13T08:42:13.730948+0000","last_active":"2024-09-13T08:42:14.868148+0000","last_peered":"2024-09-13T08:42:14.868148+0000","last_clean":"2024-09-13T08:42:14.868148+0000","last_became_active":"2024-09-13T08:42:13.728197+0000","last_became_peered":"2024-09-13T08:42:13.728197+0000","last_unstale":"2024-09-13T08:42:14.868148+0000","last_undegraded":"2024-09-13T08:42:14.868148+0000","last_fullsized":"2024-09-13T08:42:14.868148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:53.450154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720446+0000","last_change":"2024-09-13T08:42:12.708727+0000","last_active":"2024-09-13T08:42:14.720446+0000","last_peered":"2024-09-13T08:42:14.720446+0000","last_clean":"2024-09-13T08:42:14.720446+0000","last_became_active":"2024-09-13T08:42:12.708464+0000","last_became_peered":"2024-09-13T08:42:12.708464+0000","last_unstale":"2024-09-13T08:42:14.720446+0000","last_undegraded":"2024-09-13T08:42:14.720446+0000","last_fullsized":"2024-09-13T08:42:14.720446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:38.741966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720531+0000","last_change":"2024-09-13T08:42:13.709833+0000","last_active":"2024-09-13T08:42:14.720531+0000","last_peered":"2024-09-13T08:42:14.720531+0000","last_clean":"2024-09-13T08:42:14.720531+0000","last_became_active":"2024-09-13T08:42:13.709246+0000","last_became_peered":"2024-09-13T08:42:13.709246+0000","last_unstale":"2024-09-13T08:42:14.720531+0000","last_undegraded":"2024-09-13T08:42:14.720531+0000","last_fullsized":"2024-09-13T08:42:14.720531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:19.591015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719800+0000","last_change":"2024-09-13T08:42:12.703140+0000","last_active":"2024-09-13T08:42:14.719800+0000","last_peered":"2024-09-13T08:42:14.719800+0000","last_clean":"2024-09-13T08:42:14.719800+0000","last_became_active":"2024-09-13T08:42:12.702749+0000","last_became_peered":"2024-09-13T08:42:12.702749+0000","last_unstale":"2024-09-13T08:42:14.719800+0000","last_undegraded":"2024-09-13T08:42:14.719800+0000","last_fullsized":"2024-09-13T08:42:14.719800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:36.296266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721515+0000","last_change":"2024-09-13T08:42:13.724639+0000","last_active":"2024-09-13T08:42:14.721515+0000","last_peered":"2024-09-13T08:42:14.721515+0000","last_clean":"2024-09-13T08:42:14.721515+0000","last_became_active":"2024-09-13T08:42:13.724369+0000","last_became_peered":"2024-09-13T08:42:13.724369+0000","last_unstale":"2024-09-13T08:42:14.721515+0000","last_undegraded":"2024-09-13T08:42:14.721515+0000","last_fullsized":"2024-09-13T08:42:14.721515+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:23.951819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868375+0000","last_change":"2024-09-13T08:42:12.713647+0000","last_active":"2024-09-13T08:42:14.868375+0000","last_peered":"2024-09-13T08:42:14.868375+0000","last_clean":"2024-09-13T08:42:14.868375+0000","last_became_active":"2024-09-13T08:42:12.713363+0000","last_became_peered":"2024-09-13T08:42:12.713363+0000","last_unstale":"2024-09-13T08:42:14.868375+0000","last_undegraded":"2024-09-13T08:42:14.868375+0000","last_fullsized":"2024-09-13T08:42:14.868375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:09.250464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868391+0000","last_change":"2024-09-13T08:42:13.727302+0000","last_active":"2024-09-13T08:42:14.868391+0000","last_peered":"2024-09-13T08:42:14.868391+0000","last_clean":"2024-09-13T08:42:14.868391+0000","last_became_active":"2024-09-13T08:42:13.727004+0000","last_became_peered":"2024-09-13T08:42:13.727004+0000","last_unstale":"2024-09-13T08:42:14.868391+0000","last_undegraded":"2024-09-13T08:42:14.868391+0000","last_fullsized":"2024-09-13T08:42:14.868391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:08.486252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927933+0000","last_change":"2024-09-13T08:42:12.712013+0000","last_active":"2024-09-13T08:42:14.927933+0000","last_peered":"2024-09-13T08:42:14.927933+0000","last_clean":"2024-09-13T08:42:14.927933+0000","last_became_active":"2024-09-13T08:42:12.711652+0000","last_became_peered":"2024-09-13T08:42:12.711652+0000","last_unstale":"2024-09-13T08:42:14.927933+0000","last_undegraded":"2024-09-13T08:42:14.927933+0000","last_fullsized":"2024-09-13T08:42:14.927933+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:29.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927346+0000","last_change":"2024-09-13T08:42:13.733535+0000","last_active":"2024-09-13T08:42:14.927346+0000","last_peered":"2024-09-13T08:42:14.927346+0000","last_clean":"2024-09-13T08:42:14.927346+0000","last_became_active":"2024-09-13T08:42:13.733270+0000","last_became_peered":"2024-09-13T08:42:13.733270+0000","last_unstale":"2024-09-13T08:42:14.927346+0000","last_undegraded":"2024-09-13T08:42:14.927346+0000","last_fullsized":"2024-09-13T08:42:14.927346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:20.984324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"89'3","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868278+0000","last_change":"2024-09-13T08:42:12.713350+0000","last_active":"2024-09-13T08:42:14.868278+0000","last_peered":"2024-09-13T08:42:14.868278+0000","last_clean":"2024-09-13T08:42:14.868278+0000","last_became_active":"2024-09-13T08:42:12.713100+0000","last_became_peered":"2024-09-13T08:42:12.713100+0000","last_unstale":"2024-09-13T08:42:14.868278+0000","last_undegraded":"2024-09-13T08:42:14.868278+0000","last_fullsized":"2024-09-13T08:42:14.868278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:38.741966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926495+0000","last_change":"2024-09-13T08:42:13.735284+0000","last_active":"2024-09-13T08:42:14.926495+0000","last_peered":"2024-09-13T08:42:14.926495+0000","last_clean":"2024-09-13T08:42:14.926495+0000","last_became_active":"2024-09-13T08:42:13.730356+0000","last_became_peered":"2024-09-13T08:42:13.730356+0000","last_unstale":"2024-09-13T08:42:14.926495+0000","last_undegraded":"2024-09-13T08:42:14.926495+0000","last_fullsized":"2024-09-13T08:42:14.926495+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:26.822678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"89'6","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718958+0000","last_change":"2024-09-13T08:42:12.712658+0000","last_active":"2024-09-13T08:42:14.718958+0000","last_peered":"2024-09-13T08:42:14.718958+0000","last_clean":"2024-09-13T08:42:14.718958+0000","last_became_active":"2024-09-13T08:42:12.710047+0000","last_became_peered":"2024-09-13T08:42:12.710047+0000","last_unstale":"2024-09-13T08:42:14.718958+0000","last_undegraded":"2024-09-13T08:42:14.718958+0000","last_fullsized":"2024-09-13T08:42:14.718958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:10:08.826239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927719+0000","last_change":"2024-09-13T08:42:13.726453+0000","last_active":"2024-09-13T08:42:14.927719+0000","last_peered":"2024-09-13T08:42:14.927719+0000","last_clean":"2024-09-13T08:42:14.927719+0000","last_became_active":"2024-09-13T08:42:13.726170+0000","last_became_peered":"2024-09-13T08:42:13.726170+0000","last_unstale":"2024-09-13T08:42:14.927719+0000","last_undegraded":"2024-09-13T08:42:14.927719+0000","last_fullsized":"2024-09-13T08:42:14.927719+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:05.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928754+0000","last_change":"2024-09-13T08:42:12.700992+0000","last_active":"2024-09-13T08:42:14.928754+0000","last_peered":"2024-09-13T08:42:14.928754+0000","last_clean":"2024-09-13T08:42:14.928754+0000","last_became_active":"2024-09-13T08:42:12.700716+0000","last_became_peered":"2024-09-13T08:42:12.700716+0000","last_unstale":"2024-09-13T08:42:14.928754+0000","last_undegraded":"2024-09-13T08:42:14.928754+0000","last_fullsized":"2024-09-13T08:42:14.928754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:12.919139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867872+0000","last_change":"2024-09-13T08:42:13.731958+0000","last_active":"2024-09-13T08:42:14.867872+0000","last_peered":"2024-09-13T08:42:14.867872+0000","last_clean":"2024-09-13T08:42:14.867872+0000","last_became_active":"2024-09-13T08:42:13.731600+0000","last_became_peered":"2024-09-13T08:42:13.731600+0000","last_unstale":"2024-09-13T08:42:14.867872+0000","last_undegraded":"2024-09-13T08:42:14.867872+0000","last_fullsized":"2024-09-13T08:42:14.867872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:35.251021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719956+0000","last_change":"2024-09-13T08:42:12.703010+0000","last_active":"2024-09-13T08:42:14.719956+0000","last_peered":"2024-09-13T08:42:14.719956+0000","last_clean":"2024-09-13T08:42:14.719956+0000","last_became_active":"2024-09-13T08:42:12.701624+0000","last_became_peered":"2024-09-13T08:42:12.701624+0000","last_unstale":"2024-09-13T08:42:14.719956+0000","last_undegraded":"2024-09-13T08:42:14.719956+0000","last_fullsized":"2024-09-13T08:42:14.719956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:24:37.200950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719731+0000","last_change":"2024-09-13T08:42:13.716361+0000","last_active":"2024-09-13T08:42:14.719731+0000","last_peered":"2024-09-13T08:42:14.719731+0000","last_clean":"2024-09-13T08:42:14.719731+0000","last_became_active":"2024-09-13T08:42:13.716072+0000","last_became_peered":"2024-09-13T08:42:13.716072+0000","last_unstale":"2024-09-13T08:42:14.719731+0000","last_undegraded":"2024-09-13T08:42:14.719731+0000","last_fullsized":"2024-09-13T08:42:14.719731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:36.476139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927183+0000","last_change":"2024-09-13T08:42:12.708818+0000","last_active":"2024-09-13T08:42:14.927183+0000","last_peered":"2024-09-13T08:42:14.927183+0000","last_clean":"2024-09-13T08:42:14.927183+0000","last_became_active":"2024-09-13T08:42:12.708525+0000","last_became_peered":"2024-09-13T08:42:12.708525+0000","last_unstale":"2024-09-13T08:42:14.927183+0000","last_undegraded":"2024-09-13T08:42:14.927183+0000","last_fullsized":"2024-09-13T08:42:14.927183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:10:08.826239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927670+0000","last_change":"2024-09-13T08:42:13.714439+0000","last_active":"2024-09-13T08:42:14.927670+0000","last_peered":"2024-09-13T08:42:14.927670+0000","last_clean":"2024-09-13T08:42:14.927670+0000","last_became_active":"2024-09-13T08:42:13.713781+0000","last_became_peered":"2024-09-13T08:42:13.713781+0000","last_unstale":"2024-09-13T08:42:14.927670+0000","last_undegraded":"2024-09-13T08:42:14.927670+0000","last_fullsized":"2024-09-13T08:42:14.927670+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:16.744542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"89'8","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927347+0000","last_change":"2024-09-13T08:42:12.705046+0000","last_active":"2024-09-13T08:42:14.927347+0000","last_peered":"2024-09-13T08:42:14.927347+0000","last_clean":"2024-09-13T08:42:14.927347+0000","last_became_active":"2024-09-13T08:42:12.704771+0000","last_became_peered":"2024-09-13T08:42:12.704771+0000","last_unstale":"2024-09-13T08:42:14.927347+0000","last_undegraded":"2024-09-13T08:42:14.927347+0000","last_fullsized":"2024-09-13T08:42:14.927347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:49.100502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719406+0000","last_change":"2024-09-13T08:42:13.723934+0000","last_active":"2024-09-13T08:42:14.719406+0000","last_peered":"2024-09-13T08:42:14.719406+0000","last_clean":"2024-09-13T08:42:14.719406+0000","last_became_active":"2024-09-13T08:42:13.723588+0000","last_became_peered":"2024-09-13T08:42:13.723588+0000","last_unstale":"2024-09-13T08:42:14.719406+0000","last_undegraded":"2024-09-13T08:42:14.719406+0000","last_fullsized":"2024-09-13T08:42:14.719406+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:35.206566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926829+0000","last_change":"2024-09-13T08:42:12.707681+0000","last_active":"2024-09-13T08:42:14.926829+0000","last_peered":"2024-09-13T08:42:14.926829+0000","last_clean":"2024-09-13T08:42:14.926829+0000","last_became_active":"2024-09-13T08:42:12.703340+0000","last_became_peered":"2024-09-13T08:42:12.703340+0000","last_unstale":"2024-09-13T08:42:14.926829+0000","last_undegraded":"2024-09-13T08:42:14.926829+0000","last_fullsized":"2024-09-13T08:42:14.926829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:00.375320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928049+0000","last_change":"2024-09-13T08:42:13.711490+0000","last_active":"2024-09-13T08:42:14.928049+0000","last_peered":"2024-09-13T08:42:14.928049+0000","last_clean":"2024-09-13T08:42:14.928049+0000","last_became_active":"2024-09-13T08:42:13.711161+0000","last_became_peered":"2024-09-13T08:42:13.711161+0000","last_unstale":"2024-09-13T08:42:14.928049+0000","last_undegraded":"2024-09-13T08:42:14.928049+0000","last_fullsized":"2024-09-13T08:42:14.928049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:10.122330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721355+0000","last_change":"2024-09-13T08:42:12.713537+0000","last_active":"2024-09-13T08:42:14.721355+0000","last_peered":"2024-09-13T08:42:14.721355+0000","last_clean":"2024-09-13T08:42:14.721355+0000","last_became_active":"2024-09-13T08:42:12.713066+0000","last_became_peered":"2024-09-13T08:42:12.713066+0000","last_unstale":"2024-09-13T08:42:14.721355+0000","last_undegraded":"2024-09-13T08:42:14.721355+0000","last_fullsized":"2024-09-13T08:42:14.721355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:49.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718921+0000","last_change":"2024-09-13T08:42:13.734177+0000","last_active":"2024-09-13T08:42:14.718921+0000","last_peered":"2024-09-13T08:42:14.718921+0000","last_clean":"2024-09-13T08:42:14.718921+0000","last_became_active":"2024-09-13T08:42:13.725500+0000","last_became_peered":"2024-09-13T08:42:13.725500+0000","last_unstale":"2024-09-13T08:42:14.718921+0000","last_undegraded":"2024-09-13T08:42:14.718921+0000","last_fullsized":"2024-09-13T08:42:14.718921+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:02.088155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721352+0000","last_change":"2024-09-13T08:42:12.707224+0000","last_active":"2024-09-13T08:42:14.721352+0000","last_peered":"2024-09-13T08:42:14.721352+0000","last_clean":"2024-09-13T08:42:14.721352+0000","last_became_active":"2024-09-13T08:42:12.706885+0000","last_became_peered":"2024-09-13T08:42:12.706885+0000","last_unstale":"2024-09-13T08:42:14.721352+0000","last_undegraded":"2024-09-13T08:42:14.721352+0000","last_fullsized":"2024-09-13T08:42:14.721352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:19.575172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719887+0000","last_change":"2024-09-13T08:42:13.724271+0000","last_active":"2024-09-13T08:42:14.719887+0000","last_peered":"2024-09-13T08:42:14.719887+0000","last_clean":"2024-09-13T08:42:14.719887+0000","last_became_active":"2024-09-13T08:42:13.724000+0000","last_became_peered":"2024-09-13T08:42:13.724000+0000","last_unstale":"2024-09-13T08:42:14.719887+0000","last_undegraded":"2024-09-13T08:42:14.719887+0000","last_fullsized":"2024-09-13T08:42:14.719887+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:09.509192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721078+0000","last_change":"2024-09-13T08:42:12.700805+0000","last_active":"2024-09-13T08:42:14.721078+0000","last_peered":"2024-09-13T08:42:14.721078+0000","last_clean":"2024-09-13T08:42:14.721078+0000","last_became_active":"2024-09-13T08:42:12.696554+0000","last_became_peered":"2024-09-13T08:42:12.696554+0000","last_unstale":"2024-09-13T08:42:14.721078+0000","last_undegraded":"2024-09-13T08:42:14.721078+0000","last_fullsized":"2024-09-13T08:42:14.721078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:09.250464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926688+0000","last_change":"2024-09-13T08:42:13.734609+0000","last_active":"2024-09-13T08:42:14.926688+0000","last_peered":"2024-09-13T08:42:14.926688+0000","last_clean":"2024-09-13T08:42:14.926688+0000","last_became_active":"2024-09-13T08:42:13.734314+0000","last_became_peered":"2024-09-13T08:42:13.734314+0000","last_unstale":"2024-09-13T08:42:14.926688+0000","last_undegraded":"2024-09-13T08:42:14.926688+0000","last_fullsized":"2024-09-13T08:42:14.926688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:44:54.575723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720071+0000","last_change":"2024-09-13T08:42:12.700040+0000","last_active":"2024-09-13T08:42:14.720071+0000","last_peered":"2024-09-13T08:42:14.720071+0000","last_clean":"2024-09-13T08:42:14.720071+0000","last_became_active":"2024-09-13T08:42:12.697873+0000","last_became_peered":"2024-09-13T08:42:12.697873+0000","last_unstale":"2024-09-13T08:42:14.720071+0000","last_undegraded":"2024-09-13T08:42:14.720071+0000","last_fullsized":"2024-09-13T08:42:14.720071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:22.140235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928619+0000","last_change":"2024-09-13T08:42:13.735644+0000","last_active":"2024-09-13T08:42:14.928619+0000","last_peered":"2024-09-13T08:42:14.928619+0000","last_clean":"2024-09-13T08:42:14.928619+0000","last_became_active":"2024-09-13T08:42:13.735307+0000","last_became_peered":"2024-09-13T08:42:13.735307+0000","last_unstale":"2024-09-13T08:42:14.928619+0000","last_undegraded":"2024-09-13T08:42:14.928619+0000","last_fullsized":"2024-09-13T08:42:14.928619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:48.320271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927001+0000","last_change":"2024-09-13T08:42:12.714731+0000","last_active":"2024-09-13T08:42:14.927001+0000","last_peered":"2024-09-13T08:42:14.927001+0000","last_clean":"2024-09-13T08:42:14.927001+0000","last_became_active":"2024-09-13T08:42:12.714329+0000","last_became_peered":"2024-09-13T08:42:12.714329+0000","last_unstale":"2024-09-13T08:42:14.927001+0000","last_undegraded":"2024-09-13T08:42:14.927001+0000","last_fullsized":"2024-09-13T08:42:14.927001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:21.131402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718759+0000","last_change":"2024-09-13T08:42:13.719341+0000","last_active":"2024-09-13T08:42:14.718759+0000","last_peered":"2024-09-13T08:42:14.718759+0000","last_clean":"2024-09-13T08:42:14.718759+0000","last_became_active":"2024-09-13T08:42:13.719077+0000","last_became_peered":"2024-09-13T08:42:13.719077+0000","last_unstale":"2024-09-13T08:42:14.718759+0000","last_undegraded":"2024-09-13T08:42:14.718759+0000","last_fullsized":"2024-09-13T08:42:14.718759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:55:18.456339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719901+0000","last_change":"2024-09-13T08:42:12.707897+0000","last_active":"2024-09-13T08:42:14.719901+0000","last_peered":"2024-09-13T08:42:14.719901+0000","last_clean":"2024-09-13T08:42:14.719901+0000","last_became_active":"2024-09-13T08:42:12.707460+0000","last_became_peered":"2024-09-13T08:42:12.707460+0000","last_unstale":"2024-09-13T08:42:14.719901+0000","last_undegraded":"2024-09-13T08:42:14.719901+0000","last_fullsized":"2024-09-13T08:42:14.719901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:26.008882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926970+0000","last_change":"2024-09-13T08:42:13.722289+0000","last_active":"2024-09-13T08:42:14.926970+0000","last_peered":"2024-09-13T08:42:14.926970+0000","last_clean":"2024-09-13T08:42:14.926970+0000","last_became_active":"2024-09-13T08:42:13.721690+0000","last_became_peered":"2024-09-13T08:42:13.721690+0000","last_unstale":"2024-09-13T08:42:14.926970+0000","last_undegraded":"2024-09-13T08:42:14.926970+0000","last_fullsized":"2024-09-13T08:42:14.926970+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:08.814209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720620+0000","last_change":"2024-09-13T08:42:12.704047+0000","last_active":"2024-09-13T08:42:14.720620+0000","last_peered":"2024-09-13T08:42:14.720620+0000","last_clean":"2024-09-13T08:42:14.720620+0000","last_became_active":"2024-09-13T08:42:12.703727+0000","last_became_peered":"2024-09-13T08:42:12.703727+0000","last_unstale":"2024-09-13T08:42:14.720620+0000","last_undegraded":"2024-09-13T08:42:14.720620+0000","last_fullsized":"2024-09-13T08:42:14.720620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:56.762704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926566+0000","last_change":"2024-09-13T08:42:13.721259+0000","last_active":"2024-09-13T08:42:14.926566+0000","last_peered":"2024-09-13T08:42:14.926566+0000","last_clean":"2024-09-13T08:42:14.926566+0000","last_became_active":"2024-09-13T08:42:13.720830+0000","last_became_peered":"2024-09-13T08:42:13.720830+0000","last_unstale":"2024-09-13T08:42:14.926566+0000","last_undegraded":"2024-09-13T08:42:14.926566+0000","last_fullsized":"2024-09-13T08:42:14.926566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:16.055450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927320+0000","last_change":"2024-09-13T08:42:12.708197+0000","last_active":"2024-09-13T08:42:14.927320+0000","last_peered":"2024-09-13T08:42:14.927320+0000","last_clean":"2024-09-13T08:42:14.927320+0000","last_became_active":"2024-09-13T08:42:12.707866+0000","last_became_peered":"2024-09-13T08:42:12.707866+0000","last_unstale":"2024-09-13T08:42:14.927320+0000","last_undegraded":"2024-09-13T08:42:14.927320+0000","last_fullsized":"2024-09-13T08:42:14.927320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:35.467551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926935+0000","last_change":"2024-09-13T08:42:13.716732+0000","last_active":"2024-09-13T08:42:14.926935+0000","last_peered":"2024-09-13T08:42:14.926935+0000","last_clean":"2024-09-13T08:42:14.926935+0000","last_became_active":"2024-09-13T08:42:13.713391+0000","last_became_peered":"2024-09-13T08:42:13.713391+0000","last_unstale":"2024-09-13T08:42:14.926935+0000","last_undegraded":"2024-09-13T08:42:14.926935+0000","last_fullsized":"2024-09-13T08:42:14.926935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:48:23.616441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718987+0000","last_change":"2024-09-13T08:42:12.711439+0000","last_active":"2024-09-13T08:42:14.718987+0000","last_peered":"2024-09-13T08:42:14.718987+0000","last_clean":"2024-09-13T08:42:14.718987+0000","last_became_active":"2024-09-13T08:42:12.711185+0000","last_became_peered":"2024-09-13T08:42:12.711185+0000","last_unstale":"2024-09-13T08:42:14.718987+0000","last_undegraded":"2024-09-13T08:42:14.718987+0000","last_fullsized":"2024-09-13T08:42:14.718987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:16.061372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927900+0000","last_change":"2024-09-13T08:42:13.713281+0000","last_active":"2024-09-13T08:42:14.927900+0000","last_peered":"2024-09-13T08:42:14.927900+0000","last_clean":"2024-09-13T08:42:14.927900+0000","last_became_active":"2024-09-13T08:42:13.712453+0000","last_became_peered":"2024-09-13T08:42:13.712453+0000","last_unstale":"2024-09-13T08:42:14.927900+0000","last_undegraded":"2024-09-13T08:42:14.927900+0000","last_fullsized":"2024-09-13T08:42:14.927900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:53:06.151377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927615+0000","last_change":"2024-09-13T08:42:12.705295+0000","last_active":"2024-09-13T08:42:14.927615+0000","last_peered":"2024-09-13T08:42:14.927615+0000","last_clean":"2024-09-13T08:42:14.927615+0000","last_became_active":"2024-09-13T08:42:12.704995+0000","last_became_peered":"2024-09-13T08:42:12.704995+0000","last_unstale":"2024-09-13T08:42:14.927615+0000","last_undegraded":"2024-09-13T08:42:14.927615+0000","last_fullsized":"2024-09-13T08:42:14.927615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:16.061372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867774+0000","last_change":"2024-09-13T08:42:13.730098+0000","last_active":"2024-09-13T08:42:14.867774+0000","last_peered":"2024-09-13T08:42:14.867774+0000","last_clean":"2024-09-13T08:42:14.867774+0000","last_became_active":"2024-09-13T08:42:13.729598+0000","last_became_peered":"2024-09-13T08:42:13.729598+0000","last_unstale":"2024-09-13T08:42:14.867774+0000","last_undegraded":"2024-09-13T08:42:14.867774+0000","last_fullsized":"2024-09-13T08:42:14.867774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:19.591015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720567+0000","last_change":"2024-09-13T08:42:12.706098+0000","last_active":"2024-09-13T08:42:14.720567+0000","last_peered":"2024-09-13T08:42:14.720567+0000","last_clean":"2024-09-13T08:42:14.720567+0000","last_became_active":"2024-09-13T08:42:12.705718+0000","last_became_peered":"2024-09-13T08:42:12.705718+0000","last_unstale":"2024-09-13T08:42:14.720567+0000","last_undegraded":"2024-09-13T08:42:14.720567+0000","last_fullsized":"2024-09-13T08:42:14.720567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:41.518568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927385+0000","last_change":"2024-09-13T08:42:13.725508+0000","last_active":"2024-09-13T08:42:14.927385+0000","last_peered":"2024-09-13T08:42:14.927385+0000","last_clean":"2024-09-13T08:42:14.927385+0000","last_became_active":"2024-09-13T08:42:13.725231+0000","last_became_peered":"2024-09-13T08:42:13.725231+0000","last_unstale":"2024-09-13T08:42:14.927385+0000","last_undegraded":"2024-09-13T08:42:14.927385+0000","last_fullsized":"2024-09-13T08:42:14.927385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:10.827593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927403+0000","last_change":"2024-09-13T08:42:12.711334+0000","last_active":"2024-09-13T08:42:14.927403+0000","last_peered":"2024-09-13T08:42:14.927403+0000","last_clean":"2024-09-13T08:42:14.927403+0000","last_became_active":"2024-09-13T08:42:12.708192+0000","last_became_peered":"2024-09-13T08:42:12.708192+0000","last_unstale":"2024-09-13T08:42:14.927403+0000","last_undegraded":"2024-09-13T08:42:14.927403+0000","last_fullsized":"2024-09-13T08:42:14.927403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:29.994667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867775+0000","last_change":"2024-09-13T08:42:13.729205+0000","last_active":"2024-09-13T08:42:14.867775+0000","last_peered":"2024-09-13T08:42:14.867775+0000","last_clean":"2024-09-13T08:42:14.867775+0000","last_became_active":"2024-09-13T08:42:13.728059+0000","last_became_peered":"2024-09-13T08:42:13.728059+0000","last_unstale":"2024-09-13T08:42:14.867775+0000","last_undegraded":"2024-09-13T08:42:14.867775+0000","last_fullsized":"2024-09-13T08:42:14.867775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:44:34.685449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867619+0000","last_change":"2024-09-13T08:42:12.710424+0000","last_active":"2024-09-13T08:42:14.867619+0000","last_peered":"2024-09-13T08:42:14.867619+0000","last_clean":"2024-09-13T08:42:14.867619+0000","last_became_active":"2024-09-13T08:42:12.709779+0000","last_became_peered":"2024-09-13T08:42:12.709779+0000","last_unstale":"2024-09-13T08:42:14.867619+0000","last_undegraded":"2024-09-13T08:42:14.867619+0000","last_fullsized":"2024-09-13T08:42:14.867619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:49.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867847+0000","last_change":"2024-09-13T08:42:13.723867+0000","last_active":"2024-09-13T08:42:14.867847+0000","last_peered":"2024-09-13T08:42:14.867847+0000","last_clean":"2024-09-13T08:42:14.867847+0000","last_became_active":"2024-09-13T08:42:13.723608+0000","last_became_peered":"2024-09-13T08:42:13.723608+0000","last_unstale":"2024-09-13T08:42:14.867847+0000","last_undegraded":"2024-09-13T08:42:14.867847+0000","last_fullsized":"2024-09-13T08:42:14.867847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:01.754683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718637+0000","last_change":"2024-09-13T08:42:12.711454+0000","last_active":"2024-09-13T08:42:14.718637+0000","last_peered":"2024-09-13T08:42:14.718637+0000","last_clean":"2024-09-13T08:42:14.718637+0000","last_became_active":"2024-09-13T08:42:12.710883+0000","last_became_peered":"2024-09-13T08:42:12.710883+0000","last_unstale":"2024-09-13T08:42:14.718637+0000","last_undegraded":"2024-09-13T08:42:14.718637+0000","last_fullsized":"2024-09-13T08:42:14.718637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:29.994667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927518+0000","last_change":"2024-09-13T08:42:13.733533+0000","last_active":"2024-09-13T08:42:14.927518+0000","last_peered":"2024-09-13T08:42:14.927518+0000","last_clean":"2024-09-13T08:42:14.927518+0000","last_became_active":"2024-09-13T08:42:13.733228+0000","last_became_peered":"2024-09-13T08:42:13.733228+0000","last_unstale":"2024-09-13T08:42:14.927518+0000","last_undegraded":"2024-09-13T08:42:14.927518+0000","last_fullsized":"2024-09-13T08:42:14.927518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:44:24.755596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926961+0000","last_change":"2024-09-13T08:42:12.707227+0000","last_active":"2024-09-13T08:42:14.926961+0000","last_peered":"2024-09-13T08:42:14.926961+0000","last_clean":"2024-09-13T08:42:14.926961+0000","last_became_active":"2024-09-13T08:42:12.706759+0000","last_became_peered":"2024-09-13T08:42:12.706759+0000","last_unstale":"2024-09-13T08:42:14.926961+0000","last_undegraded":"2024-09-13T08:42:14.926961+0000","last_fullsized":"2024-09-13T08:42:14.926961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:39:17.508018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":259,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868008+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:42:14.868008+0000","last_peered":"2024-09-13T08:42:14.868008+0000","last_clean":"2024-09-13T08:42:14.868008+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:42:14.868008+0000","last_undegraded":"2024-09-13T08:42:14.868008+0000","last_fullsized":"2024-09-13T08:42:14.868008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509507,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1101824,"data_stored":473505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1105920,"data_stored":474107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509507,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1708032,"data_stored":1065449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542210,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36424,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335416,"statfs":{"total":96636764160,"available":96599465984,"internally_reserved":0,"allocated":1101824,"data_stored":473469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509507,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1110016,"data_stored":474109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1097728,"data_stored":473435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1695744,"data_stored":1063893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509506,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1114112,"data_stored":475069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:42:17.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:42:18.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 -- 172.21.15.31:0/951132627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70040cf850 msgr2=0x7f70040ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 --2- 172.21.15.31:0/951132627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70040cf850 0x7f70040ce3c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f70040ca470 tx=0x7f6ff402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 -- 172.21.15.31:0/951132627 shutdown_connections 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 --2- 172.21.15.31:0/951132627 >> v2:172.21.15.45:3300/0 conn(0x7f70040ce900 0x7f70040cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 --2- 172.21.15.31:0/951132627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70040cf850 0x7f70040ce3c0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 -- 172.21.15.31:0/951132627 >> 172.21.15.31:0/951132627 conn(0x7f70040cc910 msgr2=0x7f70040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 -- 172.21.15.31:0/951132627 shutdown_connections 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.154+0000 7f700c800640 1 -- 172.21.15.31:0/951132627 wait complete. 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.156+0000 7f700c800640 1 Processor -- start 2024-09-13T08:42:18.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.156+0000 7f700c800640 1 -- start start 2024-09-13T08:42:18.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70040ce900 0x7f70040700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70040cf850 0x7f70040c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70041600f0 con 0x7f700415bd70 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70040cb850 con 0x7f70040cf850 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700c800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f70040b2cc0 con 0x7f70040ce900 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57794/0 (socket says 172.21.15.31:57794) 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.157+0000 7f700a800640 1 -- 172.21.15.31:0/3309082105 learned_addr learned my addr 172.21.15.31:0/3309082105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f7009400640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70040cf850 0x7f70040c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f7009e00640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70040ce900 0x7f70040700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 -- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70040ce900 msgr2=0x7f70040700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70040ce900 0x7f70040700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 -- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70040cf850 msgr2=0x7f70040c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70040cf850 0x7f70040c2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 -- 172.21.15.31:0/3309082105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70040b2f10 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f700a800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f700000ead0 tx=0x7f700000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.158+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700000cde0 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.159+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7000004c10 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.159+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70040b31f0 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.159+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000005020 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.159+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70040b37b0 con 0x7f700415bd70 2024-09-13T08:42:18.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.160+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70040ce3c0 con 0x7f700415bd70 2024-09-13T08:42:18.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.161+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f70000051c0 con 0x7f700415bd70 2024-09-13T08:42:18.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.162+0000 7f6ffe000640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 0x7f6fe807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.162+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f700009b500 con 0x7f700415bd70 2024-09-13T08:42:18.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.162+0000 7f7009e00640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 0x7f6fe807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.163+0000 7f7009e00640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 0x7f6fe807ae30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f700409d9e0 tx=0x7f6ff8009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:18.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.166+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7000016030 con 0x7f700415bd70 2024-09-13T08:42:18.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.357+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f70040ae790 con 0x7f6fe8078970 2024-09-13T08:42:18.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.362+0000 7f6ffe000640 1 -- 172.21.15.31:0/3309082105 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252079 (secure 0 0 0) 0x7f70040ae790 con 0x7f6fe8078970 2024-09-13T08:42:18.366 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:18.367 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.365+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 msgr2=0x7f6fe807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 0x7f6fe807ae30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f700409d9e0 tx=0x7f6ff8009040 comp rx=0 tx=0).stop 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 msgr2=0x7f700409b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f700000ead0 tx=0x7f700000ef90 comp rx=0 tx=0).stop 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 shutdown_connections 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fe8078970 0x7f6fe807ae30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f700415bd70 0x7f700409b9e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70040cf850 0x7f70040c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 --2- 172.21.15.31:0/3309082105 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70040ce900 0x7f70040700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 >> 172.21.15.31:0/3309082105 conn(0x7f70040cc910 msgr2=0x7f70040acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:18.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 shutdown_connections 2024-09-13T08:42:18.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.366+0000 7f700c800640 1 -- 172.21.15.31:0/3309082105 wait complete. 2024-09-13T08:42:18.383 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-13T08:42:16.964319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263516,"kb_used_data":9800,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711204,"statfs":{"total":773094113280,"available":772824272896,"internally_reserved":0,"allocated":10035200,"data_stored":4973036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.279769"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868063+0000","last_change":"2024-09-13T08:42:13.727717+0000","last_active":"2024-09-13T08:42:14.868063+0000","last_peered":"2024-09-13T08:42:14.868063+0000","last_clean":"2024-09-13T08:42:14.868063+0000","last_became_active":"2024-09-13T08:42:13.727422+0000","last_became_peered":"2024-09-13T08:42:13.727422+0000","last_unstale":"2024-09-13T08:42:14.868063+0000","last_undegraded":"2024-09-13T08:42:14.868063+0000","last_fullsized":"2024-09-13T08:42:14.868063+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.890339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721178+0000","last_change":"2024-09-13T08:42:13.717934+0000","last_active":"2024-09-13T08:42:14.721178+0000","last_peered":"2024-09-13T08:42:14.721178+0000","last_clean":"2024-09-13T08:42:14.721178+0000","last_became_active":"2024-09-13T08:42:13.717666+0000","last_became_peered":"2024-09-13T08:42:13.717666+0000","last_unstale":"2024-09-13T08:42:14.721178+0000","last_undegraded":"2024-09-13T08:42:14.721178+0000","last_fullsized":"2024-09-13T08:42:14.721178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:03.318733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927875+0000","last_change":"2024-09-13T08:42:13.717033+0000","last_active":"2024-09-13T08:42:14.927875+0000","last_peered":"2024-09-13T08:42:14.927875+0000","last_clean":"2024-09-13T08:42:14.927875+0000","last_became_active":"2024-09-13T08:42:13.716769+0000","last_became_peered":"2024-09-13T08:42:13.716769+0000","last_unstale":"2024-09-13T08:42:14.927875+0000","last_undegraded":"2024-09-13T08:42:14.927875+0000","last_fullsized":"2024-09-13T08:42:14.927875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:13.927727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928750+0000","last_change":"2024-09-13T08:42:13.723529+0000","last_active":"2024-09-13T08:42:14.928750+0000","last_peered":"2024-09-13T08:42:14.928750+0000","last_clean":"2024-09-13T08:42:14.928750+0000","last_became_active":"2024-09-13T08:42:13.722988+0000","last_became_peered":"2024-09-13T08:42:13.722988+0000","last_unstale":"2024-09-13T08:42:14.928750+0000","last_undegraded":"2024-09-13T08:42:14.928750+0000","last_fullsized":"2024-09-13T08:42:14.928750+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:45.201238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926478+0000","last_change":"2024-09-13T08:42:13.714158+0000","last_active":"2024-09-13T08:42:14.926478+0000","last_peered":"2024-09-13T08:42:14.926478+0000","last_clean":"2024-09-13T08:42:14.926478+0000","last_became_active":"2024-09-13T08:42:13.713830+0000","last_became_peered":"2024-09-13T08:42:13.713830+0000","last_unstale":"2024-09-13T08:42:14.926478+0000","last_undegraded":"2024-09-13T08:42:14.926478+0000","last_fullsized":"2024-09-13T08:42:14.926478+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:36.860483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927608+0000","last_change":"2024-09-13T08:42:13.719723+0000","last_active":"2024-09-13T08:42:14.927608+0000","last_peered":"2024-09-13T08:42:14.927608+0000","last_clean":"2024-09-13T08:42:14.927608+0000","last_became_active":"2024-09-13T08:42:13.719445+0000","last_became_peered":"2024-09-13T08:42:13.719445+0000","last_unstale":"2024-09-13T08:42:14.927608+0000","last_undegraded":"2024-09-13T08:42:14.927608+0000","last_fullsized":"2024-09-13T08:42:14.927608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:25:04.934648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720763+0000","last_change":"2024-09-13T08:42:13.709204+0000","last_active":"2024-09-13T08:42:14.720763+0000","last_peered":"2024-09-13T08:42:14.720763+0000","last_clean":"2024-09-13T08:42:14.720763+0000","last_became_active":"2024-09-13T08:42:13.708846+0000","last_became_peered":"2024-09-13T08:42:13.708846+0000","last_unstale":"2024-09-13T08:42:14.720763+0000","last_undegraded":"2024-09-13T08:42:14.720763+0000","last_fullsized":"2024-09-13T08:42:14.720763+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:21.173905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718712+0000","last_change":"2024-09-13T08:42:13.718001+0000","last_active":"2024-09-13T08:42:14.718712+0000","last_peered":"2024-09-13T08:42:14.718712+0000","last_clean":"2024-09-13T08:42:14.718712+0000","last_became_active":"2024-09-13T08:42:13.715806+0000","last_became_peered":"2024-09-13T08:42:13.715806+0000","last_unstale":"2024-09-13T08:42:14.718712+0000","last_undegraded":"2024-09-13T08:42:14.718712+0000","last_fullsized":"2024-09-13T08:42:14.718712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:00.134424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868184+0000","last_change":"2024-09-13T08:42:13.734450+0000","last_active":"2024-09-13T08:42:14.868184+0000","last_peered":"2024-09-13T08:42:14.868184+0000","last_clean":"2024-09-13T08:42:14.868184+0000","last_became_active":"2024-09-13T08:42:13.734154+0000","last_became_peered":"2024-09-13T08:42:13.734154+0000","last_unstale":"2024-09-13T08:42:14.868184+0000","last_undegraded":"2024-09-13T08:42:14.868184+0000","last_fullsized":"2024-09-13T08:42:14.868184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:47.892780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927937+0000","last_change":"2024-09-13T08:42:13.706263+0000","last_active":"2024-09-13T08:42:14.927937+0000","last_peered":"2024-09-13T08:42:14.927937+0000","last_clean":"2024-09-13T08:42:14.927937+0000","last_became_active":"2024-09-13T08:42:13.705967+0000","last_became_peered":"2024-09-13T08:42:13.705967+0000","last_unstale":"2024-09-13T08:42:14.927937+0000","last_undegraded":"2024-09-13T08:42:14.927937+0000","last_fullsized":"2024-09-13T08:42:14.927937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:09.450278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927598+0000","last_change":"2024-09-13T08:42:13.726096+0000","last_active":"2024-09-13T08:42:14.927598+0000","last_peered":"2024-09-13T08:42:14.927598+0000","last_clean":"2024-09-13T08:42:14.927598+0000","last_became_active":"2024-09-13T08:42:13.725858+0000","last_became_peered":"2024-09-13T08:42:13.725858+0000","last_unstale":"2024-09-13T08:42:14.927598+0000","last_undegraded":"2024-09-13T08:42:14.927598+0000","last_fullsized":"2024-09-13T08:42:14.927598+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:54.237813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718893+0000","last_change":"2024-09-13T08:42:13.728350+0000","last_active":"2024-09-13T08:42:14.718893+0000","last_peered":"2024-09-13T08:42:14.718893+0000","last_clean":"2024-09-13T08:42:14.718893+0000","last_became_active":"2024-09-13T08:42:13.728128+0000","last_became_peered":"2024-09-13T08:42:13.728128+0000","last_unstale":"2024-09-13T08:42:14.718893+0000","last_undegraded":"2024-09-13T08:42:14.718893+0000","last_fullsized":"2024-09-13T08:42:14.718893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:32.371215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926889+0000","last_change":"2024-09-13T08:42:13.733438+0000","last_active":"2024-09-13T08:42:14.926889+0000","last_peered":"2024-09-13T08:42:14.926889+0000","last_clean":"2024-09-13T08:42:14.926889+0000","last_became_active":"2024-09-13T08:42:13.733182+0000","last_became_peered":"2024-09-13T08:42:13.733182+0000","last_unstale":"2024-09-13T08:42:14.926889+0000","last_undegraded":"2024-09-13T08:42:14.926889+0000","last_fullsized":"2024-09-13T08:42:14.926889+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:53:05.488068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720747+0000","last_change":"2024-09-13T08:42:13.720292+0000","last_active":"2024-09-13T08:42:14.720747+0000","last_peered":"2024-09-13T08:42:14.720747+0000","last_clean":"2024-09-13T08:42:14.720747+0000","last_became_active":"2024-09-13T08:42:13.719871+0000","last_became_peered":"2024-09-13T08:42:13.719871+0000","last_unstale":"2024-09-13T08:42:14.720747+0000","last_undegraded":"2024-09-13T08:42:14.720747+0000","last_fullsized":"2024-09-13T08:42:14.720747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:42.823706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720240+0000","last_change":"2024-09-13T08:42:13.714961+0000","last_active":"2024-09-13T08:42:14.720240+0000","last_peered":"2024-09-13T08:42:14.720240+0000","last_clean":"2024-09-13T08:42:14.720240+0000","last_became_active":"2024-09-13T08:42:13.714657+0000","last_became_peered":"2024-09-13T08:42:13.714657+0000","last_unstale":"2024-09-13T08:42:14.720240+0000","last_undegraded":"2024-09-13T08:42:14.720240+0000","last_fullsized":"2024-09-13T08:42:14.720240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:25.609683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927254+0000","last_change":"2024-09-13T08:42:13.724074+0000","last_active":"2024-09-13T08:42:14.927254+0000","last_peered":"2024-09-13T08:42:14.927254+0000","last_clean":"2024-09-13T08:42:14.927254+0000","last_became_active":"2024-09-13T08:42:13.723771+0000","last_became_peered":"2024-09-13T08:42:13.723771+0000","last_unstale":"2024-09-13T08:42:14.927254+0000","last_undegraded":"2024-09-13T08:42:14.927254+0000","last_fullsized":"2024-09-13T08:42:14.927254+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:01.053496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719339+0000","last_change":"2024-09-13T08:42:13.728598+0000","last_active":"2024-09-13T08:42:14.719339+0000","last_peered":"2024-09-13T08:42:14.719339+0000","last_clean":"2024-09-13T08:42:14.719339+0000","last_became_active":"2024-09-13T08:42:13.728266+0000","last_became_peered":"2024-09-13T08:42:13.728266+0000","last_unstale":"2024-09-13T08:42:14.719339+0000","last_undegraded":"2024-09-13T08:42:14.719339+0000","last_fullsized":"2024-09-13T08:42:14.719339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:58:17.488347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927235+0000","last_change":"2024-09-13T08:42:13.738004+0000","last_active":"2024-09-13T08:42:14.927235+0000","last_peered":"2024-09-13T08:42:14.927235+0000","last_clean":"2024-09-13T08:42:14.927235+0000","last_became_active":"2024-09-13T08:42:13.737811+0000","last_became_peered":"2024-09-13T08:42:13.737811+0000","last_unstale":"2024-09-13T08:42:14.927235+0000","last_undegraded":"2024-09-13T08:42:14.927235+0000","last_fullsized":"2024-09-13T08:42:14.927235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:28.219291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720959+0000","last_change":"2024-09-13T08:42:13.717770+0000","last_active":"2024-09-13T08:42:14.720959+0000","last_peered":"2024-09-13T08:42:14.720959+0000","last_clean":"2024-09-13T08:42:14.720959+0000","last_became_active":"2024-09-13T08:42:13.717461+0000","last_became_peered":"2024-09-13T08:42:13.717461+0000","last_unstale":"2024-09-13T08:42:14.720959+0000","last_undegraded":"2024-09-13T08:42:14.720959+0000","last_fullsized":"2024-09-13T08:42:14.720959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:35.251021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721513+0000","last_change":"2024-09-13T08:42:13.718309+0000","last_active":"2024-09-13T08:42:14.721513+0000","last_peered":"2024-09-13T08:42:14.721513+0000","last_clean":"2024-09-13T08:42:14.721513+0000","last_became_active":"2024-09-13T08:42:13.718004+0000","last_became_peered":"2024-09-13T08:42:13.718004+0000","last_unstale":"2024-09-13T08:42:14.721513+0000","last_undegraded":"2024-09-13T08:42:14.721513+0000","last_fullsized":"2024-09-13T08:42:14.721513+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.890339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928650+0000","last_change":"2024-09-13T08:42:13.726325+0000","last_active":"2024-09-13T08:42:14.928650+0000","last_peered":"2024-09-13T08:42:14.928650+0000","last_clean":"2024-09-13T08:42:14.928650+0000","last_became_active":"2024-09-13T08:42:13.723647+0000","last_became_peered":"2024-09-13T08:42:13.723647+0000","last_unstale":"2024-09-13T08:42:14.928650+0000","last_undegraded":"2024-09-13T08:42:14.928650+0000","last_fullsized":"2024-09-13T08:42:14.928650+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:36:42.725742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720962+0000","last_change":"2024-09-13T08:42:13.706853+0000","last_active":"2024-09-13T08:42:14.720962+0000","last_peered":"2024-09-13T08:42:14.720962+0000","last_clean":"2024-09-13T08:42:14.720962+0000","last_became_active":"2024-09-13T08:42:13.705968+0000","last_became_peered":"2024-09-13T08:42:13.705968+0000","last_unstale":"2024-09-13T08:42:14.720962+0000","last_undegraded":"2024-09-13T08:42:14.720962+0000","last_fullsized":"2024-09-13T08:42:14.720962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:19.266805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718710+0000","last_change":"2024-09-13T08:42:13.717922+0000","last_active":"2024-09-13T08:42:14.718710+0000","last_peered":"2024-09-13T08:42:14.718710+0000","last_clean":"2024-09-13T08:42:14.718710+0000","last_became_active":"2024-09-13T08:42:13.717663+0000","last_became_peered":"2024-09-13T08:42:13.717663+0000","last_unstale":"2024-09-13T08:42:14.718710+0000","last_undegraded":"2024-09-13T08:42:14.718710+0000","last_fullsized":"2024-09-13T08:42:14.718710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:06.306557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927267+0000","last_change":"2024-09-13T08:42:13.720025+0000","last_active":"2024-09-13T08:42:14.927267+0000","last_peered":"2024-09-13T08:42:14.927267+0000","last_clean":"2024-09-13T08:42:14.927267+0000","last_became_active":"2024-09-13T08:42:13.714339+0000","last_became_peered":"2024-09-13T08:42:13.714339+0000","last_unstale":"2024-09-13T08:42:14.927267+0000","last_undegraded":"2024-09-13T08:42:14.927267+0000","last_fullsized":"2024-09-13T08:42:14.927267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:53.575506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927687+0000","last_change":"2024-09-13T08:42:13.734026+0000","last_active":"2024-09-13T08:42:14.927687+0000","last_peered":"2024-09-13T08:42:14.927687+0000","last_clean":"2024-09-13T08:42:14.927687+0000","last_became_active":"2024-09-13T08:42:13.733254+0000","last_became_peered":"2024-09-13T08:42:13.733254+0000","last_unstale":"2024-09-13T08:42:14.927687+0000","last_undegraded":"2024-09-13T08:42:14.927687+0000","last_fullsized":"2024-09-13T08:42:14.927687+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:47.212345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926722+0000","last_change":"2024-09-13T08:42:13.721968+0000","last_active":"2024-09-13T08:42:14.926722+0000","last_peered":"2024-09-13T08:42:14.926722+0000","last_clean":"2024-09-13T08:42:14.926722+0000","last_became_active":"2024-09-13T08:42:13.721320+0000","last_became_peered":"2024-09-13T08:42:13.721320+0000","last_unstale":"2024-09-13T08:42:14.926722+0000","last_undegraded":"2024-09-13T08:42:14.926722+0000","last_fullsized":"2024-09-13T08:42:14.926722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:28:25.783453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926902+0000","last_change":"2024-09-13T08:42:13.718719+0000","last_active":"2024-09-13T08:42:14.926902+0000","last_peered":"2024-09-13T08:42:14.926902+0000","last_clean":"2024-09-13T08:42:14.926902+0000","last_became_active":"2024-09-13T08:42:13.718460+0000","last_became_peered":"2024-09-13T08:42:13.718460+0000","last_unstale":"2024-09-13T08:42:14.926902+0000","last_undegraded":"2024-09-13T08:42:14.926902+0000","last_fullsized":"2024-09-13T08:42:14.926902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:39.280717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867966+0000","last_change":"2024-09-13T08:42:13.734182+0000","last_active":"2024-09-13T08:42:14.867966+0000","last_peered":"2024-09-13T08:42:14.867966+0000","last_clean":"2024-09-13T08:42:14.867966+0000","last_became_active":"2024-09-13T08:42:13.733912+0000","last_became_peered":"2024-09-13T08:42:13.733912+0000","last_unstale":"2024-09-13T08:42:14.867966+0000","last_undegraded":"2024-09-13T08:42:14.867966+0000","last_fullsized":"2024-09-13T08:42:14.867966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:24.588964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928784+0000","last_change":"2024-09-13T08:42:13.728566+0000","last_active":"2024-09-13T08:42:14.928784+0000","last_peered":"2024-09-13T08:42:14.928784+0000","last_clean":"2024-09-13T08:42:14.928784+0000","last_became_active":"2024-09-13T08:42:13.728200+0000","last_became_peered":"2024-09-13T08:42:13.728200+0000","last_unstale":"2024-09-13T08:42:14.928784+0000","last_undegraded":"2024-09-13T08:42:14.928784+0000","last_fullsized":"2024-09-13T08:42:14.928784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:39.765773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720907+0000","last_change":"2024-09-13T08:42:13.718642+0000","last_active":"2024-09-13T08:42:14.720907+0000","last_peered":"2024-09-13T08:42:14.720907+0000","last_clean":"2024-09-13T08:42:14.720907+0000","last_became_active":"2024-09-13T08:42:13.717548+0000","last_became_peered":"2024-09-13T08:42:13.717548+0000","last_unstale":"2024-09-13T08:42:14.720907+0000","last_undegraded":"2024-09-13T08:42:14.720907+0000","last_fullsized":"2024-09-13T08:42:14.720907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:01.754683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927447+0000","last_change":"2024-09-13T08:42:13.717314+0000","last_active":"2024-09-13T08:42:14.927447+0000","last_peered":"2024-09-13T08:42:14.927447+0000","last_clean":"2024-09-13T08:42:14.927447+0000","last_became_active":"2024-09-13T08:42:13.715807+0000","last_became_peered":"2024-09-13T08:42:13.715807+0000","last_unstale":"2024-09-13T08:42:14.927447+0000","last_undegraded":"2024-09-13T08:42:14.927447+0000","last_fullsized":"2024-09-13T08:42:14.927447+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:14.609541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927259+0000","last_change":"2024-09-13T08:42:13.713984+0000","last_active":"2024-09-13T08:42:14.927259+0000","last_peered":"2024-09-13T08:42:14.927259+0000","last_clean":"2024-09-13T08:42:14.927259+0000","last_became_active":"2024-09-13T08:42:13.713707+0000","last_became_peered":"2024-09-13T08:42:13.713707+0000","last_unstale":"2024-09-13T08:42:14.927259+0000","last_undegraded":"2024-09-13T08:42:14.927259+0000","last_fullsized":"2024-09-13T08:42:14.927259+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:06.306557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719460+0000","last_change":"2024-09-13T08:42:13.738304+0000","last_active":"2024-09-13T08:42:14.719460+0000","last_peered":"2024-09-13T08:42:14.719460+0000","last_clean":"2024-09-13T08:42:14.719460+0000","last_became_active":"2024-09-13T08:42:13.736841+0000","last_became_peered":"2024-09-13T08:42:13.736841+0000","last_unstale":"2024-09-13T08:42:14.719460+0000","last_undegraded":"2024-09-13T08:42:14.719460+0000","last_fullsized":"2024-09-13T08:42:14.719460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:10.798955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"89'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926641+0000","last_change":"2024-09-13T08:42:12.710331+0000","last_active":"2024-09-13T08:42:14.926641+0000","last_peered":"2024-09-13T08:42:14.926641+0000","last_clean":"2024-09-13T08:42:14.926641+0000","last_became_active":"2024-09-13T08:42:12.707336+0000","last_became_peered":"2024-09-13T08:42:12.707336+0000","last_unstale":"2024-09-13T08:42:14.926641+0000","last_undegraded":"2024-09-13T08:42:14.926641+0000","last_fullsized":"2024-09-13T08:42:14.926641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:05.856547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868293+0000","last_change":"2024-09-13T08:42:13.733527+0000","last_active":"2024-09-13T08:42:14.868293+0000","last_peered":"2024-09-13T08:42:14.868293+0000","last_clean":"2024-09-13T08:42:14.868293+0000","last_became_active":"2024-09-13T08:42:13.733241+0000","last_became_peered":"2024-09-13T08:42:13.733241+0000","last_unstale":"2024-09-13T08:42:14.868293+0000","last_undegraded":"2024-09-13T08:42:14.868293+0000","last_fullsized":"2024-09-13T08:42:14.868293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:43:10.763388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927843+0000","last_change":"2024-09-13T08:42:12.715005+0000","last_active":"2024-09-13T08:42:14.927843+0000","last_peered":"2024-09-13T08:42:14.927843+0000","last_clean":"2024-09-13T08:42:14.927843+0000","last_became_active":"2024-09-13T08:42:12.714756+0000","last_became_peered":"2024-09-13T08:42:12.714756+0000","last_unstale":"2024-09-13T08:42:14.927843+0000","last_undegraded":"2024-09-13T08:42:14.927843+0000","last_fullsized":"2024-09-13T08:42:14.927843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:17.407632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719029+0000","last_change":"2024-09-13T08:42:13.735466+0000","last_active":"2024-09-13T08:42:14.719029+0000","last_peered":"2024-09-13T08:42:14.719029+0000","last_clean":"2024-09-13T08:42:14.719029+0000","last_became_active":"2024-09-13T08:42:13.735009+0000","last_became_peered":"2024-09-13T08:42:13.735009+0000","last_unstale":"2024-09-13T08:42:14.719029+0000","last_undegraded":"2024-09-13T08:42:14.719029+0000","last_fullsized":"2024-09-13T08:42:14.719029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:34.919273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927600+0000","last_change":"2024-09-13T08:42:12.713812+0000","last_active":"2024-09-13T08:42:14.927600+0000","last_peered":"2024-09-13T08:42:14.927600+0000","last_clean":"2024-09-13T08:42:14.927600+0000","last_became_active":"2024-09-13T08:42:12.713386+0000","last_became_peered":"2024-09-13T08:42:12.713386+0000","last_unstale":"2024-09-13T08:42:14.927600+0000","last_undegraded":"2024-09-13T08:42:14.927600+0000","last_fullsized":"2024-09-13T08:42:14.927600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:14:10.679987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868301+0000","last_change":"2024-09-13T08:42:13.728918+0000","last_active":"2024-09-13T08:42:14.868301+0000","last_peered":"2024-09-13T08:42:14.868301+0000","last_clean":"2024-09-13T08:42:14.868301+0000","last_became_active":"2024-09-13T08:42:13.727479+0000","last_became_peered":"2024-09-13T08:42:13.727479+0000","last_unstale":"2024-09-13T08:42:14.868301+0000","last_undegraded":"2024-09-13T08:42:14.868301+0000","last_fullsized":"2024-09-13T08:42:14.868301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:15.394252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928661+0000","last_change":"2024-09-13T08:42:12.715846+0000","last_active":"2024-09-13T08:42:14.928661+0000","last_peered":"2024-09-13T08:42:14.928661+0000","last_clean":"2024-09-13T08:42:14.928661+0000","last_became_active":"2024-09-13T08:42:12.715294+0000","last_became_peered":"2024-09-13T08:42:12.715294+0000","last_unstale":"2024-09-13T08:42:14.928661+0000","last_undegraded":"2024-09-13T08:42:14.928661+0000","last_fullsized":"2024-09-13T08:42:14.928661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:53:05.142788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928590+0000","last_change":"2024-09-13T08:42:13.729047+0000","last_active":"2024-09-13T08:42:14.928590+0000","last_peered":"2024-09-13T08:42:14.928590+0000","last_clean":"2024-09-13T08:42:14.928590+0000","last_became_active":"2024-09-13T08:42:13.728208+0000","last_became_peered":"2024-09-13T08:42:13.728208+0000","last_unstale":"2024-09-13T08:42:14.928590+0000","last_undegraded":"2024-09-13T08:42:14.928590+0000","last_fullsized":"2024-09-13T08:42:14.928590+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.116213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718567+0000","last_change":"2024-09-13T08:42:12.714445+0000","last_active":"2024-09-13T08:42:14.718567+0000","last_peered":"2024-09-13T08:42:14.718567+0000","last_clean":"2024-09-13T08:42:14.718567+0000","last_became_active":"2024-09-13T08:42:12.714140+0000","last_became_peered":"2024-09-13T08:42:12.714140+0000","last_unstale":"2024-09-13T08:42:14.718567+0000","last_undegraded":"2024-09-13T08:42:14.718567+0000","last_fullsized":"2024-09-13T08:42:14.718567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:29.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927635+0000","last_change":"2024-09-13T08:42:13.713892+0000","last_active":"2024-09-13T08:42:14.927635+0000","last_peered":"2024-09-13T08:42:14.927635+0000","last_clean":"2024-09-13T08:42:14.927635+0000","last_became_active":"2024-09-13T08:42:13.713564+0000","last_became_peered":"2024-09-13T08:42:13.713564+0000","last_unstale":"2024-09-13T08:42:14.927635+0000","last_undegraded":"2024-09-13T08:42:14.927635+0000","last_fullsized":"2024-09-13T08:42:14.927635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:57:30.530634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927303+0000","last_change":"2024-09-13T08:42:12.710419+0000","last_active":"2024-09-13T08:42:14.927303+0000","last_peered":"2024-09-13T08:42:14.927303+0000","last_clean":"2024-09-13T08:42:14.927303+0000","last_became_active":"2024-09-13T08:42:12.710147+0000","last_became_peered":"2024-09-13T08:42:12.710147+0000","last_unstale":"2024-09-13T08:42:14.927303+0000","last_undegraded":"2024-09-13T08:42:14.927303+0000","last_fullsized":"2024-09-13T08:42:14.927303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:07.477664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868148+0000","last_change":"2024-09-13T08:42:13.730948+0000","last_active":"2024-09-13T08:42:14.868148+0000","last_peered":"2024-09-13T08:42:14.868148+0000","last_clean":"2024-09-13T08:42:14.868148+0000","last_became_active":"2024-09-13T08:42:13.728197+0000","last_became_peered":"2024-09-13T08:42:13.728197+0000","last_unstale":"2024-09-13T08:42:14.868148+0000","last_undegraded":"2024-09-13T08:42:14.868148+0000","last_fullsized":"2024-09-13T08:42:14.868148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:53.450154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720446+0000","last_change":"2024-09-13T08:42:12.708727+0000","last_active":"2024-09-13T08:42:14.720446+0000","last_peered":"2024-09-13T08:42:14.720446+0000","last_clean":"2024-09-13T08:42:14.720446+0000","last_became_active":"2024-09-13T08:42:12.708464+0000","last_became_peered":"2024-09-13T08:42:12.708464+0000","last_unstale":"2024-09-13T08:42:14.720446+0000","last_undegraded":"2024-09-13T08:42:14.720446+0000","last_fullsized":"2024-09-13T08:42:14.720446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:38.741966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720531+0000","last_change":"2024-09-13T08:42:13.709833+0000","last_active":"2024-09-13T08:42:14.720531+0000","last_peered":"2024-09-13T08:42:14.720531+0000","last_clean":"2024-09-13T08:42:14.720531+0000","last_became_active":"2024-09-13T08:42:13.709246+0000","last_became_peered":"2024-09-13T08:42:13.709246+0000","last_unstale":"2024-09-13T08:42:14.720531+0000","last_undegraded":"2024-09-13T08:42:14.720531+0000","last_fullsized":"2024-09-13T08:42:14.720531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:19.591015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719800+0000","last_change":"2024-09-13T08:42:12.703140+0000","last_active":"2024-09-13T08:42:14.719800+0000","last_peered":"2024-09-13T08:42:14.719800+0000","last_clean":"2024-09-13T08:42:14.719800+0000","last_became_active":"2024-09-13T08:42:12.702749+0000","last_became_peered":"2024-09-13T08:42:12.702749+0000","last_unstale":"2024-09-13T08:42:14.719800+0000","last_undegraded":"2024-09-13T08:42:14.719800+0000","last_fullsized":"2024-09-13T08:42:14.719800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:36.296266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721515+0000","last_change":"2024-09-13T08:42:13.724639+0000","last_active":"2024-09-13T08:42:14.721515+0000","last_peered":"2024-09-13T08:42:14.721515+0000","last_clean":"2024-09-13T08:42:14.721515+0000","last_became_active":"2024-09-13T08:42:13.724369+0000","last_became_peered":"2024-09-13T08:42:13.724369+0000","last_unstale":"2024-09-13T08:42:14.721515+0000","last_undegraded":"2024-09-13T08:42:14.721515+0000","last_fullsized":"2024-09-13T08:42:14.721515+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:23.951819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868375+0000","last_change":"2024-09-13T08:42:12.713647+0000","last_active":"2024-09-13T08:42:14.868375+0000","last_peered":"2024-09-13T08:42:14.868375+0000","last_clean":"2024-09-13T08:42:14.868375+0000","last_became_active":"2024-09-13T08:42:12.713363+0000","last_became_peered":"2024-09-13T08:42:12.713363+0000","last_unstale":"2024-09-13T08:42:14.868375+0000","last_undegraded":"2024-09-13T08:42:14.868375+0000","last_fullsized":"2024-09-13T08:42:14.868375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:09.250464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868391+0000","last_change":"2024-09-13T08:42:13.727302+0000","last_active":"2024-09-13T08:42:14.868391+0000","last_peered":"2024-09-13T08:42:14.868391+0000","last_clean":"2024-09-13T08:42:14.868391+0000","last_became_active":"2024-09-13T08:42:13.727004+0000","last_became_peered":"2024-09-13T08:42:13.727004+0000","last_unstale":"2024-09-13T08:42:14.868391+0000","last_undegraded":"2024-09-13T08:42:14.868391+0000","last_fullsized":"2024-09-13T08:42:14.868391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:08.486252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927933+0000","last_change":"2024-09-13T08:42:12.712013+0000","last_active":"2024-09-13T08:42:14.927933+0000","last_peered":"2024-09-13T08:42:14.927933+0000","last_clean":"2024-09-13T08:42:14.927933+0000","last_became_active":"2024-09-13T08:42:12.711652+0000","last_became_peered":"2024-09-13T08:42:12.711652+0000","last_unstale":"2024-09-13T08:42:14.927933+0000","last_undegraded":"2024-09-13T08:42:14.927933+0000","last_fullsized":"2024-09-13T08:42:14.927933+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:29.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927346+0000","last_change":"2024-09-13T08:42:13.733535+0000","last_active":"2024-09-13T08:42:14.927346+0000","last_peered":"2024-09-13T08:42:14.927346+0000","last_clean":"2024-09-13T08:42:14.927346+0000","last_became_active":"2024-09-13T08:42:13.733270+0000","last_became_peered":"2024-09-13T08:42:13.733270+0000","last_unstale":"2024-09-13T08:42:14.927346+0000","last_undegraded":"2024-09-13T08:42:14.927346+0000","last_fullsized":"2024-09-13T08:42:14.927346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:20.984324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"89'3","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868278+0000","last_change":"2024-09-13T08:42:12.713350+0000","last_active":"2024-09-13T08:42:14.868278+0000","last_peered":"2024-09-13T08:42:14.868278+0000","last_clean":"2024-09-13T08:42:14.868278+0000","last_became_active":"2024-09-13T08:42:12.713100+0000","last_became_peered":"2024-09-13T08:42:12.713100+0000","last_unstale":"2024-09-13T08:42:14.868278+0000","last_undegraded":"2024-09-13T08:42:14.868278+0000","last_fullsized":"2024-09-13T08:42:14.868278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:38.741966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926495+0000","last_change":"2024-09-13T08:42:13.735284+0000","last_active":"2024-09-13T08:42:14.926495+0000","last_peered":"2024-09-13T08:42:14.926495+0000","last_clean":"2024-09-13T08:42:14.926495+0000","last_became_active":"2024-09-13T08:42:13.730356+0000","last_became_peered":"2024-09-13T08:42:13.730356+0000","last_unstale":"2024-09-13T08:42:14.926495+0000","last_undegraded":"2024-09-13T08:42:14.926495+0000","last_fullsized":"2024-09-13T08:42:14.926495+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:26.822678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"89'6","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718958+0000","last_change":"2024-09-13T08:42:12.712658+0000","last_active":"2024-09-13T08:42:14.718958+0000","last_peered":"2024-09-13T08:42:14.718958+0000","last_clean":"2024-09-13T08:42:14.718958+0000","last_became_active":"2024-09-13T08:42:12.710047+0000","last_became_peered":"2024-09-13T08:42:12.710047+0000","last_unstale":"2024-09-13T08:42:14.718958+0000","last_undegraded":"2024-09-13T08:42:14.718958+0000","last_fullsized":"2024-09-13T08:42:14.718958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:10:08.826239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927719+0000","last_change":"2024-09-13T08:42:13.726453+0000","last_active":"2024-09-13T08:42:14.927719+0000","last_peered":"2024-09-13T08:42:14.927719+0000","last_clean":"2024-09-13T08:42:14.927719+0000","last_became_active":"2024-09-13T08:42:13.726170+0000","last_became_peered":"2024-09-13T08:42:13.726170+0000","last_unstale":"2024-09-13T08:42:14.927719+0000","last_undegraded":"2024-09-13T08:42:14.927719+0000","last_fullsized":"2024-09-13T08:42:14.927719+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:05.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928754+0000","last_change":"2024-09-13T08:42:12.700992+0000","last_active":"2024-09-13T08:42:14.928754+0000","last_peered":"2024-09-13T08:42:14.928754+0000","last_clean":"2024-09-13T08:42:14.928754+0000","last_became_active":"2024-09-13T08:42:12.700716+0000","last_became_peered":"2024-09-13T08:42:12.700716+0000","last_unstale":"2024-09-13T08:42:14.928754+0000","last_undegraded":"2024-09-13T08:42:14.928754+0000","last_fullsized":"2024-09-13T08:42:14.928754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:12.919139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867872+0000","last_change":"2024-09-13T08:42:13.731958+0000","last_active":"2024-09-13T08:42:14.867872+0000","last_peered":"2024-09-13T08:42:14.867872+0000","last_clean":"2024-09-13T08:42:14.867872+0000","last_became_active":"2024-09-13T08:42:13.731600+0000","last_became_peered":"2024-09-13T08:42:13.731600+0000","last_unstale":"2024-09-13T08:42:14.867872+0000","last_undegraded":"2024-09-13T08:42:14.867872+0000","last_fullsized":"2024-09-13T08:42:14.867872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:35.251021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719956+0000","last_change":"2024-09-13T08:42:12.703010+0000","last_active":"2024-09-13T08:42:14.719956+0000","last_peered":"2024-09-13T08:42:14.719956+0000","last_clean":"2024-09-13T08:42:14.719956+0000","last_became_active":"2024-09-13T08:42:12.701624+0000","last_became_peered":"2024-09-13T08:42:12.701624+0000","last_unstale":"2024-09-13T08:42:14.719956+0000","last_undegraded":"2024-09-13T08:42:14.719956+0000","last_fullsized":"2024-09-13T08:42:14.719956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:24:37.200950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719731+0000","last_change":"2024-09-13T08:42:13.716361+0000","last_active":"2024-09-13T08:42:14.719731+0000","last_peered":"2024-09-13T08:42:14.719731+0000","last_clean":"2024-09-13T08:42:14.719731+0000","last_became_active":"2024-09-13T08:42:13.716072+0000","last_became_peered":"2024-09-13T08:42:13.716072+0000","last_unstale":"2024-09-13T08:42:14.719731+0000","last_undegraded":"2024-09-13T08:42:14.719731+0000","last_fullsized":"2024-09-13T08:42:14.719731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:36.476139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927183+0000","last_change":"2024-09-13T08:42:12.708818+0000","last_active":"2024-09-13T08:42:14.927183+0000","last_peered":"2024-09-13T08:42:14.927183+0000","last_clean":"2024-09-13T08:42:14.927183+0000","last_became_active":"2024-09-13T08:42:12.708525+0000","last_became_peered":"2024-09-13T08:42:12.708525+0000","last_unstale":"2024-09-13T08:42:14.927183+0000","last_undegraded":"2024-09-13T08:42:14.927183+0000","last_fullsized":"2024-09-13T08:42:14.927183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:10:08.826239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927670+0000","last_change":"2024-09-13T08:42:13.714439+0000","last_active":"2024-09-13T08:42:14.927670+0000","last_peered":"2024-09-13T08:42:14.927670+0000","last_clean":"2024-09-13T08:42:14.927670+0000","last_became_active":"2024-09-13T08:42:13.713781+0000","last_became_peered":"2024-09-13T08:42:13.713781+0000","last_unstale":"2024-09-13T08:42:14.927670+0000","last_undegraded":"2024-09-13T08:42:14.927670+0000","last_fullsized":"2024-09-13T08:42:14.927670+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:42:16.744542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"89'8","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927347+0000","last_change":"2024-09-13T08:42:12.705046+0000","last_active":"2024-09-13T08:42:14.927347+0000","last_peered":"2024-09-13T08:42:14.927347+0000","last_clean":"2024-09-13T08:42:14.927347+0000","last_became_active":"2024-09-13T08:42:12.704771+0000","last_became_peered":"2024-09-13T08:42:12.704771+0000","last_unstale":"2024-09-13T08:42:14.927347+0000","last_undegraded":"2024-09-13T08:42:14.927347+0000","last_fullsized":"2024-09-13T08:42:14.927347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:49.100502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719406+0000","last_change":"2024-09-13T08:42:13.723934+0000","last_active":"2024-09-13T08:42:14.719406+0000","last_peered":"2024-09-13T08:42:14.719406+0000","last_clean":"2024-09-13T08:42:14.719406+0000","last_became_active":"2024-09-13T08:42:13.723588+0000","last_became_peered":"2024-09-13T08:42:13.723588+0000","last_unstale":"2024-09-13T08:42:14.719406+0000","last_undegraded":"2024-09-13T08:42:14.719406+0000","last_fullsized":"2024-09-13T08:42:14.719406+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:35.206566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926829+0000","last_change":"2024-09-13T08:42:12.707681+0000","last_active":"2024-09-13T08:42:14.926829+0000","last_peered":"2024-09-13T08:42:14.926829+0000","last_clean":"2024-09-13T08:42:14.926829+0000","last_became_active":"2024-09-13T08:42:12.703340+0000","last_became_peered":"2024-09-13T08:42:12.703340+0000","last_unstale":"2024-09-13T08:42:14.926829+0000","last_undegraded":"2024-09-13T08:42:14.926829+0000","last_fullsized":"2024-09-13T08:42:14.926829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:00.375320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928049+0000","last_change":"2024-09-13T08:42:13.711490+0000","last_active":"2024-09-13T08:42:14.928049+0000","last_peered":"2024-09-13T08:42:14.928049+0000","last_clean":"2024-09-13T08:42:14.928049+0000","last_became_active":"2024-09-13T08:42:13.711161+0000","last_became_peered":"2024-09-13T08:42:13.711161+0000","last_unstale":"2024-09-13T08:42:14.928049+0000","last_undegraded":"2024-09-13T08:42:14.928049+0000","last_fullsized":"2024-09-13T08:42:14.928049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:10.122330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721355+0000","last_change":"2024-09-13T08:42:12.713537+0000","last_active":"2024-09-13T08:42:14.721355+0000","last_peered":"2024-09-13T08:42:14.721355+0000","last_clean":"2024-09-13T08:42:14.721355+0000","last_became_active":"2024-09-13T08:42:12.713066+0000","last_became_peered":"2024-09-13T08:42:12.713066+0000","last_unstale":"2024-09-13T08:42:14.721355+0000","last_undegraded":"2024-09-13T08:42:14.721355+0000","last_fullsized":"2024-09-13T08:42:14.721355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:49.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718921+0000","last_change":"2024-09-13T08:42:13.734177+0000","last_active":"2024-09-13T08:42:14.718921+0000","last_peered":"2024-09-13T08:42:14.718921+0000","last_clean":"2024-09-13T08:42:14.718921+0000","last_became_active":"2024-09-13T08:42:13.725500+0000","last_became_peered":"2024-09-13T08:42:13.725500+0000","last_unstale":"2024-09-13T08:42:14.718921+0000","last_undegraded":"2024-09-13T08:42:14.718921+0000","last_fullsized":"2024-09-13T08:42:14.718921+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:02.088155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721352+0000","last_change":"2024-09-13T08:42:12.707224+0000","last_active":"2024-09-13T08:42:14.721352+0000","last_peered":"2024-09-13T08:42:14.721352+0000","last_clean":"2024-09-13T08:42:14.721352+0000","last_became_active":"2024-09-13T08:42:12.706885+0000","last_became_peered":"2024-09-13T08:42:12.706885+0000","last_unstale":"2024-09-13T08:42:14.721352+0000","last_undegraded":"2024-09-13T08:42:14.721352+0000","last_fullsized":"2024-09-13T08:42:14.721352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:19.575172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719887+0000","last_change":"2024-09-13T08:42:13.724271+0000","last_active":"2024-09-13T08:42:14.719887+0000","last_peered":"2024-09-13T08:42:14.719887+0000","last_clean":"2024-09-13T08:42:14.719887+0000","last_became_active":"2024-09-13T08:42:13.724000+0000","last_became_peered":"2024-09-13T08:42:13.724000+0000","last_unstale":"2024-09-13T08:42:14.719887+0000","last_undegraded":"2024-09-13T08:42:14.719887+0000","last_fullsized":"2024-09-13T08:42:14.719887+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:09.509192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.721078+0000","last_change":"2024-09-13T08:42:12.700805+0000","last_active":"2024-09-13T08:42:14.721078+0000","last_peered":"2024-09-13T08:42:14.721078+0000","last_clean":"2024-09-13T08:42:14.721078+0000","last_became_active":"2024-09-13T08:42:12.696554+0000","last_became_peered":"2024-09-13T08:42:12.696554+0000","last_unstale":"2024-09-13T08:42:14.721078+0000","last_undegraded":"2024-09-13T08:42:14.721078+0000","last_fullsized":"2024-09-13T08:42:14.721078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:09.250464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926688+0000","last_change":"2024-09-13T08:42:13.734609+0000","last_active":"2024-09-13T08:42:14.926688+0000","last_peered":"2024-09-13T08:42:14.926688+0000","last_clean":"2024-09-13T08:42:14.926688+0000","last_became_active":"2024-09-13T08:42:13.734314+0000","last_became_peered":"2024-09-13T08:42:13.734314+0000","last_unstale":"2024-09-13T08:42:14.926688+0000","last_undegraded":"2024-09-13T08:42:14.926688+0000","last_fullsized":"2024-09-13T08:42:14.926688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:44:54.575723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720071+0000","last_change":"2024-09-13T08:42:12.700040+0000","last_active":"2024-09-13T08:42:14.720071+0000","last_peered":"2024-09-13T08:42:14.720071+0000","last_clean":"2024-09-13T08:42:14.720071+0000","last_became_active":"2024-09-13T08:42:12.697873+0000","last_became_peered":"2024-09-13T08:42:12.697873+0000","last_unstale":"2024-09-13T08:42:14.720071+0000","last_undegraded":"2024-09-13T08:42:14.720071+0000","last_fullsized":"2024-09-13T08:42:14.720071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:22.140235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.928619+0000","last_change":"2024-09-13T08:42:13.735644+0000","last_active":"2024-09-13T08:42:14.928619+0000","last_peered":"2024-09-13T08:42:14.928619+0000","last_clean":"2024-09-13T08:42:14.928619+0000","last_became_active":"2024-09-13T08:42:13.735307+0000","last_became_peered":"2024-09-13T08:42:13.735307+0000","last_unstale":"2024-09-13T08:42:14.928619+0000","last_undegraded":"2024-09-13T08:42:14.928619+0000","last_fullsized":"2024-09-13T08:42:14.928619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:48.320271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927001+0000","last_change":"2024-09-13T08:42:12.714731+0000","last_active":"2024-09-13T08:42:14.927001+0000","last_peered":"2024-09-13T08:42:14.927001+0000","last_clean":"2024-09-13T08:42:14.927001+0000","last_became_active":"2024-09-13T08:42:12.714329+0000","last_became_peered":"2024-09-13T08:42:12.714329+0000","last_unstale":"2024-09-13T08:42:14.927001+0000","last_undegraded":"2024-09-13T08:42:14.927001+0000","last_fullsized":"2024-09-13T08:42:14.927001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:21.131402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718759+0000","last_change":"2024-09-13T08:42:13.719341+0000","last_active":"2024-09-13T08:42:14.718759+0000","last_peered":"2024-09-13T08:42:14.718759+0000","last_clean":"2024-09-13T08:42:14.718759+0000","last_became_active":"2024-09-13T08:42:13.719077+0000","last_became_peered":"2024-09-13T08:42:13.719077+0000","last_unstale":"2024-09-13T08:42:14.718759+0000","last_undegraded":"2024-09-13T08:42:14.718759+0000","last_fullsized":"2024-09-13T08:42:14.718759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:55:18.456339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.719901+0000","last_change":"2024-09-13T08:42:12.707897+0000","last_active":"2024-09-13T08:42:14.719901+0000","last_peered":"2024-09-13T08:42:14.719901+0000","last_clean":"2024-09-13T08:42:14.719901+0000","last_became_active":"2024-09-13T08:42:12.707460+0000","last_became_peered":"2024-09-13T08:42:12.707460+0000","last_unstale":"2024-09-13T08:42:14.719901+0000","last_undegraded":"2024-09-13T08:42:14.719901+0000","last_fullsized":"2024-09-13T08:42:14.719901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:26.008882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926970+0000","last_change":"2024-09-13T08:42:13.722289+0000","last_active":"2024-09-13T08:42:14.926970+0000","last_peered":"2024-09-13T08:42:14.926970+0000","last_clean":"2024-09-13T08:42:14.926970+0000","last_became_active":"2024-09-13T08:42:13.721690+0000","last_became_peered":"2024-09-13T08:42:13.721690+0000","last_unstale":"2024-09-13T08:42:14.926970+0000","last_undegraded":"2024-09-13T08:42:14.926970+0000","last_fullsized":"2024-09-13T08:42:14.926970+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:08.814209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720620+0000","last_change":"2024-09-13T08:42:12.704047+0000","last_active":"2024-09-13T08:42:14.720620+0000","last_peered":"2024-09-13T08:42:14.720620+0000","last_clean":"2024-09-13T08:42:14.720620+0000","last_became_active":"2024-09-13T08:42:12.703727+0000","last_became_peered":"2024-09-13T08:42:12.703727+0000","last_unstale":"2024-09-13T08:42:14.720620+0000","last_undegraded":"2024-09-13T08:42:14.720620+0000","last_fullsized":"2024-09-13T08:42:14.720620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:56.762704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926566+0000","last_change":"2024-09-13T08:42:13.721259+0000","last_active":"2024-09-13T08:42:14.926566+0000","last_peered":"2024-09-13T08:42:14.926566+0000","last_clean":"2024-09-13T08:42:14.926566+0000","last_became_active":"2024-09-13T08:42:13.720830+0000","last_became_peered":"2024-09-13T08:42:13.720830+0000","last_unstale":"2024-09-13T08:42:14.926566+0000","last_undegraded":"2024-09-13T08:42:14.926566+0000","last_fullsized":"2024-09-13T08:42:14.926566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:11:16.055450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927320+0000","last_change":"2024-09-13T08:42:12.708197+0000","last_active":"2024-09-13T08:42:14.927320+0000","last_peered":"2024-09-13T08:42:14.927320+0000","last_clean":"2024-09-13T08:42:14.927320+0000","last_became_active":"2024-09-13T08:42:12.707866+0000","last_became_peered":"2024-09-13T08:42:12.707866+0000","last_unstale":"2024-09-13T08:42:14.927320+0000","last_undegraded":"2024-09-13T08:42:14.927320+0000","last_fullsized":"2024-09-13T08:42:14.927320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:35.467551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926935+0000","last_change":"2024-09-13T08:42:13.716732+0000","last_active":"2024-09-13T08:42:14.926935+0000","last_peered":"2024-09-13T08:42:14.926935+0000","last_clean":"2024-09-13T08:42:14.926935+0000","last_became_active":"2024-09-13T08:42:13.713391+0000","last_became_peered":"2024-09-13T08:42:13.713391+0000","last_unstale":"2024-09-13T08:42:14.926935+0000","last_undegraded":"2024-09-13T08:42:14.926935+0000","last_fullsized":"2024-09-13T08:42:14.926935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:48:23.616441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718987+0000","last_change":"2024-09-13T08:42:12.711439+0000","last_active":"2024-09-13T08:42:14.718987+0000","last_peered":"2024-09-13T08:42:14.718987+0000","last_clean":"2024-09-13T08:42:14.718987+0000","last_became_active":"2024-09-13T08:42:12.711185+0000","last_became_peered":"2024-09-13T08:42:12.711185+0000","last_unstale":"2024-09-13T08:42:14.718987+0000","last_undegraded":"2024-09-13T08:42:14.718987+0000","last_fullsized":"2024-09-13T08:42:14.718987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:16.061372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927900+0000","last_change":"2024-09-13T08:42:13.713281+0000","last_active":"2024-09-13T08:42:14.927900+0000","last_peered":"2024-09-13T08:42:14.927900+0000","last_clean":"2024-09-13T08:42:14.927900+0000","last_became_active":"2024-09-13T08:42:13.712453+0000","last_became_peered":"2024-09-13T08:42:13.712453+0000","last_unstale":"2024-09-13T08:42:14.927900+0000","last_undegraded":"2024-09-13T08:42:14.927900+0000","last_fullsized":"2024-09-13T08:42:14.927900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:53:06.151377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927615+0000","last_change":"2024-09-13T08:42:12.705295+0000","last_active":"2024-09-13T08:42:14.927615+0000","last_peered":"2024-09-13T08:42:14.927615+0000","last_clean":"2024-09-13T08:42:14.927615+0000","last_became_active":"2024-09-13T08:42:12.704995+0000","last_became_peered":"2024-09-13T08:42:12.704995+0000","last_unstale":"2024-09-13T08:42:14.927615+0000","last_undegraded":"2024-09-13T08:42:14.927615+0000","last_fullsized":"2024-09-13T08:42:14.927615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:16.061372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867774+0000","last_change":"2024-09-13T08:42:13.730098+0000","last_active":"2024-09-13T08:42:14.867774+0000","last_peered":"2024-09-13T08:42:14.867774+0000","last_clean":"2024-09-13T08:42:14.867774+0000","last_became_active":"2024-09-13T08:42:13.729598+0000","last_became_peered":"2024-09-13T08:42:13.729598+0000","last_unstale":"2024-09-13T08:42:14.867774+0000","last_undegraded":"2024-09-13T08:42:14.867774+0000","last_fullsized":"2024-09-13T08:42:14.867774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:19.591015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.720567+0000","last_change":"2024-09-13T08:42:12.706098+0000","last_active":"2024-09-13T08:42:14.720567+0000","last_peered":"2024-09-13T08:42:14.720567+0000","last_clean":"2024-09-13T08:42:14.720567+0000","last_became_active":"2024-09-13T08:42:12.705718+0000","last_became_peered":"2024-09-13T08:42:12.705718+0000","last_unstale":"2024-09-13T08:42:14.720567+0000","last_undegraded":"2024-09-13T08:42:14.720567+0000","last_fullsized":"2024-09-13T08:42:14.720567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:41.518568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927385+0000","last_change":"2024-09-13T08:42:13.725508+0000","last_active":"2024-09-13T08:42:14.927385+0000","last_peered":"2024-09-13T08:42:14.927385+0000","last_clean":"2024-09-13T08:42:14.927385+0000","last_became_active":"2024-09-13T08:42:13.725231+0000","last_became_peered":"2024-09-13T08:42:13.725231+0000","last_unstale":"2024-09-13T08:42:14.927385+0000","last_undegraded":"2024-09-13T08:42:14.927385+0000","last_fullsized":"2024-09-13T08:42:14.927385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:10.827593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927403+0000","last_change":"2024-09-13T08:42:12.711334+0000","last_active":"2024-09-13T08:42:14.927403+0000","last_peered":"2024-09-13T08:42:14.927403+0000","last_clean":"2024-09-13T08:42:14.927403+0000","last_became_active":"2024-09-13T08:42:12.708192+0000","last_became_peered":"2024-09-13T08:42:12.708192+0000","last_unstale":"2024-09-13T08:42:14.927403+0000","last_undegraded":"2024-09-13T08:42:14.927403+0000","last_fullsized":"2024-09-13T08:42:14.927403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:29.994667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867775+0000","last_change":"2024-09-13T08:42:13.729205+0000","last_active":"2024-09-13T08:42:14.867775+0000","last_peered":"2024-09-13T08:42:14.867775+0000","last_clean":"2024-09-13T08:42:14.867775+0000","last_became_active":"2024-09-13T08:42:13.728059+0000","last_became_peered":"2024-09-13T08:42:13.728059+0000","last_unstale":"2024-09-13T08:42:14.867775+0000","last_undegraded":"2024-09-13T08:42:14.867775+0000","last_fullsized":"2024-09-13T08:42:14.867775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:44:34.685449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867619+0000","last_change":"2024-09-13T08:42:12.710424+0000","last_active":"2024-09-13T08:42:14.867619+0000","last_peered":"2024-09-13T08:42:14.867619+0000","last_clean":"2024-09-13T08:42:14.867619+0000","last_became_active":"2024-09-13T08:42:12.709779+0000","last_became_peered":"2024-09-13T08:42:12.709779+0000","last_unstale":"2024-09-13T08:42:14.867619+0000","last_undegraded":"2024-09-13T08:42:14.867619+0000","last_fullsized":"2024-09-13T08:42:14.867619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:49.500754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.867847+0000","last_change":"2024-09-13T08:42:13.723867+0000","last_active":"2024-09-13T08:42:14.867847+0000","last_peered":"2024-09-13T08:42:14.867847+0000","last_clean":"2024-09-13T08:42:14.867847+0000","last_became_active":"2024-09-13T08:42:13.723608+0000","last_became_peered":"2024-09-13T08:42:13.723608+0000","last_unstale":"2024-09-13T08:42:14.867847+0000","last_undegraded":"2024-09-13T08:42:14.867847+0000","last_fullsized":"2024-09-13T08:42:14.867847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:01.754683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.718637+0000","last_change":"2024-09-13T08:42:12.711454+0000","last_active":"2024-09-13T08:42:14.718637+0000","last_peered":"2024-09-13T08:42:14.718637+0000","last_clean":"2024-09-13T08:42:14.718637+0000","last_became_active":"2024-09-13T08:42:12.710883+0000","last_became_peered":"2024-09-13T08:42:12.710883+0000","last_unstale":"2024-09-13T08:42:14.718637+0000","last_undegraded":"2024-09-13T08:42:14.718637+0000","last_fullsized":"2024-09-13T08:42:14.718637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:29.994667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.927518+0000","last_change":"2024-09-13T08:42:13.733533+0000","last_active":"2024-09-13T08:42:14.927518+0000","last_peered":"2024-09-13T08:42:14.927518+0000","last_clean":"2024-09-13T08:42:14.927518+0000","last_became_active":"2024-09-13T08:42:13.733228+0000","last_became_peered":"2024-09-13T08:42:13.733228+0000","last_unstale":"2024-09-13T08:42:14.927518+0000","last_undegraded":"2024-09-13T08:42:14.927518+0000","last_fullsized":"2024-09-13T08:42:14.927518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:12.681320+0000","last_clean_scrub_stamp":"2024-09-13T08:42:12.681320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:44:24.755596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.926961+0000","last_change":"2024-09-13T08:42:12.707227+0000","last_active":"2024-09-13T08:42:14.926961+0000","last_peered":"2024-09-13T08:42:14.926961+0000","last_clean":"2024-09-13T08:42:14.926961+0000","last_became_active":"2024-09-13T08:42:12.706759+0000","last_became_peered":"2024-09-13T08:42:12.706759+0000","last_unstale":"2024-09-13T08:42:14.926961+0000","last_undegraded":"2024-09-13T08:42:14.926961+0000","last_fullsized":"2024-09-13T08:42:14.926961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:11.672732+0000","last_clean_scrub_stamp":"2024-09-13T08:42:11.672732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:39:17.508018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":259,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T08:42:14.868008+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:42:14.868008+0000","last_peered":"2024-09-13T08:42:14.868008+0000","last_clean":"2024-09-13T08:42:14.868008+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:42:14.868008+0000","last_undegraded":"2024-09-13T08:42:14.868008+0000","last_fullsized":"2024-09-13T08:42:14.868008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509507,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1101824,"data_stored":473505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1105920,"data_stored":474107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509507,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1708032,"data_stored":1065449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542210,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36424,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335416,"statfs":{"total":96636764160,"available":96599465984,"internally_reserved":0,"allocated":1101824,"data_stored":473469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509507,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1110016,"data_stored":474109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1097728,"data_stored":473435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1695744,"data_stored":1063893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509506,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1114112,"data_stored":475069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:42:18.387 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:42:18.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:42:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 -- 172.21.15.31:0/3449690736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd70004800 msgr2=0x7efd70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 --2- 172.21.15.31:0/3449690736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd70004800 0x7efd70004c80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7efd74006f60 tx=0x7efd70005830 comp rx=0 tx=0).stop 2024-09-13T08:42:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8b400640 1 -- 172.21.15.31:0/3449690736 reap_dead start 2024-09-13T08:42:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 -- 172.21.15.31:0/3449690736 shutdown_connections 2024-09-13T08:42:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 -- 172.21.15.31:0/3449690736 >> 172.21.15.31:0/3449690736 conn(0x7efd8c0b1520 msgr2=0x7efd8c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 -- 172.21.15.31:0/3449690736 shutdown_connections 2024-09-13T08:42:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.594+0000 7efd8be00640 1 -- 172.21.15.31:0/3449690736 wait complete. 2024-09-13T08:42:18.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.596+0000 7efd8be00640 1 Processor -- start 2024-09-13T08:42:18.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.596+0000 7efd8be00640 1 -- start start 2024-09-13T08:42:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd70004800 0x7efd8c1f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd8c155420 0x7efd8c1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c06fdf0 con 0x7efd8c1585a0 2024-09-13T08:42:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c06fc70 con 0x7efd8c155420 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c06ff70 con 0x7efd70004800 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57816/0 (socket says 172.21.15.31:57816) 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8b400640 1 -- 172.21.15.31:0/759273370 learned_addr learned my addr 172.21.15.31:0/759273370 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd8aa00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd70004800 0x7efd8c1f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.597+0000 7efd82000640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd8c155420 0x7efd8c1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 -- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd70004800 msgr2=0x7efd8c1f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd70004800 0x7efd8c1f6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 -- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd8c155420 msgr2=0x7efd8c1535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd8c155420 0x7efd8c1535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 -- 172.21.15.31:0/759273370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c157e60 con 0x7efd8c1585a0 2024-09-13T08:42:18.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8b400640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0b5760 tx=0x7efd84001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84015020 con 0x7efd8c1585a0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd84010980 con 0x7efd8c1585a0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84010dd0 con 0x7efd8c1585a0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c158140 con 0x7efd8c1585a0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.598+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8c153b00 con 0x7efd8c1585a0 2024-09-13T08:42:18.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.599+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c050ee0 con 0x7efd8c1585a0 2024-09-13T08:42:18.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.601+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7efd84018020 con 0x7efd8c1585a0 2024-09-13T08:42:18.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.601+0000 7efd83e00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 0x7efd6807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:18.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.602+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7efd8409b5d0 con 0x7efd8c1585a0 2024-09-13T08:42:18.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.602+0000 7efd8aa00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 0x7efd6807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.602+0000 7efd8aa00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 0x7efd6807ae60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009a70 tx=0x7efd7c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.606+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efd8c050ee0 con 0x7efd8c1585a0 2024-09-13T08:42:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.874+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7efd8c1f5810 con 0x7efd8c1585a0 2024-09-13T08:42:18.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.877+0000 7efd83e00640 1 -- 172.21.15.31:0/759273370 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7efd8401f020 con 0x7efd8c1585a0 2024-09-13T08:42:18.878 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:42:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:42:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:42:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:42:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:42:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:42:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.879+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 msgr2=0x7efd6807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.879+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 0x7efd6807ae60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009a70 tx=0x7efd7c005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.879+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 msgr2=0x7efd8c0c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.879+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0b5760 tx=0x7efd84001b80 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 shutdown_connections 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efd680789a0 0x7efd6807ae60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd8c1585a0 0x7efd8c0c8350 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd8c155420 0x7efd8c1535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 --2- 172.21.15.31:0/759273370 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd70004800 0x7efd8c1f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 >> 172.21.15.31:0/759273370 conn(0x7efd8c0b1520 msgr2=0x7efd8c159840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 shutdown_connections 2024-09-13T08:42:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:18.880+0000 7efd8be00640 1 -- 172.21.15.31:0/759273370 wait complete. 2024-09-13T08:42:18.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 -- 172.21.15.31:0/268411274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80051c0 msgr2=0x7fc9f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 --2- 172.21.15.31:0/268411274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80051c0 0x7fc9f800b9c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001f30 tx=0x7fc9f8005830 comp rx=0 tx=0).stop 2024-09-13T08:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca13400640 1 -- 172.21.15.31:0/268411274 reap_dead start 2024-09-13T08:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 -- 172.21.15.31:0/268411274 shutdown_connections 2024-09-13T08:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 -- 172.21.15.31:0/268411274 >> 172.21.15.31:0/268411274 conn(0x7fca140cc910 msgr2=0x7fca140cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 -- 172.21.15.31:0/268411274 shutdown_connections 2024-09-13T08:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.057+0000 7fca19e00640 1 -- 172.21.15.31:0/268411274 wait complete. 2024-09-13T08:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.059+0000 7fca19e00640 1 Processor -- start 2024-09-13T08:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.059+0000 7fca19e00640 1 -- start start 2024-09-13T08:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 0x7fca140c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca140aff20 0x7fca140b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca140cc300 con 0x7fca1414f1d0 2024-09-13T08:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca140740a0 con 0x7fc9f80051c0 2024-09-13T08:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca19e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fca140cc480 con 0x7fca140aff20 2024-09-13T08:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca0a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca0a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57840/0 (socket says 172.21.15.31:57840) 2024-09-13T08:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.060+0000 7fca0a000640 1 -- 172.21.15.31:0/2746378286 learned_addr learned my addr 172.21.15.31:0/2746378286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca13400640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca140aff20 0x7fca140b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca12a00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 0x7fca140c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 -- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca140aff20 msgr2=0x7fca140b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca140aff20 0x7fca140b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 -- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 msgr2=0x7fca140c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 0x7fca140c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 -- 172.21.15.31:0/2746378286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca140ca160 con 0x7fca1414f1d0 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca12a00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 0x7fca140c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0a000640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc011a10 tx=0x7fc9fc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc01e070 con 0x7fca1414f1d0 2024-09-13T08:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9fc00fc70 con 0x7fca1414f1d0 2024-09-13T08:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc019410 con 0x7fca1414f1d0 2024-09-13T08:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.061+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1409bea0 con 0x7fca1414f1d0 2024-09-13T08:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.062+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1409c460 con 0x7fca1414f1d0 2024-09-13T08:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.063+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca140b1e80 con 0x7fca1414f1d0 2024-09-13T08:42:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.064+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fc9fc021070 con 0x7fca1414f1d0 2024-09-13T08:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.065+0000 7fca0be00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 0x7fc9f007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.065+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fc9fc09b3a0 con 0x7fca1414f1d0 2024-09-13T08:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.065+0000 7fca12a00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 0x7fc9f007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.066+0000 7fca12a00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 0x7fc9f007ae60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca04001410 tx=0x7fca04001340 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.069+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc9fc063100 con 0x7fca1414f1d0 2024-09-13T08:42:19.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.301+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca140af5d0 con 0x7fca1414f1d0 2024-09-13T08:42:19.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.301+0000 7fca0be00640 1 -- 172.21.15.31:0/2746378286 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5225 (secure 0 0 0) 0x7fc9fc06ae20 con 0x7fca1414f1d0 2024-09-13T08:42:19.302 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:19.303 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":57,"btime":"2024-09-13T08:42:16:723057+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3778871942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3778871942},{"type":"v1","addr":"172.21.15.45:6835","nonce":3778871942}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:19.303 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 57 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 msgr2=0x7fc9f007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 0x7fc9f007ae60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca04001410 tx=0x7fca04001340 comp rx=0 tx=0).stop 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 msgr2=0x7fca1409b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc011a10 tx=0x7fc9fc011ed0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 shutdown_connections 2024-09-13T08:42:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc9f00789a0 0x7fc9f007ae60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca140aff20 0x7fca140b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1414f1d0 0x7fca1409b440 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 --2- 172.21.15.31:0/2746378286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9f80051c0 0x7fca140c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 >> 172.21.15.31:0/2746378286 conn(0x7fca140cc910 msgr2=0x7fca140b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 shutdown_connections 2024-09-13T08:42:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.304+0000 7fca19e00640 1 -- 172.21.15.31:0/2746378286 wait complete. 2024-09-13T08:42:19.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:42:13.130944+0000', 'modified': '2024-09-13T08:42:16.161289+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6585}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6585': {'gid': 6585, 'name': 'c', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.31:6834/2420402248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2420402248}, {'type': 'v1', 'addr': '172.21.15.31:6834', 'nonce': 2420402248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6585, 'qdb_cluster': [6585]} 2024-09-13T08:42:19.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:42:19.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 -- 172.21.15.31:0/1246521490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b680051c0 msgr2=0x7f5b6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 --2- 172.21.15.31:0/1246521490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b680051c0 0x7f5b6800b9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c005fd0 tx=0x7f5b68005830 comp rx=0 tx=0).stop 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b82800640 1 -- 172.21.15.31:0/1246521490 reap_dead start 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 -- 172.21.15.31:0/1246521490 shutdown_connections 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 -- 172.21.15.31:0/1246521490 >> 172.21.15.31:0/1246521490 conn(0x7f5b840d3f50 msgr2=0x7f5b840d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 -- 172.21.15.31:0/1246521490 shutdown_connections 2024-09-13T08:42:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.890+0000 7f5b83200640 1 -- 172.21.15.31:0/1246521490 wait complete. 2024-09-13T08:42:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.891+0000 7f5b83200640 1 Processor -- start 2024-09-13T08:42:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 -- start start 2024-09-13T08:42:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b8414e420 0x7f5b840b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b841582e0 0x7f5b840c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84166dd0 con 0x7f5b84157e60 2024-09-13T08:42:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84166c50 con 0x7f5b841582e0 2024-09-13T08:42:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.892+0000 7f5b83200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b84166f50 con 0x7f5b8414e420 2024-09-13T08:42:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57864/0 (socket says 172.21.15.31:57864) 2024-09-13T08:42:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 -- 172.21.15.31:0/3018497641 learned_addr learned my addr 172.21.15.31:0/3018497641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 -- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b8414e420 msgr2=0x7f5b840b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b8414e420 0x7f5b840b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 -- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b841582e0 msgr2=0x7f5b840c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b841582e0 0x7f5b840c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 -- 172.21.15.31:0/3018497641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b840b72c0 con 0x7f5b84157e60 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.893+0000 7f5b81400640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5b70003e40 tx=0x7f5b70001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.894+0000 7f5b7a000640 1 -- 172.21.15.31:0/3018497641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70011700 con 0x7f5b84157e60 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.894+0000 7f5b7a000640 1 -- 172.21.15.31:0/3018497641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b70016020 con 0x7f5b84157e60 2024-09-13T08:42:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.894+0000 7f5b7a000640 1 -- 172.21.15.31:0/3018497641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70011b60 con 0x7f5b84157e60 2024-09-13T08:42:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.894+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b841f4b40 con 0x7f5b84157e60 2024-09-13T08:42:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.894+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b841f5100 con 0x7f5b84157e60 2024-09-13T08:42:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.896+0000 7f5b7a000640 1 -- 172.21.15.31:0/3018497641 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f5b70011d00 con 0x7f5b84157e60 2024-09-13T08:42:19.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.897+0000 7f5b7a000640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 0x7f5b6407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.897+0000 7f5b7a000640 1 -- 172.21.15.31:0/3018497641 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f5b7009b920 con 0x7f5b84157e60 2024-09-13T08:42:19.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.897+0000 7f5b8bdc7740 1 Processor -- start 2024-09-13T08:42:19.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.897+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 0x7f5b6407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.897+0000 7f5b8bdc7740 1 -- start start 2024-09-13T08:42:19.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 0x7f5b6407ae30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b74003670 tx=0x7f5b74000ed0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc525c10 0x55e5dc569580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc526190 0x55e5dc569ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc4a8430 con 0x55e5dc56aaa0 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc4985d0 con 0x55e5dc525c10 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.898+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55e5dc487a00 con 0x55e5dc526190 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57868/0 (socket says 172.21.15.31:57868) 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 -- 172.21.15.31:0/2278758338 learned_addr learned my addr 172.21.15.31:0/2278758338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b82800640 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc526190 0x55e5dc569ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81400640 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc525c10 0x55e5dc569580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 -- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc526190 msgr2=0x55e5dc569ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc526190 0x55e5dc569ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 -- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc525c10 msgr2=0x55e5dc569580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc525c10 0x55e5dc569580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 -- 172.21.15.31:0/2278758338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5dc450520 con 0x55e5dc56aaa0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.899+0000 7f5b81e00640 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5b74003640 tx=0x7f5b74012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9c48ee9e8282cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b6ea00640 1 -- 172.21.15.31:0/2278758338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b740119b0 con 0x55e5dc56aaa0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b6ea00640 1 -- 172.21.15.31:0/2278758338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b74023070 con 0x55e5dc56aaa0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b6ea00640 1 -- 172.21.15.31:0/2278758338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74011dd0 con 0x55e5dc56aaa0 2024-09-13T08:42:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/2278758338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 msgr2=0x55e5dc56ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5b74003640 tx=0x7f5b74012000 comp rx=0 tx=0).stop 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/2278758338 shutdown_connections 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc526190 0x55e5dc569ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc525c10 0x55e5dc569580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/2278758338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc56aaa0 0x55e5dc56ae80 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.900+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/2278758338 >> 172.21.15.31:0/2278758338 conn(0x55e5dc5211e0 msgr2=0x55e5dc524c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.901+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/2278758338 shutdown_connections 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.901+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/2278758338 wait complete. 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.901+0000 7f5b8bdc7740 1 -- start start 2024-09-13T08:42:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.901+0000 7f5b8bdc7740 1 Processor -- start 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc569ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc53f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc42e330 con 0x55e5dc53c560 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc48b810 con 0x55e5dc526190 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55e5dc1f8620 con 0x55e5dc56aaa0 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.902+0000 7f5b8bdc7740 10 client.0 _ll_register_callbacks cb 0x55e5dc433420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:42:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57878/0 (socket says 172.21.15.31:57878) 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 -- 172.21.15.31:0/1980607986 learned_addr learned my addr 172.21.15.31:0/1980607986 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 -- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 msgr2=0x55e5dc53f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc569ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b81400640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc53f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc53f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 -- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 msgr2=0x55e5dc569ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc569ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 -- 172.21.15.31:0/1980607986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5dc1e9850 con 0x55e5dc53c560 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b81400640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc53f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc569ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b82800640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00e350 tx=0x7f5b7c0092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b286d4d69a09b30 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.903+0000 7f5b63400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c033640 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b74023070 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f5b7c0337e0 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f5b7c0339e0 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 10 client.6705 fetch_fsmap learned FSMap version 57 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x55e5dc2606b0 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b74023070 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 57) ==== 3502+0+0 (secure 0 0 0) 0x7f5b7c051080 con 0x55e5dc53c560 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 10 client.6705 fetch_fsmap finished waiting for FSMap version 57 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 10 client.6705 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 20 client.6705 populate_metadata read hostname 'smithi031' 2024-09-13T08:42:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc57d430 0x55e5dc57d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 4 client.6705 mds_command: new command op to 6585 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.904+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 --> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5dc49a520 con 0x55e5dc57d430 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.905+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc57d430 0x55e5dc57d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.905+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc57d430 0x55e5dc57d810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4228c2d9fb7f2b7a server_cookie=d84123e358353893 in_seq=0 out_seq=0 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.905+0000 7f5b63400640 10 client.6705 ms_handle_connect on v2:172.21.15.31:6832/2420402248 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b63400640 1 -- 172.21.15.31:0/1980607986 <== mds.0 v2:172.21.15.31:6832/2420402248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e5dc49a520 con 0x55e5dc57d430 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b63400640 10 client.6705 handle_command_reply: tid=0 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 client.6705 shutdown 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 msgr2=0x55e5dc53ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00e350 tx=0x7f5b7c0092c0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 shutdown_connections 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc53ff20 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc57d430 0x55e5dc57d810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc53f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1980607986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc569ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b63400640 0 client.6705 ms_handle_reset on v2:172.21.15.31:6832/2420402248 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.906+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 >> 172.21.15.31:0/1980607986 conn(0x55e5dc525c10 msgr2=0x55e5dc56a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.907+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 shutdown_connections 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.907+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1980607986 wait complete. 2024-09-13T08:42:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.907+0000 7f5b8bdc7740 20 client.6705 trim_cache size 0 max 16384 2024-09-13T08:42:19.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.926+0000 7f5b8bdc7740 1 Processor -- start 2024-09-13T08:42:19.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.926+0000 7f5b8bdc7740 1 -- start start 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc57bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc54af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84200d90 con 0x55e5dc53c560 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84200120 con 0x55e5dc526190 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b84200740 con 0x55e5dc56aaa0 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57880/0 (socket says 172.21.15.31:57880) 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 -- 172.21.15.31:0/1935090883 learned_addr learned my addr 172.21.15.31:0/1935090883 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b82800640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc54af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc57bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 -- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 msgr2=0x55e5dc54af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc54af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.927+0000 7f5b81400640 1 -- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 msgr2=0x55e5dc57bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b81400640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc57bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b81400640 1 -- 172.21.15.31:0/1935090883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5dc57db40 con 0x55e5dc53c560 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b81e00640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc57bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b81400640 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5b70099a90 tx=0x7f5b70063d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b5f11756fa8471 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b6e000640 1 -- 172.21.15.31:0/1935090883 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70015040 con 0x55e5dc53c560 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b6e000640 1 -- 172.21.15.31:0/1935090883 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b7006cf60 con 0x55e5dc53c560 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.928+0000 7f5b6e000640 1 -- 172.21.15.31:0/1935090883 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70085de0 con 0x55e5dc53c560 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1935090883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 msgr2=0x55e5dc54aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5b70099a90 tx=0x7f5b70063d90 comp rx=0 tx=0).stop 2024-09-13T08:42:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1935090883 shutdown_connections 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc54af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc54aa30 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/1935090883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc57bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1935090883 >> 172.21.15.31:0/1935090883 conn(0x55e5dc525c10 msgr2=0x55e5dc51a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1935090883 shutdown_connections 2024-09-13T08:42:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/1935090883 wait complete. 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.929+0000 7f5b8bdc7740 1 -- start start 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.930+0000 7f5b8bdc7740 1 Processor -- start 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.930+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc5edff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.930+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b8bdc7740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc5ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc53d200 con 0x55e5dc53c560 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc5605c0 con 0x55e5dc526190 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b8bdc7740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55e5dc539ce0 con 0x55e5dc56aaa0 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b8bdc7740 10 client.0 _ll_register_callbacks cb 0x7f5b7c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:42:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57890/0 (socket says 172.21.15.31:57890) 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 -- 172.21.15.31:0/3371190970 learned_addr learned my addr 172.21.15.31:0/3371190970 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 -- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 msgr2=0x55e5dc5ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc5edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b82800640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc5ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc5ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 -- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 msgr2=0x55e5dc5edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc5edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 -- 172.21.15.31:0/3371190970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5dc5206f0 con 0x55e5dc53c560 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc5edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b82800640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc5ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b81400640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5b70087040 tx=0x7f5b7005fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b9f82b11c37b35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.931+0000 7f5b63400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70015070 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f5b7006a2e0 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b7006cf60 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f5b70099fb0 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 10 client.6711 fetch_fsmap learned FSMap version 57 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x55e5dc5ee590 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b7006cf60 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 57) ==== 3502+0+0 (secure 0 0 0) 0x7f5b70062830 con 0x55e5dc53c560 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 10 client.6711 fetch_fsmap finished waiting for FSMap version 57 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 10 client.6711 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:42:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 20 client.6711 populate_metadata read hostname 'smithi031' 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc58d510 0x55e5dc58d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 4 client.6711 mds_command: new command op to 6585 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.932+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 --> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] -- command(tid 0: {"prefix": "status"}) -- 0x55e5dc495fd0 con 0x55e5dc58d510 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc58d510 0x55e5dc58d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b81e00640 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc58d510 0x55e5dc58d8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4ba7e37ac82afb5 server_cookie=11f2522b3018157d in_seq=0 out_seq=0 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b63400640 10 client.6711 ms_handle_connect on v2:172.21.15.31:6832/2420402248 2024-09-13T08:42:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b63400640 1 -- 172.21.15.31:0/3371190970 <== mds.0 v2:172.21.15.31:6832/2420402248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55e5dc495fd0 con 0x55e5dc58d510 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b63400640 10 client.6711 handle_command_reply: tid=0 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b8bdc7740 1 client.6711 shutdown 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 msgr2=0x55e5dc562d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5b70087040 tx=0x7f5b7005fd50 comp rx=0 tx=0).stop 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 shutdown_connections 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.933+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:6832/2420402248,v1:172.21.15.31:6834/2420402248] conn(0x55e5dc58d510 0x55e5dc58d8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55e5dc56aaa0 0x55e5dc5ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5dc53c560 0x55e5dc562d10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 --2- 172.21.15.31:0/3371190970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5dc526190 0x55e5dc5edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 >> 172.21.15.31:0/3371190970 conn(0x55e5dc525c10 msgr2=0x55e5dc51f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b63400640 0 client.6711 ms_handle_reset on v2:172.21.15.31:6832/2420402248 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 shutdown_connections 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 1 -- 172.21.15.31:0/3371190970 wait complete. 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b8bdc7740 20 client.6711 trim_cache size 0 max 16384 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 6585, 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:42:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2372504319999997, 2024-09-13T08:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 57, 2024-09-13T08:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-09-13T08:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 90, 2024-09-13T08:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 17.303767820000001 2024-09-13T08:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 msgr2=0x7f5b6407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 0x7f5b6407ae30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b74003670 tx=0x7f5b74000ed0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 msgr2=0x7f5b840c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5b70003e40 tx=0x7f5b70001800 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 shutdown_connections 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5b64078970 0x7f5b6407ae30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b841582e0 0x7f5b840c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b84157e60 0x7f5b840c9810 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 --2- 172.21.15.31:0/3018497641 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5b8414e420 0x7f5b840b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.934+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 >> 172.21.15.31:0/3018497641 conn(0x7f5b840d3f50 msgr2=0x7f5b8406e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.935+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 shutdown_connections 2024-09-13T08:42:19.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:19.935+0000 7f5b83200640 1 -- 172.21.15.31:0/3018497641 wait complete. 2024-09-13T08:42:19.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:42:20.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 -- 172.21.15.31:0/1807263807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40ce900 msgr2=0x7fb3c40cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 --2- 172.21.15.31:0/1807263807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40ce900 0x7fb3c40cbc50 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005fd0 tx=0x7fb3bc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 -- 172.21.15.31:0/1807263807 shutdown_connections 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 --2- 172.21.15.31:0/1807263807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40ce900 0x7fb3c40cbc50 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 --2- 172.21.15.31:0/1807263807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c40cf850 0x7fb3c40ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 -- 172.21.15.31:0/1807263807 >> 172.21.15.31:0/1807263807 conn(0x7fb3c40cc910 msgr2=0x7fb3c40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 -- 172.21.15.31:0/1807263807 shutdown_connections 2024-09-13T08:42:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.110+0000 7fb3ca800640 1 -- 172.21.15.31:0/1807263807 wait complete. 2024-09-13T08:42:20.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.112+0000 7fb3ca800640 1 Processor -- start 2024-09-13T08:42:20.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.112+0000 7fb3ca800640 1 -- start start 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40ce900 0x7fb3c41f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c415bd70 0x7fb3c41ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41600f0 con 0x7fb3c40cf850 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c40cb850 con 0x7fb3c40ce900 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3ca800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c41eeb80 con 0x7fb3c415bd70 2024-09-13T08:42:20.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3baa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:20.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3baa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57916/0 (socket says 172.21.15.31:57916) 2024-09-13T08:42:20.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3baa00640 1 -- 172.21.15.31:0/3963347545 learned_addr learned my addr 172.21.15.31:0/3963347545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:20.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3c3400640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40ce900 0x7fb3c41f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:20.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.113+0000 7fb3c3e00640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c415bd70 0x7fb3c41ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 -- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c415bd70 msgr2=0x7fb3c41ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c415bd70 0x7fb3c41ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 -- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40ce900 msgr2=0x7fb3c41f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40ce900 0x7fb3c41f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 -- 172.21.15.31:0/3963347545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c415c570 con 0x7fb3c40cf850 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3baa00640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00d950 tx=0x7fb3ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac017020 con 0x7fb3c40cf850 2024-09-13T08:42:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3ac009d20 con 0x7fb3c40cf850 2024-09-13T08:42:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c41f4ea0 con 0x7fb3c40cf850 2024-09-13T08:42:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac0106c0 con 0x7fb3c40cf850 2024-09-13T08:42:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.114+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c41f5460 con 0x7fb3c40cf850 2024-09-13T08:42:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.115+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c40ce3c0 con 0x7fb3c40cf850 2024-09-13T08:42:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.117+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fb3ac01a020 con 0x7fb3c40cf850 2024-09-13T08:42:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.118+0000 7fb3bbe00640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 0x7fb3a007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.118+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fb3ac09b090 con 0x7fb3c40cf850 2024-09-13T08:42:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.118+0000 7fb3c3400640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 0x7fb3a007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.119+0000 7fb3c3400640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 0x7fb3a007ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb3c415b470 tx=0x7fb3b4007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:20.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.122+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb3ac021020 con 0x7fb3c40cf850 2024-09-13T08:42:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.360+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb3c40ae8e0 con 0x7fb3c40cf850 2024-09-13T08:42:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.360+0000 7fb3bbe00640 1 -- 172.21.15.31:0/3963347545 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) ==== 53+0+73 (secure 0 0 0) 0x7fb3ac062e80 con 0x7fb3c40cf850 2024-09-13T08:42:20.361 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 msgr2=0x7fb3a007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 0x7fb3a007ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb3c415b470 tx=0x7fb3b4007e90 comp rx=0 tx=0).stop 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 msgr2=0x7fb3c41f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00d950 tx=0x7fb3ac00de10 comp rx=0 tx=0).stop 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 shutdown_connections 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3a0078970 0x7fb3a007ae30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb3c415bd70 0x7fb3c41ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c40cf850 0x7fb3c41f4750 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 --2- 172.21.15.31:0/3963347545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40ce900 0x7fb3c41f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 >> 172.21.15.31:0/3963347545 conn(0x7fb3c40cc910 msgr2=0x7fb3c40af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 shutdown_connections 2024-09-13T08:42:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20.363+0000 7fb3ca800640 1 -- 172.21.15.31:0/3963347545 wait complete. 2024-09-13T08:42:20.379 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:42:20.379 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:42:20.379 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:42:20.379 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:42:20.379 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:20.380 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:42:20.380 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:42:20.380 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:42:20.380 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:20.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.380 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:42:20.416 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:42:20.416 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:42:20.417 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42391sec preferred_lft 42391sec 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:42:20.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:20.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.419 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:42:20.510 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:42:20.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.513 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:42:20.549 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:42:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:42:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:42:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:42:20.550 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:42:20.550 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:42:20.550 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:42:20.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:42:20.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:42:20.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:42:20.551 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:42:20.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:42:20.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:20.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.749 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:42:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:42:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:42:20.785 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:20.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:42:20.882 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:20.882 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:20.918 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:20.918 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:20.919 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:20.994 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:42:20.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:20.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:21.123 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:21.124 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:21.151 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:21.151 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:21.151 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:42:21.151 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:42:21.152 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:42:21.154 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:42:21.154 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:42:21.154 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:42:21.154 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:42:21.154 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:42:21.157 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:42:21.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:21.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:21.214 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:42:21.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:42:21.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:21.311 INFO:teuthology.orchestra.run.smithi031.stdout:client5936 2024-09-13T08:42:21.312 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:21.312 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:21.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 -- 172.21.15.31:0/3383145102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc815dfc0 msgr2=0x7f0fc80cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 --2- 172.21.15.31:0/3383145102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc815dfc0 0x7f0fc80cc300 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009830 tx=0x7f0fb802a510 comp rx=0 tx=0).stop 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 -- 172.21.15.31:0/3383145102 shutdown_connections 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 --2- 172.21.15.31:0/3383145102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc80ca5c0 0x7f0fc80ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 --2- 172.21.15.31:0/3383145102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc815dfc0 0x7f0fc80cc300 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 -- 172.21.15.31:0/3383145102 >> 172.21.15.31:0/3383145102 conn(0x7f0fc80cc910 msgr2=0x7f0fc80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.547+0000 7f0fce800640 1 -- 172.21.15.31:0/3383145102 shutdown_connections 2024-09-13T08:42:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.548+0000 7f0fce800640 1 -- 172.21.15.31:0/3383145102 wait complete. 2024-09-13T08:42:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.549+0000 7f0fce800640 1 Processor -- start 2024-09-13T08:42:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.549+0000 7f0fce800640 1 -- start start 2024-09-13T08:42:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 0x7f0fc8152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc815dfc0 0x7f0fc80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8074010 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8073e90 con 0x7f0fc80b1910 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fce800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc80b4760 con 0x7f0fc815dfc0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fbea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fbea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57934/0 (socket says 172.21.15.31:57934) 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.550+0000 7f0fbea00640 1 -- 172.21.15.31:0/1344287197 learned_addr learned my addr 172.21.15.31:0/1344287197 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fc7400640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 0x7f0fc8152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fc7e00640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc815dfc0 0x7f0fc80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 -- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc815dfc0 msgr2=0x7f0fc80b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc815dfc0 0x7f0fc80b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 -- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 msgr2=0x7f0fc8152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 0x7f0fc8152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 -- 172.21.15.31:0/1344287197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc80b49d0 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fc7400640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 0x7f0fc8152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbea00640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000d950 tx=0x7f0fb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0017070 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.551+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fb0009d20 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.552+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0004280 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.552+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc80b4cb0 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.552+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc81f5f60 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.553+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc80b20d0 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.554+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f0fb00035f0 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.555+0000 7f0fbfe00640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 0x7f0fa407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.555+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f0fb009ada0 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.555+0000 7f0fc7400640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 0x7f0fa407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.556+0000 7f0fc7400640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 0x7f0fa407ae30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8034000 tx=0x7f0fb8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.559+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0fb0063150 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.802+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0fc80ce020 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.802+0000 7f0fbfe00640 1 -- 172.21.15.31:0/1344287197 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) ==== 53+0+73 (secure 0 0 0) 0x7f0fb0062a50 con 0x7f0fc80ca5c0 2024-09-13T08:42:21.803 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 msgr2=0x7f0fa407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 0x7f0fa407ae30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8034000 tx=0x7f0fb8004730 comp rx=0 tx=0).stop 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 msgr2=0x7f0fc80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000d950 tx=0x7f0fb000de10 comp rx=0 tx=0).stop 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 shutdown_connections 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa4078970 0x7f0fa407ae30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fc815dfc0 0x7f0fc80b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fc80ca5c0 0x7f0fc80b2cc0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 --2- 172.21.15.31:0/1344287197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc80b1910 0x7f0fc8152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 >> 172.21.15.31:0/1344287197 conn(0x7f0fc80cc910 msgr2=0x7f0fc80b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 shutdown_connections 2024-09-13T08:42:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:21.805+0000 7f0fce800640 1 -- 172.21.15.31:0/1344287197 wait complete. 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:42:21.822 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:21.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:21.823 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:42:21.864 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:42:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42394sec preferred_lft 42394sec 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:42:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:21.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:21.867 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:42:21.965 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:42:21.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:21.969 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:42:22.006 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:42:22.006 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:42:22.006 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:42:22.007 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:42:22.007 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:42:22.007 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:42:22.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:42:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:22.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:22.205 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:42:22.241 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:42:22.242 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:42:22.242 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:22.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:22.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:42:22.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:22.338 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:22.373 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:22.373 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:22.373 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:22.448 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:42:22.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:22.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:22.576 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:22.576 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:22.604 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:22.604 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:22.604 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:42:22.604 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:42:22.604 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:42:22.605 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:42:22.605 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:42:22.606 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:42:22.606 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:42:22.606 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:42:22.609 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:42:22.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:22.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:22.665 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:42:22.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:42:22.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:22.754 INFO:teuthology.orchestra.run.smithi045.stdout:client6732 2024-09-13T08:42:22.755 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:22.755 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:22.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 -- 172.21.15.31:0/2479843174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 msgr2=0x7f4c980cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 --2- 172.21.15.31:0/2479843174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c980cb740 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009830 tx=0x7f4c8402a5b0 comp rx=0 tx=0).stop 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 -- 172.21.15.31:0/2479843174 shutdown_connections 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 --2- 172.21.15.31:0/2479843174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c980cb740 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 -- 172.21.15.31:0/2479843174 >> 172.21.15.31:0/2479843174 conn(0x7f4c980cfbb0 msgr2=0x7f4c980cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 -- 172.21.15.31:0/2479843174 shutdown_connections 2024-09-13T08:42:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.967+0000 7f4ca0400640 1 -- 172.21.15.31:0/2479843174 wait complete. 2024-09-13T08:42:22.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.968+0000 7f4ca0400640 1 Processor -- start 2024-09-13T08:42:22.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.968+0000 7f4ca0400640 1 -- start start 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c980cbc80 0x7f4c98156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c9815bfc0 0x7f4c980b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9806e740 con 0x7f4c98072560 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9806e5c0 con 0x7f4c980cbc80 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.969+0000 7f4ca0400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4c980b30f0 con 0x7f4c9815bfc0 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57964/0 (socket says 172.21.15.31:57964) 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 -- 172.21.15.31:0/921492875 learned_addr learned my addr 172.21.15.31:0/921492875 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9d000640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c980cbc80 0x7f4c98156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9e400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c9815bfc0 0x7f4c980b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 -- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c9815bfc0 msgr2=0x7f4c980b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c9815bfc0 0x7f4c980b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 -- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c980cbc80 msgr2=0x7f4c98156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c980cbc80 0x7f4c98156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 -- 172.21.15.31:0/921492875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c98156990 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.970+0000 7f4c9da00640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c3a0 tx=0x7f4c8c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.971+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c017070 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.971+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c8c0053c0 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.971+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c003850 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.971+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c980b3b70 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.971+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c980b4130 con 0x7f4c98072560 2024-09-13T08:42:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.972+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c980cb740 con 0x7f4c98072560 2024-09-13T08:42:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.973+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f4c8c01a070 con 0x7f4c98072560 2024-09-13T08:42:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.974+0000 7f4c95600640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 0x7f4c7807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.974+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f4c8c099e80 con 0x7f4c98072560 2024-09-13T08:42:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.974+0000 7f4c9d000640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 0x7f4c7807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.975+0000 7f4c9d000640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 0x7f4c7807ae30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c84034000 tx=0x7f4c84004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:22.978+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4c8c061c70 con 0x7f4c98072560 2024-09-13T08:42:23.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:23.216+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f4c980c9fc0 con 0x7f4c98072560 2024-09-13T08:42:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.024+0000 7f4c95600640 1 -- 172.21.15.31:0/921492875 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v58) ==== 136+0+0 (secure 0 0 0) 0x7f4c8c0666a0 con 0x7f4c98072560 2024-09-13T08:42:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 msgr2=0x7f4c7807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 0x7f4c7807ae30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c84034000 tx=0x7f4c84004730 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 msgr2=0x7f4c9814a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c3a0 tx=0x7f4c8c00c860 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 shutdown_connections 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4c78078970 0x7f4c7807ae30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4c9815bfc0 0x7f4c980b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c980cbc80 0x7f4c98156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.027+0000 7f4ca0400640 1 --2- 172.21.15.31:0/921492875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98072560 0x7f4c9814a960 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.028+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 >> 172.21.15.31:0/921492875 conn(0x7f4c980cfbb0 msgr2=0x7f4c98157740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.028+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 shutdown_connections 2024-09-13T08:42:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.028+0000 7f4ca0400640 1 -- 172.21.15.31:0/921492875 wait complete. 2024-09-13T08:42:24.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:42:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 -- 172.21.15.31:0/3557293536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab180ced20 msgr2=0x7fab180ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 --2- 172.21.15.31:0/3557293536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab180ced20 0x7fab180ce0c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fab00005fd0 tx=0x7fab0002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 -- 172.21.15.31:0/3557293536 shutdown_connections 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 --2- 172.21.15.31:0/3557293536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab180ced20 0x7fab180ce0c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 --2- 172.21.15.31:0/3557293536 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab1815be30 0x7fab180cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 -- 172.21.15.31:0/3557293536 >> 172.21.15.31:0/3557293536 conn(0x7fab180b18f0 msgr2=0x7fab180b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 -- 172.21.15.31:0/3557293536 shutdown_connections 2024-09-13T08:42:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.215+0000 7fab1f800640 1 -- 172.21.15.31:0/3557293536 wait complete. 2024-09-13T08:42:24.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.217+0000 7fab1f800640 1 Processor -- start 2024-09-13T08:42:24.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.217+0000 7fab1f800640 1 -- start start 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab180ce600 0x7fab180b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab180ced20 0x7fab180ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab18160160 con 0x7fab1815be30 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab181602e0 con 0x7fab180ce600 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1f800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab18160460 con 0x7fab180ced20 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57984/0 (socket says 172.21.15.31:57984) 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab0fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab180ced20 0x7fab180ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1d800640 1 -- 172.21.15.31:0/2664136496 learned_addr learned my addr 172.21.15.31:0/2664136496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.218+0000 7fab1ce00640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab180ce600 0x7fab180b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 -- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab180ced20 msgr2=0x7fab180ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab180ced20 0x7fab180ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 -- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab180ce600 msgr2=0x7fab180b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab180ce600 0x7fab180b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 -- 172.21.15.31:0/2664136496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181f8580 con 0x7fab1815be30 2024-09-13T08:42:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1d800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c770 tx=0x7fab1000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10003da0 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab10005070 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10005500 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.219+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181f8860 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.220+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab181f9100 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.221+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1815fa50 con 0x7fab1815be30 2024-09-13T08:42:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.222+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fab100056a0 con 0x7fab1815be30 2024-09-13T08:42:24.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.223+0000 7fab15600640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 0x7faaf407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.223+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fab1009a960 con 0x7fab1815be30 2024-09-13T08:42:24.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.223+0000 7fab1ce00640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 0x7faaf407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.224+0000 7fab1ce00640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 0x7faaf407ae30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fab08009b30 tx=0x7fab08005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.227+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab100626c0 con 0x7fab1815be30 2024-09-13T08:42:24.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.446+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fab181f8c60 con 0x7faaf4078970 2024-09-13T08:42:24.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.448+0000 7fab15600640 1 -- 172.21.15.31:0/2664136496 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fab08011be0 con 0x7faaf4078970 2024-09-13T08:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:42:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 msgr2=0x7faaf407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 0x7faaf407ae30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fab08009b30 tx=0x7fab08005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 msgr2=0x7fab180ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c770 tx=0x7fab1000cc30 comp rx=0 tx=0).stop 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 shutdown_connections 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7faaf4078970 0x7faaf407ae30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab1815be30 0x7fab180ad930 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab180ced20 0x7fab180ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 --2- 172.21.15.31:0/2664136496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab180ce600 0x7fab180b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 >> 172.21.15.31:0/2664136496 conn(0x7fab180b18f0 msgr2=0x7fab180af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.450+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 shutdown_connections 2024-09-13T08:42:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.451+0000 7fab1f800640 1 -- 172.21.15.31:0/2664136496 wait complete. 2024-09-13T08:42:24.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:42:24.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 -- 172.21.15.31:0/3282198679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 msgr2=0x7f6914003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 --2- 172.21.15.31:0/3282198679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f6914003dc0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6920009b80 tx=0x7f6914005830 comp rx=0 tx=0).stop 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6934c00640 1 -- 172.21.15.31:0/3282198679 reap_dead start 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 -- 172.21.15.31:0/3282198679 shutdown_connections 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 -- 172.21.15.31:0/3282198679 >> 172.21.15.31:0/3282198679 conn(0x7f69300b1520 msgr2=0x7f69300b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 -- 172.21.15.31:0/3282198679 shutdown_connections 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.632+0000 7f6935600640 1 -- 172.21.15.31:0/3282198679 wait complete. 2024-09-13T08:42:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.634+0000 7f6935600640 1 Processor -- start 2024-09-13T08:42:24.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.634+0000 7f6935600640 1 -- start start 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 0x7f69301e4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 0x7f69301e4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69300cc9d0 con 0x7f69140039e0 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69300cc850 con 0x7f69301f5640 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6935600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f69300ccb50 con 0x7f69301f5ac0 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58010/0 (socket says 172.21.15.31:58010) 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 -- 172.21.15.31:0/2238212467 learned_addr learned my addr 172.21.15.31:0/2238212467 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6927e00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 0x7f69301e4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6934c00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 0x7f69301e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 -- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 msgr2=0x7f69301e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 0x7f69301e4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 -- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 msgr2=0x7f69301e4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 0x7f69301e4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f692fe00640 1 -- 172.21.15.31:0/2238212467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69301f7960 con 0x7f69140039e0 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6927e00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 0x7f69301e4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.635+0000 7f6934c00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 0x7f69301e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f692fe00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f6920033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920038040 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6920004020 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693015d340 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920004450 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.636+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f693015d900 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.637+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69300ca950 con 0x7f69140039e0 2024-09-13T08:42:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.638+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f6920045020 con 0x7f69140039e0 2024-09-13T08:42:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.639+0000 7f692cc00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 0x7f690c07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.639+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7f69200bb220 con 0x7f69140039e0 2024-09-13T08:42:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.640+0000 7f6927e00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 0x7f690c07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.640+0000 7f6927e00640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 0x7f690c07ae60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f69180011a0 tx=0x7f6918006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:24.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.644+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6920083700 con 0x7f69140039e0 2024-09-13T08:42:24.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.839+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f693006f430 con 0x7f69140039e0 2024-09-13T08:42:24.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.840+0000 7f692cc00640 1 -- 172.21.15.31:0/2238212467 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f6920083000 con 0x7f69140039e0 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 msgr2=0x7f690c07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 0x7f690c07ae60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f69180011a0 tx=0x7f6918006b10 comp rx=0 tx=0).stop 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 msgr2=0x7f69301f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f6920033670 comp rx=0 tx=0).stop 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 shutdown_connections 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f690c0789a0 0x7f690c07ae60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69301f5ac0 0x7f69301e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69301f5640 0x7f69301e4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 --2- 172.21.15.31:0/2238212467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69140039e0 0x7f69301f7420 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 >> 172.21.15.31:0/2238212467 conn(0x7f69300b1520 msgr2=0x7f69300cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 shutdown_connections 2024-09-13T08:42:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:24.842+0000 7f6935600640 1 -- 172.21.15.31:0/2238212467 wait complete. 2024-09-13T08:42:24.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create 'abcd@#' 2024-09-13T08:42:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 -- 172.21.15.31:0/26891780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 msgr2=0x7fadd00d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 --2- 172.21.15.31:0/26891780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00d0690 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009830 tx=0x7fadbc02a480 comp rx=0 tx=0).stop 2024-09-13T08:42:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 -- 172.21.15.31:0/26891780 shutdown_connections 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 --2- 172.21.15.31:0/26891780 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd00ae990 0x7fadd00ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 --2- 172.21.15.31:0/26891780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00d0690 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 -- 172.21.15.31:0/26891780 >> 172.21.15.31:0/26891780 conn(0x7fadd00cfb70 msgr2=0x7fadd00cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 -- 172.21.15.31:0/26891780 shutdown_connections 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.020+0000 7fadd8200640 1 -- 172.21.15.31:0/26891780 wait complete. 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.022+0000 7fadd8200640 1 Processor -- start 2024-09-13T08:42:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.022+0000 7fadd8200640 1 -- start start 2024-09-13T08:42:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd00ae990 0x7fadd01f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd015d9c0 0x7fadd01e64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fadd006e950 con 0x7fadd00cab70 2024-09-13T08:42:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fadd006e7d0 con 0x7fadd00ae990 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd8200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fadd006ead0 con 0x7fadd015d9c0 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00b5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38172/0 (socket says 172.21.15.31:38172) 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 -- 172.21.15.31:0/624595729 learned_addr learned my addr 172.21.15.31:0/624595729 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd6200640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd015d9c0 0x7fadd01e64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd5800640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd00ae990 0x7fadd01f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 -- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd015d9c0 msgr2=0x7fadd01e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd015d9c0 0x7fadd01e64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 -- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd00ae990 msgr2=0x7fadd01f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd00ae990 0x7fadd01f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd4e00640 1 -- 172.21.15.31:0/624595729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd01e5740 con 0x7fadd00cab70 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd6200640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd015d9c0 0x7fadd01e64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.023+0000 7fadd5800640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd00ae990 0x7fadd01f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadd4e00640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd00cab70 0x7fadd00b5330 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fadbc000c00 tx=0x7fadbc032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc040070 con 0x7fadd00cab70 2024-09-13T08:42:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadbc003af0 con 0x7fadd00cab70 2024-09-13T08:42:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc045460 con 0x7fadd00cab70 2024-09-13T08:42:25.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadd8200640 1 -- 172.21.15.31:0/624595729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd00b5870 con 0x7fadd00cab70 2024-09-13T08:42:25.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.024+0000 7fadd8200640 1 -- 172.21.15.31:0/624595729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd01f5930 con 0x7fadd00cab70 2024-09-13T08:42:25.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.025+0000 7fadd8200640 1 -- 172.21.15.31:0/624595729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd00ce070 con 0x7fadd00cab70 2024-09-13T08:42:25.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.027+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fadbc039040 con 0x7fadd00cab70 2024-09-13T08:42:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.028+0000 7fadcd600640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb00789a0 0x7fadb007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.029+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fadbc0bc270 con 0x7fadd00cab70 2024-09-13T08:42:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.029+0000 7fadd5800640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb00789a0 0x7fadb007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.029+0000 7fadd5800640 1 --2- 172.21.15.31:0/624595729 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb00789a0 0x7fadb007ae60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fadd00b52d0 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.033+0000 7fadcd600640 1 -- 172.21.15.31:0/624595729 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fadbc0846b0 con 0x7fadd00cab70 2024-09-13T08:42:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-13T08:42:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:fs volume create [] : Create a CephFS volume 2024-09-13T08:42:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-09-13T08:42:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T08:42:25.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create testvol 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.424+0000 7fe7fc600640 1 -- 172.21.15.31:0/2550839927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 msgr2=0x7fe7dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.424+0000 7fe7fc600640 1 --2- 172.21.15.31:0/2550839927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7dc004c80 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009bc0 tx=0x7fe7dc005830 comp rx=0 tx=0).stop 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.424+0000 7fe7fa600640 1 -- 172.21.15.31:0/2550839927 reap_dead start 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.425+0000 7fe7fc600640 1 -- 172.21.15.31:0/2550839927 shutdown_connections 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.425+0000 7fe7fc600640 1 -- 172.21.15.31:0/2550839927 >> 172.21.15.31:0/2550839927 conn(0x7fe7f40d0c90 msgr2=0x7fe7f40d10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.425+0000 7fe7fc600640 1 -- 172.21.15.31:0/2550839927 shutdown_connections 2024-09-13T08:42:25.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.425+0000 7fe7fc600640 1 -- 172.21.15.31:0/2550839927 wait complete. 2024-09-13T08:42:25.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.426+0000 7fe7fc600640 1 Processor -- start 2024-09-13T08:42:25.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.426+0000 7fe7fc600640 1 -- start start 2024-09-13T08:42:25.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f41e9950 0x7fe7f40c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f41e6f60 0x7fe7f40c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40d17b0 con 0x7fe7dc004800 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40b6590 con 0x7fe7f41e9950 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.427+0000 7fe7fc600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f40d1930 con 0x7fe7f41e6f60 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38202/0 (socket says 172.21.15.31:38202) 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 -- 172.21.15.31:0/4122860754 learned_addr learned my addr 172.21.15.31:0/4122860754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9200640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f41e9950 0x7fe7f40c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7fa600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f41e6f60 0x7fe7f40c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 -- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f41e6f60 msgr2=0x7fe7f40c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f41e6f60 0x7fe7f40c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 -- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f41e9950 msgr2=0x7fe7f40c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:25.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f41e9950 0x7fe7f40c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 -- 172.21.15.31:0/4122860754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f41f5c10 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.428+0000 7fe7f9c00640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8003db0 tx=0x7fe7e8001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.429+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011430 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.429+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7e8016020 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.429+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011890 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.429+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f40c4030 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.429+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f40c7f70 con 0x7fe7dc004800 2024-09-13T08:42:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.430+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f415cd00 con 0x7fe7dc004800 2024-09-13T08:42:25.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.431+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe7e8011a30 con 0x7fe7dc004800 2024-09-13T08:42:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.432+0000 7fe7f1600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 0x7fe7d407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.432+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5709+0+0 (secure 0 0 0) 0x7fe7e809b890 con 0x7fe7dc004800 2024-09-13T08:42:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.432+0000 7fe7f9200640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 0x7fe7d407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:25.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.433+0000 7fe7f9200640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 0x7fe7d407ae30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0035000 tx=0x7fe7e00046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:25.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.436+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe7e8020020 con 0x7fe7dc004800 2024-09-13T08:42:25.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:25.656+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "testvol", "target": ["mon-mgr", ""]}) -- 0x7fe7f40c75f0 con 0x7fe7d4078970 2024-09-13T08:42:28.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.064+0000 7fe7f1600640 1 -- 172.21.15.31:0/4122860754 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) ==== 60+0+0 (secure 0 0 0) 0x7fe7f40c75f0 con 0x7fe7d4078970 2024-09-13T08:42:28.065 INFO:teuthology.orchestra.run.smithi031.stderr:Volume created successfully (no MDS daemons created) 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 msgr2=0x7fe7d407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 0x7fe7d407ae30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0035000 tx=0x7fe7e00046c0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 msgr2=0x7fe7f41f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8003db0 tx=0x7fe7e8001800 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 shutdown_connections 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7d4078970 0x7fe7d407ae30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f41e6f60 0x7fe7f40c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f41e9950 0x7fe7f40c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 --2- 172.21.15.31:0/4122860754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7dc004800 0x7fe7f41f56d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 >> 172.21.15.31:0/4122860754 conn(0x7fe7f40d0c90 msgr2=0x7fe7f40b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:28.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 shutdown_connections 2024-09-13T08:42:28.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.068+0000 7fe7fc600640 1 -- 172.21.15.31:0/4122860754 wait complete. 2024-09-13T08:42:28.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create testvol 'abcd@#' 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 -- 172.21.15.31:0/4066383864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 msgr2=0x7fb5f8004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 --2- 172.21.15.31:0/4066383864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 0x7fb5f8004880 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb60000c290 tx=0x7fb5f8005830 comp rx=0 tx=0).stop 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb616800640 1 -- 172.21.15.31:0/4066383864 reap_dead start 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 -- 172.21.15.31:0/4066383864 shutdown_connections 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 -- 172.21.15.31:0/4066383864 >> 172.21.15.31:0/4066383864 conn(0x7fb6100cfbb0 msgr2=0x7fb6100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 -- 172.21.15.31:0/4066383864 shutdown_connections 2024-09-13T08:42:28.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.262+0000 7fb618800640 1 -- 172.21.15.31:0/4066383864 wait complete. 2024-09-13T08:42:28.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.263+0000 7fb618800640 1 Processor -- start 2024-09-13T08:42:28.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.264+0000 7fb618800640 1 -- start start 2024-09-13T08:42:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.264+0000 7fb618800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 0x7fb610157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb618800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6100ce8c0 0x7fb6101e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb618800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb610153af0 0x7fb610153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb618800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6100cbfd0 con 0x7fb5f80043b0 2024-09-13T08:42:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb618800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6100cbe50 con 0x7fb610153af0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb618800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb6100b29e0 con 0x7fb6100ce8c0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 0x7fb610157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 0x7fb610157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38218/0 (socket says 172.21.15.31:38218) 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 -- 172.21.15.31:0/986958170 learned_addr learned my addr 172.21.15.31:0/986958170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615400640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6100ce8c0 0x7fb6101e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb616800640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb610153af0 0x7fb610153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 -- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6100ce8c0 msgr2=0x7fb6101e8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6100ce8c0 0x7fb6101e8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 -- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb610153af0 msgr2=0x7fb610153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb610153af0 0x7fb610153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb615e00640 1 -- 172.21.15.31:0/986958170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6100cd820 con 0x7fb5f80043b0 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.265+0000 7fb616800640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb610153af0 0x7fb610153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:28.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb615e00640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5f80043b0 0x7fb610157af0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb604011ae0 tx=0x7fb604009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60401e070 con 0x7fb5f80043b0 2024-09-13T08:42:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb60400fc70 con 0x7fb5f80043b0 2024-09-13T08:42:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb618800640 1 -- 172.21.15.31:0/986958170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6100cdb00 con 0x7fb5f80043b0 2024-09-13T08:42:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604019410 con 0x7fb5f80043b0 2024-09-13T08:42:28.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.266+0000 7fb618800640 1 -- 172.21.15.31:0/986958170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6100ce0c0 con 0x7fb5f80043b0 2024-09-13T08:42:28.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.267+0000 7fb618800640 1 -- 172.21.15.31:0/986958170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61006fec0 con 0x7fb5f80043b0 2024-09-13T08:42:28.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.268+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fb604021070 con 0x7fb5f80043b0 2024-09-13T08:42:28.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.269+0000 7fb60a000640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb5f4078970 0x7fb5f407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.269+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6543+0+0 (secure 0 0 0) 0x7fb60409b760 con 0x7fb5f80043b0 2024-09-13T08:42:28.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.270+0000 7fb615400640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb5f4078970 0x7fb5f407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.270+0000 7fb615400640 1 --2- 172.21.15.31:0/986958170 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb5f4078970 0x7fb5f407ae30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb600033000 tx=0x7fb600038040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:28.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.274+0000 7fb60a000640 1 -- 172.21.15.31:0/986958170 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb604063210 con 0x7fb5f80043b0 2024-09-13T08:42:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-13T08:42:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:fs subvolumegroup create [] [] [] [] [] : Create a CephFS subvolume group in a volume, and optionally, with a specific data pool layout, and a specific numeric mode 2024-09-13T08:42:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-09-13T08:42:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T08:42:28.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create testvol 'abcd@#' 2024-09-13T08:42:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 -- 172.21.15.31:0/983983577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b5430 msgr2=0x7f6ddc0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 --2- 172.21.15.31:0/983983577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0d24a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8005fd0 tx=0x7f6dc802a4b0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 -- 172.21.15.31:0/983983577 shutdown_connections 2024-09-13T08:42:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 --2- 172.21.15.31:0/983983577 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc0b1480 0x7f6ddc0b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 --2- 172.21.15.31:0/983983577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0d24a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 -- 172.21.15.31:0/983983577 >> 172.21.15.31:0/983983577 conn(0x7f6ddc0d3e90 msgr2=0x7f6ddc0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 -- 172.21.15.31:0/983983577 shutdown_connections 2024-09-13T08:42:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.667+0000 7f6de1200640 1 -- 172.21.15.31:0/983983577 wait complete. 2024-09-13T08:42:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.669+0000 7f6de1200640 1 Processor -- start 2024-09-13T08:42:28.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.669+0000 7f6de1200640 1 -- start start 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b1480 0x7f6ddc0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc161470 0x7f6ddc0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b5220 con 0x7f6ddc0b1480 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b50a0 con 0x7f6ddc0b5430 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6de1200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b1e50 con 0x7f6ddc161470 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b1480 0x7f6ddc0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b1480 0x7f6ddc0c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38238/0 (socket says 172.21.15.31:38238) 2024-09-13T08:42:28.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 -- 172.21.15.31:0/2729185639 learned_addr learned my addr 172.21.15.31:0/2729185639 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddaa00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddbe00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc161470 0x7f6ddc0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 -- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc161470 msgr2=0x7f6ddc0cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc161470 0x7f6ddc0cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 -- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ddc0b5430 msgr2=0x7f6ddc0cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.670+0000 7f6ddb400640 1 -- 172.21.15.31:0/2729185639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc0cd1f0 con 0x7f6ddc0b1480 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6ddbe00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ddc161470 0x7f6ddc0cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6ddaa00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ddc0b5430 0x7f6ddc0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6ddb400640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ddc0b1480 0x7f6ddc0c9ee0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00c3d0 tx=0x7f6dcc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc017070 con 0x7f6ddc0b1480 2024-09-13T08:42:28.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dcc0053c0 con 0x7f6ddc0b1480 2024-09-13T08:42:28.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6de1200640 1 -- 172.21.15.31:0/2729185639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc0cd460 con 0x7f6ddc0b1480 2024-09-13T08:42:28.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc003850 con 0x7f6ddc0b1480 2024-09-13T08:42:28.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.671+0000 7f6de1200640 1 -- 172.21.15.31:0/2729185639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc150ed0 con 0x7f6ddc0b1480 2024-09-13T08:42:28.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.672+0000 7f6de1200640 1 -- 172.21.15.31:0/2729185639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ddc0b1910 con 0x7f6ddc0b1480 2024-09-13T08:42:28.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.674+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f6dcc01a070 con 0x7f6ddc0b1480 2024-09-13T08:42:28.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.675+0000 7f6dd3400640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dbc078970 0x7f6dbc07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:28.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.675+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6543+0+0 (secure 0 0 0) 0x7f6dcc09a2f0 con 0x7f6ddc0b1480 2024-09-13T08:42:28.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.675+0000 7f6ddaa00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dbc078970 0x7f6dbc07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:28.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.676+0000 7f6ddaa00640 1 --2- 172.21.15.31:0/2729185639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dbc078970 0x7f6dbc07ae30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8033000 tx=0x7f6dc8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:28.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:28.680+0000 7f6dd3400640 1 -- 172.21.15.31:0/2729185639 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6dcc062360 con 0x7f6ddc0b1480 2024-09-13T08:42:28.906 INFO:teuthology.orchestra.run.smithi031.stderr:Invalid command: invalid chars @# in abcd@# 2024-09-13T08:42:28.906 INFO:teuthology.orchestra.run.smithi031.stderr:fs subvolume create [] [] [] [] [] [] [--namespace-isolated] [] : Create a CephFS subvolume in a volume, and optionally, with a specific size (in bytes), a specific data pool layout, a specific mode, in a specific subvolume group and in separate RADOS namespace 2024-09-13T08:42:28.906 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-09-13T08:42:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T08:42:28.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm testvol --yes-i-really-mean-it 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 -- 172.21.15.31:0/2803494693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f80d1d90 msgr2=0x7f57f80d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 --2- 172.21.15.31:0/2803494693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f80d1d90 0x7f57f80d21d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f57f80b6a80 tx=0x7f57e402a4d0 comp rx=0 tx=0).stop 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 -- 172.21.15.31:0/2803494693 shutdown_connections 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 --2- 172.21.15.31:0/2803494693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f80d1d90 0x7f57f80d21d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 --2- 172.21.15.31:0/2803494693 >> v2:172.21.15.45:3300/0 conn(0x7f57f8161470 0x7f57f80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 -- 172.21.15.31:0/2803494693 >> 172.21.15.31:0/2803494693 conn(0x7f57f80d3e90 msgr2=0x7f57f80d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 -- 172.21.15.31:0/2803494693 shutdown_connections 2024-09-13T08:42:29.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.091+0000 7f57fee00640 1 -- 172.21.15.31:0/2803494693 wait complete. 2024-09-13T08:42:29.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.092+0000 7f57fee00640 1 Processor -- start 2024-09-13T08:42:29.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.093+0000 7f57fee00640 1 -- start start 2024-09-13T08:42:29.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.093+0000 7f57fee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d1d90 0x7f57f80cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:29.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.093+0000 7f57fee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f57f80d2710 0x7f57f8155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80b2610 con 0x7f57f8161470 2024-09-13T08:42:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80b2490 con 0x7f57f80d1d90 2024-09-13T08:42:29.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f57f80b2790 con 0x7f57f80d2710 2024-09-13T08:42:29.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:29.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38250/0 (socket says 172.21.15.31:38250) 2024-09-13T08:42:29.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 -- 172.21.15.31:0/2924046140 learned_addr learned my addr 172.21.15.31:0/2924046140 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fd000640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f57f80d2710 0x7f57f8155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fda00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d1d90 0x7f57f80cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 -- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f57f80d2710 msgr2=0x7f57f8155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f57f80d2710 0x7f57f8155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 -- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d1d90 msgr2=0x7f57f80cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d1d90 0x7f57f80cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.094+0000 7f57fe400640 1 -- 172.21.15.31:0/2924046140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f80cd0d0 con 0x7f57f8161470 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57fe400640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f57f000c7a0 tx=0x7f57f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0003b70 con 0x7f57f8161470 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57f0003d10 con 0x7f57f8161470 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f80ccb10 con 0x7f57f8161470 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f000aab0 con 0x7f57f8161470 2024-09-13T08:42:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.095+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f81f7f20 con 0x7f57f8161470 2024-09-13T08:42:29.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.096+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f81e77b0 con 0x7f57f8161470 2024-09-13T08:42:29.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.097+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f57f000ac50 con 0x7f57f8161470 2024-09-13T08:42:29.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.098+0000 7f57f5600640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 0x7f57d807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:29.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.098+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6543+0+0 (secure 0 0 0) 0x7f57f009ae40 con 0x7f57f8161470 2024-09-13T08:42:29.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.099+0000 7f57fda00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 0x7f57d807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:29.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.099+0000 7f57fda00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 0x7f57d807ae30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009ac0 tx=0x7f57ec005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:29.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.103+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f57f81e77b0 con 0x7f57f8161470 2024-09-13T08:42:29.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:29.320+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "testvol", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) -- 0x7f57f80d21d0 con 0x7f57d8078970 2024-09-13T08:42:29.325 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:42:29.324+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:42:29.378 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:42:32.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.341+0000 7f57f5600640 1 -- 172.21.15.31:0/2924046140 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+78 (secure 0 0 0) 0x7f57f80d21d0 con 0x7f57d8078970 2024-09-13T08:42:32.342 INFO:teuthology.orchestra.run.smithi031.stdout:metadata pool: cephfs.testvol.meta data pool: ['cephfs.testvol.data'] removed 2024-09-13T08:42:32.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 msgr2=0x7f57d807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 0x7f57d807ae30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009ac0 tx=0x7f57ec005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 msgr2=0x7f57f815c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f57f000c7a0 tx=0x7f57f000cc60 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 shutdown_connections 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f57d8078970 0x7f57d807ae30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57f8161470 0x7f57f815c260 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f57f80d2710 0x7f57f8155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 --2- 172.21.15.31:0/2924046140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f80d1d90 0x7f57f80cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.345+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 >> 172.21.15.31:0/2924046140 conn(0x7f57f80d3e90 msgr2=0x7f57f81e6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.346+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 shutdown_connections 2024-09-13T08:42:32.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.346+0000 7f57fee00640 1 -- 172.21.15.31:0/2924046140 wait complete. 2024-09-13T08:42:32.366 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:42:32.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:32.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:32.561 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:42:32.562 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:42:32.562 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:32.562 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:32.599 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:32.599 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:32.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:42:32.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 -- 172.21.15.31:0/678868393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86140ced20 msgr2=0x7f86140ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 --2- 172.21.15.31:0/678868393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86140ced20 0x7f86140ce290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8600009830 tx=0x7f860002a510 comp rx=0 tx=0).stop 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 -- 172.21.15.31:0/678868393 shutdown_connections 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 --2- 172.21.15.31:0/678868393 >> v2:172.21.15.45:3300/0 conn(0x7f8614073860 0x7f8614073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 --2- 172.21.15.31:0/678868393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86140ced20 0x7f86140ce290 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 -- 172.21.15.31:0/678868393 >> 172.21.15.31:0/678868393 conn(0x7f86140b18f0 msgr2=0x7f86140b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 -- 172.21.15.31:0/678868393 shutdown_connections 2024-09-13T08:42:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.880+0000 7f861a800640 1 -- 172.21.15.31:0/678868393 wait complete. 2024-09-13T08:42:32.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.882+0000 7f861a800640 1 Processor -- start 2024-09-13T08:42:32.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.882+0000 7f861a800640 1 -- start start 2024-09-13T08:42:32.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8614073860 0x7f861414c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:32.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f861415be30 0x7f861414cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86140733e0 con 0x7f86140ced20 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f86140ce9e0 con 0x7f861415be30 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f861a800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8614074230 con 0x7f8614073860 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f8612a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f8612a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38256/0 (socket says 172.21.15.31:38256) 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f8612a00640 1 -- 172.21.15.31:0/4290501070 learned_addr learned my addr 172.21.15.31:0/4290501070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f8613400640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8614073860 0x7f861414c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.883+0000 7f8613e00640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f861415be30 0x7f861414cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 -- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8614073860 msgr2=0x7f861414c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8614073860 0x7f861414c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 -- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f861415be30 msgr2=0x7f861414cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:32.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f861415be30 0x7f861414cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:32.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 -- 172.21.15.31:0/4290501070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861414cca0 con 0x7f86140ced20 2024-09-13T08:42:32.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f8612a00640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8600000c00 tx=0x7f8600032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:32.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600040070 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86000038e0 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600005c90 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.884+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86140ca8b0 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.885+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861415a570 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.886+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861415a080 con 0x7f86140ced20 2024-09-13T08:42:32.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.887+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f8600039040 con 0x7f86140ced20 2024-09-13T08:42:32.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.888+0000 7f860b400640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 0x7f85f407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:32.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.888+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f86000bc140 con 0x7f86140ced20 2024-09-13T08:42:32.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.888+0000 7f8613400640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 0x7f85f407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:32.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.889+0000 7f8613400640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 0x7f85f407ae30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8604000f60 tx=0x7f8604009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:32.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:32.892+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f861414cca0 con 0x7f86140ced20 2024-09-13T08:42:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.099+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8614074720 con 0x7f86140ced20 2024-09-13T08:42:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.099+0000 7f860b400640 1 -- 172.21.15.31:0/4290501070 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v97) ==== 80+0+122 (secure 0 0 0) 0x7f8600083e50 con 0x7f86140ced20 2024-09-13T08:42:33.100 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.45:6835/3778871942 2024-09-14T08:42:29.324924+0000 2024-09-13T08:42:33.100 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.45:6834/3778871942 2024-09-14T08:42:29.324924+0000 2024-09-13T08:42:33.101 INFO:teuthology.orchestra.run.smithi031.stderr:listed 2 entries 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 msgr2=0x7f85f407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 0x7f85f407ae30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8604000f60 tx=0x7f8604009040 comp rx=0 tx=0).stop 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 msgr2=0x7f86140c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8600000c00 tx=0x7f8600032c10 comp rx=0 tx=0).stop 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 shutdown_connections 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85f4078970 0x7f85f407ae30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f861415be30 0x7f861414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86140ced20 0x7f86140c9e60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.101+0000 7f861a800640 1 --2- 172.21.15.31:0/4290501070 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8614073860 0x7f861414c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.102+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 >> 172.21.15.31:0/4290501070 conn(0x7f86140b18f0 msgr2=0x7f86140b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:33.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.102+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 shutdown_connections 2024-09-13T08:42:33.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.102+0000 7f861a800640 1 -- 172.21.15.31:0/4290501070 wait complete. 2024-09-13T08:42:33.118 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:42:33.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:33.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:33.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:42:33.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:33.182 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:42:33.218 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:42:33.218 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:33.218 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:33.258 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:33.258 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:33.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 -- 172.21.15.31:0/1000854509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 msgr2=0x7f70c015bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 --2- 172.21.15.31:0/1000854509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c015bdc0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f70b0005fd0 tx=0x7f70b002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 -- 172.21.15.31:0/1000854509 shutdown_connections 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 --2- 172.21.15.31:0/1000854509 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c00cf3a0 0x7f70c015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 --2- 172.21.15.31:0/1000854509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c015bdc0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 -- 172.21.15.31:0/1000854509 >> 172.21.15.31:0/1000854509 conn(0x7f70c00b18f0 msgr2=0x7f70c00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 -- 172.21.15.31:0/1000854509 shutdown_connections 2024-09-13T08:42:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.503+0000 7f70c7a00640 1 -- 172.21.15.31:0/1000854509 wait complete. 2024-09-13T08:42:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.505+0000 7f70c7a00640 1 Processor -- start 2024-09-13T08:42:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.505+0000 7f70c7a00640 1 -- start start 2024-09-13T08:42:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c00cf3a0 0x7f70c009c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c015e930 0x7f70c013d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70c006e7e0 con 0x7f70c01446e0 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70c006e660 con 0x7f70c00cf3a0 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70c7a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f70c006e960 con 0x7f70c015e930 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70b7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70b7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38260/0 (socket says 172.21.15.31:38260) 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.506+0000 7f70b7e00640 1 -- 172.21.15.31:0/2121051262 learned_addr learned my addr 172.21.15.31:0/2121051262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70c5a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c015e930 0x7f70c013d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70c5000640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c00cf3a0 0x7f70c009c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 -- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c015e930 msgr2=0x7f70c013d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c015e930 0x7f70c013d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 -- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c00cf3a0 msgr2=0x7f70c009c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c00cf3a0 0x7f70c009c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 -- 172.21.15.31:0/2121051262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c009ca80 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70b7e00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f70a800c3a0 tx=0x7f70a800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a8017070 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70a80053c0 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c013e100 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.507+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a8003850 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.508+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c009bd40 con 0x7f70c01446e0 2024-09-13T08:42:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.508+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c015ed70 con 0x7f70c01446e0 2024-09-13T08:42:33.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.510+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f70a801a070 con 0x7f70c01446e0 2024-09-13T08:42:33.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.511+0000 7f70bd600640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 0x7f709c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:33.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.511+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f70a8099f70 con 0x7f70c01446e0 2024-09-13T08:42:33.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.511+0000 7f70c5000640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 0x7f709c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:33.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.512+0000 7f70c5000640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 0x7f709c07ae30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70b0033000 tx=0x7f70b0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.515+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f70c015b340 con 0x7f70c01446e0 2024-09-13T08:42:33.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.726+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f70c0142940 con 0x7f70c01446e0 2024-09-13T08:42:33.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.727+0000 7f70bd600640 1 -- 172.21.15.31:0/2121051262 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v97) ==== 80+0+122 (secure 0 0 0) 0x7f70a8061c70 con 0x7f70c01446e0 2024-09-13T08:42:33.728 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.45:6835/3778871942 2024-09-14T08:42:29.324924+0000 2024-09-13T08:42:33.728 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.45:6834/3778871942 2024-09-14T08:42:29.324924+0000 2024-09-13T08:42:33.728 INFO:teuthology.orchestra.run.smithi031.stderr:listed 2 entries 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 msgr2=0x7f709c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 0x7f709c07ae30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70b0033000 tx=0x7f70b0004730 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 msgr2=0x7f70c01efea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f70a800c3a0 tx=0x7f70a800c860 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 shutdown_connections 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f709c078970 0x7f709c07ae30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f70c015e930 0x7f70c013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70c01446e0 0x7f70c01efea0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 --2- 172.21.15.31:0/2121051262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c00cf3a0 0x7f70c009c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.729+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 >> 172.21.15.31:0/2121051262 conn(0x7f70c00b18f0 msgr2=0x7f70c0142e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:33.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.730+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 shutdown_connections 2024-09-13T08:42:33.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:33.730+0000 7f70c7a00640 1 -- 172.21.15.31:0/2121051262 wait complete. 2024-09-13T08:42:33.745 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:42:33.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:33.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:33.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:42:33.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:33.816 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:42:33.854 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:42:33.854 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:33.854 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:33.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:42:33.891 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:42:33.891 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:33.891 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:33.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:42:33.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 -- 172.21.15.31:0/4024315371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 msgr2=0x7f5e940cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 --2- 172.21.15.31:0/4024315371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e940cf850 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5e84005fd0 tx=0x7f5e8402aa90 comp rx=0 tx=0).stop 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 -- 172.21.15.31:0/4024315371 shutdown_connections 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 --2- 172.21.15.31:0/4024315371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e9415f9e0 0x7f5e9415fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 --2- 172.21.15.31:0/4024315371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e940cf850 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 -- 172.21.15.31:0/4024315371 >> 172.21.15.31:0/4024315371 conn(0x7f5e940cc910 msgr2=0x7f5e940cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:34.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 -- 172.21.15.31:0/4024315371 shutdown_connections 2024-09-13T08:42:34.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.091+0000 7f5e9b000640 1 -- 172.21.15.31:0/4024315371 wait complete. 2024-09-13T08:42:34.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.093+0000 7f5e9b000640 1 Processor -- start 2024-09-13T08:42:34.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.093+0000 7f5e9b000640 1 -- start start 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e940ce3c0 0x7f5e9409bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e9415f9e0 0x7f5e940b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5e940cb9d0 con 0x7f5e9415bd70 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e940cb850 con 0x7f5e9415f9e0 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e9b000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5e940cc360 con 0x7f5e940ce3c0 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38280/0 (socket says 172.21.15.31:38280) 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 -- 172.21.15.31:0/3213619925 learned_addr learned my addr 172.21.15.31:0/3213619925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e99000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e9415f9e0 0x7f5e940b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93e00640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e940ce3c0 0x7f5e9409bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 -- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e940ce3c0 msgr2=0x7f5e9409bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e940ce3c0 0x7f5e9409bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 -- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e9415f9e0 msgr2=0x7f5e940b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e9415f9e0 0x7f5e940b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.094+0000 7f5e93400640 1 -- 172.21.15.31:0/3213619925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9409b6d0 con 0x7f5e9415bd70 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e93400640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000d780 tx=0x7f5e8000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80017070 con 0x7f5e9415bd70 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e8000de90 con 0x7f5e9415bd70 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800099d0 con 0x7f5e9415bd70 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9409b9b0 con 0x7f5e9415bd70 2024-09-13T08:42:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.095+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e941e5a10 con 0x7f5e9415bd70 2024-09-13T08:42:34.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.096+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e94160150 con 0x7f5e9415bd70 2024-09-13T08:42:34.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.097+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f5e80009b70 con 0x7f5e9415bd70 2024-09-13T08:42:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.098+0000 7f5e8be00640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 0x7f5e7407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.098+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f5e8009ab60 con 0x7f5e9415bd70 2024-09-13T08:42:34.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.098+0000 7f5e93e00640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 0x7f5e7407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.099+0000 7f5e93e00640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 0x7f5e7407ae30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5e84035000 tx=0x7f5e84039040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.103+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5e94160150 con 0x7f5e9415bd70 2024-09-13T08:42:34.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.340+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e940cf850 con 0x7f5e9415bd70 2024-09-13T08:42:34.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.340+0000 7f5e8be00640 1 -- 172.21.15.31:0/3213619925 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5225 (secure 0 0 0) 0x7f5e800628f0 con 0x7f5e9415bd70 2024-09-13T08:42:34.341 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:34.342 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"btime":"2024-09-13T08:42:30:327258+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6481,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3602175668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3602175668},{"type":"v1","addr":"172.21.15.45:6835","nonce":3602175668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:34.342 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 msgr2=0x7f5e7407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 0x7f5e7407ae30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5e84035000 tx=0x7f5e84039040 comp rx=0 tx=0).stop 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 msgr2=0x7f5e9415af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000d780 tx=0x7f5e8000dc40 comp rx=0 tx=0).stop 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 shutdown_connections 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5e74078970 0x7f5e7407ae30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e9415f9e0 0x7f5e940b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e9415bd70 0x7f5e9415af90 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 --2- 172.21.15.31:0/3213619925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5e940ce3c0 0x7f5e9409bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 >> 172.21.15.31:0/3213619925 conn(0x7f5e940cc910 msgr2=0x7f5e940cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:34.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 shutdown_connections 2024-09-13T08:42:34.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.343+0000 7f5e9b000640 1 -- 172.21.15.31:0/3213619925 wait complete. 2024-09-13T08:42:34.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.520+0000 7fd9fa600640 1 -- 172.21.15.31:0/1140648875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415ffb0 msgr2=0x7fd9f4160460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.520+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1140648875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415ffb0 0x7fd9f4160460 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4075320 tx=0x7fd9ec02a910 comp rx=0 tx=0).stop 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 -- 172.21.15.31:0/1140648875 shutdown_connections 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1140648875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415ffb0 0x7fd9f4160460 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1140648875 >> v2:172.21.15.45:3300/0 conn(0x7fd9f415f530 0x7fd9f415fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1140648875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f150 0x7fd9f40b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 -- 172.21.15.31:0/1140648875 >> 172.21.15.31:0/1140648875 conn(0x7fd9f40d1c70 msgr2=0x7fd9f40d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 -- 172.21.15.31:0/1140648875 shutdown_connections 2024-09-13T08:42:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.521+0000 7fd9fa600640 1 -- 172.21.15.31:0/1140648875 wait complete. 2024-09-13T08:42:34.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.522+0000 7fd9fa600640 1 Processor -- start 2024-09-13T08:42:34.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.522+0000 7fd9fa600640 1 -- start start 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f530 0x7fd9f4154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9f415ffb0 0x7fd9f40afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f40cff00 con 0x7fd9f415f150 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f40cfd80 con 0x7fd9f415ffb0 2024-09-13T08:42:34.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9fa600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f40d0080 con 0x7fd9f415f530 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.523+0000 7fd9f9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38288/0 (socket says 172.21.15.31:38288) 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 -- 172.21.15.31:0/1944179841 learned_addr learned my addr 172.21.15.31:0/1944179841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9c00640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9f415ffb0 0x7fd9f40afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f3e00640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f530 0x7fd9f4154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 -- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f530 msgr2=0x7fd9f4154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f530 0x7fd9f4154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 -- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9f415ffb0 msgr2=0x7fd9f40afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9f415ffb0 0x7fd9f40afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 -- 172.21.15.31:0/1944179841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f4155e70 con 0x7fd9f415f150 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f9200640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d720 tx=0x7fd9e800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.524+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8017070 con 0x7fd9f415f150 2024-09-13T08:42:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.525+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9e8009d20 con 0x7fd9f415f150 2024-09-13T08:42:34.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.525+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8004280 con 0x7fd9f415f150 2024-09-13T08:42:34.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.525+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f4156150 con 0x7fd9f415f150 2024-09-13T08:42:34.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.525+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f4156710 con 0x7fd9f415f150 2024-09-13T08:42:34.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.526+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f415fa70 con 0x7fd9f415f150 2024-09-13T08:42:34.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.527+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fd9e8005020 con 0x7fd9f415f150 2024-09-13T08:42:34.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.528+0000 7fd9f0c00640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 0x7fd9d407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.528+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fd9e809afe0 con 0x7fd9f415f150 2024-09-13T08:42:34.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.528+0000 7fd9f3e00640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 0x7fd9d407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.529+0000 7fd9f3e00640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 0x7fd9d407ae30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0005fd0 tx=0x7fd9e0005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.532+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd9e8020020 con 0x7fd9f415f150 2024-09-13T08:42:34.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.764+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9f4074940 con 0x7fd9f415f150 2024-09-13T08:42:34.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.765+0000 7fd9f0c00640 1 -- 172.21.15.31:0/1944179841 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5225 (secure 0 0 0) 0x7fd9e8063350 con 0x7fd9f415f150 2024-09-13T08:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"btime":"2024-09-13T08:42:30:327258+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6481,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3602175668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3602175668},{"type":"v1","addr":"172.21.15.45:6835","nonce":3602175668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:34.766 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.767+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 msgr2=0x7fd9d407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.767+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 0x7fd9d407ae30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0005fd0 tx=0x7fd9e0005850 comp rx=0 tx=0).stop 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.767+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 msgr2=0x7fd9f41579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.767+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d720 tx=0x7fd9e800dbe0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 shutdown_connections 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd9d4078970 0x7fd9d407ae30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9f415ffb0 0x7fd9f40afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd9f415f530 0x7fd9f4154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 --2- 172.21.15.31:0/1944179841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9f415f150 0x7fd9f41579b0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 >> 172.21.15.31:0/1944179841 conn(0x7fd9f40d1c70 msgr2=0x7fd9f41552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:34.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 shutdown_connections 2024-09-13T08:42:34.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.768+0000 7fd9fa600640 1 -- 172.21.15.31:0/1944179841 wait complete. 2024-09-13T08:42:34.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:34.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 -- 172.21.15.31:0/2449190673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cbc50 msgr2=0x7fba240cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 --2- 172.21.15.31:0/2449190673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cbc50 0x7fba240cc060 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fba240cfe60 tx=0x7fba1c02a920 comp rx=0 tx=0).stop 2024-09-13T08:42:34.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 -- 172.21.15.31:0/2449190673 shutdown_connections 2024-09-13T08:42:34.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 --2- 172.21.15.31:0/2449190673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cbc50 0x7fba240cc060 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 -- 172.21.15.31:0/2449190673 >> 172.21.15.31:0/2449190673 conn(0x7fba240cc910 msgr2=0x7fba240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:34.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 -- 172.21.15.31:0/2449190673 shutdown_connections 2024-09-13T08:42:34.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.947+0000 7fba2a400640 1 -- 172.21.15.31:0/2449190673 wait complete. 2024-09-13T08:42:34.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.948+0000 7fba2a400640 1 Processor -- start 2024-09-13T08:42:34.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.948+0000 7fba2a400640 1 -- start start 2024-09-13T08:42:34.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba240cbc50 0x7fba241504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba2415bd70 0x7fba241f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba241600f0 con 0x7fba240cf850 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba240cead0 con 0x7fba2415bd70 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.949+0000 7fba2a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fba240c50f0 con 0x7fba240cbc50 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38518/0 (socket says 172.21.15.31:38518) 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 -- 172.21.15.31:0/3484031224 learned_addr learned my addr 172.21.15.31:0/3484031224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba23e00640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba2415bd70 0x7fba241f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba23400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba240cbc50 0x7fba241504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 -- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba240cbc50 msgr2=0x7fba241504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba240cbc50 0x7fba241504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 -- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba2415bd70 msgr2=0x7fba241f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba2415bd70 0x7fba241f85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:34.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 -- 172.21.15.31:0/3484031224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba240c4ae0 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.950+0000 7fba1aa00640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c500 tx=0x7fba0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.951+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c016020 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.951+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba0c012920 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.951+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2409be70 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.951+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c012d90 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.951+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba24150a00 con 0x7fba240cf850 2024-09-13T08:42:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.952+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba240aea40 con 0x7fba240cf850 2024-09-13T08:42:34.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.953+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fba0c019020 con 0x7fba240cf850 2024-09-13T08:42:34.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.954+0000 7fba1be00640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 0x7fba0007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:34.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.955+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fba0c09a740 con 0x7fba240cf850 2024-09-13T08:42:34.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.955+0000 7fba23400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 0x7fba0007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:34.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.955+0000 7fba23400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 0x7fba0007ae60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fba2409d4f0 tx=0x7fba14009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:34.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:34.959+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fba240aea40 con 0x7fba240cf850 2024-09-13T08:42:35.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.157+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba240cb850 con 0x7fba240cf850 2024-09-13T08:42:35.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.158+0000 7fba1be00640 1 -- 172.21.15.31:0/3484031224 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16813 (secure 0 0 0) 0x7fba0c06b030 con 0x7fba240cf850 2024-09-13T08:42:35.159 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:35.159 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:32.335290+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:10.935693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:12.097514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/3778871942":"2024-09-14T08:42:29.324924+0000","172.21.15.45:6834/3778871942":"2024-09-14T08:42:29.324924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:35.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 msgr2=0x7fba0007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 0x7fba0007ae60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fba2409d4f0 tx=0x7fba14009040 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 msgr2=0x7fba2409b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c500 tx=0x7fba0c00c9c0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 shutdown_connections 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fba000789a0 0x7fba0007ae60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba2415bd70 0x7fba241f85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba240cf850 0x7fba2409b420 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 --2- 172.21.15.31:0/3484031224 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba240cbc50 0x7fba241504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.160+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 >> 172.21.15.31:0/3484031224 conn(0x7fba240cc910 msgr2=0x7fba240ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.161+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 shutdown_connections 2024-09-13T08:42:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.161+0000 7fba2a400640 1 -- 172.21.15.31:0/3484031224 wait complete. 2024-09-13T08:42:35.177 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:42:35.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 -- 172.21.15.31:0/649421543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15ad60 msgr2=0x7fc30c15f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 --2- 172.21.15.31:0/649421543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15ad60 0x7fc30c15f3d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc300005fd0 tx=0x7fc30002a4a0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 -- 172.21.15.31:0/649421543 shutdown_connections 2024-09-13T08:42:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 --2- 172.21.15.31:0/649421543 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c06f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 --2- 172.21.15.31:0/649421543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15ad60 0x7fc30c15f3d0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 -- 172.21.15.31:0/649421543 >> 172.21.15.31:0/649421543 conn(0x7fc30c0cc910 msgr2=0x7fc30c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 -- 172.21.15.31:0/649421543 shutdown_connections 2024-09-13T08:42:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.337+0000 7fc314800640 1 -- 172.21.15.31:0/649421543 wait complete. 2024-09-13T08:42:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.339+0000 7fc314800640 1 Processor -- start 2024-09-13T08:42:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.339+0000 7fc314800640 1 -- start start 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 0x7fc30c1f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c09d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c06fe70 con 0x7fc30c15b140 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c06fcf0 con 0x7fc30c15ad60 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc314800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c0ca4e0 con 0x7fc30c15b5b0 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38526/0 (socket says 172.21.15.31:38526) 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 -- 172.21.15.31:0/4235041033 learned_addr learned my addr 172.21.15.31:0/4235041033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc312800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c09d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311e00640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 0x7fc30c1f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 -- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 msgr2=0x7fc30c09d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c09d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 -- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 msgr2=0x7fc30c1f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 0x7fc30c1f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311400640 1 -- 172.21.15.31:0/4235041033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c1f8790 con 0x7fc30c15b140 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc311e00640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 0x7fc30c1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.340+0000 7fc312800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c09d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc311400640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c430 tx=0x7fc2fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc011790 con 0x7fc30c15b140 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2fc017070 con 0x7fc30c15b140 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c1f8a70 con 0x7fc30c15b140 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc011d00 con 0x7fc30c15b140 2024-09-13T08:42:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.341+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1f4d20 con 0x7fc30c15b140 2024-09-13T08:42:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.342+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c1f8f90 con 0x7fc30c15b140 2024-09-13T08:42:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.343+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fc2fc01c460 con 0x7fc30c15b140 2024-09-13T08:42:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.344+0000 7fc306000640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 0x7fc2f007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.345+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fc2fc09ab60 con 0x7fc30c15b140 2024-09-13T08:42:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.345+0000 7fc311e00640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 0x7fc2f007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.345+0000 7fc311e00640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 0x7fc2f007ae30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc300033000 tx=0x7fc300004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:35.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.349+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc30c1f8f90 con 0x7fc30c15b140 2024-09-13T08:42:35.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.582+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc30c1f4b10 con 0x7fc30c15b140 2024-09-13T08:42:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.583+0000 7fc306000640 1 -- 172.21.15.31:0/4235041033 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5225 (secure 0 0 0) 0x7fc2fc062eb0 con 0x7fc30c15b140 2024-09-13T08:42:35.584 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:35.584 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"btime":"2024-09-13T08:42:30:327258+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6481,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3602175668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3602175668},{"type":"v1","addr":"172.21.15.45:6835","nonce":3602175668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:35.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 msgr2=0x7fc2f007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 0x7fc2f007ae30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc300033000 tx=0x7fc300004730 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 msgr2=0x7fc30c1585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c430 tx=0x7fc2fc00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 shutdown_connections 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2f0078970 0x7fc2f007ae30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc30c15b5b0 0x7fc30c09d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30c15b140 0x7fc30c1585d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 --2- 172.21.15.31:0/4235041033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30c15ad60 0x7fc30c1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 >> 172.21.15.31:0/4235041033 conn(0x7fc30c0cc910 msgr2=0x7fc30c15e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 shutdown_connections 2024-09-13T08:42:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.586+0000 7fc314800640 1 -- 172.21.15.31:0/4235041033 wait complete. 2024-09-13T08:42:35.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 -- 172.21.15.31:0/767979455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40d42e0 msgr2=0x7fe5e40b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 --2- 172.21.15.31:0/767979455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40d42e0 0x7fe5e40b0f80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc005fd0 tx=0x7fe5dc02aa90 comp rx=0 tx=0).stop 2024-09-13T08:42:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 -- 172.21.15.31:0/767979455 shutdown_connections 2024-09-13T08:42:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 --2- 172.21.15.31:0/767979455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40d42e0 0x7fe5e40b0f80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 --2- 172.21.15.31:0/767979455 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5e415dfa0 0x7fe5e40d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 -- 172.21.15.31:0/767979455 >> 172.21.15.31:0/767979455 conn(0x7fe5e408d8b0 msgr2=0x7fe5e409d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 -- 172.21.15.31:0/767979455 shutdown_connections 2024-09-13T08:42:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.763+0000 7fe5ea400640 1 -- 172.21.15.31:0/767979455 wait complete. 2024-09-13T08:42:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.765+0000 7fe5ea400640 1 Processor -- start 2024-09-13T08:42:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.765+0000 7fe5ea400640 1 -- start start 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5e40d42e0 0x7fe5e4160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e415dfa0 0x7fe5e409de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e40b1910 con 0x7fe5e40b47f0 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e40b1790 con 0x7fe5e415dfa0 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5ea400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe5e40b1a90 con 0x7fe5e40d42e0 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38532/0 (socket says 172.21.15.31:38532) 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 -- 172.21.15.31:0/1737505 learned_addr learned my addr 172.21.15.31:0/1737505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 -- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5e40d42e0 msgr2=0x7fe5e4160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5e40d42e0 0x7fe5e4160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 -- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e415dfa0 msgr2=0x7fe5e409de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e415dfa0 0x7fe5e409de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.766+0000 7fe5e9000640 1 -- 172.21.15.31:0/1737505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e41f4f10 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5e9000640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e790 tx=0x7fe5d400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400eea0 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5d400cdb0 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e41f51f0 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4010690 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.767+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e41f57b0 con 0x7fe5e40b47f0 2024-09-13T08:42:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.768+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e41f55e0 con 0x7fe5e40b47f0 2024-09-13T08:42:35.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.769+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe5d4010830 con 0x7fe5e40b47f0 2024-09-13T08:42:35.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.770+0000 7fe5e1600640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 0x7fe5c007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:35.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.770+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fe5d409b690 con 0x7fe5e40b47f0 2024-09-13T08:42:35.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.770+0000 7fe5dbe00640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 0x7fe5c007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:35.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.771+0000 7fe5dbe00640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 0x7fe5c007ae30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc000f60 tx=0x7fe5cc0079b0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:35.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:35.774+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe5d40639e0 con 0x7fe5e40b47f0 2024-09-13T08:42:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.006+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5e40b1f40 con 0x7fe5e40b47f0 2024-09-13T08:42:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.007+0000 7fe5e1600640 1 -- 172.21.15.31:0/1737505 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5225 (secure 0 0 0) 0x7fe5d40632e0 con 0x7fe5e40b47f0 2024-09-13T08:42:36.008 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:36.008 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"btime":"2024-09-13T08:42:30:327258+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6481,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3602175668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3602175668},{"type":"v1","addr":"172.21.15.45:6835","nonce":3602175668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:13.130944+0000","modified":"2024-09-13T08:42:16.161289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6585},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6585":{"gid":6585,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6834/2420402248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2420402248},{"type":"v1","addr":"172.21.15.31:6834","nonce":2420402248}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6585,"qdb_cluster":[6585]},"id":6}]} 2024-09-13T08:42:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 62 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.009+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 msgr2=0x7fe5c007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.009+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 0x7fe5c007ae30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc000f60 tx=0x7fe5cc0079b0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.009+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 msgr2=0x7fe5e41f48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e790 tx=0x7fe5d400ec50 comp rx=0 tx=0).stop 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 shutdown_connections 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe5c0078970 0x7fe5c007ae30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e415dfa0 0x7fe5e409de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5e40d42e0 0x7fe5e4160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 --2- 172.21.15.31:0/1737505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e40b47f0 0x7fe5e41f48c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 >> 172.21.15.31:0/1737505 conn(0x7fe5e408d8b0 msgr2=0x7fe5e40b8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 shutdown_connections 2024-09-13T08:42:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.010+0000 7fe5ea400640 1 -- 172.21.15.31:0/1737505 wait complete. 2024-09-13T08:42:36.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:36.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 -- 172.21.15.31:0/614442779 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880b1910 msgr2=0x7f4e880b1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/614442779 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880b1910 0x7f4e880b1d60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4e80005fd0 tx=0x7f4e8002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 -- 172.21.15.31:0/614442779 shutdown_connections 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/614442779 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880b1910 0x7f4e880b1d60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/614442779 >> v2:172.21.15.45:3300/0 conn(0x7f4e880ca5c0 0x7f4e880cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 -- 172.21.15.31:0/614442779 >> 172.21.15.31:0/614442779 conn(0x7f4e880cc910 msgr2=0x7f4e880cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 -- 172.21.15.31:0/614442779 shutdown_connections 2024-09-13T08:42:36.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.187+0000 7f4e8ea00640 1 -- 172.21.15.31:0/614442779 wait complete. 2024-09-13T08:42:36.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.189+0000 7f4e8ea00640 1 Processor -- start 2024-09-13T08:42:36.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.189+0000 7f4e8ea00640 1 -- start start 2024-09-13T08:42:36.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880ca5c0 0x7f4e881e7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e8815dfc0 0x7f4e880afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88073b10 con 0x7f4e880b1910 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88073990 con 0x7f4e8815dfc0 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4e88073c90 con 0x7f4e880ca5c0 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38546/0 (socket says 172.21.15.31:38546) 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e87e00640 1 -- 172.21.15.31:0/2568946002 learned_addr learned my addr 172.21.15.31:0/2568946002 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e7fe00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880ca5c0 0x7f4e881e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.190+0000 7f4e8ca00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e8815dfc0 0x7f4e880afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 -- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880ca5c0 msgr2=0x7f4e881e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880ca5c0 0x7f4e881e7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 -- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e8815dfc0 msgr2=0x7f4e880afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e8815dfc0 0x7f4e880afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 -- 172.21.15.31:0/2568946002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e881e7d90 con 0x7f4e880b1910 2024-09-13T08:42:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e87e00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c760 tx=0x7f4e7800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78016020 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e7800a950 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7800ada0 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e881e8070 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.191+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e881e8630 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.192+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e880b1d60 con 0x7f4e880b1910 2024-09-13T08:42:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.193+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f4e78019020 con 0x7f4e880b1910 2024-09-13T08:42:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.194+0000 7f4e84c00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 0x7f4e6407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.194+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f4e7809ab30 con 0x7f4e880b1910 2024-09-13T08:42:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.195+0000 7f4e7fe00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 0x7f4e6407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.195+0000 7f4e7fe00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 0x7f4e6407ae30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809d0c0 tx=0x7f4e70005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.198+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4e78020020 con 0x7f4e880b1910 2024-09-13T08:42:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.399+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e880af5d0 con 0x7f4e880b1910 2024-09-13T08:42:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.400+0000 7f4e84c00640 1 -- 172.21.15.31:0/2568946002 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16813 (secure 0 0 0) 0x7f4e880cbe40 con 0x7f4e880b1910 2024-09-13T08:42:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:36.401 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:32.335290+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:10.935693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:12.097514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/3778871942":"2024-09-14T08:42:29.324924+0000","172.21.15.45:6834/3778871942":"2024-09-14T08:42:29.324924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 msgr2=0x7f4e6407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 0x7f4e6407ae30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809d0c0 tx=0x7f4e70005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 msgr2=0x7f4e8815d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c760 tx=0x7f4e7800cc20 comp rx=0 tx=0).stop 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 shutdown_connections 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4e64078970 0x7f4e6407ae30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e8815dfc0 0x7f4e880afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4e880ca5c0 0x7f4e881e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 --2- 172.21.15.31:0/2568946002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e880b1910 0x7f4e8815d500 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 >> 172.21.15.31:0/2568946002 conn(0x7f4e880cc910 msgr2=0x7f4e880ad400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 shutdown_connections 2024-09-13T08:42:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.403+0000 7f4e8ea00640 1 -- 172.21.15.31:0/2568946002 wait complete. 2024-09-13T08:42:36.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:42:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 -- 172.21.15.31:0/1646393516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 msgr2=0x7f0dc80d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1646393516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80d1110 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0dc806d3f0 tx=0x7f0dc002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 -- 172.21.15.31:0/1646393516 shutdown_connections 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1646393516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80d1110 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1646393516 >> v2:172.21.15.45:3300/0 conn(0x7f0dc815e160 0x7f0dc80d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 -- 172.21.15.31:0/1646393516 >> 172.21.15.31:0/1646393516 conn(0x7f0dc80d3f50 msgr2=0x7f0dc80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 -- 172.21.15.31:0/1646393516 shutdown_connections 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.581+0000 7f0dcf000640 1 -- 172.21.15.31:0/1646393516 wait complete. 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.583+0000 7f0dcf000640 1 Processor -- start 2024-09-13T08:42:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.583+0000 7f0dcf000640 1 -- start start 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 0x7f0dc815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 0x7f0dc815abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80c8bd0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80c8a50 con 0x7f0dc815e160 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcf000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc80c8d50 con 0x7f0dc80d0cd0 2024-09-13T08:42:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38570/0 (socket says 172.21.15.31:38570) 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 -- 172.21.15.31:0/1681294507 learned_addr learned my addr 172.21.15.31:0/1681294507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dce600640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 0x7f0dc815abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 -- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 msgr2=0x7f0dc815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcd200640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 0x7f0dc815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 0x7f0dc815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 -- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 msgr2=0x7f0dc815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 0x7f0dc815abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcdc00640 1 -- 172.21.15.31:0/1681294507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc8158cc0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dce600640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 0x7f0dc815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.584+0000 7f0dcd200640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 0x7f0dc815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dcdc00640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e790 tx=0x7f0dbc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00eea0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dbc00cdb0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc010690 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc8158fa0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.585+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc80cd2e0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.586+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc80d1110 con 0x7f0dc80cfa70 2024-09-13T08:42:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.587+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f0dbc010830 con 0x7f0dc80cfa70 2024-09-13T08:42:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.588+0000 7f0dc5600640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 0x7f0da807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.589+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f0dbc09b5f0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.589+0000 7f0dcd200640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 0x7f0da807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.589+0000 7f0dcd200640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 0x7f0da807ae30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0db40013f0 tx=0x7f0db4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.593+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0dbc0639e0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:42:36.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:42:36.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.831+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0dc806d810 con 0x7f0dc80cfa70 2024-09-13T08:42:36.832 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:42:36.832+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:42:36.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.834+0000 7f0dc5600640 1 -- 172.21.15.31:0/1681294507 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) ==== 161+0+0 (secure 0 0 0) 0x7f0dbc0632e0 con 0x7f0dc80cfa70 2024-09-13T08:42:36.835 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:42:36.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 msgr2=0x7f0da807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 0x7f0da807ae30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0db40013f0 tx=0x7f0db4009040 comp rx=0 tx=0).stop 2024-09-13T08:42:36.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 msgr2=0x7f0dc80cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:36.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e790 tx=0x7f0dbc00ec50 comp rx=0 tx=0).stop 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 shutdown_connections 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078970 0x7f0da807ae30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dc815e160 0x7f0dc815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dc80d0cd0 0x7f0dc815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 --2- 172.21.15.31:0/1681294507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dc80cfa70 0x7f0dc80cd9b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.837+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 >> 172.21.15.31:0/1681294507 conn(0x7f0dc80d3f50 msgr2=0x7f0dc806e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.838+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 shutdown_connections 2024-09-13T08:42:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:36.838+0000 7f0dcf000640 1 -- 172.21.15.31:0/1681294507 wait complete. 2024-09-13T08:42:36.844 INFO:tasks.ceph.mds.c.smithi031.stderr: -2688> 2024-09-13T08:42:17.880+0000 7f0456000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:42:36.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:42:36.891 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 -- 172.21.15.31:0/1932606660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71200cde50 msgr2=0x7f71200ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 --2- 172.21.15.31:0/1932606660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71200cde50 0x7f71200ca1c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7108009860 tx=0x7f710802a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 -- 172.21.15.31:0/1932606660 shutdown_connections 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 --2- 172.21.15.31:0/1932606660 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f71200cde50 0x7f71200ca1c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 -- 172.21.15.31:0/1932606660 >> 172.21.15.31:0/1932606660 conn(0x7f71200cfb70 msgr2=0x7f71200cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 -- 172.21.15.31:0/1932606660 shutdown_connections 2024-09-13T08:42:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.022+0000 7f7125c00640 1 -- 172.21.15.31:0/1932606660 wait complete. 2024-09-13T08:42:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.023+0000 7f7125c00640 1 Processor -- start 2024-09-13T08:42:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 -- start start 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71200cde50 0x7f712015fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 0x7f712015f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71200cb9d0 con 0x7f71200cc0e0 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71200cb850 con 0x7f71200cde50 2024-09-13T08:42:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.024+0000 7f7125c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f712015e4a0 con 0x7f712015df30 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38594/0 (socket says 172.21.15.31:38594) 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 -- 172.21.15.31:0/2989554486 learned_addr learned my addr 172.21.15.31:0/2989554486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f7116000640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71200cde50 0x7f712015fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711f400640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 0x7f712015f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 -- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 msgr2=0x7f712015f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 0x7f712015f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 -- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71200cde50 msgr2=0x7f712015fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:37.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71200cde50 0x7f712015fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711ea00640 1 -- 172.21.15.31:0/2989554486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71201608d0 con 0x7f71200cc0e0 2024-09-13T08:42:37.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.025+0000 7f711f400640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 0x7f712015f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:37.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f711ea00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f711000c410 tx=0x7f711000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110017070 con 0x7f71200cc0e0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71100035f0 con 0x7f71200cc0e0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110003a40 con 0x7f71200cc0e0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712015e840 con 0x7f71200cc0e0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.026+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f712015ee00 con 0x7f71200cc0e0 2024-09-13T08:42:37.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.027+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71200cb130 con 0x7f71200cc0e0 2024-09-13T08:42:37.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.028+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f711001a070 con 0x7f71200cc0e0 2024-09-13T08:42:37.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.030+0000 7f7117400640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 0x7f70fc07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:37.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.030+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f7110099e80 con 0x7f71200cc0e0 2024-09-13T08:42:37.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.030+0000 7f7116000640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 0x7f70fc07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:37.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.031+0000 7f7116000640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 0x7f70fc07ae30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7108034000 tx=0x7f7108004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:37.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.034+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f71100622b0 con 0x7f71200cc0e0 2024-09-13T08:42:37.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.276+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f71200b5af0 con 0x7f71200cc0e0 2024-09-13T08:42:37.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.840+0000 7f7117400640 1 -- 172.21.15.31:0/2989554486 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) ==== 104+0+0 (secure 0 0 0) 0x7f7110061bb0 con 0x7f71200cc0e0 2024-09-13T08:42:37.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 msgr2=0x7f70fc07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:37.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 0x7f70fc07ae30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7108034000 tx=0x7f7108004730 comp rx=0 tx=0).stop 2024-09-13T08:42:37.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 msgr2=0x7f712014eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:37.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f711000c410 tx=0x7f711000c8d0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 shutdown_connections 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70fc078970 0x7f70fc07ae30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f712015df30 0x7f712015f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71200cde50 0x7f712015fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 --2- 172.21.15.31:0/2989554486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71200cc0e0 0x7f712014eed0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 >> 172.21.15.31:0/2989554486 conn(0x7f71200cfb70 msgr2=0x7f7120073bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 shutdown_connections 2024-09-13T08:42:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:37.843+0000 7f7125c00640 1 -- 172.21.15.31:0/2989554486 wait complete. 2024-09-13T08:42:37.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:42:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 -- 172.21.15.31:0/1091142507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac072560 msgr2=0x7f84ac0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1091142507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac072560 0x7f84ac0cb740 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f84ac06e410 tx=0x7f849402a530 comp rx=0 tx=0).stop 2024-09-13T08:42:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 -- 172.21.15.31:0/1091142507 shutdown_connections 2024-09-13T08:42:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1091142507 >> v2:172.21.15.45:3300/0 conn(0x7f84ac0cbc80 0x7f84ac0b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1091142507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac072560 0x7f84ac0cb740 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 -- 172.21.15.31:0/1091142507 >> 172.21.15.31:0/1091142507 conn(0x7f84ac0cfbb0 msgr2=0x7f84ac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 -- 172.21.15.31:0/1091142507 shutdown_connections 2024-09-13T08:42:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.067+0000 7f84b1c00640 1 -- 172.21.15.31:0/1091142507 wait complete. 2024-09-13T08:42:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.069+0000 7f84b1c00640 1 Processor -- start 2024-09-13T08:42:38.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.069+0000 7f84b1c00640 1 -- start start 2024-09-13T08:42:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84ac072560 0x7f84ac152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84ac0cbc80 0x7f84ac1e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:38.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac06e740 con 0x7f84ac15bfc0 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac06e5c0 con 0x7f84ac0cbc80 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84b1c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f84ac0b2f20 con 0x7f84ac072560 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38614/0 (socket says 172.21.15.31:38614) 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 -- 172.21.15.31:0/1684496249 learned_addr learned my addr 172.21.15.31:0/1684496249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84a2000640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84ac0cbc80 0x7f84ac1e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84aaa00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84ac072560 0x7f84ac152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 -- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84ac072560 msgr2=0x7f84ac152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84ac072560 0x7f84ac152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 -- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84ac0cbc80 msgr2=0x7f84ac1e5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84ac0cbc80 0x7f84ac1e5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.070+0000 7f84ab400640 1 -- 172.21.15.31:0/1684496249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ac1f63a0 con 0x7f84ac15bfc0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84ab400640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b970 tx=0x7f84a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4019070 con 0x7f84ac15bfc0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84a400cc50 con 0x7f84ac15bfc0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ac1f6680 con 0x7f84ac15bfc0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40146a0 con 0x7f84ac15bfc0 2024-09-13T08:42:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.071+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ac1f6c40 con 0x7f84ac15bfc0 2024-09-13T08:42:38.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.072+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ac0b3340 con 0x7f84ac15bfc0 2024-09-13T08:42:38.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.073+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f84a401c070 con 0x7f84ac15bfc0 2024-09-13T08:42:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.074+0000 7f84a3400640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 0x7f848807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.074+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f84a409aa60 con 0x7f84ac15bfc0 2024-09-13T08:42:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.074+0000 7f84aaa00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 0x7f848807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:38.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.075+0000 7f84aaa00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 0x7f848807ae30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f849c007fd0 tx=0x7f849c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:38.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.078+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f84a4062d60 con 0x7f84ac15bfc0 2024-09-13T08:42:38.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.276+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f84ac073f60 con 0x7f84ac15bfc0 2024-09-13T08:42:38.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.842+0000 7f84a3400640 1 -- 172.21.15.31:0/1684496249 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) ==== 181+0+0 (secure 0 0 0) 0x7f84a4062660 con 0x7f84ac15bfc0 2024-09-13T08:42:38.843 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.845+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 msgr2=0x7f848807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.845+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 0x7f848807ae30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f849c007fd0 tx=0x7f849c009040 comp rx=0 tx=0).stop 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.845+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 msgr2=0x7f84ac1e6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.845+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b970 tx=0x7f84a400be30 comp rx=0 tx=0).stop 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 shutdown_connections 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8488078970 0x7f848807ae30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84ac15bfc0 0x7f84ac1e6380 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84ac0cbc80 0x7f84ac1e5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 --2- 172.21.15.31:0/1684496249 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84ac072560 0x7f84ac152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:38.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 >> 172.21.15.31:0/1684496249 conn(0x7f84ac0cfbb0 msgr2=0x7f84ac0ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:38.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 shutdown_connections 2024-09-13T08:42:38.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:38.846+0000 7f84b1c00640 1 -- 172.21.15.31:0/1684496249 wait complete. 2024-09-13T08:42:38.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.037+0000 7f4602400640 1 -- 172.21.15.31:0/981242503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0cf550 msgr2=0x7f45fc0cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.037+0000 7f4602400640 1 --2- 172.21.15.31:0/981242503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0cf550 0x7f45fc0cf990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009830 tx=0x7f45e402a510 comp rx=0 tx=0).stop 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.038+0000 7f4602400640 1 -- 172.21.15.31:0/981242503 shutdown_connections 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.038+0000 7f4602400640 1 --2- 172.21.15.31:0/981242503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0cf550 0x7f45fc0cf990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.038+0000 7f4602400640 1 -- 172.21.15.31:0/981242503 >> 172.21.15.31:0/981242503 conn(0x7f45fc0d0440 msgr2=0x7f45fc0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.038+0000 7f4602400640 1 -- 172.21.15.31:0/981242503 shutdown_connections 2024-09-13T08:42:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.038+0000 7f4602400640 1 -- 172.21.15.31:0/981242503 wait complete. 2024-09-13T08:42:39.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.039+0000 7f4602400640 1 Processor -- start 2024-09-13T08:42:39.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.039+0000 7f4602400640 1 -- start start 2024-09-13T08:42:39.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45fc0cf550 0x7f45fc1f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f45fc15e210 0x7f45fc0c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc0cd7f0 con 0x7f45fc0b3290 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc0cd670 con 0x7f45fc0cf550 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.040+0000 7f4602400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f45fc0b3e30 con 0x7f45fc15e210 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38638/0 (socket says 172.21.15.31:38638) 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 -- 172.21.15.31:0/3356954188 learned_addr learned my addr 172.21.15.31:0/3356954188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45f2a00640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45fc0cf550 0x7f45fc1f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fbe00640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f45fc15e210 0x7f45fc0c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 -- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f45fc15e210 msgr2=0x7f45fc0c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f45fc15e210 0x7f45fc0c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 -- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45fc0cf550 msgr2=0x7f45fc1f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45fc0cf550 0x7f45fc1f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.041+0000 7f45fb400640 1 -- 172.21.15.31:0/3356954188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45fc0c9810 con 0x7f45fc0b3290 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f45fb400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c3f0 tx=0x7f45ec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec017070 con 0x7f45fc0b3290 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45ec0035f0 con 0x7f45fc0b3290 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec003a40 con 0x7f45fc0b3290 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45fc0c9af0 con 0x7f45fc0b3290 2024-09-13T08:42:39.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.042+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45fc0ca0b0 con 0x7f45fc0b3290 2024-09-13T08:42:39.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.043+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45fc0cf990 con 0x7f45fc0b3290 2024-09-13T08:42:39.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.044+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f45ec01a070 con 0x7f45fc0b3290 2024-09-13T08:42:39.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.045+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 0x7f45d807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:39.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.045+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5443+0+0 (secure 0 0 0) 0x7f45ec099db0 con 0x7f45fc0b3290 2024-09-13T08:42:39.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.045+0000 7f45f2a00640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 0x7f45d807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:39.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.046+0000 7f45f2a00640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 0x7f45d807ae30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45e4034000 tx=0x7f45e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:39.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.049+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f45ec09e070 con 0x7f45fc0b3290 2024-09-13T08:42:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.251+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f45fc06f250 con 0x7f45fc0b3290 2024-09-13T08:42:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.849+0000 7f45f3e00640 1 -- 172.21.15.31:0/3356954188 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) ==== 169+0+0 (secure 0 0 0) 0x7f45ec061c10 con 0x7f45fc0b3290 2024-09-13T08:42:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:42:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 msgr2=0x7f45d807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 0x7f45d807ae30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45e4034000 tx=0x7f45e4004730 comp rx=0 tx=0).stop 2024-09-13T08:42:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 msgr2=0x7f45fc099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c3f0 tx=0x7f45ec00c8b0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 shutdown_connections 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f45d8078970 0x7f45d807ae30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f45fc15e210 0x7f45fc0c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45fc0cf550 0x7f45fc1f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 --2- 172.21.15.31:0/3356954188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45fc0b3290 0x7f45fc099af0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.852+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 >> 172.21.15.31:0/3356954188 conn(0x7f45fc0d0440 msgr2=0x7f45fc06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.853+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 shutdown_connections 2024-09-13T08:42:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:39.853+0000 7f4602400640 1 -- 172.21.15.31:0/3356954188 wait complete. 2024-09-13T08:42:39.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:42:40.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 -- 172.21.15.31:0/1707476006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 msgr2=0x7fab9c0ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 --2- 172.21.15.31:0/1707476006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c0ce0c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fab9c06e610 tx=0x7fab8c02a540 comp rx=0 tx=0).stop 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 -- 172.21.15.31:0/1707476006 shutdown_connections 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 --2- 172.21.15.31:0/1707476006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c0ce0c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 --2- 172.21.15.31:0/1707476006 >> v2:172.21.15.45:3300/0 conn(0x7fab9c15be30 0x7fab9c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 -- 172.21.15.31:0/1707476006 >> 172.21.15.31:0/1707476006 conn(0x7fab9c0b18f0 msgr2=0x7fab9c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 -- 172.21.15.31:0/1707476006 shutdown_connections 2024-09-13T08:42:40.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.035+0000 7faba4800640 1 -- 172.21.15.31:0/1707476006 wait complete. 2024-09-13T08:42:40.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.037+0000 7faba4800640 1 Processor -- start 2024-09-13T08:42:40.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.037+0000 7faba4800640 1 -- start start 2024-09-13T08:42:40.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 0x7fab9c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 0x7fab9c1561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0749f0 con 0x7fab9c0ced20 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c074870 con 0x7fab9c0ce600 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba4800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c160160 con 0x7fab9c15be30 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38650/0 (socket says 172.21.15.31:38650) 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 -- 172.21.15.31:0/2309498361 learned_addr learned my addr 172.21.15.31:0/2309498361 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 -- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 msgr2=0x7fab9c1561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba2800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 0x7fab9c1561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1e00640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 0x7fab9c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 0x7fab9c1561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 -- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 msgr2=0x7fab9c150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.038+0000 7faba1400640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 0x7fab9c150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba1400640 1 -- 172.21.15.31:0/2309498361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c1f92c0 con 0x7fab9c0ced20 2024-09-13T08:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba1e00640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 0x7fab9c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba2800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 0x7fab9c1561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:40.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba1400640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009920 tx=0x7fab8c032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c040070 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab8c032e70 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c0055b0 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c153900 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.039+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c153ec0 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.040+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c1605c0 con 0x7fab9c0ced20 2024-09-13T08:42:40.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.042+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fab8c039040 con 0x7fab9c0ced20 2024-09-13T08:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.043+0000 7fab96000640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 0x7fab8007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.043+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fab8c0bba90 con 0x7fab9c0ced20 2024-09-13T08:42:40.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.043+0000 7faba1e00640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 0x7fab8007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.044+0000 7faba1e00640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 0x7fab8007ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab90009aa0 tx=0x7fab90005db0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:40.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.048+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab9c1f92c0 con 0x7fab9c0ced20 2024-09-13T08:42:40.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.240+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fab9c1598a0 con 0x7fab9c0ced20 2024-09-13T08:42:40.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.244+0000 7fab96000640 1 -- 172.21.15.31:0/2309498361 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v37) ==== 71+0+0 (secure 0 0 0) 0x7fab8c084170 con 0x7fab9c0ced20 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 msgr2=0x7fab8007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 0x7fab8007ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab90009aa0 tx=0x7fab90005db0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 msgr2=0x7fab9c1f8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009920 tx=0x7fab8c032c80 comp rx=0 tx=0).stop 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 shutdown_connections 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab80078970 0x7fab8007ae30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab9c15be30 0x7fab9c1561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c0ced20 0x7fab9c1f8d80 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 --2- 172.21.15.31:0/2309498361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab9c0ce600 0x7fab9c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 >> 172.21.15.31:0/2309498361 conn(0x7fab9c0b18f0 msgr2=0x7fab9c1582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 shutdown_connections 2024-09-13T08:42:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.247+0000 7faba4800640 1 -- 172.21.15.31:0/2309498361 wait complete. 2024-09-13T08:42:40.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.422+0000 7fb45be00640 1 -- 172.21.15.31:0/834287595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e820 msgr2=0x7fb4540b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.422+0000 7fb45be00640 1 --2- 172.21.15.31:0/834287595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e820 0x7fb4540b1b30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb440005fd0 tx=0x7fb44002a540 comp rx=0 tx=0).stop 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 -- 172.21.15.31:0/834287595 shutdown_connections 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 --2- 172.21.15.31:0/834287595 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 0x7fb4540ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 --2- 172.21.15.31:0/834287595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e820 0x7fb4540b1b30 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 -- 172.21.15.31:0/834287595 >> 172.21.15.31:0/834287595 conn(0x7fb454074260 msgr2=0x7fb454074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 -- 172.21.15.31:0/834287595 shutdown_connections 2024-09-13T08:42:40.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.423+0000 7fb45be00640 1 -- 172.21.15.31:0/834287595 wait complete. 2024-09-13T08:42:40.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.424+0000 7fb45be00640 1 Processor -- start 2024-09-13T08:42:40.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.424+0000 7fb45be00640 1 -- start start 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 0x7fb4540c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 0x7fb4540c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4540ae4c0 con 0x7fb45415e440 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4540ae340 con 0x7fb45415e820 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb45be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb4540ae640 con 0x7fb4540b1600 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.425+0000 7fb458a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38678/0 (socket says 172.21.15.31:38678) 2024-09-13T08:42:40.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 -- 172.21.15.31:0/1099691484 learned_addr learned my addr 172.21.15.31:0/1099691484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb459e00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 0x7fb4540c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb459400640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 0x7fb4540c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 -- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 msgr2=0x7fb4540c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 0x7fb4540c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 -- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 msgr2=0x7fb4540c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 0x7fb4540c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 -- 172.21.15.31:0/1099691484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4540c8b80 con 0x7fb45415e440 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb459e00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 0x7fb4540c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.426+0000 7fb458a00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb440006010 tx=0x7fb4400038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.427+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44003e030 con 0x7fb45415e440 2024-09-13T08:42:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.427+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb440003d90 con 0x7fb45415e440 2024-09-13T08:42:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.427+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440035040 con 0x7fb45415e440 2024-09-13T08:42:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.427+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4540c8e60 con 0x7fb45415e440 2024-09-13T08:42:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.427+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45409dd80 con 0x7fb45415e440 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.428+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb454050fa0 con 0x7fb45415e440 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.429+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fb440041030 con 0x7fb45415e440 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.430+0000 7fb451600640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 0x7fb43407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.430+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fb4400baf30 con 0x7fb45415e440 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.430+0000 7fb459400640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 0x7fb43407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.431+0000 7fb459400640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 0x7fb43407ae30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb448001680 tx=0x7fb448009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.434+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb454050fa0 con 0x7fb45415e440 2024-09-13T08:42:40.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.709+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7fb4540af3f0 con 0x7fb45415e440 2024-09-13T08:42:40.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.851+0000 7fb451600640 1 -- 172.21.15.31:0/1099691484 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v304) ==== 146+0+0 (secure 0 0 0) 0x7fb4540af3f0 con 0x7fb45415e440 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 msgr2=0x7fb43407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 0x7fb43407ae30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb448001680 tx=0x7fb448009040 comp rx=0 tx=0).stop 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 msgr2=0x7fb45415b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb440006010 tx=0x7fb4400038a0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 shutdown_connections 2024-09-13T08:42:40.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb434078970 0x7fb43407ae30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45415e820 0x7fb4540c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb45415e440 0x7fb45415b610 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 --2- 172.21.15.31:0/1099691484 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4540b1600 0x7fb4540c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 >> 172.21.15.31:0/1099691484 conn(0x7fb454074260 msgr2=0x7fb45406ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 shutdown_connections 2024-09-13T08:42:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:40.854+0000 7fb45be00640 1 -- 172.21.15.31:0/1099691484 wait complete. 2024-09-13T08:42:40.871 INFO:tasks.cephfs_test_runner:Test the creating vols, subvols subvolgroups fails when their names uses ... ok 2024-09-13T08:42:40.871 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-13T08:42:40.871 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:42:40.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 -- 172.21.15.31:0/3855610976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584406d210 msgr2=0x7f58440ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 --2- 172.21.15.31:0/3855610976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584406d210 0x7f58440ceea0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f583c005fd0 tx=0x7f583c02a940 comp rx=0 tx=0).stop 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 -- 172.21.15.31:0/3855610976 shutdown_connections 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 --2- 172.21.15.31:0/3855610976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584406d210 0x7f58440ceea0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 -- 172.21.15.31:0/3855610976 >> 172.21.15.31:0/3855610976 conn(0x7f58440cda20 msgr2=0x7f58440cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 -- 172.21.15.31:0/3855610976 shutdown_connections 2024-09-13T08:42:41.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.031+0000 7f584aa00640 1 -- 172.21.15.31:0/3855610976 wait complete. 2024-09-13T08:42:41.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.032+0000 7f584aa00640 1 Processor -- start 2024-09-13T08:42:41.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.033+0000 7f584aa00640 1 -- start start 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.033+0000 7f584aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 0x7f5844070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f584aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 0x7f584414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f584aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f584aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5844074020 con 0x7f584415bc00 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f584aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5844073ea0 con 0x7f58440cb5f0 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f584aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f58440741a0 con 0x7f584406d210 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38692/0 (socket says 172.21.15.31:38692) 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 -- 172.21.15.31:0/1100307394 learned_addr learned my addr 172.21.15.31:0/1100307394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:41.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5843400640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 0x7f584414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5843e00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 0x7f5844070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 -- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 msgr2=0x7f5844070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 0x7f5844070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 -- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 msgr2=0x7f584414e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 0x7f584414e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5848a00640 1 -- 172.21.15.31:0/1100307394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58440c8210 con 0x7f584415bc00 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5843e00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 0x7f5844070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.034+0000 7f5843400640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 0x7f584414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f5848a00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f583c02af30 tx=0x7f583c031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c03e070 con 0x7f584415bc00 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f583c004560 con 0x7f584415bc00 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c043460 con 0x7f584415bc00 2024-09-13T08:42:41.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58440ca860 con 0x7f584415bc00 2024-09-13T08:42:41.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.035+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58440cab20 con 0x7f584415bc00 2024-09-13T08:42:41.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.036+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5844074650 con 0x7f584415bc00 2024-09-13T08:42:41.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.037+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f583c03a440 con 0x7f584415bc00 2024-09-13T08:42:41.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.039+0000 7f583be00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 0x7f582407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.039+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f583c0bc360 con 0x7f584415bc00 2024-09-13T08:42:41.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.039+0000 7f5843e00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 0x7f582407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.040+0000 7f5843e00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 0x7f582407ae30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58340046b0 tx=0x7f5834009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.043+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f583c084a40 con 0x7f584415bc00 2024-09-13T08:42:41.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.238+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58440cacf0 con 0x7f584415bc00 2024-09-13T08:42:41.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.238+0000 7f583be00640 1 -- 172.21.15.31:0/1100307394 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+13116 (secure 0 0 0) 0x7f583c084340 con 0x7f584415bc00 2024-09-13T08:42:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:41.240 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":100,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:39.843318+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6834/2420402248":"2024-09-14T08:42:36.832722+0000","172.21.15.31:6832/2420402248":"2024-09-14T08:42:36.832722+0000","172.21.15.45:6835/3778871942":"2024-09-14T08:42:29.324924+0000","172.21.15.45:6834/3778871942":"2024-09-14T08:42:29.324924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 msgr2=0x7f582407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 0x7f582407ae30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58340046b0 tx=0x7f5834009040 comp rx=0 tx=0).stop 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 msgr2=0x7f584414ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f583c02af30 tx=0x7f583c031ea0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 shutdown_connections 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5824078970 0x7f582407ae30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f584415bc00 0x7f584414ae20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58440cb5f0 0x7f584414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 --2- 172.21.15.31:0/1100307394 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f584406d210 0x7f5844070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 >> 172.21.15.31:0/1100307394 conn(0x7f58440cda20 msgr2=0x7f58440af780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 shutdown_connections 2024-09-13T08:42:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.241+0000 7f584aa00640 1 -- 172.21.15.31:0/1100307394 wait complete. 2024-09-13T08:42:41.258 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:42:41.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:42:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.420+0000 7fed53400640 1 -- 172.21.15.31:0/1023656347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074870 msgr2=0x7fed4c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.420+0000 7fed53400640 1 --2- 172.21.15.31:0/1023656347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074870 0x7fed4c0ce550 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fed4c072b20 tx=0x7fed4402aaa0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 -- 172.21.15.31:0/1023656347 shutdown_connections 2024-09-13T08:42:41.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 --2- 172.21.15.31:0/1023656347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 0x7fed4c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 --2- 172.21.15.31:0/1023656347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074870 0x7fed4c0ce550 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 --2- 172.21.15.31:0/1023656347 >> v2:172.21.15.45:3300/0 conn(0x7fed4c074490 0x7fed4c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 -- 172.21.15.31:0/1023656347 >> 172.21.15.31:0/1023656347 conn(0x7fed4c0d0440 msgr2=0x7fed4c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 -- 172.21.15.31:0/1023656347 shutdown_connections 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.421+0000 7fed53400640 1 -- 172.21.15.31:0/1023656347 wait complete. 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.422+0000 7fed53400640 1 Processor -- start 2024-09-13T08:42:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.422+0000 7fed53400640 1 -- start start 2024-09-13T08:42:41.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 0x7fed4c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 0x7fed4c1f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c15f240 con 0x7fed4c074490 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c15f0c0 con 0x7fed4c074870 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.423+0000 7fed53400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fed4c15f3c0 con 0x7fed4c0b1a40 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38712/0 (socket says 172.21.15.31:38712) 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 -- 172.21.15.31:0/3441747305 learned_addr learned my addr 172.21.15.31:0/3441747305 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed4be00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 0x7fed4c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed51400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 0x7fed4c1f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 -- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 msgr2=0x7fed4c1f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 0x7fed4c1f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 -- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 msgr2=0x7fed4c0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 0x7fed4c0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 -- 172.21.15.31:0/3441747305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c0c4dc0 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed4be00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 0x7fed4c0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed50a00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fed4000e9f0 tx=0x7fed4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.424+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4000ccc0 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.425+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed40010430 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.425+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40010840 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.425+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c15dc50 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.425+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c15e210 con 0x7fed4c074490 2024-09-13T08:42:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.426+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4c0b1500 con 0x7fed4c074490 2024-09-13T08:42:41.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.427+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fed400109e0 con 0x7fed4c074490 2024-09-13T08:42:41.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.428+0000 7fed48c00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 0x7fed2c07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.428+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fed4009b400 con 0x7fed4c074490 2024-09-13T08:42:41.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.428+0000 7fed4be00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 0x7fed2c07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.429+0000 7fed4be00640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 0x7fed2c07ae60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed4c166060 tx=0x7fed38005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.432+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fed40063ae0 con 0x7fed4c074490 2024-09-13T08:42:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.628+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed4c099e60 con 0x7fed4c074490 2024-09-13T08:42:41.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.629+0000 7fed48c00640 1 -- 172.21.15.31:0/3441747305 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7fed4006c020 con 0x7fed4c074490 2024-09-13T08:42:41.630 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:42:41.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 msgr2=0x7fed2c07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 0x7fed2c07ae60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed4c166060 tx=0x7fed38005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 msgr2=0x7fed4c15ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fed4000e9f0 tx=0x7fed4000eeb0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 shutdown_connections 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed2c0789a0 0x7fed2c07ae60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed4c0b1a40 0x7fed4c1f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed4c074870 0x7fed4c0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 --2- 172.21.15.31:0/3441747305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4c074490 0x7fed4c15ea10 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 >> 172.21.15.31:0/3441747305 conn(0x7fed4c0d0440 msgr2=0x7fed4c0b5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 shutdown_connections 2024-09-13T08:42:41.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.631+0000 7fed53400640 1 -- 172.21.15.31:0/3441747305 wait complete. 2024-09-13T08:42:41.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 -- 172.21.15.31:0/2095042777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a815bf80 msgr2=0x7f38a80cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 --2- 172.21.15.31:0/2095042777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a815bf80 0x7f38a80cfe10 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f389c009be0 tx=0x7f389c02a850 comp rx=0 tx=0).stop 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 -- 172.21.15.31:0/2095042777 shutdown_connections 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 --2- 172.21.15.31:0/2095042777 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a80c80c0 0x7f38a80cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 --2- 172.21.15.31:0/2095042777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a815bf80 0x7f38a80cfe10 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 -- 172.21.15.31:0/2095042777 >> 172.21.15.31:0/2095042777 conn(0x7f38a80d1e20 msgr2=0x7f38a80d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 -- 172.21.15.31:0/2095042777 shutdown_connections 2024-09-13T08:42:41.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.807+0000 7f38b0000640 1 -- 172.21.15.31:0/2095042777 wait complete. 2024-09-13T08:42:41.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.808+0000 7f38b0000640 1 Processor -- start 2024-09-13T08:42:41.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.808+0000 7f38b0000640 1 -- start start 2024-09-13T08:42:41.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38a80c80c0 0x7f38a814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a815bf80 0x7f38a80cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38a80cd230 con 0x7f38a80d2940 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f38a80cd0b0 con 0x7f38a80c80c0 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.809+0000 7f38b0000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f38a80cd3b0 con 0x7f38a815bf80 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38730/0 (socket says 172.21.15.31:38730) 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 -- 172.21.15.31:0/2610918363 learned_addr learned my addr 172.21.15.31:0/2610918363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38ad600640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38a80c80c0 0x7f38a814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38ae000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a815bf80 0x7f38a80cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 -- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a815bf80 msgr2=0x7f38a80cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a815bf80 0x7f38a80cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 -- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38a80c80c0 msgr2=0x7f38a814b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38a80c80c0 0x7f38a814b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.810+0000 7f38acc00640 1 -- 172.21.15.31:0/2610918363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a80cbe60 con 0x7f38a80d2940 2024-09-13T08:42:41.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38acc00640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f389400c450 tx=0x7f389400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894016020 con 0x7f38a80d2940 2024-09-13T08:42:41.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3894012920 con 0x7f38a80d2940 2024-09-13T08:42:41.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a815c730 con 0x7f38a80d2940 2024-09-13T08:42:41.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894012d70 con 0x7f38a80d2940 2024-09-13T08:42:41.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.811+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a80cbb80 con 0x7f38a80d2940 2024-09-13T08:42:41.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.812+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a815ec20 con 0x7f38a80d2940 2024-09-13T08:42:41.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.814+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f3894019020 con 0x7f38a80d2940 2024-09-13T08:42:41.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.814+0000 7f38a5600640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 0x7f388807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:41.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.815+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f389409a4e0 con 0x7f38a80d2940 2024-09-13T08:42:41.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.815+0000 7f38ad600640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 0x7f388807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:41.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.815+0000 7f38ad600640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 0x7f388807ae60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f389c009bb0 tx=0x7f389c004ab0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:41.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:41.819+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3894020030 con 0x7f38a80d2940 2024-09-13T08:42:42.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.093+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f38a80cd940 con 0x7f38a80d2940 2024-09-13T08:42:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.859+0000 7f38a5600640 1 -- 172.21.15.31:0/2610918363 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v306) ==== 152+0+0 (secure 0 0 0) 0x7f3894062570 con 0x7f38a80d2940 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 msgr2=0x7f388807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 0x7f388807ae60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f389c009bb0 tx=0x7f389c004ab0 comp rx=0 tx=0).stop 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 msgr2=0x7f38a80c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f389400c450 tx=0x7f389400c910 comp rx=0 tx=0).stop 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 shutdown_connections 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38880789a0 0x7f388807ae60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f38a815bf80 0x7f38a80cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a80d2940 0x7f38a80c9070 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 --2- 172.21.15.31:0/2610918363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38a80c80c0 0x7f38a814b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 >> 172.21.15.31:0/2610918363 conn(0x7f38a80d1e20 msgr2=0x7f38a815a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 shutdown_connections 2024-09-13T08:42:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:42.862+0000 7f38b0000640 1 -- 172.21.15.31:0/2610918363 wait complete. 2024-09-13T08:42:42.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 -- 172.21.15.31:0/165506204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 msgr2=0x7efca40cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 --2- 172.21.15.31:0/165506204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca40cbc50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7efc9c005fd0 tx=0x7efc9c02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 -- 172.21.15.31:0/165506204 shutdown_connections 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 --2- 172.21.15.31:0/165506204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca40cbc50 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 --2- 172.21.15.31:0/165506204 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 0x7efca40cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 -- 172.21.15.31:0/165506204 >> 172.21.15.31:0/165506204 conn(0x7efca40cc910 msgr2=0x7efca40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 -- 172.21.15.31:0/165506204 shutdown_connections 2024-09-13T08:42:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.047+0000 7efca9200640 1 -- 172.21.15.31:0/165506204 wait complete. 2024-09-13T08:42:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.048+0000 7efca9200640 1 Processor -- start 2024-09-13T08:42:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 -- start start 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 0x7efca409d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efca415f9e0 0x7efca406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efca40cec50 con 0x7efca415fe20 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efca40cead0 con 0x7efca415f9e0 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.049+0000 7efca9200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efca40cb850 con 0x7efca415bd70 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38750/0 (socket says 172.21.15.31:38750) 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 -- 172.21.15.31:0/1762376024 learned_addr learned my addr 172.21.15.31:0/1762376024 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca1e00640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 0x7efca409d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca1400640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efca415f9e0 0x7efca406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 -- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 msgr2=0x7efca409d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 0x7efca409d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 -- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efca415f9e0 msgr2=0x7efca406f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efca415f9e0 0x7efca406f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.050+0000 7efca2800640 1 -- 172.21.15.31:0/1762376024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca41f4410 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efca2800640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efc9c006010 tx=0x7efc9c031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c040070 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc9c004040 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c004450 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca40b31c0 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.051+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca41f46d0 con 0x7efca415fe20 2024-09-13T08:42:43.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.052+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca40cf850 con 0x7efca415fe20 2024-09-13T08:42:43.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.053+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7efc9c039040 con 0x7efca415fe20 2024-09-13T08:42:43.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.054+0000 7efc9a000640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 0x7efc8407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.054+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7efc9c0bbcc0 con 0x7efca415fe20 2024-09-13T08:42:43.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.055+0000 7efca1e00640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 0x7efc8407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.055+0000 7efca1e00640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 0x7efc8407ae30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efca40c6540 tx=0x7efc94005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.058+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efc9c0c2080 con 0x7efca415fe20 2024-09-13T08:42:43.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.253+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7efca41f48a0 con 0x7efca415fe20 2024-09-13T08:42:43.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.254+0000 7efc9a000640 1 -- 172.21.15.31:0/1762376024 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+1998 (secure 0 0 0) 0x7efc9c083d50 con 0x7efca415fe20 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:42:43.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:42:43.256 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:42:43.257 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 msgr2=0x7efc8407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 0x7efc8407ae30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efca40c6540 tx=0x7efc94005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 msgr2=0x7efca41f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efc9c006010 tx=0x7efc9c031670 comp rx=0 tx=0).stop 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 shutdown_connections 2024-09-13T08:42:43.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc84078970 0x7efc8407ae30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca415fe20 0x7efca41f4bf0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efca415f9e0 0x7efca406f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 --2- 172.21.15.31:0/1762376024 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efca415bd70 0x7efca409d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 >> 172.21.15.31:0/1762376024 conn(0x7efca40cc910 msgr2=0x7efca40ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 shutdown_connections 2024-09-13T08:42:43.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.256+0000 7efca9200640 1 -- 172.21.15.31:0/1762376024 wait complete. 2024-09-13T08:42:43.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:42:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 -- 172.21.15.31:0/1553253759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f815bfc0 msgr2=0x7fe1f8072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 --2- 172.21.15.31:0/1553253759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f815bfc0 0x7fe1f8072560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009860 tx=0x7fe1e802a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 -- 172.21.15.31:0/1553253759 shutdown_connections 2024-09-13T08:42:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 --2- 172.21.15.31:0/1553253759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f815bfc0 0x7fe1f8072560 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 -- 172.21.15.31:0/1553253759 >> 172.21.15.31:0/1553253759 conn(0x7fe1f80cfbb0 msgr2=0x7fe1f80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 -- 172.21.15.31:0/1553253759 shutdown_connections 2024-09-13T08:42:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.437+0000 7fe1fe600640 1 -- 172.21.15.31:0/1553253759 wait complete. 2024-09-13T08:42:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.438+0000 7fe1fe600640 1 Processor -- start 2024-09-13T08:42:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 -- start start 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 0x7fe1f80b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f80c86d0 0x7fe1f80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f806e760 con 0x7fe1f815bfc0 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80cc060 con 0x7fe1f806fa60 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.439+0000 7fe1fe600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f80b30f0 con 0x7fe1f80c86d0 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38762/0 (socket says 172.21.15.31:38762) 2024-09-13T08:42:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 -- 172.21.15.31:0/3143461387 learned_addr learned my addr 172.21.15.31:0/3143461387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1f7400640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 0x7fe1f80b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1f7e00640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f80c86d0 0x7fe1f80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 -- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f80c86d0 msgr2=0x7fe1f80cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f80c86d0 0x7fe1f80cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 -- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 msgr2=0x7fe1f80b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 0x7fe1f80b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 -- 172.21.15.31:0/3143461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f80c6e00 con 0x7fe1f815bfc0 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1f7400640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 0x7fe1f80b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.440+0000 7fe1eea00640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000c430 tx=0x7fe1e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.441+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0011790 con 0x7fe1f815bfc0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.441+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1e0017070 con 0x7fe1f815bfc0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.441+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0011cc0 con 0x7fe1f815bfc0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.441+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f80c70e0 con 0x7fe1f815bfc0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.441+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f80c76a0 con 0x7fe1f815bfc0 2024-09-13T08:42:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.442+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f806a180 con 0x7fe1f815bfc0 2024-09-13T08:42:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.443+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe1e001c460 con 0x7fe1f815bfc0 2024-09-13T08:42:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.444+0000 7fe1efe00640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 0x7fe1d407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.444+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fe1e009a6d0 con 0x7fe1f815bfc0 2024-09-13T08:42:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.444+0000 7fe1f7400640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 0x7fe1d407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.445+0000 7fe1f7400640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 0x7fe1d407ae60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8034000 tx=0x7fe1e8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.449+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe1f806a180 con 0x7fe1f815bfc0 2024-09-13T08:42:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.644+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe1f8072560 con 0x7fe1f815bfc0 2024-09-13T08:42:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.647+0000 7fe1efe00640 1 -- 172.21.15.31:0/3143461387 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7fe1e0062db0 con 0x7fe1f815bfc0 2024-09-13T08:42:43.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 msgr2=0x7fe1d407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 0x7fe1d407ae60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8034000 tx=0x7fe1e8004730 comp rx=0 tx=0).stop 2024-09-13T08:42:43.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 msgr2=0x7fe1f80ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:43.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000c430 tx=0x7fe1e000c8f0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 shutdown_connections 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe1d40789a0 0x7fe1d407ae60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe1f80c86d0 0x7fe1f80cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f815bfc0 0x7fe1f80ce9c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 --2- 172.21.15.31:0/3143461387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f806fa60 0x7fe1f80b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 >> 172.21.15.31:0/3143461387 conn(0x7fe1f80cfbb0 msgr2=0x7fe1f815abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 shutdown_connections 2024-09-13T08:42:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:43.650+0000 7fe1fe600640 1 -- 172.21.15.31:0/3143461387 wait complete. 2024-09-13T08:42:43.668 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:42:43.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:43.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:42:43.841 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:42:43.841 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:43.841 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:43.877 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:42:43.878 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:43.878 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:43.915 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:42:43.915 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:43.915 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:43.953 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:42:43.953 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:43.953 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:43.991 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:42:43.991 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:42:43.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 -- 172.21.15.31:0/2175626226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f2415dfc0 msgr2=0x7f1f240cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2175626226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f2415dfc0 0x7f1f240cc300 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1f24070b80 tx=0x7f1f1402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 -- 172.21.15.31:0/2175626226 shutdown_connections 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2175626226 >> v2:172.21.15.45:3300/0 conn(0x7f1f240ca5c0 0x7f1f240ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2175626226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f2415dfc0 0x7f1f240cc300 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.151+0000 7f1f2a800640 1 -- 172.21.15.31:0/2175626226 >> 172.21.15.31:0/2175626226 conn(0x7f1f240cc910 msgr2=0x7f1f240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.152+0000 7f1f2a800640 1 -- 172.21.15.31:0/2175626226 shutdown_connections 2024-09-13T08:42:44.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.152+0000 7f1f2a800640 1 -- 172.21.15.31:0/2175626226 wait complete. 2024-09-13T08:42:44.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.153+0000 7f1f2a800640 1 Processor -- start 2024-09-13T08:42:44.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.153+0000 7f1f2a800640 1 -- start start 2024-09-13T08:42:44.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f240ca5c0 0x7f1f2415caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f2415dfc0 0x7f1f240ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24074010 con 0x7f1f240b1910 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24073e90 con 0x7f1f240ca5c0 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f2a800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1f241f6050 con 0x7f1f2415dfc0 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f23400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f23400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38788/0 (socket says 172.21.15.31:38788) 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.154+0000 7f1f23400640 1 -- 172.21.15.31:0/2541454386 learned_addr learned my addr 172.21.15.31:0/2541454386 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f240ca5c0 0x7f1f2415caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23e00640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f2415dfc0 0x7f1f240ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 -- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f2415dfc0 msgr2=0x7f1f240ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f2415dfc0 0x7f1f240ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 -- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f240ca5c0 msgr2=0x7f1f2415caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f240ca5c0 0x7f1f2415caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 -- 172.21.15.31:0/2541454386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2415cfe0 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f23400640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1f140099c0 tx=0x7f1f140039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1403e030 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f140047a0 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14035040 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.155+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f241f57b0 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.156+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f241f5d70 con 0x7f1f240b1910 2024-09-13T08:42:44.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.156+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f240b2070 con 0x7f1f240b1910 2024-09-13T08:42:44.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.158+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f1f14041030 con 0x7f1f240b1910 2024-09-13T08:42:44.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.158+0000 7f1f1be00640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 0x7f1f0007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.159+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f1f140bae80 con 0x7f1f240b1910 2024-09-13T08:42:44.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.159+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 0x7f1f0007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.159+0000 7f1f1aa00640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 0x7f1f0007ae30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c005fd0 tx=0x7f1f0c005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:44.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.163+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1f14083600 con 0x7f1f240b1910 2024-09-13T08:42:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.393+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f240af5d0 con 0x7f1f240b1910 2024-09-13T08:42:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.394+0000 7f1f1be00640 1 -- 172.21.15.31:0/2541454386 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+3911 (secure 0 0 0) 0x7f1f14082f00 con 0x7f1f240b1910 2024-09-13T08:42:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:44.395 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"btime":"2024-09-13T08:42:37:835497+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1337597966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1337597966},{"type":"v1","addr":"172.21.15.31:6835","nonce":1337597966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6307,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/2697560219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2697560219},{"type":"v1","addr":"172.21.15.45:6801","nonce":2697560219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6481,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3602175668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3602175668},{"type":"v1","addr":"172.21.15.45:6835","nonce":3602175668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3380692422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3380692422},{"type":"v1","addr":"172.21.15.31:6834","nonce":3380692422}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[]} 2024-09-13T08:42:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 64 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 msgr2=0x7f1f0007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 0x7f1f0007ae30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c005fd0 tx=0x7f1f0c005850 comp rx=0 tx=0).stop 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 msgr2=0x7f1f241f5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1f140099c0 tx=0x7f1f140039e0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 shutdown_connections 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f00078970 0x7f1f0007ae30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f2415dfc0 0x7f1f240ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f240ca5c0 0x7f1f2415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 --2- 172.21.15.31:0/2541454386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f240b1910 0x7f1f241f5110 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 >> 172.21.15.31:0/2541454386 conn(0x7f1f240cc910 msgr2=0x7f1f24073710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:44.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 shutdown_connections 2024-09-13T08:42:44.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.396+0000 7f1f2a800640 1 -- 172.21.15.31:0/2541454386 wait complete. 2024-09-13T08:42:44.413 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:42:44.413 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:42:44.413 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:42:44.413 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:44.414 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:42:44.414 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:42:44.414 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:42:44.414 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:44.414 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:42:44.414 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:42:44.415 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:42:44.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:44.415 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:42:44.415 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:42:44.415 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:42:44.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:42:44.428 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:42:44.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:42:44.430 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:42:44.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:42:44.432 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:42:44.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:42:44.434 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:42:44.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:42:44.474 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:42:44.475 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:44.477 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:42:44.478 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:44.478 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:42:44.479 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:44.481 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:42:44.481 DEBUG:teuthology.parallel:result is None 2024-09-13T08:42:44.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:42:44.603 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:42:44.606 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:42:44.619 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:42:44.619 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:42:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 -- 172.21.15.31:0/1252971284 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccba0 msgr2=0x7f15080b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 --2- 172.21.15.31:0/1252971284 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccba0 0x7f15080b3dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00a9a0 tx=0x7f14fc02b8f0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 -- 172.21.15.31:0/1252971284 shutdown_connections 2024-09-13T08:42:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 --2- 172.21.15.31:0/1252971284 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccba0 0x7f15080b3dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 -- 172.21.15.31:0/1252971284 >> 172.21.15.31:0/1252971284 conn(0x7f15080b18f0 msgr2=0x7f15080b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 -- 172.21.15.31:0/1252971284 shutdown_connections 2024-09-13T08:42:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.648+0000 7f1510800640 1 -- 172.21.15.31:0/1252971284 wait complete. 2024-09-13T08:42:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.650+0000 7f1510800640 1 Processor -- start 2024-09-13T08:42:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.650+0000 7f1510800640 1 -- start start 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15080ccba0 0x7f150809c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 0x7f1508154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1508153ef0 con 0x7f15080cf770 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f15080704b0 con 0x7f15080ccba0 2024-09-13T08:42:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f1510800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1508154070 con 0x7f15080ccf80 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38818/0 (socket says 172.21.15.31:38818) 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 -- 172.21.15.31:0/460232697 learned_addr learned my addr 172.21.15.31:0/460232697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150d400640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 0x7f1508154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150de00640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15080ccba0 0x7f150809c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 -- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 msgr2=0x7f1508154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 0x7f1508154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 -- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15080ccba0 msgr2=0x7f150809c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15080ccba0 0x7f150809c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150e800640 1 -- 172.21.15.31:0/460232697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150809d500 con 0x7f15080cf770 2024-09-13T08:42:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.651+0000 7f150d400640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 0x7f1508154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f150e800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f150400c6e0 tx=0x7f150400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504003b70 con 0x7f15080cf770 2024-09-13T08:42:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1508154d40 con 0x7f15080cf770 2024-09-13T08:42:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1504003d10 con 0x7f15080cf770 2024-09-13T08:42:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504005330 con 0x7f15080cf770 2024-09-13T08:42:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.652+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150809d220 con 0x7f15080cf770 2024-09-13T08:42:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.653+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15080b3dc0 con 0x7f15080cf770 2024-09-13T08:42:44.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.654+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f15040054d0 con 0x7f15080cf770 2024-09-13T08:42:44.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.655+0000 7f1502000640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 0x7f14ec07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:44.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.655+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f150409a6d0 con 0x7f15080cf770 2024-09-13T08:42:44.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.655+0000 7f150de00640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 0x7f14ec07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.656+0000 7f150de00640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 0x7f14ec07ae30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14fc036000 tx=0x7f14fc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.660+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1504062d30 con 0x7f15080cf770 2024-09-13T08:42:44.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.868+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f15080749f0 con 0x7f15080cf770 2024-09-13T08:42:44.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.874+0000 7f1502000640 1 -- 172.21.15.31:0/460232697 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) ==== 97+0+0 (secure 0 0 0) 0x7f1504062630 con 0x7f15080cf770 2024-09-13T08:42:44.875 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.877+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 msgr2=0x7f14ec07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.877+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 0x7f14ec07ae30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14fc036000 tx=0x7f14fc004730 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.877+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 msgr2=0x7f15081f8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.877+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f150400c6e0 tx=0x7f150400cba0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 shutdown_connections 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f14ec078970 0x7f14ec07ae30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15080cf770 0x7f15081f8430 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15080ccf80 0x7f1508154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 --2- 172.21.15.31:0/460232697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15080ccba0 0x7f150809c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 >> 172.21.15.31:0/460232697 conn(0x7f15080b18f0 msgr2=0x7f15080cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 shutdown_connections 2024-09-13T08:42:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:44.878+0000 7f1510800640 1 -- 172.21.15.31:0/460232697 wait complete. 2024-09-13T08:42:44.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 -- 172.21.15.31:0/2073429347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415f9e0 msgr2=0x7f91e415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 --2- 172.21.15.31:0/2073429347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415f9e0 0x7f91e415fe20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009830 tx=0x7f91cc02a510 comp rx=0 tx=0).stop 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 -- 172.21.15.31:0/2073429347 shutdown_connections 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 --2- 172.21.15.31:0/2073429347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415f9e0 0x7f91e415fe20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 --2- 172.21.15.31:0/2073429347 >> v2:172.21.15.45:3300/0 conn(0x7f91e415bd70 0x7f91e40cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.057+0000 7f91e9200640 1 -- 172.21.15.31:0/2073429347 >> 172.21.15.31:0/2073429347 conn(0x7f91e40cc910 msgr2=0x7f91e40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.058+0000 7f91e9200640 1 -- 172.21.15.31:0/2073429347 shutdown_connections 2024-09-13T08:42:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.058+0000 7f91e9200640 1 -- 172.21.15.31:0/2073429347 wait complete. 2024-09-13T08:42:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.059+0000 7f91e9200640 1 Processor -- start 2024-09-13T08:42:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.059+0000 7f91e9200640 1 -- start start 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415bd70 0x7f91e40c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 0x7f91e40c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91e40cb9d0 con 0x7f91e415f9e0 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91e40cb850 con 0x7f91e40ce3c0 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.060+0000 7f91e9200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f91e40cc360 con 0x7f91e415bd70 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 0x7f91e40c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91d9400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415bd70 0x7f91e40c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:44026/0 (socket says 172.21.15.31:44026) 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 -- 172.21.15.31:0/1809975728 learned_addr learned my addr 172.21.15.31:0/1809975728 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 -- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415bd70 msgr2=0x7f91e40c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415bd70 0x7f91e40c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 -- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 msgr2=0x7f91e40c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 0x7f91e40c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 -- 172.21.15.31:0/1809975728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e40b5a20 con 0x7f91e40ce3c0 2024-09-13T08:42:45.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e2800640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 0x7f91e40c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:42:45.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.061+0000 7f91e1e00640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e9c0 tx=0x7f91d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.062+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d400cde0 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.062+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91d4004020 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.062+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4004450 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.062+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e40c8970 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.062+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e40c8f30 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.063+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e40aeed0 con 0x7f91e40ce3c0 2024-09-13T08:42:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.064+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f91d40035f0 con 0x7f91e40ce3c0 2024-09-13T08:42:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.065+0000 7f91daa00640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 0x7f91c007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.065+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4857+0+0 (secure 0 0 0) 0x7f91d409b170 con 0x7f91e40ce3c0 2024-09-13T08:42:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.065+0000 7f91d9400640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 0x7f91c007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.066+0000 7f91d9400640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 0x7f91c007ae30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f91cc034000 tx=0x7f91cc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.070+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f91e40aeed0 con 0x7f91e40ce3c0 2024-09-13T08:42:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.341+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f91e4072710 con 0x7f91e40ce3c0 2024-09-13T08:42:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.342+0000 7f91daa00640 1 -- 172.21.15.31:0/1809975728 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f91d40632b0 con 0x7f91e40ce3c0 2024-09-13T08:42:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:45.343 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:42:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:42:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:42:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:42:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:42:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:42:45.358 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:42:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 msgr2=0x7f91c007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 0x7f91c007ae30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f91cc034000 tx=0x7f91cc004730 comp rx=0 tx=0).stop 2024-09-13T08:42:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 msgr2=0x7f91e40b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e9c0 tx=0x7f91d400ee80 comp rx=0 tx=0).stop 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 shutdown_connections 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f91c0078970 0x7f91c007ae30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91e415f9e0 0x7f91e40c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f91e415bd70 0x7f91e40c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 --2- 172.21.15.31:0/1809975728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e40ce3c0 0x7f91e40b3070 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 >> 172.21.15.31:0/1809975728 conn(0x7f91e40cc910 msgr2=0x7f91e40ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 shutdown_connections 2024-09-13T08:42:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.345+0000 7f91e9200640 1 -- 172.21.15.31:0/1809975728 wait complete. 2024-09-13T08:42:45.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:42:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.520+0000 7f17ed800640 1 -- 172.21.15.31:0/3967156541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 msgr2=0x7f17e806f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.520+0000 7f17ed800640 1 --2- 172.21.15.31:0/3967156541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e806f9b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009830 tx=0x7f17d802a510 comp rx=0 tx=0).stop 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 -- 172.21.15.31:0/3967156541 shutdown_connections 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 --2- 172.21.15.31:0/3967156541 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80ccb00 0x7f17e80b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 --2- 172.21.15.31:0/3967156541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e806f9b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 -- 172.21.15.31:0/3967156541 >> 172.21.15.31:0/3967156541 conn(0x7f17e806cb00 msgr2=0x7f17e806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 -- 172.21.15.31:0/3967156541 shutdown_connections 2024-09-13T08:42:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.521+0000 7f17ed800640 1 -- 172.21.15.31:0/3967156541 wait complete. 2024-09-13T08:42:45.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.522+0000 7f17ed800640 1 Processor -- start 2024-09-13T08:42:45.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.522+0000 7f17ed800640 1 -- start start 2024-09-13T08:42:45.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 0x7f17e8152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80ccb00 0x7f17e815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17e80b5490 con 0x7f17e80cc720 2024-09-13T08:42:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17e80b5310 con 0x7f17e80ccb00 2024-09-13T08:42:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17ed800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f17e8152d90 con 0x7f17e80701a0 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.523+0000 7f17e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42414/0 (socket says 172.21.15.31:42414) 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 -- 172.21.15.31:0/1823703036 learned_addr learned my addr 172.21.15.31:0/1823703036 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7e00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 0x7f17e8152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17ece00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80ccb00 0x7f17e815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 -- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 msgr2=0x7f17e8152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 0x7f17e8152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 -- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80ccb00 msgr2=0x7f17e815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80ccb00 0x7f17e815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 -- 172.21.15.31:0/1823703036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e815cf80 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7e00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 0x7f17e8152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17e7400640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f17d400d700 tx=0x7f17d400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.524+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d4017070 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.525+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17d4004310 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.525+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e8150730 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.525+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d4009a80 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.525+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e8150cf0 con 0x7f17e80cc720 2024-09-13T08:42:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.526+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e8153260 con 0x7f17e80cc720 2024-09-13T08:42:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.527+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f17d4009c20 con 0x7f17e80cc720 2024-09-13T08:42:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.528+0000 7f17dfe00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 0x7f17c807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.529+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4857+0+0 (secure 0 0 0) 0x7f17d409a8c0 con 0x7f17e80cc720 2024-09-13T08:42:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.529+0000 7f17e7e00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 0x7f17c807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.529+0000 7f17e7e00640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 0x7f17c807ae30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17d8034000 tx=0x7f17d8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.533+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f17d4062970 con 0x7f17e80cc720 2024-09-13T08:42:45.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.729+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f17e80b36c0 con 0x7f17e80cc720 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.730+0000 7f17dfe00640 1 -- 172.21.15.31:0/1823703036 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) ==== 84+0+65 (secure 0 0 0) 0x7f17d40673a0 con 0x7f17e80cc720 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:45.731 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:42:45.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 msgr2=0x7f17c807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 0x7f17c807ae30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17d8034000 tx=0x7f17d8004730 comp rx=0 tx=0).stop 2024-09-13T08:42:45.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 msgr2=0x7f17e815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f17d400d700 tx=0x7f17d400dbc0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 shutdown_connections 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f17c8078970 0x7f17c807ae30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80ccb00 0x7f17e815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17e80cc720 0x7f17e815ae40 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 --2- 172.21.15.31:0/1823703036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f17e80701a0 0x7f17e8152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.732+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 >> 172.21.15.31:0/1823703036 conn(0x7f17e806cb00 msgr2=0x7f17e81ea7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.733+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 shutdown_connections 2024-09-13T08:42:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.733+0000 7f17ed800640 1 -- 172.21.15.31:0/1823703036 wait complete. 2024-09-13T08:42:45.749 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:42:45.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4166304840 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 msgr2=0x7ffaec0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4166304840 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec0cb740 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffad4009830 tx=0x7ffad402a520 comp rx=0 tx=0).stop 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4166304840 shutdown_connections 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4166304840 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec0cb740 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4166304840 >> v2:172.21.15.45:3300/0 conn(0x7ffaec15bfc0 0x7ffaec072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4166304840 >> 172.21.15.31:0/4166304840 conn(0x7ffaec0cfbb0 msgr2=0x7ffaec0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4166304840 shutdown_connections 2024-09-13T08:42:45.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.916+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4166304840 wait complete. 2024-09-13T08:42:45.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.918+0000 7ffaf0e00640 1 Processor -- start 2024-09-13T08:42:45.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.918+0000 7ffaf0e00640 1 -- start start 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaec15cc00 0x7ffaec15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec0b3270 con 0x7ffaec15bfc0 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec0b30f0 con 0x7ffaec15cc00 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaf0e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffaec0b33f0 con 0x7ffaec06fa60 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffae1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffae1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42426/0 (socket says 172.21.15.31:42426) 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffae1400640 1 -- 172.21.15.31:0/4179183099 learned_addr learned my addr 172.21.15.31:0/4179183099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffae9e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.919+0000 7ffaea800640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaec15cc00 0x7ffaec15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 -- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 msgr2=0x7ffaec156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 -- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaec15cc00 msgr2=0x7ffaec15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:45.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaec15cc00 0x7ffaec15e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:45.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 -- 172.21.15.31:0/4179183099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec0c8780 con 0x7ffaec15bfc0 2024-09-13T08:42:45.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae1400640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ffad4009950 tx=0x7ffad4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4040070 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffad4005310 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4005c90 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.920+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec0c8a60 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.921+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec0c9020 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.922+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaec050fc0 con 0x7ffaec15bfc0 2024-09-13T08:42:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.923+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7ffad4039040 con 0x7ffaec15bfc0 2024-09-13T08:42:45.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.924+0000 7ffae2a00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 0x7ffac807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:45.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.924+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4857+0+0 (secure 0 0 0) 0x7ffad40bb7f0 con 0x7ffaec15bfc0 2024-09-13T08:42:45.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.924+0000 7ffae9e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 0x7ffac807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:45.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.925+0000 7ffae9e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 0x7ffac807ae30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0095f0 tx=0x7ffadc005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:45.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:45.928+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffaec050fc0 con 0x7ffaec15bfc0 2024-09-13T08:42:46.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.122+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffaec0cbf20 con 0x7ffaec15bfc0 2024-09-13T08:42:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.887+0000 7ffae2a00640 1 -- 172.21.15.31:0/4179183099 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) ==== 138+0+0 (secure 0 0 0) 0x7ffad40838a0 con 0x7ffaec15bfc0 2024-09-13T08:42:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:42:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 msgr2=0x7ffac807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 0x7ffac807ae30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0095f0 tx=0x7ffadc005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 msgr2=0x7ffaec15c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ffad4009950 tx=0x7ffad4032c10 comp rx=0 tx=0).stop 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 shutdown_connections 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffac8078970 0x7ffac807ae30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaec15cc00 0x7ffaec15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffaec15bfc0 0x7ffaec15c6c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 --2- 172.21.15.31:0/4179183099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffaec06fa60 0x7ffaec156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.891+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 >> 172.21.15.31:0/4179183099 conn(0x7ffaec0cfbb0 msgr2=0x7ffaec158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.892+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 shutdown_connections 2024-09-13T08:42:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:46.892+0000 7ffaf0e00640 1 -- 172.21.15.31:0/4179183099 wait complete. 2024-09-13T08:42:46.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 -- 172.21.15.31:0/2655377484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 msgr2=0x7fab6c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 --2- 172.21.15.31:0/2655377484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab6c004800 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fab74009b50 tx=0x7fab6c005830 comp rx=0 tx=0).stop 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8ca00640 1 -- 172.21.15.31:0/2655377484 reap_dead start 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 -- 172.21.15.31:0/2655377484 shutdown_connections 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 -- 172.21.15.31:0/2655377484 >> 172.21.15.31:0/2655377484 conn(0x7fab880d0ae0 msgr2=0x7fab880d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 -- 172.21.15.31:0/2655377484 shutdown_connections 2024-09-13T08:42:47.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.091+0000 7fab8d400640 1 -- 172.21.15.31:0/2655377484 wait complete. 2024-09-13T08:42:47.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.092+0000 7fab8d400640 1 Processor -- start 2024-09-13T08:42:47.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 -- start start 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab880c7bb0 0x7fab881f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 0x7fab881f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab880713e0 con 0x7fab6c004000 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab88071260 con 0x7fab881f60b0 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.093+0000 7fab8d400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab880d1600 con 0x7fab880c7bb0 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42454/0 (socket says 172.21.15.31:42454) 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 -- 172.21.15.31:0/3667442331 learned_addr learned my addr 172.21.15.31:0/3667442331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab880c7bb0 0x7fab881f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab8ca00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 0x7fab881f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:47.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 -- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab880c7bb0 msgr2=0x7fab881f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:47.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab880c7bb0 0x7fab881f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 -- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 msgr2=0x7fab881f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:47.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 0x7fab881f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 -- 172.21.15.31:0/3667442331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab880c8120 con 0x7fab6c004000 2024-09-13T08:42:47.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab8ca00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 0x7fab881f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.094+0000 7fab87e00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fab78001a70 tx=0x7fab780039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.095+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab78015070 con 0x7fab6c004000 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.095+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab78011920 con 0x7fab6c004000 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.095+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab881583a0 con 0x7fab6c004000 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.095+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab78011d70 con 0x7fab6c004000 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.095+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab88157090 con 0x7fab6c004000 2024-09-13T08:42:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.096+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab88073510 con 0x7fab6c004000 2024-09-13T08:42:47.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.098+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fab78025440 con 0x7fab6c004000 2024-09-13T08:42:47.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.099+0000 7fab7fe00640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 0x7fab6807ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:47.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.099+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5227+0+0 (secure 0 0 0) 0x7fab7809bc40 con 0x7fab6c004000 2024-09-13T08:42:47.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.099+0000 7fab87400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 0x7fab6807ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:47.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.100+0000 7fab87400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 0x7fab6807ae90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fab74035000 tx=0x7fab740046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:47.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.102+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab880cb630 con 0x7fab6c004000 2024-09-13T08:42:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.300+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fab88157260 con 0x7fab6c004000 2024-09-13T08:42:47.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.896+0000 7fab7fe00640 1 -- 172.21.15.31:0/3667442331 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) ==== 170+0+0 (secure 0 0 0) 0x7fab78063c10 con 0x7fab6c004000 2024-09-13T08:42:47.897 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 msgr2=0x7fab6807ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 0x7fab6807ae90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fab74035000 tx=0x7fab740046c0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 msgr2=0x7fab8815e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fab78001a70 tx=0x7fab780039c0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 shutdown_connections 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab680789d0 0x7fab6807ae90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881f60b0 0x7fab881f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab880c7bb0 0x7fab881f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 --2- 172.21.15.31:0/3667442331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6c004000 0x7fab8815e310 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:47.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 >> 172.21.15.31:0/3667442331 conn(0x7fab880d0ae0 msgr2=0x7fab880b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:47.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 shutdown_connections 2024-09-13T08:42:47.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:47.899+0000 7fab8d400640 1 -- 172.21.15.31:0/3667442331 wait complete. 2024-09-13T08:42:47.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:42:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 -- 172.21.15.31:0/1926777284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1415bfc0 msgr2=0x7ffb14072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:48.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 --2- 172.21.15.31:0/1926777284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1415bfc0 0x7ffb14072560 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ffb04005fd0 tx=0x7ffb0402a530 comp rx=0 tx=0).stop 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 -- 172.21.15.31:0/1926777284 shutdown_connections 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 --2- 172.21.15.31:0/1926777284 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb1406fec0 0x7ffb140b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 --2- 172.21.15.31:0/1926777284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1415bfc0 0x7ffb14072560 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 -- 172.21.15.31:0/1926777284 >> 172.21.15.31:0/1926777284 conn(0x7ffb140cfbb0 msgr2=0x7ffb140cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 -- 172.21.15.31:0/1926777284 shutdown_connections 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.101+0000 7ffb18c00640 1 -- 172.21.15.31:0/1926777284 wait complete. 2024-09-13T08:42:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.103+0000 7ffb18c00640 1 Processor -- start 2024-09-13T08:42:48.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.103+0000 7ffb18c00640 1 -- start start 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1415bfc0 0x7ffb140cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb14142d60 0x7ffb140c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140cbfd0 con 0x7ffb1406fa60 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140cbe50 con 0x7ffb1415bfc0 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb18c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffb140b2f20 con 0x7ffb14142d60 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb11e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb11e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42458/0 (socket says 172.21.15.31:42458) 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb11e00640 1 -- 172.21.15.31:0/4120672842 learned_addr learned my addr 172.21.15.31:0/4120672842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb09400640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1415bfc0 0x7ffb140cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.104+0000 7ffb12800640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb14142d60 0x7ffb140c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 -- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb14142d60 msgr2=0x7ffb140c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb14142d60 0x7ffb140c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 -- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1415bfc0 msgr2=0x7ffb140cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1415bfc0 0x7ffb140cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 -- 172.21.15.31:0/4120672842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb14151310 con 0x7ffb1406fa60 2024-09-13T08:42:48.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb11e00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffb04034040 tx=0x7ffb04003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0403e030 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb040047a0 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.105+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04035040 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.106+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb14159610 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.106+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb141501f0 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.107+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb140b3410 con 0x7ffb1406fa60 2024-09-13T08:42:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.108+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7ffb04041030 con 0x7ffb1406fa60 2024-09-13T08:42:48.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.109+0000 7ffb0aa00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 0x7ffaf007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:48.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.109+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5609+0+0 (secure 0 0 0) 0x7ffb040bb2a0 con 0x7ffb1406fa60 2024-09-13T08:42:48.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.109+0000 7ffb09400640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 0x7ffaf007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:48.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.110+0000 7ffb09400640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 0x7ffaf007ae60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb141eac90 tx=0x7ffafc007e30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:48.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.114+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffb04082fd0 con 0x7ffb1406fa60 2024-09-13T08:42:48.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.346+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ffb140cb310 con 0x7ffb1406fa60 2024-09-13T08:42:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.898+0000 7ffb0aa00640 1 -- 172.21.15.31:0/4120672842 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v67) ==== 174+0+0 (secure 0 0 0) 0x7ffb04087a00 con 0x7ffb1406fa60 2024-09-13T08:42:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 16 and data pool 17 2024-09-13T08:42:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.903+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 msgr2=0x7ffaf007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.903+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 0x7ffaf007ae60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb141eac90 tx=0x7ffafc007e30 comp rx=0 tx=0).stop 2024-09-13T08:42:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.903+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 msgr2=0x7ffb140c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.903+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffb04034040 tx=0x7ffb04003970 comp rx=0 tx=0).stop 2024-09-13T08:42:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 shutdown_connections 2024-09-13T08:42:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffaf00789a0 0x7ffaf007ae60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb14142d60 0x7ffb140c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1415bfc0 0x7ffb140cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 --2- 172.21.15.31:0/4120672842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb1406fa60 0x7ffb140c6c80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.904+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 >> 172.21.15.31:0/4120672842 conn(0x7ffb140cfbb0 msgr2=0x7ffb14152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.905+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 shutdown_connections 2024-09-13T08:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:48.905+0000 7ffb18c00640 1 -- 172.21.15.31:0/4120672842 wait complete. 2024-09-13T08:42:48.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 -- 172.21.15.31:0/1966731066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c15dfc0 msgr2=0x7fe86c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 --2- 172.21.15.31:0/1966731066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c15dfc0 0x7fe86c0cc300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009860 tx=0x7fe85c02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 -- 172.21.15.31:0/1966731066 shutdown_connections 2024-09-13T08:42:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 --2- 172.21.15.31:0/1966731066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c15dfc0 0x7fe86c0cc300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 -- 172.21.15.31:0/1966731066 >> 172.21.15.31:0/1966731066 conn(0x7fe86c0cc910 msgr2=0x7fe86c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:49.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 -- 172.21.15.31:0/1966731066 shutdown_connections 2024-09-13T08:42:49.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.111+0000 7fe871a00640 1 -- 172.21.15.31:0/1966731066 wait complete. 2024-09-13T08:42:49.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.113+0000 7fe871a00640 1 Processor -- start 2024-09-13T08:42:49.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.113+0000 7fe871a00640 1 -- start start 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c0b1910 0x7fe86c0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe86c0ca5c0 0x7fe86c0ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0b1da0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0740a0 con 0x7fe86c0ca5c0 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe871a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0b1f20 con 0x7fe86c0b1910 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42470/0 (socket says 172.21.15.31:42470) 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 -- 172.21.15.31:0/2506562115 learned_addr learned my addr 172.21.15.31:0/2506562115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86aa00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c0b1910 0x7fe86c0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe862000640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe86c0ca5c0 0x7fe86c0ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 -- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c0b1910 msgr2=0x7fe86c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c0b1910 0x7fe86c0b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 -- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe86c0ca5c0 msgr2=0x7fe86c0ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe86c0ca5c0 0x7fe86c0ad660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.114+0000 7fe86b400640 1 -- 172.21.15.31:0/2506562115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c1f5350 con 0x7fe86c15dfc0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe86b400640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c770 tx=0x7fe86400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864017070 con 0x7fe86c15dfc0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe864005070 con 0x7fe86c15dfc0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c1f5630 con 0x7fe86c15dfc0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8640054c0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.115+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c1f5bf0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.116+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c0cc300 con 0x7fe86c15dfc0 2024-09-13T08:42:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.117+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe86401a070 con 0x7fe86c15dfc0 2024-09-13T08:42:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.118+0000 7fe863400640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 0x7fe84807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.118+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fe86409a240 con 0x7fe86c15dfc0 2024-09-13T08:42:49.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.119+0000 7fe86aa00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 0x7fe84807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.119+0000 7fe86aa00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 0x7fe84807ae30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe85c034000 tx=0x7fe85c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.123+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe864061fa0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.319+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe86c0ce2c0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.320+0000 7fe863400640 1 -- 172.21.15.31:0/2506562115 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+16708 (secure 0 0 0) 0x7fe8640669d0 con 0x7fe86c15dfc0 2024-09-13T08:42:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:49.322 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:48.347453+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:46.132965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:47.310441+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 msgr2=0x7fe84807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 0x7fe84807ae30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe85c034000 tx=0x7fe85c004730 comp rx=0 tx=0).stop 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 msgr2=0x7fe86c09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c770 tx=0x7fe86400cc30 comp rx=0 tx=0).stop 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 shutdown_connections 2024-09-13T08:42:49.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe848078970 0x7fe84807ae30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c15dfc0 0x7fe86c09aa10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe86c0ca5c0 0x7fe86c0ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 --2- 172.21.15.31:0/2506562115 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe86c0b1910 0x7fe86c0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 >> 172.21.15.31:0/2506562115 conn(0x7fe86c0cc910 msgr2=0x7fe86c0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 shutdown_connections 2024-09-13T08:42:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.323+0000 7fe871a00640 1 -- 172.21.15.31:0/2506562115 wait complete. 2024-09-13T08:42:49.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 -- 172.21.15.31:0/1492580697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a015dfc0 msgr2=0x7f89a00cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1492580697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a015dfc0 0x7f89a00cc300 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f89a0070b80 tx=0x7f899002a540 comp rx=0 tx=0).stop 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 -- 172.21.15.31:0/1492580697 shutdown_connections 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1492580697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a00b1910 0x7f89a00b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1492580697 >> v2:172.21.15.45:3300/0 conn(0x7f89a00ca5c0 0x7f89a00ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1492580697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a015dfc0 0x7f89a00cc300 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 -- 172.21.15.31:0/1492580697 >> 172.21.15.31:0/1492580697 conn(0x7f89a00cc910 msgr2=0x7f89a00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 -- 172.21.15.31:0/1492580697 shutdown_connections 2024-09-13T08:42:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.500+0000 7f89a4a00640 1 -- 172.21.15.31:0/1492580697 wait complete. 2024-09-13T08:42:49.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.502+0000 7f89a4a00640 1 Processor -- start 2024-09-13T08:42:49.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.502+0000 7f89a4a00640 1 -- start start 2024-09-13T08:42:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00ca5c0 0x7f89a0152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a015dfc0 0x7f89a00b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89a0073900 con 0x7f89a00b1910 2024-09-13T08:42:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89a0073780 con 0x7f89a00ca5c0 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f89a4a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f89a0073a80 con 0x7f89a015dfc0 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42486/0 (socket says 172.21.15.31:42486) 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 -- 172.21.15.31:0/1365591459 learned_addr learned my addr 172.21.15.31:0/1365591459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899e800640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a015dfc0 0x7f89a00b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f8997e00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00ca5c0 0x7f89a0152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 -- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a015dfc0 msgr2=0x7f89a00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a015dfc0 0x7f89a00b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 -- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00ca5c0 msgr2=0x7f89a0152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00ca5c0 0x7f89a0152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.503+0000 7f899de00640 1 -- 172.21.15.31:0/1365591459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a009ce00 con 0x7f89a00b1910 2024-09-13T08:42:49.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f899de00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f89900099c0 tx=0x7f89900039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899003e030 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89900047a0 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990035040 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a009d0e0 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.504+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a00b4250 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.505+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a0073f80 con 0x7f89a00b1910 2024-09-13T08:42:49.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.507+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f8990041030 con 0x7f89a00b1910 2024-09-13T08:42:49.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.507+0000 7f8996000640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 0x7f897c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.508+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f89900bb1e0 con 0x7f89a00b1910 2024-09-13T08:42:49.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.508+0000 7f8997e00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 0x7f897c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.508+0000 7f8997e00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 0x7f897c07ae30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8988005fd0 tx=0x7f8988005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.512+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f89900836c0 con 0x7f89a00b1910 2024-09-13T08:42:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.707+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89a00af5d0 con 0x7f89a00b1910 2024-09-13T08:42:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.709+0000 7f8996000640 1 -- 172.21.15.31:0/1365591459 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+16708 (secure 0 0 0) 0x7f8990082fc0 con 0x7f89a00b1910 2024-09-13T08:42:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:48.347453+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:46.132965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:47.310441+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 msgr2=0x7f897c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 0x7f897c07ae30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8988005fd0 tx=0x7f8988005850 comp rx=0 tx=0).stop 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 msgr2=0x7f89a01f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f89900099c0 tx=0x7f89900039e0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 shutdown_connections 2024-09-13T08:42:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f897c078970 0x7f897c07ae30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89a015dfc0 0x7f89a00b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00ca5c0 0x7f89a0152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 --2- 172.21.15.31:0/1365591459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a00b1910 0x7f89a01f59a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 >> 172.21.15.31:0/1365591459 conn(0x7f89a00cc910 msgr2=0x7f89a00d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 shutdown_connections 2024-09-13T08:42:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.711+0000 7f89a4a00640 1 -- 172.21.15.31:0/1365591459 wait complete. 2024-09-13T08:42:49.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:42:49.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 -- 172.21.15.31:0/870561535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc0d4930 msgr2=0x7f3bbc0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 --2- 172.21.15.31:0/870561535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc0b1070 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc08cdd0 tx=0x7f3bb402a980 comp rx=0 tx=0).stop 2024-09-13T08:42:49.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 -- 172.21.15.31:0/870561535 shutdown_connections 2024-09-13T08:42:49.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 --2- 172.21.15.31:0/870561535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc0b1070 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 --2- 172.21.15.31:0/870561535 >> v2:172.21.15.45:3300/0 conn(0x7f3bbc1613f0 0x7f3bbc0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 -- 172.21.15.31:0/870561535 >> 172.21.15.31:0/870561535 conn(0x7f3bbc0d3e10 msgr2=0x7f3bbc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 -- 172.21.15.31:0/870561535 shutdown_connections 2024-09-13T08:42:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.891+0000 7f3bc1800640 1 -- 172.21.15.31:0/870561535 wait complete. 2024-09-13T08:42:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.892+0000 7f3bc1800640 1 Processor -- start 2024-09-13T08:42:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.893+0000 7f3bc1800640 1 -- start start 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.893+0000 7f3bc1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc1f8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.893+0000 7f3bc1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bc1800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3bbc1617d0 0x7f3bbc0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bc1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc0b1940 con 0x7f3bbc1613f0 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bc1800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc0b17c0 con 0x7f3bbc0d4930 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bc1800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3bbc0b1ac0 con 0x7f3bbc1617d0 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42510/0 (socket says 172.21.15.31:42510) 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 -- 172.21.15.31:0/579556030 learned_addr learned my addr 172.21.15.31:0/579556030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbbe00640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc1f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bc0e00640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3bbc1617d0 0x7f3bbc0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 -- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3bbc1617d0 msgr2=0x7f3bbc0caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3bbc1617d0 0x7f3bbc0caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 -- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc0d4930 msgr2=0x7f3bbc1f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc1f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.894+0000 7f3bbb400640 1 -- 172.21.15.31:0/579556030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc14e660 con 0x7f3bbc1613f0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bbb400640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800b3c0 tx=0x7f3ba800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8011830 con 0x7f3bbc1613f0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ba8016070 con 0x7f3bbc1613f0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bbc1515e0 con 0x7f3bbc1613f0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8011da0 con 0x7f3bbc1613f0 2024-09-13T08:42:49.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.895+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bbc1f7c60 con 0x7f3bbc1613f0 2024-09-13T08:42:49.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.896+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bbc0b1070 con 0x7f3bbc1613f0 2024-09-13T08:42:49.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.897+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f3ba801b460 con 0x7f3bbc1613f0 2024-09-13T08:42:49.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.898+0000 7f3bb3e00640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 0x7f3b9c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:49.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.898+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f3ba809a550 con 0x7f3bbc1613f0 2024-09-13T08:42:49.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.898+0000 7f3bbbe00640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 0x7f3b9c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:49.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.899+0000 7f3bbbe00640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 0x7f3b9c07ae30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3bac005fd0 tx=0x7f3bac00b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:49.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:49.902+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3ba8020020 con 0x7f3bbc1613f0 2024-09-13T08:42:50.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.140+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3bbc06dff0 con 0x7f3bbc1613f0 2024-09-13T08:42:50.882 INFO:tasks.ceph.mds.d.smithi045.stderr:2024-09-13T08:42:50.882+0000 7f2120000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:42:50.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.923+0000 7f3bb3e00640 1 -- 172.21.15.31:0/579556030 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v70) ==== 118+0+0 (secure 0 0 0) 0x7f3ba80622b0 con 0x7f3bbc1613f0 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 msgr2=0x7f3b9c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 0x7f3b9c07ae30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3bac005fd0 tx=0x7f3bac00b390 comp rx=0 tx=0).stop 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 msgr2=0x7f3bbc14e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800b3c0 tx=0x7f3ba800b880 comp rx=0 tx=0).stop 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 shutdown_connections 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3b9c078970 0x7f3b9c07ae30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3bbc1617d0 0x7f3bbc0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bbc1613f0 0x7f3bbc14e120 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 --2- 172.21.15.31:0/579556030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc0d4930 0x7f3bbc1f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 >> 172.21.15.31:0/579556030 conn(0x7f3bbc0d3e10 msgr2=0x7f3bbc0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 shutdown_connections 2024-09-13T08:42:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:50.926+0000 7f3bc1800640 1 -- 172.21.15.31:0/579556030 wait complete. 2024-09-13T08:42:50.947 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:42:50.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 -- 172.21.15.31:0/1159408471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f705806fa60 msgr2=0x7f70580cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 --2- 172.21.15.31:0/1159408471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f705806fa60 0x7f70580cb740 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7044005fd0 tx=0x7f704402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 -- 172.21.15.31:0/1159408471 shutdown_connections 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 --2- 172.21.15.31:0/1159408471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f705806fa60 0x7f70580cb740 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 --2- 172.21.15.31:0/1159408471 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 0x7f7058072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.119+0000 7f705f000640 1 -- 172.21.15.31:0/1159408471 >> 172.21.15.31:0/1159408471 conn(0x7f70580cfbb0 msgr2=0x7f70580cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.120+0000 7f705f000640 1 -- 172.21.15.31:0/1159408471 shutdown_connections 2024-09-13T08:42:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.120+0000 7f705f000640 1 -- 172.21.15.31:0/1159408471 wait complete. 2024-09-13T08:42:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.121+0000 7f705f000640 1 Processor -- start 2024-09-13T08:42:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.121+0000 7f705f000640 1 -- start start 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806fa60 0x7f7058142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 0x7f70580c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70580b3270 con 0x7f7058149720 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70580b30f0 con 0x7f705806fa60 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.122+0000 7f705f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f70580b33f0 con 0x7f705815bfc0 2024-09-13T08:42:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42522/0 (socket says 172.21.15.31:42522) 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 -- 172.21.15.31:0/3189352396 learned_addr learned my addr 172.21.15.31:0/3189352396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f7057e00640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806fa60 0x7f7058142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f7057400640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 0x7f70580c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 -- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 msgr2=0x7f70580c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 0x7f70580c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 -- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806fa60 msgr2=0x7f7058142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806fa60 0x7f7058142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 -- 172.21.15.31:0/3189352396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70580c9e30 con 0x7f7058149720 2024-09-13T08:42:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.123+0000 7f705d000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f705000c7a0 tx=0x7f705000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.124+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050003b70 con 0x7f7058149720 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.124+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7050003d30 con 0x7f7058149720 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.124+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70581437d0 con 0x7f7058149720 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.124+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705000aab0 con 0x7f7058149720 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.124+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7058143d90 con 0x7f7058149720 2024-09-13T08:42:51.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.125+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70580ca000 con 0x7f7058149720 2024-09-13T08:42:51.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.126+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f705000ac50 con 0x7f7058149720 2024-09-13T08:42:51.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.127+0000 7f704fe00640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 0x7f703807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:51.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.127+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f705009a8e0 con 0x7f7058149720 2024-09-13T08:42:51.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.127+0000 7f7057e00640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 0x7f703807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:51.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.128+0000 7f7057e00640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 0x7f703807ae60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f704800bb50 tx=0x7f704800b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:51.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.132+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f70580ca000 con 0x7f7058149720 2024-09-13T08:42:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.364+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f70580b2f10 con 0x7f7058149720 2024-09-13T08:42:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.924+0000 7f704fe00640 1 -- 172.21.15.31:0/3189352396 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v71) ==== 122+0+0 (secure 0 0 0) 0x7f70580b2f10 con 0x7f7058149720 2024-09-13T08:42:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 msgr2=0x7f703807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 0x7f703807ae60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f704800bb50 tx=0x7f704800b2e0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 msgr2=0x7f70580c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f705000c7a0 tx=0x7f705000cc60 comp rx=0 tx=0).stop 2024-09-13T08:42:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 shutdown_connections 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f70380789a0 0x7f703807ae60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7058149720 0x7f70580c6f00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f705815bfc0 0x7f70580c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 --2- 172.21.15.31:0/3189352396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f705806fa60 0x7f7058142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 >> 172.21.15.31:0/3189352396 conn(0x7f70580cfbb0 msgr2=0x7f7058073ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.927+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 shutdown_connections 2024-09-13T08:42:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:51.928+0000 7f705f000640 1 -- 172.21.15.31:0/3189352396 wait complete. 2024-09-13T08:42:51.947 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:42:51.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.109+0000 7f0d9f200640 1 -- 172.21.15.31:0/949593605 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d9815dfc0 msgr2=0x7f0d980cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.109+0000 7f0d9f200640 1 --2- 172.21.15.31:0/949593605 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d9815dfc0 0x7f0d980cc300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009860 tx=0x7f0d8802a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.110+0000 7f0d9f200640 1 -- 172.21.15.31:0/949593605 shutdown_connections 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.110+0000 7f0d9f200640 1 --2- 172.21.15.31:0/949593605 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d9815dfc0 0x7f0d980cc300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.110+0000 7f0d9f200640 1 -- 172.21.15.31:0/949593605 >> 172.21.15.31:0/949593605 conn(0x7f0d980cc910 msgr2=0x7f0d980cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.110+0000 7f0d9f200640 1 -- 172.21.15.31:0/949593605 shutdown_connections 2024-09-13T08:42:52.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.110+0000 7f0d9f200640 1 -- 172.21.15.31:0/949593605 wait complete. 2024-09-13T08:42:52.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.111+0000 7f0d9f200640 1 Processor -- start 2024-09-13T08:42:52.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.111+0000 7f0d9f200640 1 -- start start 2024-09-13T08:42:52.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d980b1910 0x7f0d98150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d980ca5c0 0x7f0d980b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d980b1da0 con 0x7f0d9815dfc0 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d980740a0 con 0x7f0d980b1910 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0d980b1f20 con 0x7f0d980ca5c0 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.112+0000 7f0d9d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d97e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d980b1910 0x7f0d98150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42532/0 (socket says 172.21.15.31:42532) 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 -- 172.21.15.31:0/2324310683 learned_addr learned my addr 172.21.15.31:0/2324310683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d97400640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d980ca5c0 0x7f0d980b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 -- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d980ca5c0 msgr2=0x7f0d980b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d980ca5c0 0x7f0d980b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 -- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d980b1910 msgr2=0x7f0d98150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d980b1910 0x7f0d98150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 -- 172.21.15.31:0/2324310683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d980b4c60 con 0x7f0d9815dfc0 2024-09-13T08:42:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d9d200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000cbc0 tx=0x7f0d9000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.113+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90016020 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.114+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d980b4f20 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.114+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d9000ae30 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.114+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90012680 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.114+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9809ae70 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.115+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d980cc300 con 0x7f0d9815dfc0 2024-09-13T08:42:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.116+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f0d90019020 con 0x7f0d9815dfc0 2024-09-13T08:42:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.117+0000 7f0d8fe00640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 0x7f0d7807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.117+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f0d9009b0c0 con 0x7f0d9815dfc0 2024-09-13T08:42:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.117+0000 7f0d97e00640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 0x7f0d7807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.118+0000 7f0d97e00640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 0x7f0d7807ae60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0d88034000 tx=0x7f0d88004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.121+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0d90020020 con 0x7f0d9815dfc0 2024-09-13T08:42:52.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.356+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d980b0750 con 0x7f0d9815dfc0 2024-09-13T08:42:52.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.357+0000 7f0d8fe00640 1 -- 172.21.15.31:0/2324310683 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5222 (secure 0 0 0) 0x7f0d980cbe40 con 0x7f0d9815dfc0 2024-09-13T08:42:52.358 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:52.358 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"btime":"2024-09-13T08:42:51:923681+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:42:52.358 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 msgr2=0x7f0d7807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 0x7f0d7807ae60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0d88034000 tx=0x7f0d88004730 comp rx=0 tx=0).stop 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 msgr2=0x7f0d980b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000cbc0 tx=0x7f0d9000a430 comp rx=0 tx=0).stop 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 shutdown_connections 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d780789a0 0x7f0d7807ae60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d9815dfc0 0x7f0d980b3200 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d980ca5c0 0x7f0d980b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 --2- 172.21.15.31:0/2324310683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d980b1910 0x7f0d98150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 >> 172.21.15.31:0/2324310683 conn(0x7f0d980cc910 msgr2=0x7f0d980b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 shutdown_connections 2024-09-13T08:42:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.359+0000 7f0d9f200640 1 -- 172.21.15.31:0/2324310683 wait complete. 2024-09-13T08:42:52.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:42:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 -- 172.21.15.31:0/1865508856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c0cf850 msgr2=0x7ffb4c0ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1865508856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c0cf850 0x7ffb4c0ce3c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0ca470 tx=0x7ffb3802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 -- 172.21.15.31:0/1865508856 shutdown_connections 2024-09-13T08:42:52.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1865508856 >> v2:172.21.15.45:3300/0 conn(0x7ffb4c0ce900 0x7ffb4c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1865508856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c0cf850 0x7ffb4c0ce3c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 -- 172.21.15.31:0/1865508856 >> 172.21.15.31:0/1865508856 conn(0x7ffb4c0cc910 msgr2=0x7ffb4c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 -- 172.21.15.31:0/1865508856 shutdown_connections 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.536+0000 7ffb53c00640 1 -- 172.21.15.31:0/1865508856 wait complete. 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.538+0000 7ffb53c00640 1 Processor -- start 2024-09-13T08:42:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.538+0000 7ffb53c00640 1 -- start start 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.538+0000 7ffb53c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb4c0ce900 0x7ffb4c14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb53c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb4c0cf850 0x7ffb4c1f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb53c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb53c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4c1600f0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb53c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4c0cb850 con 0x7ffb4c0ce900 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb53c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffb4c09d700 con 0x7ffb4c0cf850 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42550/0 (socket says 172.21.15.31:42550) 2024-09-13T08:42:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 -- 172.21.15.31:0/1469645898 learned_addr learned my addr 172.21.15.31:0/1469645898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51200640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb4c0ce900 0x7ffb4c14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb4be00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb4c0cf850 0x7ffb4c1f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 -- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb4c0cf850 msgr2=0x7ffb4c1f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb4c0cf850 0x7ffb4c1f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 -- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb4c0ce900 msgr2=0x7ffb4c14dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb4c0ce900 0x7ffb4c14dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.539+0000 7ffb51c00640 1 -- 172.21.15.31:0/1469645898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4c09d3d0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb51c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400eb10 tx=0x7ffb4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44004b20 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb44004ce0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44005280 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4c0b5600 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.540+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb4c0b2cc0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.541+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb4c0ce3c0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.542+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7ffb44005420 con 0x7ffb4c15bd70 2024-09-13T08:42:52.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.543+0000 7ffb48c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 0x7ffb2c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.543+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7ffb4409b7b0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.543+0000 7ffb51200640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 0x7ffb2c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.544+0000 7ffb51200640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 0x7ffb2c07ae30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb40001720 tx=0x7ffb40009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.547+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffb440635a0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.743+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb4c09d880 con 0x7ffb4c15bd70 2024-09-13T08:42:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.745+0000 7ffb48c00640 1 -- 172.21.15.31:0/1469645898 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+16695 (secure 0 0 0) 0x7ffb44067fd0 con 0x7ffb4c15bd70 2024-09-13T08:42:52.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:52.746 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":108,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:42:49.909939+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:46.132965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:47.310441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 msgr2=0x7ffb2c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 0x7ffb2c07ae30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb40001720 tx=0x7ffb40009040 comp rx=0 tx=0).stop 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 msgr2=0x7ffb4c1f8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400eb10 tx=0x7ffb4400c2d0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 shutdown_connections 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffb2c078970 0x7ffb2c07ae30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb4c15bd70 0x7ffb4c1f8670 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffb4c0cf850 0x7ffb4c1f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 --2- 172.21.15.31:0/1469645898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb4c0ce900 0x7ffb4c14dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 >> 172.21.15.31:0/1469645898 conn(0x7ffb4c0cc910 msgr2=0x7ffb4c0cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 shutdown_connections 2024-09-13T08:42:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.747+0000 7ffb53c00640 1 -- 172.21.15.31:0/1469645898 wait complete. 2024-09-13T08:42:52.764 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:42:52.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.925+0000 7f541be00640 1 -- 172.21.15.31:0/1445549771 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 msgr2=0x7f541c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.925+0000 7f541be00640 1 --2- 172.21.15.31:0/1445549771 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c0cf950 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f540c005fd0 tx=0x7f540c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 -- 172.21.15.31:0/1445549771 shutdown_connections 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 --2- 172.21.15.31:0/1445549771 >> v2:172.21.15.45:3300/0 conn(0x7f541c0ce470 0x7f541c0728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 --2- 172.21.15.31:0/1445549771 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c0cf950 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 -- 172.21.15.31:0/1445549771 >> 172.21.15.31:0/1445549771 conn(0x7f541c0b1980 msgr2=0x7f541c0b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 -- 172.21.15.31:0/1445549771 shutdown_connections 2024-09-13T08:42:52.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.926+0000 7f541be00640 1 -- 172.21.15.31:0/1445549771 wait complete. 2024-09-13T08:42:52.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.927+0000 7f541be00640 1 Processor -- start 2024-09-13T08:42:52.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.927+0000 7f541be00640 1 -- start start 2024-09-13T08:42:52.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 0x7f541c1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0ae7f0 con 0x7f541c072de0 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0ae670 con 0x7f541c0ce470 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.928+0000 7f541be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f541c0ae970 con 0x7f541c15dfc0 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f541aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f541aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42560/0 (socket says 172.21.15.31:42560) 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f541aa00640 1 -- 172.21.15.31:0/4052166336 learned_addr learned my addr 172.21.15.31:0/4052166336 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 0x7f541c1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f541b400640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 -- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 msgr2=0x7f541c155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 -- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 msgr2=0x7f541c15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f5412000640 1 -- 172.21.15.31:0/4052166336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c155c70 con 0x7f541c0ce470 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.929+0000 7f541aa00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f5412000640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 0x7f541c1551f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f540400e9c0 tx=0x7f540400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540400ccc0 con 0x7f541c0ce470 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5404004020 con 0x7f541c0ce470 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c0c6f60 con 0x7f541c0ce470 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540400ce60 con 0x7f541c0ce470 2024-09-13T08:42:52.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.930+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c09af40 con 0x7f541c0ce470 2024-09-13T08:42:52.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.931+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c0728a0 con 0x7f541c0ce470 2024-09-13T08:42:52.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.936+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f54040035f0 con 0x7f541c0ce470 2024-09-13T08:42:52.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.937+0000 7f5413400640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 0x7f53f807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:52.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.937+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f540409b5f0 con 0x7f541c0ce470 2024-09-13T08:42:52.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.937+0000 7f541aa00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 0x7f53f807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:52.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.938+0000 7f541aa00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 0x7f53f807ae30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f540c009230 tx=0x7f540c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:52.940+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5404063460 con 0x7f541c0ce470 2024-09-13T08:42:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.132+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f541c0b5cd0 con 0x7f53f8078970 2024-09-13T08:42:53.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.138+0000 7f5413400640 1 -- 172.21.15.31:0/4052166336 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7f541c0b5cd0 con 0x7f53f8078970 2024-09-13T08:42:53.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:53.143 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 msgr2=0x7f53f807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 0x7f53f807ae30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f540c009230 tx=0x7f540c004730 comp rx=0 tx=0).stop 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 msgr2=0x7f541c1551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 0x7f541c1551f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f540400e9c0 tx=0x7f540400ee80 comp rx=0 tx=0).stop 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 shutdown_connections 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53f8078970 0x7f53f807ae30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f541c15dfc0 0x7f541c155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f541c0ce470 0x7f541c1551f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 --2- 172.21.15.31:0/4052166336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c072de0 0x7f541c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 >> 172.21.15.31:0/4052166336 conn(0x7f541c0b1980 msgr2=0x7f541c070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 shutdown_connections 2024-09-13T08:42:53.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.142+0000 7f541be00640 1 -- 172.21.15.31:0/4052166336 wait complete. 2024-09-13T08:42:53.158 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T08:42:52.971049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265352,"kb_used_data":11656,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709368,"statfs":{"total":773094113280,"available":772822392832,"internally_reserved":0,"allocated":11935744,"data_stored":5873564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001120"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215326+0000","last_change":"2024-09-13T08:42:48.929961+0000","last_active":"2024-09-13T08:42:50.215326+0000","last_peered":"2024-09-13T08:42:50.215326+0000","last_clean":"2024-09-13T08:42:50.215326+0000","last_became_active":"2024-09-13T08:42:48.928616+0000","last_became_peered":"2024-09-13T08:42:48.928616+0000","last_unstale":"2024-09-13T08:42:50.215326+0000","last_undegraded":"2024-09-13T08:42:50.215326+0000","last_fullsized":"2024-09-13T08:42:50.215326+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:07.847124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683351+0000","last_change":"2024-09-13T08:42:48.929056+0000","last_active":"2024-09-13T08:42:50.683351+0000","last_peered":"2024-09-13T08:42:50.683351+0000","last_clean":"2024-09-13T08:42:50.683351+0000","last_became_active":"2024-09-13T08:42:48.928297+0000","last_became_peered":"2024-09-13T08:42:48.928297+0000","last_unstale":"2024-09-13T08:42:50.683351+0000","last_undegraded":"2024-09-13T08:42:50.683351+0000","last_fullsized":"2024-09-13T08:42:50.683351+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:04.844276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715977+0000","last_change":"2024-09-13T08:42:48.919137+0000","last_active":"2024-09-13T08:42:50.715977+0000","last_peered":"2024-09-13T08:42:50.715977+0000","last_clean":"2024-09-13T08:42:50.715977+0000","last_became_active":"2024-09-13T08:42:48.918876+0000","last_became_peered":"2024-09-13T08:42:48.918876+0000","last_unstale":"2024-09-13T08:42:50.715977+0000","last_undegraded":"2024-09-13T08:42:50.715977+0000","last_fullsized":"2024-09-13T08:42:50.715977+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:44.983616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919881+0000","last_change":"2024-09-13T08:42:48.933418+0000","last_active":"2024-09-13T08:42:49.919881+0000","last_peered":"2024-09-13T08:42:49.919881+0000","last_clean":"2024-09-13T08:42:49.919881+0000","last_became_active":"2024-09-13T08:42:48.927797+0000","last_became_peered":"2024-09-13T08:42:48.927797+0000","last_unstale":"2024-09-13T08:42:49.919881+0000","last_undegraded":"2024-09-13T08:42:49.919881+0000","last_fullsized":"2024-09-13T08:42:49.919881+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:53.609294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921533+0000","last_change":"2024-09-13T08:42:48.929725+0000","last_active":"2024-09-13T08:42:49.921533+0000","last_peered":"2024-09-13T08:42:49.921533+0000","last_clean":"2024-09-13T08:42:49.921533+0000","last_became_active":"2024-09-13T08:42:48.929450+0000","last_became_peered":"2024-09-13T08:42:48.929450+0000","last_unstale":"2024-09-13T08:42:49.921533+0000","last_undegraded":"2024-09-13T08:42:49.921533+0000","last_fullsized":"2024-09-13T08:42:49.921533+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:56:42.644926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189384+0000","last_change":"2024-09-13T08:42:48.931621+0000","last_active":"2024-09-13T08:42:50.189384+0000","last_peered":"2024-09-13T08:42:50.189384+0000","last_clean":"2024-09-13T08:42:50.189384+0000","last_became_active":"2024-09-13T08:42:48.931364+0000","last_became_peered":"2024-09-13T08:42:48.931364+0000","last_unstale":"2024-09-13T08:42:50.189384+0000","last_undegraded":"2024-09-13T08:42:50.189384+0000","last_fullsized":"2024-09-13T08:42:50.189384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:24.122788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715569+0000","last_change":"2024-09-13T08:42:48.924322+0000","last_active":"2024-09-13T08:42:50.715569+0000","last_peered":"2024-09-13T08:42:50.715569+0000","last_clean":"2024-09-13T08:42:50.715569+0000","last_became_active":"2024-09-13T08:42:48.922743+0000","last_became_peered":"2024-09-13T08:42:48.922743+0000","last_unstale":"2024-09-13T08:42:50.715569+0000","last_undegraded":"2024-09-13T08:42:50.715569+0000","last_fullsized":"2024-09-13T08:42:50.715569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:14.685145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716093+0000","last_change":"2024-09-13T08:42:48.924769+0000","last_active":"2024-09-13T08:42:50.716093+0000","last_peered":"2024-09-13T08:42:50.716093+0000","last_clean":"2024-09-13T08:42:50.716093+0000","last_became_active":"2024-09-13T08:42:48.922797+0000","last_became_peered":"2024-09-13T08:42:48.922797+0000","last_unstale":"2024-09-13T08:42:50.716093+0000","last_undegraded":"2024-09-13T08:42:50.716093+0000","last_fullsized":"2024-09-13T08:42:50.716093+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:27.340931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215331+0000","last_change":"2024-09-13T08:42:48.937822+0000","last_active":"2024-09-13T08:42:50.215331+0000","last_peered":"2024-09-13T08:42:50.215331+0000","last_clean":"2024-09-13T08:42:50.215331+0000","last_became_active":"2024-09-13T08:42:48.935614+0000","last_became_peered":"2024-09-13T08:42:48.935614+0000","last_unstale":"2024-09-13T08:42:50.215331+0000","last_undegraded":"2024-09-13T08:42:50.215331+0000","last_fullsized":"2024-09-13T08:42:50.215331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:04.844276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921544+0000","last_change":"2024-09-13T08:42:48.937696+0000","last_active":"2024-09-13T08:42:49.921544+0000","last_peered":"2024-09-13T08:42:49.921544+0000","last_clean":"2024-09-13T08:42:49.921544+0000","last_became_active":"2024-09-13T08:42:48.935883+0000","last_became_peered":"2024-09-13T08:42:48.935883+0000","last_unstale":"2024-09-13T08:42:49.921544+0000","last_undegraded":"2024-09-13T08:42:49.921544+0000","last_fullsized":"2024-09-13T08:42:49.921544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:50.493189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715996+0000","last_change":"2024-09-13T08:42:48.922633+0000","last_active":"2024-09-13T08:42:50.715996+0000","last_peered":"2024-09-13T08:42:50.715996+0000","last_clean":"2024-09-13T08:42:50.715996+0000","last_became_active":"2024-09-13T08:42:48.922285+0000","last_became_peered":"2024-09-13T08:42:48.922285+0000","last_unstale":"2024-09-13T08:42:50.715996+0000","last_undegraded":"2024-09-13T08:42:50.715996+0000","last_fullsized":"2024-09-13T08:42:50.715996+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:56.090927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921633+0000","last_change":"2024-09-13T08:42:48.922921+0000","last_active":"2024-09-13T08:42:49.921633+0000","last_peered":"2024-09-13T08:42:49.921633+0000","last_clean":"2024-09-13T08:42:49.921633+0000","last_became_active":"2024-09-13T08:42:48.921548+0000","last_became_peered":"2024-09-13T08:42:48.921548+0000","last_unstale":"2024-09-13T08:42:49.921633+0000","last_undegraded":"2024-09-13T08:42:49.921633+0000","last_fullsized":"2024-09-13T08:42:49.921633+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:48.610803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683322+0000","last_change":"2024-09-13T08:42:48.928575+0000","last_active":"2024-09-13T08:42:50.683322+0000","last_peered":"2024-09-13T08:42:50.683322+0000","last_clean":"2024-09-13T08:42:50.683322+0000","last_became_active":"2024-09-13T08:42:48.928236+0000","last_became_peered":"2024-09-13T08:42:48.928236+0000","last_unstale":"2024-09-13T08:42:50.683322+0000","last_undegraded":"2024-09-13T08:42:50.683322+0000","last_fullsized":"2024-09-13T08:42:50.683322+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:24.645996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917595+0000","last_change":"2024-09-13T08:42:48.923449+0000","last_active":"2024-09-13T08:42:49.917595+0000","last_peered":"2024-09-13T08:42:49.917595+0000","last_clean":"2024-09-13T08:42:49.917595+0000","last_became_active":"2024-09-13T08:42:48.923154+0000","last_became_peered":"2024-09-13T08:42:48.923154+0000","last_unstale":"2024-09-13T08:42:49.917595+0000","last_undegraded":"2024-09-13T08:42:49.917595+0000","last_fullsized":"2024-09-13T08:42:49.917595+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:56.117626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214902+0000","last_change":"2024-09-13T08:42:48.931242+0000","last_active":"2024-09-13T08:42:50.214902+0000","last_peered":"2024-09-13T08:42:50.214902+0000","last_clean":"2024-09-13T08:42:50.214902+0000","last_became_active":"2024-09-13T08:42:48.928133+0000","last_became_peered":"2024-09-13T08:42:48.928133+0000","last_unstale":"2024-09-13T08:42:50.214902+0000","last_undegraded":"2024-09-13T08:42:50.214902+0000","last_fullsized":"2024-09-13T08:42:50.214902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:11.489289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919617+0000","last_change":"2024-09-13T08:42:48.922248+0000","last_active":"2024-09-13T08:42:49.919617+0000","last_peered":"2024-09-13T08:42:49.919617+0000","last_clean":"2024-09-13T08:42:49.919617+0000","last_became_active":"2024-09-13T08:42:48.921913+0000","last_became_peered":"2024-09-13T08:42:48.921913+0000","last_unstale":"2024-09-13T08:42:49.919617+0000","last_undegraded":"2024-09-13T08:42:49.919617+0000","last_fullsized":"2024-09-13T08:42:49.919617+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:20:00.412578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715714+0000","last_change":"2024-09-13T08:42:48.924914+0000","last_active":"2024-09-13T08:42:50.715714+0000","last_peered":"2024-09-13T08:42:50.715714+0000","last_clean":"2024-09-13T08:42:50.715714+0000","last_became_active":"2024-09-13T08:42:48.924543+0000","last_became_peered":"2024-09-13T08:42:48.924543+0000","last_unstale":"2024-09-13T08:42:50.715714+0000","last_undegraded":"2024-09-13T08:42:50.715714+0000","last_fullsized":"2024-09-13T08:42:50.715714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:16:01.045632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715941+0000","last_change":"2024-09-13T08:42:48.929226+0000","last_active":"2024-09-13T08:42:50.715941+0000","last_peered":"2024-09-13T08:42:50.715941+0000","last_clean":"2024-09-13T08:42:50.715941+0000","last_became_active":"2024-09-13T08:42:48.928937+0000","last_became_peered":"2024-09-13T08:42:48.928937+0000","last_unstale":"2024-09-13T08:42:50.715941+0000","last_undegraded":"2024-09-13T08:42:50.715941+0000","last_fullsized":"2024-09-13T08:42:50.715941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:54.429540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715893+0000","last_change":"2024-09-13T08:42:48.924814+0000","last_active":"2024-09-13T08:42:50.715893+0000","last_peered":"2024-09-13T08:42:50.715893+0000","last_clean":"2024-09-13T08:42:50.715893+0000","last_became_active":"2024-09-13T08:42:48.923990+0000","last_became_peered":"2024-09-13T08:42:48.923990+0000","last_unstale":"2024-09-13T08:42:50.715893+0000","last_undegraded":"2024-09-13T08:42:50.715893+0000","last_fullsized":"2024-09-13T08:42:50.715893+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:20:33.022118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715618+0000","last_change":"2024-09-13T08:42:48.927994+0000","last_active":"2024-09-13T08:42:50.715618+0000","last_peered":"2024-09-13T08:42:50.715618+0000","last_clean":"2024-09-13T08:42:50.715618+0000","last_became_active":"2024-09-13T08:42:48.927674+0000","last_became_peered":"2024-09-13T08:42:48.927674+0000","last_unstale":"2024-09-13T08:42:50.715618+0000","last_undegraded":"2024-09-13T08:42:50.715618+0000","last_fullsized":"2024-09-13T08:42:50.715618+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:35.885431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716233+0000","last_change":"2024-09-13T08:42:48.917950+0000","last_active":"2024-09-13T08:42:50.716233+0000","last_peered":"2024-09-13T08:42:50.716233+0000","last_clean":"2024-09-13T08:42:50.716233+0000","last_became_active":"2024-09-13T08:42:48.917613+0000","last_became_peered":"2024-09-13T08:42:48.917613+0000","last_unstale":"2024-09-13T08:42:50.716233+0000","last_undegraded":"2024-09-13T08:42:50.716233+0000","last_fullsized":"2024-09-13T08:42:50.716233+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:34.374825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189604+0000","last_change":"2024-09-13T08:42:48.923079+0000","last_active":"2024-09-13T08:42:50.189604+0000","last_peered":"2024-09-13T08:42:50.189604+0000","last_clean":"2024-09-13T08:42:50.189604+0000","last_became_active":"2024-09-13T08:42:48.922813+0000","last_became_peered":"2024-09-13T08:42:48.922813+0000","last_unstale":"2024-09-13T08:42:50.189604+0000","last_undegraded":"2024-09-13T08:42:50.189604+0000","last_fullsized":"2024-09-13T08:42:50.189604+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:35.241616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917613+0000","last_change":"2024-09-13T08:42:48.925871+0000","last_active":"2024-09-13T08:42:49.917613+0000","last_peered":"2024-09-13T08:42:49.917613+0000","last_clean":"2024-09-13T08:42:49.917613+0000","last_became_active":"2024-09-13T08:42:48.925464+0000","last_became_peered":"2024-09-13T08:42:48.925464+0000","last_unstale":"2024-09-13T08:42:49.917613+0000","last_undegraded":"2024-09-13T08:42:49.917613+0000","last_fullsized":"2024-09-13T08:42:49.917613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:11.489289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919088+0000","last_change":"2024-09-13T08:42:48.927441+0000","last_active":"2024-09-13T08:42:49.919088+0000","last_peered":"2024-09-13T08:42:49.919088+0000","last_clean":"2024-09-13T08:42:49.919088+0000","last_became_active":"2024-09-13T08:42:48.926848+0000","last_became_peered":"2024-09-13T08:42:48.926848+0000","last_unstale":"2024-09-13T08:42:49.919088+0000","last_undegraded":"2024-09-13T08:42:49.919088+0000","last_fullsized":"2024-09-13T08:42:49.919088+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:49.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715760+0000","last_change":"2024-09-13T08:42:48.913934+0000","last_active":"2024-09-13T08:42:50.715760+0000","last_peered":"2024-09-13T08:42:50.715760+0000","last_clean":"2024-09-13T08:42:50.715760+0000","last_became_active":"2024-09-13T08:42:48.911894+0000","last_became_peered":"2024-09-13T08:42:48.911894+0000","last_unstale":"2024-09-13T08:42:50.715760+0000","last_undegraded":"2024-09-13T08:42:50.715760+0000","last_fullsized":"2024-09-13T08:42:50.715760+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:47.541292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189747+0000","last_change":"2024-09-13T08:42:48.921453+0000","last_active":"2024-09-13T08:42:50.189747+0000","last_peered":"2024-09-13T08:42:50.189747+0000","last_clean":"2024-09-13T08:42:50.189747+0000","last_became_active":"2024-09-13T08:42:48.921160+0000","last_became_peered":"2024-09-13T08:42:48.921160+0000","last_unstale":"2024-09-13T08:42:50.189747+0000","last_undegraded":"2024-09-13T08:42:50.189747+0000","last_fullsized":"2024-09-13T08:42:50.189747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:05:25.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920299+0000","last_change":"2024-09-13T08:42:48.933960+0000","last_active":"2024-09-13T08:42:49.920299+0000","last_peered":"2024-09-13T08:42:49.920299+0000","last_clean":"2024-09-13T08:42:49.920299+0000","last_became_active":"2024-09-13T08:42:48.933648+0000","last_became_peered":"2024-09-13T08:42:48.933648+0000","last_unstale":"2024-09-13T08:42:49.920299+0000","last_undegraded":"2024-09-13T08:42:49.920299+0000","last_fullsized":"2024-09-13T08:42:49.920299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:39.551390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189473+0000","last_change":"2024-09-13T08:42:48.935046+0000","last_active":"2024-09-13T08:42:50.189473+0000","last_peered":"2024-09-13T08:42:50.189473+0000","last_clean":"2024-09-13T08:42:50.189473+0000","last_became_active":"2024-09-13T08:42:48.934787+0000","last_became_peered":"2024-09-13T08:42:48.934787+0000","last_unstale":"2024-09-13T08:42:50.189473+0000","last_undegraded":"2024-09-13T08:42:50.189473+0000","last_fullsized":"2024-09-13T08:42:50.189473+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:48.984880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917981+0000","last_change":"2024-09-13T08:42:48.922522+0000","last_active":"2024-09-13T08:42:49.917981+0000","last_peered":"2024-09-13T08:42:49.917981+0000","last_clean":"2024-09-13T08:42:49.917981+0000","last_became_active":"2024-09-13T08:42:48.922219+0000","last_became_peered":"2024-09-13T08:42:48.922219+0000","last_unstale":"2024-09-13T08:42:49.917981+0000","last_undegraded":"2024-09-13T08:42:49.917981+0000","last_fullsized":"2024-09-13T08:42:49.917981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:56.062750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715895+0000","last_change":"2024-09-13T08:42:48.917723+0000","last_active":"2024-09-13T08:42:50.715895+0000","last_peered":"2024-09-13T08:42:50.715895+0000","last_clean":"2024-09-13T08:42:50.715895+0000","last_became_active":"2024-09-13T08:42:48.917377+0000","last_became_peered":"2024-09-13T08:42:48.917377+0000","last_unstale":"2024-09-13T08:42:50.715895+0000","last_undegraded":"2024-09-13T08:42:50.715895+0000","last_fullsized":"2024-09-13T08:42:50.715895+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:54.222967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215011+0000","last_change":"2024-09-13T08:42:48.928732+0000","last_active":"2024-09-13T08:42:50.215011+0000","last_peered":"2024-09-13T08:42:50.215011+0000","last_clean":"2024-09-13T08:42:50.215011+0000","last_became_active":"2024-09-13T08:42:48.928455+0000","last_became_peered":"2024-09-13T08:42:48.928455+0000","last_unstale":"2024-09-13T08:42:50.215011+0000","last_undegraded":"2024-09-13T08:42:50.215011+0000","last_fullsized":"2024-09-13T08:42:50.215011+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:56.062750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920305+0000","last_change":"2024-09-13T08:42:48.931967+0000","last_active":"2024-09-13T08:42:49.920305+0000","last_peered":"2024-09-13T08:42:49.920305+0000","last_clean":"2024-09-13T08:42:49.920305+0000","last_became_active":"2024-09-13T08:42:48.931704+0000","last_became_peered":"2024-09-13T08:42:48.931704+0000","last_unstale":"2024-09-13T08:42:49.920305+0000","last_undegraded":"2024-09-13T08:42:49.920305+0000","last_fullsized":"2024-09-13T08:42:49.920305+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:13.636965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215482+0000","last_change":"2024-09-13T08:42:48.932873+0000","last_active":"2024-09-13T08:42:50.215482+0000","last_peered":"2024-09-13T08:42:50.215482+0000","last_clean":"2024-09-13T08:42:50.215482+0000","last_became_active":"2024-09-13T08:42:48.932619+0000","last_became_peered":"2024-09-13T08:42:48.932619+0000","last_unstale":"2024-09-13T08:42:50.215482+0000","last_undegraded":"2024-09-13T08:42:50.215482+0000","last_fullsized":"2024-09-13T08:42:50.215482+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:29.023419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":19,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918153+0000","last_change":"2024-09-13T08:42:47.919293+0000","last_active":"2024-09-13T08:42:49.918153+0000","last_peered":"2024-09-13T08:42:49.918153+0000","last_clean":"2024-09-13T08:42:49.918153+0000","last_became_active":"2024-09-13T08:42:47.918970+0000","last_became_peered":"2024-09-13T08:42:47.918970+0000","last_unstale":"2024-09-13T08:42:49.918153+0000","last_undegraded":"2024-09-13T08:42:49.918153+0000","last_fullsized":"2024-09-13T08:42:49.918153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918316+0000","last_change":"2024-09-13T08:42:48.923056+0000","last_active":"2024-09-13T08:42:49.918316+0000","last_peered":"2024-09-13T08:42:49.918316+0000","last_clean":"2024-09-13T08:42:49.918316+0000","last_became_active":"2024-09-13T08:42:48.919206+0000","last_became_peered":"2024-09-13T08:42:48.919206+0000","last_unstale":"2024-09-13T08:42:49.918316+0000","last_undegraded":"2024-09-13T08:42:49.918316+0000","last_fullsized":"2024-09-13T08:42:49.918316+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:23.606208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683354+0000","last_change":"2024-09-13T08:42:47.918010+0000","last_active":"2024-09-13T08:42:50.683354+0000","last_peered":"2024-09-13T08:42:50.683354+0000","last_clean":"2024-09-13T08:42:50.683354+0000","last_became_active":"2024-09-13T08:42:47.917655+0000","last_became_peered":"2024-09-13T08:42:47.917655+0000","last_unstale":"2024-09-13T08:42:50.683354+0000","last_undegraded":"2024-09-13T08:42:50.683354+0000","last_fullsized":"2024-09-13T08:42:50.683354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:42.297105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189616+0000","last_change":"2024-09-13T08:42:48.917530+0000","last_active":"2024-09-13T08:42:50.189616+0000","last_peered":"2024-09-13T08:42:50.189616+0000","last_clean":"2024-09-13T08:42:50.189616+0000","last_became_active":"2024-09-13T08:42:48.917266+0000","last_became_peered":"2024-09-13T08:42:48.917266+0000","last_unstale":"2024-09-13T08:42:50.189616+0000","last_undegraded":"2024-09-13T08:42:50.189616+0000","last_fullsized":"2024-09-13T08:42:50.189616+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:26.850441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920122+0000","last_change":"2024-09-13T08:42:47.917801+0000","last_active":"2024-09-13T08:42:49.920122+0000","last_peered":"2024-09-13T08:42:49.920122+0000","last_clean":"2024-09-13T08:42:49.920122+0000","last_became_active":"2024-09-13T08:42:47.917374+0000","last_became_peered":"2024-09-13T08:42:47.917374+0000","last_unstale":"2024-09-13T08:42:49.920122+0000","last_undegraded":"2024-09-13T08:42:49.920122+0000","last_fullsized":"2024-09-13T08:42:49.920122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:51:37.702500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715596+0000","last_change":"2024-09-13T08:42:48.924729+0000","last_active":"2024-09-13T08:42:50.715596+0000","last_peered":"2024-09-13T08:42:50.715596+0000","last_clean":"2024-09-13T08:42:50.715596+0000","last_became_active":"2024-09-13T08:42:48.924355+0000","last_became_peered":"2024-09-13T08:42:48.924355+0000","last_unstale":"2024-09-13T08:42:50.715596+0000","last_undegraded":"2024-09-13T08:42:50.715596+0000","last_fullsized":"2024-09-13T08:42:50.715596+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:01.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919766+0000","last_change":"2024-09-13T08:42:47.918285+0000","last_active":"2024-09-13T08:42:49.919766+0000","last_peered":"2024-09-13T08:42:49.919766+0000","last_clean":"2024-09-13T08:42:49.919766+0000","last_became_active":"2024-09-13T08:42:47.917957+0000","last_became_peered":"2024-09-13T08:42:47.917957+0000","last_unstale":"2024-09-13T08:42:49.919766+0000","last_undegraded":"2024-09-13T08:42:49.919766+0000","last_fullsized":"2024-09-13T08:42:49.919766+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:06:24.941563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919780+0000","last_change":"2024-09-13T08:42:48.931997+0000","last_active":"2024-09-13T08:42:49.919780+0000","last_peered":"2024-09-13T08:42:49.919780+0000","last_clean":"2024-09-13T08:42:49.919780+0000","last_became_active":"2024-09-13T08:42:48.931719+0000","last_became_peered":"2024-09-13T08:42:48.931719+0000","last_unstale":"2024-09-13T08:42:49.919780+0000","last_undegraded":"2024-09-13T08:42:49.919780+0000","last_fullsized":"2024-09-13T08:42:49.919780+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:52.408508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715174+0000","last_change":"2024-09-13T08:42:47.925152+0000","last_active":"2024-09-13T08:42:50.715174+0000","last_peered":"2024-09-13T08:42:50.715174+0000","last_clean":"2024-09-13T08:42:50.715174+0000","last_became_active":"2024-09-13T08:42:47.924078+0000","last_became_peered":"2024-09-13T08:42:47.924078+0000","last_unstale":"2024-09-13T08:42:50.715174+0000","last_undegraded":"2024-09-13T08:42:50.715174+0000","last_fullsized":"2024-09-13T08:42:50.715174+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:19.406011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716193+0000","last_change":"2024-09-13T08:42:48.928511+0000","last_active":"2024-09-13T08:42:50.716193+0000","last_peered":"2024-09-13T08:42:50.716193+0000","last_clean":"2024-09-13T08:42:50.716193+0000","last_became_active":"2024-09-13T08:42:48.928088+0000","last_became_peered":"2024-09-13T08:42:48.928088+0000","last_unstale":"2024-09-13T08:42:50.716193+0000","last_undegraded":"2024-09-13T08:42:50.716193+0000","last_fullsized":"2024-09-13T08:42:50.716193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:19.260415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214856+0000","last_change":"2024-09-13T08:42:47.921240+0000","last_active":"2024-09-13T08:42:50.214856+0000","last_peered":"2024-09-13T08:42:50.214856+0000","last_clean":"2024-09-13T08:42:50.214856+0000","last_became_active":"2024-09-13T08:42:47.920848+0000","last_became_peered":"2024-09-13T08:42:47.920848+0000","last_unstale":"2024-09-13T08:42:50.214856+0000","last_undegraded":"2024-09-13T08:42:50.214856+0000","last_fullsized":"2024-09-13T08:42:50.214856+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:30.087222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920630+0000","last_change":"2024-09-13T08:42:48.938972+0000","last_active":"2024-09-13T08:42:49.920630+0000","last_peered":"2024-09-13T08:42:49.920630+0000","last_clean":"2024-09-13T08:42:49.920630+0000","last_became_active":"2024-09-13T08:42:48.936269+0000","last_became_peered":"2024-09-13T08:42:48.936269+0000","last_unstale":"2024-09-13T08:42:49.920630+0000","last_undegraded":"2024-09-13T08:42:49.920630+0000","last_fullsized":"2024-09-13T08:42:49.920630+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:36.591513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716007+0000","last_change":"2024-09-13T08:42:47.921139+0000","last_active":"2024-09-13T08:42:50.716007+0000","last_peered":"2024-09-13T08:42:50.716007+0000","last_clean":"2024-09-13T08:42:50.716007+0000","last_became_active":"2024-09-13T08:42:47.920668+0000","last_became_peered":"2024-09-13T08:42:47.920668+0000","last_unstale":"2024-09-13T08:42:50.716007+0000","last_undegraded":"2024-09-13T08:42:50.716007+0000","last_fullsized":"2024-09-13T08:42:50.716007+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:54.292629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716478+0000","last_change":"2024-09-13T08:42:48.913227+0000","last_active":"2024-09-13T08:42:50.716478+0000","last_peered":"2024-09-13T08:42:50.716478+0000","last_clean":"2024-09-13T08:42:50.716478+0000","last_became_active":"2024-09-13T08:42:48.912918+0000","last_became_peered":"2024-09-13T08:42:48.912918+0000","last_unstale":"2024-09-13T08:42:50.716478+0000","last_undegraded":"2024-09-13T08:42:50.716478+0000","last_fullsized":"2024-09-13T08:42:50.716478+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:46.559943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920770+0000","last_change":"2024-09-13T08:42:47.921384+0000","last_active":"2024-09-13T08:42:49.920770+0000","last_peered":"2024-09-13T08:42:49.920770+0000","last_clean":"2024-09-13T08:42:49.920770+0000","last_became_active":"2024-09-13T08:42:47.920811+0000","last_became_peered":"2024-09-13T08:42:47.920811+0000","last_unstale":"2024-09-13T08:42:49.920770+0000","last_undegraded":"2024-09-13T08:42:49.920770+0000","last_fullsized":"2024-09-13T08:42:49.920770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.124047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683313+0000","last_change":"2024-09-13T08:42:48.920834+0000","last_active":"2024-09-13T08:42:50.683313+0000","last_peered":"2024-09-13T08:42:50.683313+0000","last_clean":"2024-09-13T08:42:50.683313+0000","last_became_active":"2024-09-13T08:42:48.920374+0000","last_became_peered":"2024-09-13T08:42:48.920374+0000","last_unstale":"2024-09-13T08:42:50.683313+0000","last_undegraded":"2024-09-13T08:42:50.683313+0000","last_fullsized":"2024-09-13T08:42:50.683313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920066+0000","last_change":"2024-09-13T08:42:47.917210+0000","last_active":"2024-09-13T08:42:49.920066+0000","last_peered":"2024-09-13T08:42:49.920066+0000","last_clean":"2024-09-13T08:42:49.920066+0000","last_became_active":"2024-09-13T08:42:47.916867+0000","last_became_peered":"2024-09-13T08:42:47.916867+0000","last_unstale":"2024-09-13T08:42:49.920066+0000","last_undegraded":"2024-09-13T08:42:49.920066+0000","last_fullsized":"2024-09-13T08:42:49.920066+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:53.838545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683533+0000","last_change":"2024-09-13T08:42:48.924445+0000","last_active":"2024-09-13T08:42:50.683533+0000","last_peered":"2024-09-13T08:42:50.683533+0000","last_clean":"2024-09-13T08:42:50.683533+0000","last_became_active":"2024-09-13T08:42:48.924122+0000","last_became_peered":"2024-09-13T08:42:48.924122+0000","last_unstale":"2024-09-13T08:42:50.683533+0000","last_undegraded":"2024-09-13T08:42:50.683533+0000","last_fullsized":"2024-09-13T08:42:50.683533+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:09.464850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921328+0000","last_change":"2024-09-13T08:42:47.925885+0000","last_active":"2024-09-13T08:42:49.921328+0000","last_peered":"2024-09-13T08:42:49.921328+0000","last_clean":"2024-09-13T08:42:49.921328+0000","last_became_active":"2024-09-13T08:42:47.925509+0000","last_became_peered":"2024-09-13T08:42:47.925509+0000","last_unstale":"2024-09-13T08:42:49.921328+0000","last_undegraded":"2024-09-13T08:42:49.921328+0000","last_fullsized":"2024-09-13T08:42:49.921328+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:55:52.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189665+0000","last_change":"2024-09-13T08:42:48.920397+0000","last_active":"2024-09-13T08:42:50.189665+0000","last_peered":"2024-09-13T08:42:50.189665+0000","last_clean":"2024-09-13T08:42:50.189665+0000","last_became_active":"2024-09-13T08:42:48.920066+0000","last_became_peered":"2024-09-13T08:42:48.920066+0000","last_unstale":"2024-09-13T08:42:50.189665+0000","last_undegraded":"2024-09-13T08:42:50.189665+0000","last_fullsized":"2024-09-13T08:42:50.189665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:57:16.907452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921529+0000","last_change":"2024-09-13T08:42:47.916457+0000","last_active":"2024-09-13T08:42:49.921529+0000","last_peered":"2024-09-13T08:42:49.921529+0000","last_clean":"2024-09-13T08:42:49.921529+0000","last_became_active":"2024-09-13T08:42:47.916092+0000","last_became_peered":"2024-09-13T08:42:47.916092+0000","last_unstale":"2024-09-13T08:42:49.921529+0000","last_undegraded":"2024-09-13T08:42:49.921529+0000","last_fullsized":"2024-09-13T08:42:49.921529+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:57:06.575988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716194+0000","last_change":"2024-09-13T08:42:48.919573+0000","last_active":"2024-09-13T08:42:50.716194+0000","last_peered":"2024-09-13T08:42:50.716194+0000","last_clean":"2024-09-13T08:42:50.716194+0000","last_became_active":"2024-09-13T08:42:48.919312+0000","last_became_peered":"2024-09-13T08:42:48.919312+0000","last_unstale":"2024-09-13T08:42:50.716194+0000","last_undegraded":"2024-09-13T08:42:50.716194+0000","last_fullsized":"2024-09-13T08:42:50.716194+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:40.140756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":21,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919905+0000","last_change":"2024-09-13T08:42:47.918152+0000","last_active":"2024-09-13T08:42:49.919905+0000","last_peered":"2024-09-13T08:42:49.919905+0000","last_clean":"2024-09-13T08:42:49.919905+0000","last_became_active":"2024-09-13T08:42:47.916644+0000","last_became_peered":"2024-09-13T08:42:47.916644+0000","last_unstale":"2024-09-13T08:42:49.919905+0000","last_undegraded":"2024-09-13T08:42:49.919905+0000","last_fullsized":"2024-09-13T08:42:49.919905+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:28.579773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921047+0000","last_change":"2024-09-13T08:42:48.925074+0000","last_active":"2024-09-13T08:42:49.921047+0000","last_peered":"2024-09-13T08:42:49.921047+0000","last_clean":"2024-09-13T08:42:49.921047+0000","last_became_active":"2024-09-13T08:42:48.924774+0000","last_became_peered":"2024-09-13T08:42:48.924774+0000","last_unstale":"2024-09-13T08:42:49.921047+0000","last_undegraded":"2024-09-13T08:42:49.921047+0000","last_fullsized":"2024-09-13T08:42:49.921047+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:42.072741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921360+0000","last_change":"2024-09-13T08:42:47.918868+0000","last_active":"2024-09-13T08:42:49.921360+0000","last_peered":"2024-09-13T08:42:49.921360+0000","last_clean":"2024-09-13T08:42:49.921360+0000","last_became_active":"2024-09-13T08:42:47.918489+0000","last_became_peered":"2024-09-13T08:42:47.918489+0000","last_unstale":"2024-09-13T08:42:49.921360+0000","last_undegraded":"2024-09-13T08:42:49.921360+0000","last_fullsized":"2024-09-13T08:42:49.921360+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:58:51.693121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919275+0000","last_change":"2024-09-13T08:42:48.923950+0000","last_active":"2024-09-13T08:42:49.919275+0000","last_peered":"2024-09-13T08:42:49.919275+0000","last_clean":"2024-09-13T08:42:49.919275+0000","last_became_active":"2024-09-13T08:42:48.923646+0000","last_became_peered":"2024-09-13T08:42:48.923646+0000","last_unstale":"2024-09-13T08:42:49.919275+0000","last_undegraded":"2024-09-13T08:42:49.919275+0000","last_fullsized":"2024-09-13T08:42:49.919275+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:48:23.745822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715335+0000","last_change":"2024-09-13T08:42:47.927451+0000","last_active":"2024-09-13T08:42:50.715335+0000","last_peered":"2024-09-13T08:42:50.715335+0000","last_clean":"2024-09-13T08:42:50.715335+0000","last_became_active":"2024-09-13T08:42:47.927172+0000","last_became_peered":"2024-09-13T08:42:47.927172+0000","last_unstale":"2024-09-13T08:42:50.715335+0000","last_undegraded":"2024-09-13T08:42:50.715335+0000","last_fullsized":"2024-09-13T08:42:50.715335+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:08:13.970546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919469+0000","last_change":"2024-09-13T08:42:48.919556+0000","last_active":"2024-09-13T08:42:49.919469+0000","last_peered":"2024-09-13T08:42:49.919469+0000","last_clean":"2024-09-13T08:42:49.919469+0000","last_became_active":"2024-09-13T08:42:48.912870+0000","last_became_peered":"2024-09-13T08:42:48.912870+0000","last_unstale":"2024-09-13T08:42:49.919469+0000","last_undegraded":"2024-09-13T08:42:49.919469+0000","last_fullsized":"2024-09-13T08:42:49.919469+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:16.946594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":23,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214802+0000","last_change":"2024-09-13T08:42:47.924945+0000","last_active":"2024-09-13T08:42:50.214802+0000","last_peered":"2024-09-13T08:42:50.214802+0000","last_clean":"2024-09-13T08:42:50.214802+0000","last_became_active":"2024-09-13T08:42:47.923813+0000","last_became_peered":"2024-09-13T08:42:47.923813+0000","last_unstale":"2024-09-13T08:42:50.214802+0000","last_undegraded":"2024-09-13T08:42:50.214802+0000","last_fullsized":"2024-09-13T08:42:50.214802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:43:10.521549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215574+0000","last_change":"2024-09-13T08:42:48.929937+0000","last_active":"2024-09-13T08:42:50.215574+0000","last_peered":"2024-09-13T08:42:50.215574+0000","last_clean":"2024-09-13T08:42:50.215574+0000","last_became_active":"2024-09-13T08:42:48.929452+0000","last_became_peered":"2024-09-13T08:42:48.929452+0000","last_unstale":"2024-09-13T08:42:50.215574+0000","last_undegraded":"2024-09-13T08:42:50.215574+0000","last_fullsized":"2024-09-13T08:42:50.215574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:23.606208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189426+0000","last_change":"2024-09-13T08:42:47.924495+0000","last_active":"2024-09-13T08:42:50.189426+0000","last_peered":"2024-09-13T08:42:50.189426+0000","last_clean":"2024-09-13T08:42:50.189426+0000","last_became_active":"2024-09-13T08:42:47.922905+0000","last_became_peered":"2024-09-13T08:42:47.922905+0000","last_unstale":"2024-09-13T08:42:50.189426+0000","last_undegraded":"2024-09-13T08:42:50.189426+0000","last_fullsized":"2024-09-13T08:42:50.189426+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:20.346046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215580+0000","last_change":"2024-09-13T08:42:48.935881+0000","last_active":"2024-09-13T08:42:50.215580+0000","last_peered":"2024-09-13T08:42:50.215580+0000","last_clean":"2024-09-13T08:42:50.215580+0000","last_became_active":"2024-09-13T08:42:48.931944+0000","last_became_peered":"2024-09-13T08:42:48.931944+0000","last_unstale":"2024-09-13T08:42:50.215580+0000","last_undegraded":"2024-09-13T08:42:50.215580+0000","last_fullsized":"2024-09-13T08:42:50.215580+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:14:39.849271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921532+0000","last_change":"2024-09-13T08:42:47.921550+0000","last_active":"2024-09-13T08:42:49.921532+0000","last_peered":"2024-09-13T08:42:49.921532+0000","last_clean":"2024-09-13T08:42:49.921532+0000","last_became_active":"2024-09-13T08:42:47.921246+0000","last_became_peered":"2024-09-13T08:42:47.921246+0000","last_unstale":"2024-09-13T08:42:49.921532+0000","last_undegraded":"2024-09-13T08:42:49.921532+0000","last_fullsized":"2024-09-13T08:42:49.921532+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:36.292928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715869+0000","last_change":"2024-09-13T08:42:48.917043+0000","last_active":"2024-09-13T08:42:50.715869+0000","last_peered":"2024-09-13T08:42:50.715869+0000","last_clean":"2024-09-13T08:42:50.715869+0000","last_became_active":"2024-09-13T08:42:48.916730+0000","last_became_peered":"2024-09-13T08:42:48.916730+0000","last_unstale":"2024-09-13T08:42:50.715869+0000","last_undegraded":"2024-09-13T08:42:50.715869+0000","last_fullsized":"2024-09-13T08:42:50.715869+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:44:06.469897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918228+0000","last_change":"2024-09-13T08:42:47.918580+0000","last_active":"2024-09-13T08:42:49.918228+0000","last_peered":"2024-09-13T08:42:49.918228+0000","last_clean":"2024-09-13T08:42:49.918228+0000","last_became_active":"2024-09-13T08:42:47.918284+0000","last_became_peered":"2024-09-13T08:42:47.918284+0000","last_unstale":"2024-09-13T08:42:49.918228+0000","last_undegraded":"2024-09-13T08:42:49.918228+0000","last_fullsized":"2024-09-13T08:42:49.918228+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:00.623413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918159+0000","last_change":"2024-09-13T08:42:48.923844+0000","last_active":"2024-09-13T08:42:49.918159+0000","last_peered":"2024-09-13T08:42:49.918159+0000","last_clean":"2024-09-13T08:42:49.918159+0000","last_became_active":"2024-09-13T08:42:48.923266+0000","last_became_peered":"2024-09-13T08:42:48.923266+0000","last_unstale":"2024-09-13T08:42:49.918159+0000","last_undegraded":"2024-09-13T08:42:49.918159+0000","last_fullsized":"2024-09-13T08:42:49.918159+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:58.849052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917881+0000","last_change":"2024-09-13T08:42:47.913763+0000","last_active":"2024-09-13T08:42:49.917881+0000","last_peered":"2024-09-13T08:42:49.917881+0000","last_clean":"2024-09-13T08:42:49.917881+0000","last_became_active":"2024-09-13T08:42:47.913459+0000","last_became_peered":"2024-09-13T08:42:47.913459+0000","last_unstale":"2024-09-13T08:42:49.917881+0000","last_undegraded":"2024-09-13T08:42:49.917881+0000","last_fullsized":"2024-09-13T08:42:49.917881+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:34.882599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715273+0000","last_change":"2024-09-13T08:42:48.917417+0000","last_active":"2024-09-13T08:42:50.715273+0000","last_peered":"2024-09-13T08:42:50.715273+0000","last_clean":"2024-09-13T08:42:50.715273+0000","last_became_active":"2024-09-13T08:42:48.917137+0000","last_became_peered":"2024-09-13T08:42:48.917137+0000","last_unstale":"2024-09-13T08:42:50.715273+0000","last_undegraded":"2024-09-13T08:42:50.715273+0000","last_fullsized":"2024-09-13T08:42:50.715273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:48.610803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921504+0000","last_change":"2024-09-13T08:42:47.919027+0000","last_active":"2024-09-13T08:42:49.921504+0000","last_peered":"2024-09-13T08:42:49.921504+0000","last_clean":"2024-09-13T08:42:49.921504+0000","last_became_active":"2024-09-13T08:42:47.918716+0000","last_became_peered":"2024-09-13T08:42:47.918716+0000","last_unstale":"2024-09-13T08:42:49.921504+0000","last_undegraded":"2024-09-13T08:42:49.921504+0000","last_fullsized":"2024-09-13T08:42:49.921504+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:09.411340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215198+0000","last_change":"2024-09-13T08:42:48.922264+0000","last_active":"2024-09-13T08:42:50.215198+0000","last_peered":"2024-09-13T08:42:50.215198+0000","last_clean":"2024-09-13T08:42:50.215198+0000","last_became_active":"2024-09-13T08:42:48.920392+0000","last_became_peered":"2024-09-13T08:42:48.920392+0000","last_unstale":"2024-09-13T08:42:50.215198+0000","last_undegraded":"2024-09-13T08:42:50.215198+0000","last_fullsized":"2024-09-13T08:42:50.215198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214913+0000","last_change":"2024-09-13T08:42:47.922612+0000","last_active":"2024-09-13T08:42:50.214913+0000","last_peered":"2024-09-13T08:42:50.214913+0000","last_clean":"2024-09-13T08:42:50.214913+0000","last_became_active":"2024-09-13T08:42:47.922293+0000","last_became_peered":"2024-09-13T08:42:47.922293+0000","last_unstale":"2024-09-13T08:42:50.214913+0000","last_undegraded":"2024-09-13T08:42:50.214913+0000","last_fullsized":"2024-09-13T08:42:50.214913+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:04.539183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917505+0000","last_change":"2024-09-13T08:42:48.930218+0000","last_active":"2024-09-13T08:42:49.917505+0000","last_peered":"2024-09-13T08:42:49.917505+0000","last_clean":"2024-09-13T08:42:49.917505+0000","last_became_active":"2024-09-13T08:42:48.929933+0000","last_became_peered":"2024-09-13T08:42:48.929933+0000","last_unstale":"2024-09-13T08:42:49.917505+0000","last_undegraded":"2024-09-13T08:42:49.917505+0000","last_fullsized":"2024-09-13T08:42:49.917505+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:06.685353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919328+0000","last_change":"2024-09-13T08:42:47.924679+0000","last_active":"2024-09-13T08:42:49.919328+0000","last_peered":"2024-09-13T08:42:49.919328+0000","last_clean":"2024-09-13T08:42:49.919328+0000","last_became_active":"2024-09-13T08:42:47.922407+0000","last_became_peered":"2024-09-13T08:42:47.922407+0000","last_unstale":"2024-09-13T08:42:49.919328+0000","last_undegraded":"2024-09-13T08:42:49.919328+0000","last_fullsized":"2024-09-13T08:42:49.919328+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:01.241800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921180+0000","last_change":"2024-09-13T08:42:48.929703+0000","last_active":"2024-09-13T08:42:49.921180+0000","last_peered":"2024-09-13T08:42:49.921180+0000","last_clean":"2024-09-13T08:42:49.921180+0000","last_became_active":"2024-09-13T08:42:48.929295+0000","last_became_peered":"2024-09-13T08:42:48.929295+0000","last_unstale":"2024-09-13T08:42:49.921180+0000","last_undegraded":"2024-09-13T08:42:49.921180+0000","last_fullsized":"2024-09-13T08:42:49.921180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:39:53.724211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919257+0000","last_change":"2024-09-13T08:42:47.924294+0000","last_active":"2024-09-13T08:42:49.919257+0000","last_peered":"2024-09-13T08:42:49.919257+0000","last_clean":"2024-09-13T08:42:49.919257+0000","last_became_active":"2024-09-13T08:42:47.923952+0000","last_became_peered":"2024-09-13T08:42:47.923952+0000","last_unstale":"2024-09-13T08:42:49.919257+0000","last_undegraded":"2024-09-13T08:42:49.919257+0000","last_fullsized":"2024-09-13T08:42:49.919257+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:38.324867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918071+0000","last_change":"2024-09-13T08:42:48.931869+0000","last_active":"2024-09-13T08:42:49.918071+0000","last_peered":"2024-09-13T08:42:49.918071+0000","last_clean":"2024-09-13T08:42:49.918071+0000","last_became_active":"2024-09-13T08:42:48.931566+0000","last_became_peered":"2024-09-13T08:42:48.931566+0000","last_unstale":"2024-09-13T08:42:49.918071+0000","last_undegraded":"2024-09-13T08:42:49.918071+0000","last_fullsized":"2024-09-13T08:42:49.918071+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:07.847124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920200+0000","last_change":"2024-09-13T08:42:47.915299+0000","last_active":"2024-09-13T08:42:49.920200+0000","last_peered":"2024-09-13T08:42:49.920200+0000","last_clean":"2024-09-13T08:42:49.920200+0000","last_became_active":"2024-09-13T08:42:47.914079+0000","last_became_peered":"2024-09-13T08:42:47.914079+0000","last_unstale":"2024-09-13T08:42:49.920200+0000","last_undegraded":"2024-09-13T08:42:49.920200+0000","last_fullsized":"2024-09-13T08:42:49.920200+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:43.361006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189767+0000","last_change":"2024-09-13T08:42:48.917993+0000","last_active":"2024-09-13T08:42:50.189767+0000","last_peered":"2024-09-13T08:42:50.189767+0000","last_clean":"2024-09-13T08:42:50.189767+0000","last_became_active":"2024-09-13T08:42:48.917614+0000","last_became_peered":"2024-09-13T08:42:48.917614+0000","last_unstale":"2024-09-13T08:42:50.189767+0000","last_undegraded":"2024-09-13T08:42:50.189767+0000","last_fullsized":"2024-09-13T08:42:50.189767+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:41.424198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.909728+0000","last_change":"2024-09-13T08:42:47.923979+0000","last_active":"2024-09-13T08:42:50.909728+0000","last_peered":"2024-09-13T08:42:50.909728+0000","last_clean":"2024-09-13T08:42:50.909728+0000","last_became_active":"2024-09-13T08:42:47.923665+0000","last_became_peered":"2024-09-13T08:42:47.923665+0000","last_unstale":"2024-09-13T08:42:50.909728+0000","last_undegraded":"2024-09-13T08:42:50.909728+0000","last_fullsized":"2024-09-13T08:42:50.909728+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:45.135574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215617+0000","last_change":"2024-09-13T08:42:48.928863+0000","last_active":"2024-09-13T08:42:50.215617+0000","last_peered":"2024-09-13T08:42:50.215617+0000","last_clean":"2024-09-13T08:42:50.215617+0000","last_became_active":"2024-09-13T08:42:48.928507+0000","last_became_peered":"2024-09-13T08:42:48.928507+0000","last_unstale":"2024-09-13T08:42:50.215617+0000","last_undegraded":"2024-09-13T08:42:50.215617+0000","last_fullsized":"2024-09-13T08:42:50.215617+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:04.109657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920252+0000","last_change":"2024-09-13T08:42:47.927904+0000","last_active":"2024-09-13T08:42:49.920252+0000","last_peered":"2024-09-13T08:42:49.920252+0000","last_clean":"2024-09-13T08:42:49.920252+0000","last_became_active":"2024-09-13T08:42:47.927227+0000","last_became_peered":"2024-09-13T08:42:47.927227+0000","last_unstale":"2024-09-13T08:42:49.920252+0000","last_undegraded":"2024-09-13T08:42:49.920252+0000","last_fullsized":"2024-09-13T08:42:49.920252+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:05.255239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918198+0000","last_change":"2024-09-13T08:42:48.934552+0000","last_active":"2024-09-13T08:42:49.918198+0000","last_peered":"2024-09-13T08:42:49.918198+0000","last_clean":"2024-09-13T08:42:49.918198+0000","last_became_active":"2024-09-13T08:42:48.934285+0000","last_became_peered":"2024-09-13T08:42:48.934285+0000","last_unstale":"2024-09-13T08:42:49.918198+0000","last_undegraded":"2024-09-13T08:42:49.918198+0000","last_fullsized":"2024-09-13T08:42:49.918198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:29.023419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919965+0000","last_change":"2024-09-13T08:42:47.920084+0000","last_active":"2024-09-13T08:42:49.919965+0000","last_peered":"2024-09-13T08:42:49.919965+0000","last_clean":"2024-09-13T08:42:49.919965+0000","last_became_active":"2024-09-13T08:42:47.918598+0000","last_became_peered":"2024-09-13T08:42:47.918598+0000","last_unstale":"2024-09-13T08:42:49.919965+0000","last_undegraded":"2024-09-13T08:42:49.919965+0000","last_fullsized":"2024-09-13T08:42:49.919965+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:58.276766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715313+0000","last_change":"2024-09-13T08:42:48.931267+0000","last_active":"2024-09-13T08:42:50.715313+0000","last_peered":"2024-09-13T08:42:50.715313+0000","last_clean":"2024-09-13T08:42:50.715313+0000","last_became_active":"2024-09-13T08:42:48.930886+0000","last_became_peered":"2024-09-13T08:42:48.930886+0000","last_unstale":"2024-09-13T08:42:50.715313+0000","last_undegraded":"2024-09-13T08:42:50.715313+0000","last_fullsized":"2024-09-13T08:42:50.715313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:36.746413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716300+0000","last_change":"2024-09-13T08:42:47.918161+0000","last_active":"2024-09-13T08:42:50.716300+0000","last_peered":"2024-09-13T08:42:50.716300+0000","last_clean":"2024-09-13T08:42:50.716300+0000","last_became_active":"2024-09-13T08:42:47.917855+0000","last_became_peered":"2024-09-13T08:42:47.917855+0000","last_unstale":"2024-09-13T08:42:50.716300+0000","last_undegraded":"2024-09-13T08:42:50.716300+0000","last_fullsized":"2024-09-13T08:42:50.716300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:40.421365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917989+0000","last_change":"2024-09-13T08:42:48.939097+0000","last_active":"2024-09-13T08:42:49.917989+0000","last_peered":"2024-09-13T08:42:49.917989+0000","last_clean":"2024-09-13T08:42:49.917989+0000","last_became_active":"2024-09-13T08:42:48.938830+0000","last_became_peered":"2024-09-13T08:42:48.938830+0000","last_unstale":"2024-09-13T08:42:49.917989+0000","last_undegraded":"2024-09-13T08:42:49.917989+0000","last_fullsized":"2024-09-13T08:42:49.917989+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917709+0000","last_change":"2024-09-13T08:42:47.917137+0000","last_active":"2024-09-13T08:42:49.917709+0000","last_peered":"2024-09-13T08:42:49.917709+0000","last_clean":"2024-09-13T08:42:49.917709+0000","last_became_active":"2024-09-13T08:42:47.916303+0000","last_became_peered":"2024-09-13T08:42:47.916303+0000","last_unstale":"2024-09-13T08:42:49.917709+0000","last_undegraded":"2024-09-13T08:42:49.917709+0000","last_fullsized":"2024-09-13T08:42:49.917709+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:16:00.042800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917570+0000","last_change":"2024-09-13T08:42:48.929443+0000","last_active":"2024-09-13T08:42:49.917570+0000","last_peered":"2024-09-13T08:42:49.917570+0000","last_clean":"2024-09-13T08:42:49.917570+0000","last_became_active":"2024-09-13T08:42:48.926428+0000","last_became_peered":"2024-09-13T08:42:48.926428+0000","last_unstale":"2024-09-13T08:42:49.917570+0000","last_undegraded":"2024-09-13T08:42:49.917570+0000","last_fullsized":"2024-09-13T08:42:49.917570+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.153278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215248+0000","last_change":"2024-09-13T08:42:47.924426+0000","last_active":"2024-09-13T08:42:50.215248+0000","last_peered":"2024-09-13T08:42:50.215248+0000","last_clean":"2024-09-13T08:42:50.215248+0000","last_became_active":"2024-09-13T08:42:47.924140+0000","last_became_peered":"2024-09-13T08:42:47.924140+0000","last_unstale":"2024-09-13T08:42:50.215248+0000","last_undegraded":"2024-09-13T08:42:50.215248+0000","last_fullsized":"2024-09-13T08:42:50.215248+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:11.065256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189456+0000","last_change":"2024-09-13T08:42:48.931667+0000","last_active":"2024-09-13T08:42:50.189456+0000","last_peered":"2024-09-13T08:42:50.189456+0000","last_clean":"2024-09-13T08:42:50.189456+0000","last_became_active":"2024-09-13T08:42:48.927482+0000","last_became_peered":"2024-09-13T08:42:48.927482+0000","last_unstale":"2024-09-13T08:42:50.189456+0000","last_undegraded":"2024-09-13T08:42:50.189456+0000","last_fullsized":"2024-09-13T08:42:50.189456+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:24.571274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189047+0000","last_change":"2024-09-13T08:42:47.924508+0000","last_active":"2024-09-13T08:42:50.189047+0000","last_peered":"2024-09-13T08:42:50.189047+0000","last_clean":"2024-09-13T08:42:50.189047+0000","last_became_active":"2024-09-13T08:42:47.923507+0000","last_became_peered":"2024-09-13T08:42:47.923507+0000","last_unstale":"2024-09-13T08:42:50.189047+0000","last_undegraded":"2024-09-13T08:42:50.189047+0000","last_fullsized":"2024-09-13T08:42:50.189047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:14.035680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919130+0000","last_change":"2024-09-13T08:42:48.927133+0000","last_active":"2024-09-13T08:42:49.919130+0000","last_peered":"2024-09-13T08:42:49.919130+0000","last_clean":"2024-09-13T08:42:49.919130+0000","last_became_active":"2024-09-13T08:42:48.926484+0000","last_became_peered":"2024-09-13T08:42:48.926484+0000","last_unstale":"2024-09-13T08:42:49.919130+0000","last_undegraded":"2024-09-13T08:42:49.919130+0000","last_fullsized":"2024-09-13T08:42:49.919130+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:05.635794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683100+0000","last_change":"2024-09-13T08:42:47.920152+0000","last_active":"2024-09-13T08:42:50.683100+0000","last_peered":"2024-09-13T08:42:50.683100+0000","last_clean":"2024-09-13T08:42:50.683100+0000","last_became_active":"2024-09-13T08:42:47.916985+0000","last_became_peered":"2024-09-13T08:42:47.916985+0000","last_unstale":"2024-09-13T08:42:50.683100+0000","last_undegraded":"2024-09-13T08:42:50.683100+0000","last_fullsized":"2024-09-13T08:42:50.683100+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:47.642097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":293,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715127+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:42:50.715127+0000","last_peered":"2024-09-13T08:42:50.715127+0000","last_clean":"2024-09-13T08:42:50.715127+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:42:50.715127+0000","last_undegraded":"2024-09-13T08:42:50.715127+0000","last_fullsized":"2024-09-13T08:42:50.715127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509514,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1355776,"data_stored":587737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1335296,"data_stored":586001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1937408,"data_stored":1176461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542217,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1351680,"data_stored":587703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509514,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1335296,"data_stored":586001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1347584,"data_stored":586639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509515,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1929216,"data_stored":1176369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509513,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1343488,"data_stored":586653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:42:53.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.358+0000 7f5a2d800640 1 -- 172.21.15.31:0/3091685272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a0c004800 msgr2=0x7f5a0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.358+0000 7f5a2d800640 1 --2- 172.21.15.31:0/3091685272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a0c004800 0x7f5a0c004c80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5a10001790 tx=0x7f5a0c005830 comp rx=0 tx=0).stop 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.358+0000 7f5a2ce00640 1 -- 172.21.15.31:0/3091685272 reap_dead start 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.359+0000 7f5a2d800640 1 -- 172.21.15.31:0/3091685272 shutdown_connections 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.359+0000 7f5a2d800640 1 -- 172.21.15.31:0/3091685272 >> 172.21.15.31:0/3091685272 conn(0x7f5a280d3e90 msgr2=0x7f5a280d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.359+0000 7f5a2d800640 1 -- 172.21.15.31:0/3091685272 shutdown_connections 2024-09-13T08:42:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.359+0000 7f5a2d800640 1 -- 172.21.15.31:0/3091685272 wait complete. 2024-09-13T08:42:53.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.360+0000 7f5a2d800640 1 Processor -- start 2024-09-13T08:42:53.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.360+0000 7f5a2d800640 1 -- start start 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 0x7f5a280cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 0x7f5a281f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a280b3be0 con 0x7f5a280b7d10 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a280d15a0 con 0x7f5a0c004800 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.361+0000 7f5a2d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5a280b3d60 con 0x7f5a281f3dc0 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42578/0 (socket says 172.21.15.31:42578) 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 -- 172.21.15.31:0/1606117704 learned_addr learned my addr 172.21.15.31:0/1606117704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 -- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 msgr2=0x7f5a281f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a2ce00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 0x7f5a281f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 0x7f5a281f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a27e00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 0x7f5a280cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 -- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 msgr2=0x7f5a280cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 0x7f5a280cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 -- 172.21.15.31:0/1606117704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a280c7ee0 con 0x7f5a280b7d10 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a2ce00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 0x7f5a281f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a27e00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 0x7f5a280cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.362+0000 7f5a1fe00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002aa20 tx=0x7f5a100060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.363+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10037040 con 0x7f5a280b7d10 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.363+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a10041070 con 0x7f5a280b7d10 2024-09-13T08:42:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.363+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10003850 con 0x7f5a280b7d10 2024-09-13T08:42:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.363+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a281f54c0 con 0x7f5a280b7d10 2024-09-13T08:42:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.363+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a281f51e0 con 0x7f5a280b7d10 2024-09-13T08:42:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.364+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a280ce290 con 0x7f5a280b7d10 2024-09-13T08:42:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.365+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f5a100039f0 con 0x7f5a280b7d10 2024-09-13T08:42:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.366+0000 7f5a24c00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 0x7f5a0407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.366+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f5a100baf70 con 0x7f5a280b7d10 2024-09-13T08:42:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.366+0000 7f5a27e00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 0x7f5a0407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.367+0000 7f5a27e00640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 0x7f5a0407ae60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a18004840 tx=0x7f5a18000ed0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:53.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.371+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5a280ce290 con 0x7f5a280b7d10 2024-09-13T08:42:53.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.561+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a281f4f90 con 0x7f5a040789a0 2024-09-13T08:42:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.566+0000 7f5a24c00640 1 -- 172.21.15.31:0/1606117704 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7f5a281f4f90 con 0x7f5a040789a0 2024-09-13T08:42:53.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:42:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 msgr2=0x7f5a0407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 0x7f5a0407ae60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a18004840 tx=0x7f5a18000ed0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 msgr2=0x7f5a280c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002aa20 tx=0x7f5a100060e0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 shutdown_connections 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a040789a0 0x7f5a0407ae60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a281f3dc0 0x7f5a281f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a280b7d10 0x7f5a280c79a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 --2- 172.21.15.31:0/1606117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a0c004800 0x7f5a280cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.569+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 >> 172.21.15.31:0/1606117704 conn(0x7f5a280d3e90 msgr2=0x7f5a2806df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.570+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 shutdown_connections 2024-09-13T08:42:53.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.570+0000 7f5a2d800640 1 -- 172.21.15.31:0/1606117704 wait complete. 2024-09-13T08:42:53.587 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T08:42:52.971049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265352,"kb_used_data":11656,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709368,"statfs":{"total":773094113280,"available":772822392832,"internally_reserved":0,"allocated":11935744,"data_stored":5873564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001120"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215326+0000","last_change":"2024-09-13T08:42:48.929961+0000","last_active":"2024-09-13T08:42:50.215326+0000","last_peered":"2024-09-13T08:42:50.215326+0000","last_clean":"2024-09-13T08:42:50.215326+0000","last_became_active":"2024-09-13T08:42:48.928616+0000","last_became_peered":"2024-09-13T08:42:48.928616+0000","last_unstale":"2024-09-13T08:42:50.215326+0000","last_undegraded":"2024-09-13T08:42:50.215326+0000","last_fullsized":"2024-09-13T08:42:50.215326+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:07.847124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683351+0000","last_change":"2024-09-13T08:42:48.929056+0000","last_active":"2024-09-13T08:42:50.683351+0000","last_peered":"2024-09-13T08:42:50.683351+0000","last_clean":"2024-09-13T08:42:50.683351+0000","last_became_active":"2024-09-13T08:42:48.928297+0000","last_became_peered":"2024-09-13T08:42:48.928297+0000","last_unstale":"2024-09-13T08:42:50.683351+0000","last_undegraded":"2024-09-13T08:42:50.683351+0000","last_fullsized":"2024-09-13T08:42:50.683351+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:04.844276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715977+0000","last_change":"2024-09-13T08:42:48.919137+0000","last_active":"2024-09-13T08:42:50.715977+0000","last_peered":"2024-09-13T08:42:50.715977+0000","last_clean":"2024-09-13T08:42:50.715977+0000","last_became_active":"2024-09-13T08:42:48.918876+0000","last_became_peered":"2024-09-13T08:42:48.918876+0000","last_unstale":"2024-09-13T08:42:50.715977+0000","last_undegraded":"2024-09-13T08:42:50.715977+0000","last_fullsized":"2024-09-13T08:42:50.715977+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:44.983616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919881+0000","last_change":"2024-09-13T08:42:48.933418+0000","last_active":"2024-09-13T08:42:49.919881+0000","last_peered":"2024-09-13T08:42:49.919881+0000","last_clean":"2024-09-13T08:42:49.919881+0000","last_became_active":"2024-09-13T08:42:48.927797+0000","last_became_peered":"2024-09-13T08:42:48.927797+0000","last_unstale":"2024-09-13T08:42:49.919881+0000","last_undegraded":"2024-09-13T08:42:49.919881+0000","last_fullsized":"2024-09-13T08:42:49.919881+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:27:53.609294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921533+0000","last_change":"2024-09-13T08:42:48.929725+0000","last_active":"2024-09-13T08:42:49.921533+0000","last_peered":"2024-09-13T08:42:49.921533+0000","last_clean":"2024-09-13T08:42:49.921533+0000","last_became_active":"2024-09-13T08:42:48.929450+0000","last_became_peered":"2024-09-13T08:42:48.929450+0000","last_unstale":"2024-09-13T08:42:49.921533+0000","last_undegraded":"2024-09-13T08:42:49.921533+0000","last_fullsized":"2024-09-13T08:42:49.921533+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:56:42.644926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189384+0000","last_change":"2024-09-13T08:42:48.931621+0000","last_active":"2024-09-13T08:42:50.189384+0000","last_peered":"2024-09-13T08:42:50.189384+0000","last_clean":"2024-09-13T08:42:50.189384+0000","last_became_active":"2024-09-13T08:42:48.931364+0000","last_became_peered":"2024-09-13T08:42:48.931364+0000","last_unstale":"2024-09-13T08:42:50.189384+0000","last_undegraded":"2024-09-13T08:42:50.189384+0000","last_fullsized":"2024-09-13T08:42:50.189384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:24.122788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715569+0000","last_change":"2024-09-13T08:42:48.924322+0000","last_active":"2024-09-13T08:42:50.715569+0000","last_peered":"2024-09-13T08:42:50.715569+0000","last_clean":"2024-09-13T08:42:50.715569+0000","last_became_active":"2024-09-13T08:42:48.922743+0000","last_became_peered":"2024-09-13T08:42:48.922743+0000","last_unstale":"2024-09-13T08:42:50.715569+0000","last_undegraded":"2024-09-13T08:42:50.715569+0000","last_fullsized":"2024-09-13T08:42:50.715569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:14.685145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716093+0000","last_change":"2024-09-13T08:42:48.924769+0000","last_active":"2024-09-13T08:42:50.716093+0000","last_peered":"2024-09-13T08:42:50.716093+0000","last_clean":"2024-09-13T08:42:50.716093+0000","last_became_active":"2024-09-13T08:42:48.922797+0000","last_became_peered":"2024-09-13T08:42:48.922797+0000","last_unstale":"2024-09-13T08:42:50.716093+0000","last_undegraded":"2024-09-13T08:42:50.716093+0000","last_fullsized":"2024-09-13T08:42:50.716093+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:27.340931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215331+0000","last_change":"2024-09-13T08:42:48.937822+0000","last_active":"2024-09-13T08:42:50.215331+0000","last_peered":"2024-09-13T08:42:50.215331+0000","last_clean":"2024-09-13T08:42:50.215331+0000","last_became_active":"2024-09-13T08:42:48.935614+0000","last_became_peered":"2024-09-13T08:42:48.935614+0000","last_unstale":"2024-09-13T08:42:50.215331+0000","last_undegraded":"2024-09-13T08:42:50.215331+0000","last_fullsized":"2024-09-13T08:42:50.215331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:04.844276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921544+0000","last_change":"2024-09-13T08:42:48.937696+0000","last_active":"2024-09-13T08:42:49.921544+0000","last_peered":"2024-09-13T08:42:49.921544+0000","last_clean":"2024-09-13T08:42:49.921544+0000","last_became_active":"2024-09-13T08:42:48.935883+0000","last_became_peered":"2024-09-13T08:42:48.935883+0000","last_unstale":"2024-09-13T08:42:49.921544+0000","last_undegraded":"2024-09-13T08:42:49.921544+0000","last_fullsized":"2024-09-13T08:42:49.921544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:50.493189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715996+0000","last_change":"2024-09-13T08:42:48.922633+0000","last_active":"2024-09-13T08:42:50.715996+0000","last_peered":"2024-09-13T08:42:50.715996+0000","last_clean":"2024-09-13T08:42:50.715996+0000","last_became_active":"2024-09-13T08:42:48.922285+0000","last_became_peered":"2024-09-13T08:42:48.922285+0000","last_unstale":"2024-09-13T08:42:50.715996+0000","last_undegraded":"2024-09-13T08:42:50.715996+0000","last_fullsized":"2024-09-13T08:42:50.715996+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:56.090927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921633+0000","last_change":"2024-09-13T08:42:48.922921+0000","last_active":"2024-09-13T08:42:49.921633+0000","last_peered":"2024-09-13T08:42:49.921633+0000","last_clean":"2024-09-13T08:42:49.921633+0000","last_became_active":"2024-09-13T08:42:48.921548+0000","last_became_peered":"2024-09-13T08:42:48.921548+0000","last_unstale":"2024-09-13T08:42:49.921633+0000","last_undegraded":"2024-09-13T08:42:49.921633+0000","last_fullsized":"2024-09-13T08:42:49.921633+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:48.610803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683322+0000","last_change":"2024-09-13T08:42:48.928575+0000","last_active":"2024-09-13T08:42:50.683322+0000","last_peered":"2024-09-13T08:42:50.683322+0000","last_clean":"2024-09-13T08:42:50.683322+0000","last_became_active":"2024-09-13T08:42:48.928236+0000","last_became_peered":"2024-09-13T08:42:48.928236+0000","last_unstale":"2024-09-13T08:42:50.683322+0000","last_undegraded":"2024-09-13T08:42:50.683322+0000","last_fullsized":"2024-09-13T08:42:50.683322+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:24.645996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917595+0000","last_change":"2024-09-13T08:42:48.923449+0000","last_active":"2024-09-13T08:42:49.917595+0000","last_peered":"2024-09-13T08:42:49.917595+0000","last_clean":"2024-09-13T08:42:49.917595+0000","last_became_active":"2024-09-13T08:42:48.923154+0000","last_became_peered":"2024-09-13T08:42:48.923154+0000","last_unstale":"2024-09-13T08:42:49.917595+0000","last_undegraded":"2024-09-13T08:42:49.917595+0000","last_fullsized":"2024-09-13T08:42:49.917595+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:56.117626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214902+0000","last_change":"2024-09-13T08:42:48.931242+0000","last_active":"2024-09-13T08:42:50.214902+0000","last_peered":"2024-09-13T08:42:50.214902+0000","last_clean":"2024-09-13T08:42:50.214902+0000","last_became_active":"2024-09-13T08:42:48.928133+0000","last_became_peered":"2024-09-13T08:42:48.928133+0000","last_unstale":"2024-09-13T08:42:50.214902+0000","last_undegraded":"2024-09-13T08:42:50.214902+0000","last_fullsized":"2024-09-13T08:42:50.214902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:11.489289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919617+0000","last_change":"2024-09-13T08:42:48.922248+0000","last_active":"2024-09-13T08:42:49.919617+0000","last_peered":"2024-09-13T08:42:49.919617+0000","last_clean":"2024-09-13T08:42:49.919617+0000","last_became_active":"2024-09-13T08:42:48.921913+0000","last_became_peered":"2024-09-13T08:42:48.921913+0000","last_unstale":"2024-09-13T08:42:49.919617+0000","last_undegraded":"2024-09-13T08:42:49.919617+0000","last_fullsized":"2024-09-13T08:42:49.919617+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:20:00.412578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715714+0000","last_change":"2024-09-13T08:42:48.924914+0000","last_active":"2024-09-13T08:42:50.715714+0000","last_peered":"2024-09-13T08:42:50.715714+0000","last_clean":"2024-09-13T08:42:50.715714+0000","last_became_active":"2024-09-13T08:42:48.924543+0000","last_became_peered":"2024-09-13T08:42:48.924543+0000","last_unstale":"2024-09-13T08:42:50.715714+0000","last_undegraded":"2024-09-13T08:42:50.715714+0000","last_fullsized":"2024-09-13T08:42:50.715714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:16:01.045632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715941+0000","last_change":"2024-09-13T08:42:48.929226+0000","last_active":"2024-09-13T08:42:50.715941+0000","last_peered":"2024-09-13T08:42:50.715941+0000","last_clean":"2024-09-13T08:42:50.715941+0000","last_became_active":"2024-09-13T08:42:48.928937+0000","last_became_peered":"2024-09-13T08:42:48.928937+0000","last_unstale":"2024-09-13T08:42:50.715941+0000","last_undegraded":"2024-09-13T08:42:50.715941+0000","last_fullsized":"2024-09-13T08:42:50.715941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:54.429540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715893+0000","last_change":"2024-09-13T08:42:48.924814+0000","last_active":"2024-09-13T08:42:50.715893+0000","last_peered":"2024-09-13T08:42:50.715893+0000","last_clean":"2024-09-13T08:42:50.715893+0000","last_became_active":"2024-09-13T08:42:48.923990+0000","last_became_peered":"2024-09-13T08:42:48.923990+0000","last_unstale":"2024-09-13T08:42:50.715893+0000","last_undegraded":"2024-09-13T08:42:50.715893+0000","last_fullsized":"2024-09-13T08:42:50.715893+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:20:33.022118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715618+0000","last_change":"2024-09-13T08:42:48.927994+0000","last_active":"2024-09-13T08:42:50.715618+0000","last_peered":"2024-09-13T08:42:50.715618+0000","last_clean":"2024-09-13T08:42:50.715618+0000","last_became_active":"2024-09-13T08:42:48.927674+0000","last_became_peered":"2024-09-13T08:42:48.927674+0000","last_unstale":"2024-09-13T08:42:50.715618+0000","last_undegraded":"2024-09-13T08:42:50.715618+0000","last_fullsized":"2024-09-13T08:42:50.715618+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:35.885431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716233+0000","last_change":"2024-09-13T08:42:48.917950+0000","last_active":"2024-09-13T08:42:50.716233+0000","last_peered":"2024-09-13T08:42:50.716233+0000","last_clean":"2024-09-13T08:42:50.716233+0000","last_became_active":"2024-09-13T08:42:48.917613+0000","last_became_peered":"2024-09-13T08:42:48.917613+0000","last_unstale":"2024-09-13T08:42:50.716233+0000","last_undegraded":"2024-09-13T08:42:50.716233+0000","last_fullsized":"2024-09-13T08:42:50.716233+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:34.374825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189604+0000","last_change":"2024-09-13T08:42:48.923079+0000","last_active":"2024-09-13T08:42:50.189604+0000","last_peered":"2024-09-13T08:42:50.189604+0000","last_clean":"2024-09-13T08:42:50.189604+0000","last_became_active":"2024-09-13T08:42:48.922813+0000","last_became_peered":"2024-09-13T08:42:48.922813+0000","last_unstale":"2024-09-13T08:42:50.189604+0000","last_undegraded":"2024-09-13T08:42:50.189604+0000","last_fullsized":"2024-09-13T08:42:50.189604+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:35.241616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917613+0000","last_change":"2024-09-13T08:42:48.925871+0000","last_active":"2024-09-13T08:42:49.917613+0000","last_peered":"2024-09-13T08:42:49.917613+0000","last_clean":"2024-09-13T08:42:49.917613+0000","last_became_active":"2024-09-13T08:42:48.925464+0000","last_became_peered":"2024-09-13T08:42:48.925464+0000","last_unstale":"2024-09-13T08:42:49.917613+0000","last_undegraded":"2024-09-13T08:42:49.917613+0000","last_fullsized":"2024-09-13T08:42:49.917613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:11.489289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919088+0000","last_change":"2024-09-13T08:42:48.927441+0000","last_active":"2024-09-13T08:42:49.919088+0000","last_peered":"2024-09-13T08:42:49.919088+0000","last_clean":"2024-09-13T08:42:49.919088+0000","last_became_active":"2024-09-13T08:42:48.926848+0000","last_became_peered":"2024-09-13T08:42:48.926848+0000","last_unstale":"2024-09-13T08:42:49.919088+0000","last_undegraded":"2024-09-13T08:42:49.919088+0000","last_fullsized":"2024-09-13T08:42:49.919088+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:49.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715760+0000","last_change":"2024-09-13T08:42:48.913934+0000","last_active":"2024-09-13T08:42:50.715760+0000","last_peered":"2024-09-13T08:42:50.715760+0000","last_clean":"2024-09-13T08:42:50.715760+0000","last_became_active":"2024-09-13T08:42:48.911894+0000","last_became_peered":"2024-09-13T08:42:48.911894+0000","last_unstale":"2024-09-13T08:42:50.715760+0000","last_undegraded":"2024-09-13T08:42:50.715760+0000","last_fullsized":"2024-09-13T08:42:50.715760+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:47.541292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189747+0000","last_change":"2024-09-13T08:42:48.921453+0000","last_active":"2024-09-13T08:42:50.189747+0000","last_peered":"2024-09-13T08:42:50.189747+0000","last_clean":"2024-09-13T08:42:50.189747+0000","last_became_active":"2024-09-13T08:42:48.921160+0000","last_became_peered":"2024-09-13T08:42:48.921160+0000","last_unstale":"2024-09-13T08:42:50.189747+0000","last_undegraded":"2024-09-13T08:42:50.189747+0000","last_fullsized":"2024-09-13T08:42:50.189747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:05:25.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920299+0000","last_change":"2024-09-13T08:42:48.933960+0000","last_active":"2024-09-13T08:42:49.920299+0000","last_peered":"2024-09-13T08:42:49.920299+0000","last_clean":"2024-09-13T08:42:49.920299+0000","last_became_active":"2024-09-13T08:42:48.933648+0000","last_became_peered":"2024-09-13T08:42:48.933648+0000","last_unstale":"2024-09-13T08:42:49.920299+0000","last_undegraded":"2024-09-13T08:42:49.920299+0000","last_fullsized":"2024-09-13T08:42:49.920299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:39.551390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189473+0000","last_change":"2024-09-13T08:42:48.935046+0000","last_active":"2024-09-13T08:42:50.189473+0000","last_peered":"2024-09-13T08:42:50.189473+0000","last_clean":"2024-09-13T08:42:50.189473+0000","last_became_active":"2024-09-13T08:42:48.934787+0000","last_became_peered":"2024-09-13T08:42:48.934787+0000","last_unstale":"2024-09-13T08:42:50.189473+0000","last_undegraded":"2024-09-13T08:42:50.189473+0000","last_fullsized":"2024-09-13T08:42:50.189473+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:48.984880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917981+0000","last_change":"2024-09-13T08:42:48.922522+0000","last_active":"2024-09-13T08:42:49.917981+0000","last_peered":"2024-09-13T08:42:49.917981+0000","last_clean":"2024-09-13T08:42:49.917981+0000","last_became_active":"2024-09-13T08:42:48.922219+0000","last_became_peered":"2024-09-13T08:42:48.922219+0000","last_unstale":"2024-09-13T08:42:49.917981+0000","last_undegraded":"2024-09-13T08:42:49.917981+0000","last_fullsized":"2024-09-13T08:42:49.917981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:56.062750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715895+0000","last_change":"2024-09-13T08:42:48.917723+0000","last_active":"2024-09-13T08:42:50.715895+0000","last_peered":"2024-09-13T08:42:50.715895+0000","last_clean":"2024-09-13T08:42:50.715895+0000","last_became_active":"2024-09-13T08:42:48.917377+0000","last_became_peered":"2024-09-13T08:42:48.917377+0000","last_unstale":"2024-09-13T08:42:50.715895+0000","last_undegraded":"2024-09-13T08:42:50.715895+0000","last_fullsized":"2024-09-13T08:42:50.715895+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:54.222967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215011+0000","last_change":"2024-09-13T08:42:48.928732+0000","last_active":"2024-09-13T08:42:50.215011+0000","last_peered":"2024-09-13T08:42:50.215011+0000","last_clean":"2024-09-13T08:42:50.215011+0000","last_became_active":"2024-09-13T08:42:48.928455+0000","last_became_peered":"2024-09-13T08:42:48.928455+0000","last_unstale":"2024-09-13T08:42:50.215011+0000","last_undegraded":"2024-09-13T08:42:50.215011+0000","last_fullsized":"2024-09-13T08:42:50.215011+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:56.062750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920305+0000","last_change":"2024-09-13T08:42:48.931967+0000","last_active":"2024-09-13T08:42:49.920305+0000","last_peered":"2024-09-13T08:42:49.920305+0000","last_clean":"2024-09-13T08:42:49.920305+0000","last_became_active":"2024-09-13T08:42:48.931704+0000","last_became_peered":"2024-09-13T08:42:48.931704+0000","last_unstale":"2024-09-13T08:42:49.920305+0000","last_undegraded":"2024-09-13T08:42:49.920305+0000","last_fullsized":"2024-09-13T08:42:49.920305+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:12:13.636965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215482+0000","last_change":"2024-09-13T08:42:48.932873+0000","last_active":"2024-09-13T08:42:50.215482+0000","last_peered":"2024-09-13T08:42:50.215482+0000","last_clean":"2024-09-13T08:42:50.215482+0000","last_became_active":"2024-09-13T08:42:48.932619+0000","last_became_peered":"2024-09-13T08:42:48.932619+0000","last_unstale":"2024-09-13T08:42:50.215482+0000","last_undegraded":"2024-09-13T08:42:50.215482+0000","last_fullsized":"2024-09-13T08:42:50.215482+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:29.023419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":19,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918153+0000","last_change":"2024-09-13T08:42:47.919293+0000","last_active":"2024-09-13T08:42:49.918153+0000","last_peered":"2024-09-13T08:42:49.918153+0000","last_clean":"2024-09-13T08:42:49.918153+0000","last_became_active":"2024-09-13T08:42:47.918970+0000","last_became_peered":"2024-09-13T08:42:47.918970+0000","last_unstale":"2024-09-13T08:42:49.918153+0000","last_undegraded":"2024-09-13T08:42:49.918153+0000","last_fullsized":"2024-09-13T08:42:49.918153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:35.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918316+0000","last_change":"2024-09-13T08:42:48.923056+0000","last_active":"2024-09-13T08:42:49.918316+0000","last_peered":"2024-09-13T08:42:49.918316+0000","last_clean":"2024-09-13T08:42:49.918316+0000","last_became_active":"2024-09-13T08:42:48.919206+0000","last_became_peered":"2024-09-13T08:42:48.919206+0000","last_unstale":"2024-09-13T08:42:49.918316+0000","last_undegraded":"2024-09-13T08:42:49.918316+0000","last_fullsized":"2024-09-13T08:42:49.918316+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:23.606208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683354+0000","last_change":"2024-09-13T08:42:47.918010+0000","last_active":"2024-09-13T08:42:50.683354+0000","last_peered":"2024-09-13T08:42:50.683354+0000","last_clean":"2024-09-13T08:42:50.683354+0000","last_became_active":"2024-09-13T08:42:47.917655+0000","last_became_peered":"2024-09-13T08:42:47.917655+0000","last_unstale":"2024-09-13T08:42:50.683354+0000","last_undegraded":"2024-09-13T08:42:50.683354+0000","last_fullsized":"2024-09-13T08:42:50.683354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:42.297105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189616+0000","last_change":"2024-09-13T08:42:48.917530+0000","last_active":"2024-09-13T08:42:50.189616+0000","last_peered":"2024-09-13T08:42:50.189616+0000","last_clean":"2024-09-13T08:42:50.189616+0000","last_became_active":"2024-09-13T08:42:48.917266+0000","last_became_peered":"2024-09-13T08:42:48.917266+0000","last_unstale":"2024-09-13T08:42:50.189616+0000","last_undegraded":"2024-09-13T08:42:50.189616+0000","last_fullsized":"2024-09-13T08:42:50.189616+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:26.850441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920122+0000","last_change":"2024-09-13T08:42:47.917801+0000","last_active":"2024-09-13T08:42:49.920122+0000","last_peered":"2024-09-13T08:42:49.920122+0000","last_clean":"2024-09-13T08:42:49.920122+0000","last_became_active":"2024-09-13T08:42:47.917374+0000","last_became_peered":"2024-09-13T08:42:47.917374+0000","last_unstale":"2024-09-13T08:42:49.920122+0000","last_undegraded":"2024-09-13T08:42:49.920122+0000","last_fullsized":"2024-09-13T08:42:49.920122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:51:37.702500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715596+0000","last_change":"2024-09-13T08:42:48.924729+0000","last_active":"2024-09-13T08:42:50.715596+0000","last_peered":"2024-09-13T08:42:50.715596+0000","last_clean":"2024-09-13T08:42:50.715596+0000","last_became_active":"2024-09-13T08:42:48.924355+0000","last_became_peered":"2024-09-13T08:42:48.924355+0000","last_unstale":"2024-09-13T08:42:50.715596+0000","last_undegraded":"2024-09-13T08:42:50.715596+0000","last_fullsized":"2024-09-13T08:42:50.715596+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:01.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919766+0000","last_change":"2024-09-13T08:42:47.918285+0000","last_active":"2024-09-13T08:42:49.919766+0000","last_peered":"2024-09-13T08:42:49.919766+0000","last_clean":"2024-09-13T08:42:49.919766+0000","last_became_active":"2024-09-13T08:42:47.917957+0000","last_became_peered":"2024-09-13T08:42:47.917957+0000","last_unstale":"2024-09-13T08:42:49.919766+0000","last_undegraded":"2024-09-13T08:42:49.919766+0000","last_fullsized":"2024-09-13T08:42:49.919766+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:06:24.941563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919780+0000","last_change":"2024-09-13T08:42:48.931997+0000","last_active":"2024-09-13T08:42:49.919780+0000","last_peered":"2024-09-13T08:42:49.919780+0000","last_clean":"2024-09-13T08:42:49.919780+0000","last_became_active":"2024-09-13T08:42:48.931719+0000","last_became_peered":"2024-09-13T08:42:48.931719+0000","last_unstale":"2024-09-13T08:42:49.919780+0000","last_undegraded":"2024-09-13T08:42:49.919780+0000","last_fullsized":"2024-09-13T08:42:49.919780+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:52.408508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715174+0000","last_change":"2024-09-13T08:42:47.925152+0000","last_active":"2024-09-13T08:42:50.715174+0000","last_peered":"2024-09-13T08:42:50.715174+0000","last_clean":"2024-09-13T08:42:50.715174+0000","last_became_active":"2024-09-13T08:42:47.924078+0000","last_became_peered":"2024-09-13T08:42:47.924078+0000","last_unstale":"2024-09-13T08:42:50.715174+0000","last_undegraded":"2024-09-13T08:42:50.715174+0000","last_fullsized":"2024-09-13T08:42:50.715174+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:19.406011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716193+0000","last_change":"2024-09-13T08:42:48.928511+0000","last_active":"2024-09-13T08:42:50.716193+0000","last_peered":"2024-09-13T08:42:50.716193+0000","last_clean":"2024-09-13T08:42:50.716193+0000","last_became_active":"2024-09-13T08:42:48.928088+0000","last_became_peered":"2024-09-13T08:42:48.928088+0000","last_unstale":"2024-09-13T08:42:50.716193+0000","last_undegraded":"2024-09-13T08:42:50.716193+0000","last_fullsized":"2024-09-13T08:42:50.716193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:19.260415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214856+0000","last_change":"2024-09-13T08:42:47.921240+0000","last_active":"2024-09-13T08:42:50.214856+0000","last_peered":"2024-09-13T08:42:50.214856+0000","last_clean":"2024-09-13T08:42:50.214856+0000","last_became_active":"2024-09-13T08:42:47.920848+0000","last_became_peered":"2024-09-13T08:42:47.920848+0000","last_unstale":"2024-09-13T08:42:50.214856+0000","last_undegraded":"2024-09-13T08:42:50.214856+0000","last_fullsized":"2024-09-13T08:42:50.214856+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:30.087222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920630+0000","last_change":"2024-09-13T08:42:48.938972+0000","last_active":"2024-09-13T08:42:49.920630+0000","last_peered":"2024-09-13T08:42:49.920630+0000","last_clean":"2024-09-13T08:42:49.920630+0000","last_became_active":"2024-09-13T08:42:48.936269+0000","last_became_peered":"2024-09-13T08:42:48.936269+0000","last_unstale":"2024-09-13T08:42:49.920630+0000","last_undegraded":"2024-09-13T08:42:49.920630+0000","last_fullsized":"2024-09-13T08:42:49.920630+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:36.591513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716007+0000","last_change":"2024-09-13T08:42:47.921139+0000","last_active":"2024-09-13T08:42:50.716007+0000","last_peered":"2024-09-13T08:42:50.716007+0000","last_clean":"2024-09-13T08:42:50.716007+0000","last_became_active":"2024-09-13T08:42:47.920668+0000","last_became_peered":"2024-09-13T08:42:47.920668+0000","last_unstale":"2024-09-13T08:42:50.716007+0000","last_undegraded":"2024-09-13T08:42:50.716007+0000","last_fullsized":"2024-09-13T08:42:50.716007+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:54.292629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716478+0000","last_change":"2024-09-13T08:42:48.913227+0000","last_active":"2024-09-13T08:42:50.716478+0000","last_peered":"2024-09-13T08:42:50.716478+0000","last_clean":"2024-09-13T08:42:50.716478+0000","last_became_active":"2024-09-13T08:42:48.912918+0000","last_became_peered":"2024-09-13T08:42:48.912918+0000","last_unstale":"2024-09-13T08:42:50.716478+0000","last_undegraded":"2024-09-13T08:42:50.716478+0000","last_fullsized":"2024-09-13T08:42:50.716478+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:46.559943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920770+0000","last_change":"2024-09-13T08:42:47.921384+0000","last_active":"2024-09-13T08:42:49.920770+0000","last_peered":"2024-09-13T08:42:49.920770+0000","last_clean":"2024-09-13T08:42:49.920770+0000","last_became_active":"2024-09-13T08:42:47.920811+0000","last_became_peered":"2024-09-13T08:42:47.920811+0000","last_unstale":"2024-09-13T08:42:49.920770+0000","last_undegraded":"2024-09-13T08:42:49.920770+0000","last_fullsized":"2024-09-13T08:42:49.920770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.124047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683313+0000","last_change":"2024-09-13T08:42:48.920834+0000","last_active":"2024-09-13T08:42:50.683313+0000","last_peered":"2024-09-13T08:42:50.683313+0000","last_clean":"2024-09-13T08:42:50.683313+0000","last_became_active":"2024-09-13T08:42:48.920374+0000","last_became_peered":"2024-09-13T08:42:48.920374+0000","last_unstale":"2024-09-13T08:42:50.683313+0000","last_undegraded":"2024-09-13T08:42:50.683313+0000","last_fullsized":"2024-09-13T08:42:50.683313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920066+0000","last_change":"2024-09-13T08:42:47.917210+0000","last_active":"2024-09-13T08:42:49.920066+0000","last_peered":"2024-09-13T08:42:49.920066+0000","last_clean":"2024-09-13T08:42:49.920066+0000","last_became_active":"2024-09-13T08:42:47.916867+0000","last_became_peered":"2024-09-13T08:42:47.916867+0000","last_unstale":"2024-09-13T08:42:49.920066+0000","last_undegraded":"2024-09-13T08:42:49.920066+0000","last_fullsized":"2024-09-13T08:42:49.920066+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:53.838545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683533+0000","last_change":"2024-09-13T08:42:48.924445+0000","last_active":"2024-09-13T08:42:50.683533+0000","last_peered":"2024-09-13T08:42:50.683533+0000","last_clean":"2024-09-13T08:42:50.683533+0000","last_became_active":"2024-09-13T08:42:48.924122+0000","last_became_peered":"2024-09-13T08:42:48.924122+0000","last_unstale":"2024-09-13T08:42:50.683533+0000","last_undegraded":"2024-09-13T08:42:50.683533+0000","last_fullsized":"2024-09-13T08:42:50.683533+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:09.464850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921328+0000","last_change":"2024-09-13T08:42:47.925885+0000","last_active":"2024-09-13T08:42:49.921328+0000","last_peered":"2024-09-13T08:42:49.921328+0000","last_clean":"2024-09-13T08:42:49.921328+0000","last_became_active":"2024-09-13T08:42:47.925509+0000","last_became_peered":"2024-09-13T08:42:47.925509+0000","last_unstale":"2024-09-13T08:42:49.921328+0000","last_undegraded":"2024-09-13T08:42:49.921328+0000","last_fullsized":"2024-09-13T08:42:49.921328+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:55:52.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189665+0000","last_change":"2024-09-13T08:42:48.920397+0000","last_active":"2024-09-13T08:42:50.189665+0000","last_peered":"2024-09-13T08:42:50.189665+0000","last_clean":"2024-09-13T08:42:50.189665+0000","last_became_active":"2024-09-13T08:42:48.920066+0000","last_became_peered":"2024-09-13T08:42:48.920066+0000","last_unstale":"2024-09-13T08:42:50.189665+0000","last_undegraded":"2024-09-13T08:42:50.189665+0000","last_fullsized":"2024-09-13T08:42:50.189665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:57:16.907452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921529+0000","last_change":"2024-09-13T08:42:47.916457+0000","last_active":"2024-09-13T08:42:49.921529+0000","last_peered":"2024-09-13T08:42:49.921529+0000","last_clean":"2024-09-13T08:42:49.921529+0000","last_became_active":"2024-09-13T08:42:47.916092+0000","last_became_peered":"2024-09-13T08:42:47.916092+0000","last_unstale":"2024-09-13T08:42:49.921529+0000","last_undegraded":"2024-09-13T08:42:49.921529+0000","last_fullsized":"2024-09-13T08:42:49.921529+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:57:06.575988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716194+0000","last_change":"2024-09-13T08:42:48.919573+0000","last_active":"2024-09-13T08:42:50.716194+0000","last_peered":"2024-09-13T08:42:50.716194+0000","last_clean":"2024-09-13T08:42:50.716194+0000","last_became_active":"2024-09-13T08:42:48.919312+0000","last_became_peered":"2024-09-13T08:42:48.919312+0000","last_unstale":"2024-09-13T08:42:50.716194+0000","last_undegraded":"2024-09-13T08:42:50.716194+0000","last_fullsized":"2024-09-13T08:42:50.716194+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:40.140756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":21,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919905+0000","last_change":"2024-09-13T08:42:47.918152+0000","last_active":"2024-09-13T08:42:49.919905+0000","last_peered":"2024-09-13T08:42:49.919905+0000","last_clean":"2024-09-13T08:42:49.919905+0000","last_became_active":"2024-09-13T08:42:47.916644+0000","last_became_peered":"2024-09-13T08:42:47.916644+0000","last_unstale":"2024-09-13T08:42:49.919905+0000","last_undegraded":"2024-09-13T08:42:49.919905+0000","last_fullsized":"2024-09-13T08:42:49.919905+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:33:28.579773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921047+0000","last_change":"2024-09-13T08:42:48.925074+0000","last_active":"2024-09-13T08:42:49.921047+0000","last_peered":"2024-09-13T08:42:49.921047+0000","last_clean":"2024-09-13T08:42:49.921047+0000","last_became_active":"2024-09-13T08:42:48.924774+0000","last_became_peered":"2024-09-13T08:42:48.924774+0000","last_unstale":"2024-09-13T08:42:49.921047+0000","last_undegraded":"2024-09-13T08:42:49.921047+0000","last_fullsized":"2024-09-13T08:42:49.921047+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:42.072741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921360+0000","last_change":"2024-09-13T08:42:47.918868+0000","last_active":"2024-09-13T08:42:49.921360+0000","last_peered":"2024-09-13T08:42:49.921360+0000","last_clean":"2024-09-13T08:42:49.921360+0000","last_became_active":"2024-09-13T08:42:47.918489+0000","last_became_peered":"2024-09-13T08:42:47.918489+0000","last_unstale":"2024-09-13T08:42:49.921360+0000","last_undegraded":"2024-09-13T08:42:49.921360+0000","last_fullsized":"2024-09-13T08:42:49.921360+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:58:51.693121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919275+0000","last_change":"2024-09-13T08:42:48.923950+0000","last_active":"2024-09-13T08:42:49.919275+0000","last_peered":"2024-09-13T08:42:49.919275+0000","last_clean":"2024-09-13T08:42:49.919275+0000","last_became_active":"2024-09-13T08:42:48.923646+0000","last_became_peered":"2024-09-13T08:42:48.923646+0000","last_unstale":"2024-09-13T08:42:49.919275+0000","last_undegraded":"2024-09-13T08:42:49.919275+0000","last_fullsized":"2024-09-13T08:42:49.919275+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:48:23.745822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715335+0000","last_change":"2024-09-13T08:42:47.927451+0000","last_active":"2024-09-13T08:42:50.715335+0000","last_peered":"2024-09-13T08:42:50.715335+0000","last_clean":"2024-09-13T08:42:50.715335+0000","last_became_active":"2024-09-13T08:42:47.927172+0000","last_became_peered":"2024-09-13T08:42:47.927172+0000","last_unstale":"2024-09-13T08:42:50.715335+0000","last_undegraded":"2024-09-13T08:42:50.715335+0000","last_fullsized":"2024-09-13T08:42:50.715335+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:08:13.970546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919469+0000","last_change":"2024-09-13T08:42:48.919556+0000","last_active":"2024-09-13T08:42:49.919469+0000","last_peered":"2024-09-13T08:42:49.919469+0000","last_clean":"2024-09-13T08:42:49.919469+0000","last_became_active":"2024-09-13T08:42:48.912870+0000","last_became_peered":"2024-09-13T08:42:48.912870+0000","last_unstale":"2024-09-13T08:42:49.919469+0000","last_undegraded":"2024-09-13T08:42:49.919469+0000","last_fullsized":"2024-09-13T08:42:49.919469+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:16.946594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":23,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214802+0000","last_change":"2024-09-13T08:42:47.924945+0000","last_active":"2024-09-13T08:42:50.214802+0000","last_peered":"2024-09-13T08:42:50.214802+0000","last_clean":"2024-09-13T08:42:50.214802+0000","last_became_active":"2024-09-13T08:42:47.923813+0000","last_became_peered":"2024-09-13T08:42:47.923813+0000","last_unstale":"2024-09-13T08:42:50.214802+0000","last_undegraded":"2024-09-13T08:42:50.214802+0000","last_fullsized":"2024-09-13T08:42:50.214802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:43:10.521549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215574+0000","last_change":"2024-09-13T08:42:48.929937+0000","last_active":"2024-09-13T08:42:50.215574+0000","last_peered":"2024-09-13T08:42:50.215574+0000","last_clean":"2024-09-13T08:42:50.215574+0000","last_became_active":"2024-09-13T08:42:48.929452+0000","last_became_peered":"2024-09-13T08:42:48.929452+0000","last_unstale":"2024-09-13T08:42:50.215574+0000","last_undegraded":"2024-09-13T08:42:50.215574+0000","last_fullsized":"2024-09-13T08:42:50.215574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:23.606208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189426+0000","last_change":"2024-09-13T08:42:47.924495+0000","last_active":"2024-09-13T08:42:50.189426+0000","last_peered":"2024-09-13T08:42:50.189426+0000","last_clean":"2024-09-13T08:42:50.189426+0000","last_became_active":"2024-09-13T08:42:47.922905+0000","last_became_peered":"2024-09-13T08:42:47.922905+0000","last_unstale":"2024-09-13T08:42:50.189426+0000","last_undegraded":"2024-09-13T08:42:50.189426+0000","last_fullsized":"2024-09-13T08:42:50.189426+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:20.346046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215580+0000","last_change":"2024-09-13T08:42:48.935881+0000","last_active":"2024-09-13T08:42:50.215580+0000","last_peered":"2024-09-13T08:42:50.215580+0000","last_clean":"2024-09-13T08:42:50.215580+0000","last_became_active":"2024-09-13T08:42:48.931944+0000","last_became_peered":"2024-09-13T08:42:48.931944+0000","last_unstale":"2024-09-13T08:42:50.215580+0000","last_undegraded":"2024-09-13T08:42:50.215580+0000","last_fullsized":"2024-09-13T08:42:50.215580+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:14:39.849271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921532+0000","last_change":"2024-09-13T08:42:47.921550+0000","last_active":"2024-09-13T08:42:49.921532+0000","last_peered":"2024-09-13T08:42:49.921532+0000","last_clean":"2024-09-13T08:42:49.921532+0000","last_became_active":"2024-09-13T08:42:47.921246+0000","last_became_peered":"2024-09-13T08:42:47.921246+0000","last_unstale":"2024-09-13T08:42:49.921532+0000","last_undegraded":"2024-09-13T08:42:49.921532+0000","last_fullsized":"2024-09-13T08:42:49.921532+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:36.292928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715869+0000","last_change":"2024-09-13T08:42:48.917043+0000","last_active":"2024-09-13T08:42:50.715869+0000","last_peered":"2024-09-13T08:42:50.715869+0000","last_clean":"2024-09-13T08:42:50.715869+0000","last_became_active":"2024-09-13T08:42:48.916730+0000","last_became_peered":"2024-09-13T08:42:48.916730+0000","last_unstale":"2024-09-13T08:42:50.715869+0000","last_undegraded":"2024-09-13T08:42:50.715869+0000","last_fullsized":"2024-09-13T08:42:50.715869+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:44:06.469897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918228+0000","last_change":"2024-09-13T08:42:47.918580+0000","last_active":"2024-09-13T08:42:49.918228+0000","last_peered":"2024-09-13T08:42:49.918228+0000","last_clean":"2024-09-13T08:42:49.918228+0000","last_became_active":"2024-09-13T08:42:47.918284+0000","last_became_peered":"2024-09-13T08:42:47.918284+0000","last_unstale":"2024-09-13T08:42:49.918228+0000","last_undegraded":"2024-09-13T08:42:49.918228+0000","last_fullsized":"2024-09-13T08:42:49.918228+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:00.623413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918159+0000","last_change":"2024-09-13T08:42:48.923844+0000","last_active":"2024-09-13T08:42:49.918159+0000","last_peered":"2024-09-13T08:42:49.918159+0000","last_clean":"2024-09-13T08:42:49.918159+0000","last_became_active":"2024-09-13T08:42:48.923266+0000","last_became_peered":"2024-09-13T08:42:48.923266+0000","last_unstale":"2024-09-13T08:42:49.918159+0000","last_undegraded":"2024-09-13T08:42:49.918159+0000","last_fullsized":"2024-09-13T08:42:49.918159+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:58.849052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917881+0000","last_change":"2024-09-13T08:42:47.913763+0000","last_active":"2024-09-13T08:42:49.917881+0000","last_peered":"2024-09-13T08:42:49.917881+0000","last_clean":"2024-09-13T08:42:49.917881+0000","last_became_active":"2024-09-13T08:42:47.913459+0000","last_became_peered":"2024-09-13T08:42:47.913459+0000","last_unstale":"2024-09-13T08:42:49.917881+0000","last_undegraded":"2024-09-13T08:42:49.917881+0000","last_fullsized":"2024-09-13T08:42:49.917881+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:34.882599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715273+0000","last_change":"2024-09-13T08:42:48.917417+0000","last_active":"2024-09-13T08:42:50.715273+0000","last_peered":"2024-09-13T08:42:50.715273+0000","last_clean":"2024-09-13T08:42:50.715273+0000","last_became_active":"2024-09-13T08:42:48.917137+0000","last_became_peered":"2024-09-13T08:42:48.917137+0000","last_unstale":"2024-09-13T08:42:50.715273+0000","last_undegraded":"2024-09-13T08:42:50.715273+0000","last_fullsized":"2024-09-13T08:42:50.715273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:48.610803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921504+0000","last_change":"2024-09-13T08:42:47.919027+0000","last_active":"2024-09-13T08:42:49.921504+0000","last_peered":"2024-09-13T08:42:49.921504+0000","last_clean":"2024-09-13T08:42:49.921504+0000","last_became_active":"2024-09-13T08:42:47.918716+0000","last_became_peered":"2024-09-13T08:42:47.918716+0000","last_unstale":"2024-09-13T08:42:49.921504+0000","last_undegraded":"2024-09-13T08:42:49.921504+0000","last_fullsized":"2024-09-13T08:42:49.921504+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:09.411340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215198+0000","last_change":"2024-09-13T08:42:48.922264+0000","last_active":"2024-09-13T08:42:50.215198+0000","last_peered":"2024-09-13T08:42:50.215198+0000","last_clean":"2024-09-13T08:42:50.215198+0000","last_became_active":"2024-09-13T08:42:48.920392+0000","last_became_peered":"2024-09-13T08:42:48.920392+0000","last_unstale":"2024-09-13T08:42:50.215198+0000","last_undegraded":"2024-09-13T08:42:50.215198+0000","last_fullsized":"2024-09-13T08:42:50.215198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.214913+0000","last_change":"2024-09-13T08:42:47.922612+0000","last_active":"2024-09-13T08:42:50.214913+0000","last_peered":"2024-09-13T08:42:50.214913+0000","last_clean":"2024-09-13T08:42:50.214913+0000","last_became_active":"2024-09-13T08:42:47.922293+0000","last_became_peered":"2024-09-13T08:42:47.922293+0000","last_unstale":"2024-09-13T08:42:50.214913+0000","last_undegraded":"2024-09-13T08:42:50.214913+0000","last_fullsized":"2024-09-13T08:42:50.214913+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:26:04.539183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917505+0000","last_change":"2024-09-13T08:42:48.930218+0000","last_active":"2024-09-13T08:42:49.917505+0000","last_peered":"2024-09-13T08:42:49.917505+0000","last_clean":"2024-09-13T08:42:49.917505+0000","last_became_active":"2024-09-13T08:42:48.929933+0000","last_became_peered":"2024-09-13T08:42:48.929933+0000","last_unstale":"2024-09-13T08:42:49.917505+0000","last_undegraded":"2024-09-13T08:42:49.917505+0000","last_fullsized":"2024-09-13T08:42:49.917505+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:06.685353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919328+0000","last_change":"2024-09-13T08:42:47.924679+0000","last_active":"2024-09-13T08:42:49.919328+0000","last_peered":"2024-09-13T08:42:49.919328+0000","last_clean":"2024-09-13T08:42:49.919328+0000","last_became_active":"2024-09-13T08:42:47.922407+0000","last_became_peered":"2024-09-13T08:42:47.922407+0000","last_unstale":"2024-09-13T08:42:49.919328+0000","last_undegraded":"2024-09-13T08:42:49.919328+0000","last_fullsized":"2024-09-13T08:42:49.919328+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:01.241800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.921180+0000","last_change":"2024-09-13T08:42:48.929703+0000","last_active":"2024-09-13T08:42:49.921180+0000","last_peered":"2024-09-13T08:42:49.921180+0000","last_clean":"2024-09-13T08:42:49.921180+0000","last_became_active":"2024-09-13T08:42:48.929295+0000","last_became_peered":"2024-09-13T08:42:48.929295+0000","last_unstale":"2024-09-13T08:42:49.921180+0000","last_undegraded":"2024-09-13T08:42:49.921180+0000","last_fullsized":"2024-09-13T08:42:49.921180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:39:53.724211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919257+0000","last_change":"2024-09-13T08:42:47.924294+0000","last_active":"2024-09-13T08:42:49.919257+0000","last_peered":"2024-09-13T08:42:49.919257+0000","last_clean":"2024-09-13T08:42:49.919257+0000","last_became_active":"2024-09-13T08:42:47.923952+0000","last_became_peered":"2024-09-13T08:42:47.923952+0000","last_unstale":"2024-09-13T08:42:49.919257+0000","last_undegraded":"2024-09-13T08:42:49.919257+0000","last_fullsized":"2024-09-13T08:42:49.919257+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:38.324867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918071+0000","last_change":"2024-09-13T08:42:48.931869+0000","last_active":"2024-09-13T08:42:49.918071+0000","last_peered":"2024-09-13T08:42:49.918071+0000","last_clean":"2024-09-13T08:42:49.918071+0000","last_became_active":"2024-09-13T08:42:48.931566+0000","last_became_peered":"2024-09-13T08:42:48.931566+0000","last_unstale":"2024-09-13T08:42:49.918071+0000","last_undegraded":"2024-09-13T08:42:49.918071+0000","last_fullsized":"2024-09-13T08:42:49.918071+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:26:07.847124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920200+0000","last_change":"2024-09-13T08:42:47.915299+0000","last_active":"2024-09-13T08:42:49.920200+0000","last_peered":"2024-09-13T08:42:49.920200+0000","last_clean":"2024-09-13T08:42:49.920200+0000","last_became_active":"2024-09-13T08:42:47.914079+0000","last_became_peered":"2024-09-13T08:42:47.914079+0000","last_unstale":"2024-09-13T08:42:49.920200+0000","last_undegraded":"2024-09-13T08:42:49.920200+0000","last_fullsized":"2024-09-13T08:42:49.920200+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:43.361006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189767+0000","last_change":"2024-09-13T08:42:48.917993+0000","last_active":"2024-09-13T08:42:50.189767+0000","last_peered":"2024-09-13T08:42:50.189767+0000","last_clean":"2024-09-13T08:42:50.189767+0000","last_became_active":"2024-09-13T08:42:48.917614+0000","last_became_peered":"2024-09-13T08:42:48.917614+0000","last_unstale":"2024-09-13T08:42:50.189767+0000","last_undegraded":"2024-09-13T08:42:50.189767+0000","last_fullsized":"2024-09-13T08:42:50.189767+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:41.424198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.909728+0000","last_change":"2024-09-13T08:42:47.923979+0000","last_active":"2024-09-13T08:42:50.909728+0000","last_peered":"2024-09-13T08:42:50.909728+0000","last_clean":"2024-09-13T08:42:50.909728+0000","last_became_active":"2024-09-13T08:42:47.923665+0000","last_became_peered":"2024-09-13T08:42:47.923665+0000","last_unstale":"2024-09-13T08:42:50.909728+0000","last_undegraded":"2024-09-13T08:42:50.909728+0000","last_fullsized":"2024-09-13T08:42:50.909728+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:14:45.135574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215617+0000","last_change":"2024-09-13T08:42:48.928863+0000","last_active":"2024-09-13T08:42:50.215617+0000","last_peered":"2024-09-13T08:42:50.215617+0000","last_clean":"2024-09-13T08:42:50.215617+0000","last_became_active":"2024-09-13T08:42:48.928507+0000","last_became_peered":"2024-09-13T08:42:48.928507+0000","last_unstale":"2024-09-13T08:42:50.215617+0000","last_undegraded":"2024-09-13T08:42:50.215617+0000","last_fullsized":"2024-09-13T08:42:50.215617+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:04.109657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.920252+0000","last_change":"2024-09-13T08:42:47.927904+0000","last_active":"2024-09-13T08:42:49.920252+0000","last_peered":"2024-09-13T08:42:49.920252+0000","last_clean":"2024-09-13T08:42:49.920252+0000","last_became_active":"2024-09-13T08:42:47.927227+0000","last_became_peered":"2024-09-13T08:42:47.927227+0000","last_unstale":"2024-09-13T08:42:49.920252+0000","last_undegraded":"2024-09-13T08:42:49.920252+0000","last_fullsized":"2024-09-13T08:42:49.920252+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:50:05.255239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.918198+0000","last_change":"2024-09-13T08:42:48.934552+0000","last_active":"2024-09-13T08:42:49.918198+0000","last_peered":"2024-09-13T08:42:49.918198+0000","last_clean":"2024-09-13T08:42:49.918198+0000","last_became_active":"2024-09-13T08:42:48.934285+0000","last_became_peered":"2024-09-13T08:42:48.934285+0000","last_unstale":"2024-09-13T08:42:49.918198+0000","last_undegraded":"2024-09-13T08:42:49.918198+0000","last_fullsized":"2024-09-13T08:42:49.918198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:03:29.023419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919965+0000","last_change":"2024-09-13T08:42:47.920084+0000","last_active":"2024-09-13T08:42:49.919965+0000","last_peered":"2024-09-13T08:42:49.919965+0000","last_clean":"2024-09-13T08:42:49.919965+0000","last_became_active":"2024-09-13T08:42:47.918598+0000","last_became_peered":"2024-09-13T08:42:47.918598+0000","last_unstale":"2024-09-13T08:42:49.919965+0000","last_undegraded":"2024-09-13T08:42:49.919965+0000","last_fullsized":"2024-09-13T08:42:49.919965+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:58.276766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715313+0000","last_change":"2024-09-13T08:42:48.931267+0000","last_active":"2024-09-13T08:42:50.715313+0000","last_peered":"2024-09-13T08:42:50.715313+0000","last_clean":"2024-09-13T08:42:50.715313+0000","last_became_active":"2024-09-13T08:42:48.930886+0000","last_became_peered":"2024-09-13T08:42:48.930886+0000","last_unstale":"2024-09-13T08:42:50.715313+0000","last_undegraded":"2024-09-13T08:42:50.715313+0000","last_fullsized":"2024-09-13T08:42:50.715313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:36.746413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.716300+0000","last_change":"2024-09-13T08:42:47.918161+0000","last_active":"2024-09-13T08:42:50.716300+0000","last_peered":"2024-09-13T08:42:50.716300+0000","last_clean":"2024-09-13T08:42:50.716300+0000","last_became_active":"2024-09-13T08:42:47.917855+0000","last_became_peered":"2024-09-13T08:42:47.917855+0000","last_unstale":"2024-09-13T08:42:50.716300+0000","last_undegraded":"2024-09-13T08:42:50.716300+0000","last_fullsized":"2024-09-13T08:42:50.716300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:40.421365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917989+0000","last_change":"2024-09-13T08:42:48.939097+0000","last_active":"2024-09-13T08:42:49.917989+0000","last_peered":"2024-09-13T08:42:49.917989+0000","last_clean":"2024-09-13T08:42:49.917989+0000","last_became_active":"2024-09-13T08:42:48.938830+0000","last_became_peered":"2024-09-13T08:42:48.938830+0000","last_unstale":"2024-09-13T08:42:49.917989+0000","last_undegraded":"2024-09-13T08:42:49.917989+0000","last_fullsized":"2024-09-13T08:42:49.917989+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:07.749566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917709+0000","last_change":"2024-09-13T08:42:47.917137+0000","last_active":"2024-09-13T08:42:49.917709+0000","last_peered":"2024-09-13T08:42:49.917709+0000","last_clean":"2024-09-13T08:42:49.917709+0000","last_became_active":"2024-09-13T08:42:47.916303+0000","last_became_peered":"2024-09-13T08:42:47.916303+0000","last_unstale":"2024-09-13T08:42:49.917709+0000","last_undegraded":"2024-09-13T08:42:49.917709+0000","last_fullsized":"2024-09-13T08:42:49.917709+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:16:00.042800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.917570+0000","last_change":"2024-09-13T08:42:48.929443+0000","last_active":"2024-09-13T08:42:49.917570+0000","last_peered":"2024-09-13T08:42:49.917570+0000","last_clean":"2024-09-13T08:42:49.917570+0000","last_became_active":"2024-09-13T08:42:48.926428+0000","last_became_peered":"2024-09-13T08:42:48.926428+0000","last_unstale":"2024-09-13T08:42:49.917570+0000","last_undegraded":"2024-09-13T08:42:49.917570+0000","last_fullsized":"2024-09-13T08:42:49.917570+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.153278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.215248+0000","last_change":"2024-09-13T08:42:47.924426+0000","last_active":"2024-09-13T08:42:50.215248+0000","last_peered":"2024-09-13T08:42:50.215248+0000","last_clean":"2024-09-13T08:42:50.215248+0000","last_became_active":"2024-09-13T08:42:47.924140+0000","last_became_peered":"2024-09-13T08:42:47.924140+0000","last_unstale":"2024-09-13T08:42:50.215248+0000","last_undegraded":"2024-09-13T08:42:50.215248+0000","last_fullsized":"2024-09-13T08:42:50.215248+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:11.065256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189456+0000","last_change":"2024-09-13T08:42:48.931667+0000","last_active":"2024-09-13T08:42:50.189456+0000","last_peered":"2024-09-13T08:42:50.189456+0000","last_clean":"2024-09-13T08:42:50.189456+0000","last_became_active":"2024-09-13T08:42:48.927482+0000","last_became_peered":"2024-09-13T08:42:48.927482+0000","last_unstale":"2024-09-13T08:42:50.189456+0000","last_undegraded":"2024-09-13T08:42:50.189456+0000","last_fullsized":"2024-09-13T08:42:50.189456+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:24.571274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.189047+0000","last_change":"2024-09-13T08:42:47.924508+0000","last_active":"2024-09-13T08:42:50.189047+0000","last_peered":"2024-09-13T08:42:50.189047+0000","last_clean":"2024-09-13T08:42:50.189047+0000","last_became_active":"2024-09-13T08:42:47.923507+0000","last_became_peered":"2024-09-13T08:42:47.923507+0000","last_unstale":"2024-09-13T08:42:50.189047+0000","last_undegraded":"2024-09-13T08:42:50.189047+0000","last_fullsized":"2024-09-13T08:42:50.189047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:14.035680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:49.919130+0000","last_change":"2024-09-13T08:42:48.927133+0000","last_active":"2024-09-13T08:42:49.919130+0000","last_peered":"2024-09-13T08:42:49.919130+0000","last_clean":"2024-09-13T08:42:49.919130+0000","last_became_active":"2024-09-13T08:42:48.926484+0000","last_became_peered":"2024-09-13T08:42:48.926484+0000","last_unstale":"2024-09-13T08:42:49.919130+0000","last_undegraded":"2024-09-13T08:42:49.919130+0000","last_fullsized":"2024-09-13T08:42:49.919130+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:47.888926+0000","last_clean_scrub_stamp":"2024-09-13T08:42:47.888926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:50:05.635794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.683100+0000","last_change":"2024-09-13T08:42:47.920152+0000","last_active":"2024-09-13T08:42:50.683100+0000","last_peered":"2024-09-13T08:42:50.683100+0000","last_clean":"2024-09-13T08:42:50.683100+0000","last_became_active":"2024-09-13T08:42:47.916985+0000","last_became_peered":"2024-09-13T08:42:47.916985+0000","last_unstale":"2024-09-13T08:42:50.683100+0000","last_undegraded":"2024-09-13T08:42:50.683100+0000","last_fullsized":"2024-09-13T08:42:50.683100+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:42:46.886094+0000","last_clean_scrub_stamp":"2024-09-13T08:42:46.886094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:10:47.642097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":293,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T08:42:50.715127+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:42:50.715127+0000","last_peered":"2024-09-13T08:42:50.715127+0000","last_clean":"2024-09-13T08:42:50.715127+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:42:50.715127+0000","last_undegraded":"2024-09-13T08:42:50.715127+0000","last_fullsized":"2024-09-13T08:42:50.715127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509514,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1355776,"data_stored":587737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1335296,"data_stored":586001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1937408,"data_stored":1176461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542217,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1351680,"data_stored":587703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509514,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1335296,"data_stored":586001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1347584,"data_stored":586639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509515,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1929216,"data_stored":1176369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509513,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1343488,"data_stored":586653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:42:53.591 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:42:53.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:42:53.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 -- 172.21.15.31:0/3066459868 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c15bfc0 msgr2=0x7f010c072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 --2- 172.21.15.31:0/3066459868 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c15bfc0 0x7f010c072560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009860 tx=0x7f00fc02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 -- 172.21.15.31:0/3066459868 shutdown_connections 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 --2- 172.21.15.31:0/3066459868 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c15bfc0 0x7f010c072560 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 -- 172.21.15.31:0/3066459868 >> 172.21.15.31:0/3066459868 conn(0x7f010c0cfbb0 msgr2=0x7f010c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 -- 172.21.15.31:0/3066459868 shutdown_connections 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.795+0000 7f0110e00640 1 -- 172.21.15.31:0/3066459868 wait complete. 2024-09-13T08:42:53.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.796+0000 7f0110e00640 1 Processor -- start 2024-09-13T08:42:53.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 -- start start 2024-09-13T08:42:53.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c06fa60 0x7f010c152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 0x7f010c1e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f010c06e760 con 0x7f010c1e8d90 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f010c0cc060 con 0x7f010c15bfc0 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.797+0000 7f0110e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f010c0b30f0 con 0x7f010c06fa60 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42586/0 (socket says 172.21.15.31:42586) 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 -- 172.21.15.31:0/529613531 learned_addr learned my addr 172.21.15.31:0/529613531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:53.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f0109e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c06fa60 0x7f010c152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f0101400640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 0x7f010c1e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 -- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c06fa60 msgr2=0x7f010c152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c06fa60 0x7f010c152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 -- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 msgr2=0x7f010c1e8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 0x7f010c1e8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f010a800640 1 -- 172.21.15.31:0/529613531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c153d10 con 0x7f010c1e8d90 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.798+0000 7f0101400640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 0x7f010c1e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f010a800640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f010400cbc0 tx=0x7f010400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:53.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104016020 con 0x7f010c1e8d90 2024-09-13T08:42:53.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f010400ae30 con 0x7f010c1e8d90 2024-09-13T08:42:53.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104012660 con 0x7f010c1e8d90 2024-09-13T08:42:53.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c153ff0 con 0x7f010c1e8d90 2024-09-13T08:42:53.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.799+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c1545b0 con 0x7f010c1e8d90 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.800+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c153aa0 con 0x7f010c1e8d90 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.801+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f0104019020 con 0x7f010c1e8d90 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.802+0000 7f0102a00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 0x7f00e807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.802+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f010409af80 con 0x7f010c1e8d90 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.802+0000 7f0109e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 0x7f00e807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.803+0000 7f0109e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 0x7f00e807ae30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f00fc034000 tx=0x7f00fc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:53.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:53.807+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0104020030 con 0x7f010c1e8d90 2024-09-13T08:42:54.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.074+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f010c072560 con 0x7f010c1e8d90 2024-09-13T08:42:54.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.076+0000 7f0102a00640 1 -- 172.21.15.31:0/529613531 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f01040633c0 con 0x7f010c1e8d90 2024-09-13T08:42:54.078 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:42:54.079 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:42:54.079 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:42:54.079 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:42:54.079 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:42:54.079 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 msgr2=0x7f00e807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 0x7f00e807ae30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f00fc034000 tx=0x7f00fc004730 comp rx=0 tx=0).stop 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 msgr2=0x7f010c155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f010400cbc0 tx=0x7f010400a430 comp rx=0 tx=0).stop 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 shutdown_connections 2024-09-13T08:42:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00e8078970 0x7f00e807ae30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f010c1e8d90 0x7f010c155a30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f010c15bfc0 0x7f010c1e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 --2- 172.21.15.31:0/529613531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f010c06fa60 0x7f010c152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 >> 172.21.15.31:0/529613531 conn(0x7f010c0cfbb0 msgr2=0x7f010c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 shutdown_connections 2024-09-13T08:42:54.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.079+0000 7f0110e00640 1 -- 172.21.15.31:0/529613531 wait complete. 2024-09-13T08:42:54.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.255+0000 7fc582a00640 1 -- 172.21.15.31:0/526901570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5640041e0 msgr2=0x7fc564004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.255+0000 7fc582a00640 1 --2- 172.21.15.31:0/526901570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5640041e0 0x7fc564004800 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc568009b50 tx=0x7fc564005830 comp rx=0 tx=0).stop 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.256+0000 7fc582000640 1 -- 172.21.15.31:0/526901570 reap_dead start 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.256+0000 7fc582a00640 1 -- 172.21.15.31:0/526901570 shutdown_connections 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.256+0000 7fc582a00640 1 -- 172.21.15.31:0/526901570 >> 172.21.15.31:0/526901570 conn(0x7fc57c08d8b0 msgr2=0x7fc57c09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.256+0000 7fc582a00640 1 -- 172.21.15.31:0/526901570 shutdown_connections 2024-09-13T08:42:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.256+0000 7fc582a00640 1 -- 172.21.15.31:0/526901570 wait complete. 2024-09-13T08:42:54.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.257+0000 7fc582a00640 1 Processor -- start 2024-09-13T08:42:54.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.257+0000 7fc582a00640 1 -- start start 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 0x7fc57c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57c15c8b0 0x7fc57c1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0b1690 con 0x7fc57c15cd30 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0d3ef0 con 0x7fc57c15c8b0 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.258+0000 7fc582a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc57c0b1810 con 0x7fc5640041e0 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42602/0 (socket says 172.21.15.31:42602) 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 -- 172.21.15.31:0/3885471236 learned_addr learned my addr 172.21.15.31:0/3885471236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc580c00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57c15c8b0 0x7fc57c1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc581600640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 0x7fc57c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 -- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 msgr2=0x7fc57c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 0x7fc57c0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 -- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57c15c8b0 msgr2=0x7fc57c1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57c15c8b0 0x7fc57c1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 -- 172.21.15.31:0/3885471236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57c0a1030 con 0x7fc57c15cd30 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc581600640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 0x7fc57c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.259+0000 7fc582000640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0d38c0 tx=0x7fc574001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.260+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574018070 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.260+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc574011960 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.260+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574011d70 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.260+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57c15f4f0 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.260+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc57c15fab0 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.261+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57c0b0f80 con 0x7fc57c15cd30 2024-09-13T08:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.262+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fc57401b070 con 0x7fc57c15cd30 2024-09-13T08:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.263+0000 7fc579600640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 0x7fc55c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.264+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fc57409a970 con 0x7fc57c15cd30 2024-09-13T08:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.264+0000 7fc581600640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 0x7fc55c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.264+0000 7fc581600640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 0x7fc55c07ae30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc570000fe0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:54.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.268+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc5740626d0 con 0x7fc57c15cd30 2024-09-13T08:42:54.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.498+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc57c0d4cf0 con 0x7fc57c15cd30 2024-09-13T08:42:54.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.499+0000 7fc579600640 1 -- 172.21.15.31:0/3885471236 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5222 (secure 0 0 0) 0x7fc574067100 con 0x7fc57c15cd30 2024-09-13T08:42:54.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:42:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"btime":"2024-09-13T08:42:51:923681+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:42:54.501 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2024-09-13T08:42:54.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 msgr2=0x7fc55c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 0x7fc55c07ae30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc570000fe0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 msgr2=0x7fc57c1607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0d38c0 tx=0x7fc574001b80 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 shutdown_connections 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc55c078970 0x7fc55c07ae30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57c15cd30 0x7fc57c1607f0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57c15c8b0 0x7fc57c1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 --2- 172.21.15.31:0/3885471236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5640041e0 0x7fc57c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.501+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 >> 172.21.15.31:0/3885471236 conn(0x7fc57c08d8b0 msgr2=0x7fc57c0d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.502+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 shutdown_connections 2024-09-13T08:42:54.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:54.502+0000 7fc582a00640 1 -- 172.21.15.31:0/3885471236 wait complete. 2024-09-13T08:42:54.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:42:48.347422+0000', 'modified': '2024-09-13T08:42:51.365032+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6963}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6963': {'gid': 6963, 'name': 'd', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6801/3014570213', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 3014570213}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 3014570213}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6963, 'qdb_cluster': [6963]} 2024-09-13T08:42:54.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:42:54.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T08:42:55.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 -- 172.21.15.31:0/558500372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 msgr2=0x7efcb80cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 --2- 172.21.15.31:0/558500372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb80cdb80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efcac005fd0 tx=0x7efcac02a530 comp rx=0 tx=0).stop 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 -- 172.21.15.31:0/558500372 shutdown_connections 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 --2- 172.21.15.31:0/558500372 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb8073860 0x7efcb8073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 --2- 172.21.15.31:0/558500372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb80cdb80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 -- 172.21.15.31:0/558500372 >> 172.21.15.31:0/558500372 conn(0x7efcb80b18f0 msgr2=0x7efcb80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 -- 172.21.15.31:0/558500372 shutdown_connections 2024-09-13T08:42:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.084+0000 7efcc0000640 1 -- 172.21.15.31:0/558500372 wait complete. 2024-09-13T08:42:55.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.086+0000 7efcc0000640 1 Processor -- start 2024-09-13T08:42:55.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.086+0000 7efcc0000640 1 -- start start 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb80ced20 0x7efcb80b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb81f8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcb80733e0 con 0x7efcb815be30 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcb80ce9e0 con 0x7efcb8073860 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcc0000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efcb8074230 con 0x7efcb80ced20 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcbd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcbe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb81f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcbd600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:51076/0 (socket says 172.21.15.31:51076) 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcbd600640 1 -- 172.21.15.31:0/4241393362 learned_addr learned my addr 172.21.15.31:0/4241393362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.087+0000 7efcbcc00640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb80ced20 0x7efcb80b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 -- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb80ced20 msgr2=0x7efcb80b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb80ced20 0x7efcb80b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 -- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 msgr2=0x7efcb81f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb81f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 -- 172.21.15.31:0/4241393362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb809d120 con 0x7efcb8073860 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbe000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb81f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:42:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.088+0000 7efcbd600640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efcac034040 tx=0x7efcac0039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.089+0000 7efcb5600640 1 -- 172.21.15.31:0/4241393362 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac03e030 con 0x7efcb8073860 2024-09-13T08:42:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.089+0000 7efcb5600640 1 -- 172.21.15.31:0/4241393362 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcac0047a0 con 0x7efcb8073860 2024-09-13T08:42:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.089+0000 7efcb5600640 1 -- 172.21.15.31:0/4241393362 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac035040 con 0x7efcb8073860 2024-09-13T08:42:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.089+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb8158680 con 0x7efcb8073860 2024-09-13T08:42:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.089+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb8158c40 con 0x7efcb8073860 2024-09-13T08:42:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.091+0000 7efcb5600640 1 -- 172.21.15.31:0/4241393362 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7efcac041030 con 0x7efcb8073860 2024-09-13T08:42:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.092+0000 7efcc3011740 1 Processor -- start 2024-09-13T08:42:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.092+0000 7efcb5600640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 0x7efc9807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.092+0000 7efcc3011740 1 -- start start 2024-09-13T08:42:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.092+0000 7efcb5600640 1 -- 172.21.15.31:0/4241393362 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7efcac0bb2d0 con 0x7efcb8073860 2024-09-13T08:42:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.092+0000 7efcbe000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 0x7efc9807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 0x55c303673390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcbe000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 0x7efc9807ae30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a730 tx=0x7efcb000d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 0x55c3036738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c3035b2430 con 0x55c3036748b0 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c3035a25d0 con 0x55c30362ffa0 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55c303591a00 con 0x55c30362fa20 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcbd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcbd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40688/0 (socket says 172.21.15.31:40688) 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcbd600640 1 -- 172.21.15.31:0/2356330992 learned_addr learned my addr 172.21.15.31:0/2356330992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.093+0000 7efcbe000640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 0x55c3036738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbcc00640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 0x55c303673390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 -- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 msgr2=0x55c303673390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 0x55c303673390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 -- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 msgr2=0x55c3036738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 0x55c3036738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 -- 172.21.15.31:0/2356330992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c30355a520 con 0x55c3036748b0 2024-09-13T08:42:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbe000640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 0x55c3036738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbcc00640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 0x55c303673390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efcbd600640 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efcac082d30 tx=0x7efcac009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f46c8ccc36a1a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efca9c00640 1 -- 172.21.15.31:0/2356330992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac048040 con 0x55c3036748b0 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efca9c00640 1 -- 172.21.15.31:0/2356330992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcac083030 con 0x55c3036748b0 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.094+0000 7efca9c00640 1 -- 172.21.15.31:0/2356330992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac08c6c0 con 0x55c3036748b0 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 -- 172.21.15.31:0/2356330992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 msgr2=0x55c303674c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efcac082d30 tx=0x7efcac009680 comp rx=0 tx=0).stop 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 -- 172.21.15.31:0/2356330992 shutdown_connections 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c30362ffa0 0x55c3036738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362fa20 0x55c303673390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 --2- 172.21.15.31:0/2356330992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3036748b0 0x55c303674c90 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 -- 172.21.15.31:0/2356330992 >> 172.21.15.31:0/2356330992 conn(0x55c30362aff0 msgr2=0x55c30362ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 -- 172.21.15.31:0/2356330992 shutdown_connections 2024-09-13T08:42:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.095+0000 7efcc3011740 1 -- 172.21.15.31:0/2356330992 wait complete. 2024-09-13T08:42:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.096+0000 7efcc3011740 1 -- start start 2024-09-13T08:42:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.096+0000 7efcc3011740 1 Processor -- start 2024-09-13T08:42:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036739b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c303673ef0 0x55c303686c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c303538330 con 0x55c30362ffa0 2024-09-13T08:42:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c303595810 con 0x55c3036748b0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55c303302620 con 0x55c303673ef0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcc3011740 10 client.0 _ll_register_callbacks cb 0x55c30353d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcbd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcbd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40698/0 (socket says 172.21.15.31:40698) 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcbd600640 1 -- 172.21.15.31:0/1830190886 learned_addr learned my addr 172.21.15.31:0/1830190886 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcbcc00640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.097+0000 7efcbe000640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c303673ef0 0x55c303686c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 -- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c303673ef0 msgr2=0x55c303686c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c303673ef0 0x55c303686c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 -- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 msgr2=0x55c3036739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 -- 172.21.15.31:0/1830190886 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3032f3850 con 0x55c30362ffa0 2024-09-13T08:42:55.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcbd600640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efcac082fc0 tx=0x7efcac0b9960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c29ca37a69091bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac048040 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7efcac08c690 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcac083030 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7efcac087a60 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcc3011740 10 client.7071 fetch_fsmap learned FSMap version 71 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) -- 0x55c30336a6b0 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcac083030 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.098+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 71) ==== 3502+0+0 (secure 0 0 0) 0x7efcac0b9bc0 con 0x55c30362ffa0 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 10 client.7071 fetch_fsmap finished waiting for FSMap version 71 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 10 client.7071 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T08:42:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 20 client.7071 populate_metadata read hostname 'smithi031' 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c303684720 0x55c303684b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 4 client.7071 mds_command: new command op to 6963 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 --> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3035a4520 con 0x55c303684720 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.099+0000 7efcbcc00640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c303684720 0x55c303684b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.100+0000 7efcbcc00640 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c303684720 0x55c303684b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e4441e52d73dd8a server_cookie=6a260584a3d21aed in_seq=0 out_seq=0 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.100+0000 7efc9e000640 10 client.7071 ms_handle_connect on v2:172.21.15.45:6800/3014570213 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efc9e000640 1 -- 172.21.15.31:0/1830190886 <== mds.0 v2:172.21.15.45:6800/3014570213 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c3035a4520 con 0x55c303684720 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efc9e000640 10 client.7071 handle_command_reply: tid=0 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 client.7071 shutdown 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 msgr2=0x55c30366dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efcac082fc0 tx=0x7efcac0b9960 comp rx=0 tx=0).stop 2024-09-13T08:42:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 shutdown_connections 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c303684720 0x55c303684b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c303673ef0 0x55c303686c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efc9e000640 0 client.7071 ms_handle_reset on v2:172.21.15.45:6800/3014570213 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 --2- 172.21.15.31:0/1830190886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30362ffa0 0x55c30366dbc0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 >> 172.21.15.31:0/1830190886 conn(0x55c30362fa20 msgr2=0x55c303629b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 shutdown_connections 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 1 -- 172.21.15.31:0/1830190886 wait complete. 2024-09-13T08:42:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.101+0000 7efcc3011740 20 client.7071 trim_cache size 0 max 16384 2024-09-13T08:42:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.120+0000 7efcc3011740 1 Processor -- start 2024-09-13T08:42:55.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.120+0000 7efcc3011740 1 -- start start 2024-09-13T08:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c303650790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036285b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81f5990 con 0x55c303673ef0 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcb809ccb0 con 0x55c3036748b0 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efcb81f5340 con 0x55c30362ffa0 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.121+0000 7efcbcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40704/0 (socket says 172.21.15.31:40704) 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 -- 172.21.15.31:0/2922025529 learned_addr learned my addr 172.21.15.31:0/2922025529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbe000640 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036285b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbd600640 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c303650790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 -- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 msgr2=0x55c303650790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c303650790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 -- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 msgr2=0x55c3036285b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036285b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 -- 172.21.15.31:0/2922025529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3036746e0 con 0x55c303673ef0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efcbcc00640 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x55c303658dd0 tx=0x7efca4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff75e5012c47cf1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efca9200640 1 -- 172.21.15.31:0/2922025529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4005360 con 0x55c303673ef0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efca9200640 1 -- 172.21.15.31:0/2922025529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efca4023070 con 0x55c303673ef0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.122+0000 7efca9200640 1 -- 172.21.15.31:0/2922025529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4007050 con 0x55c303673ef0 2024-09-13T08:42:55.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 -- 172.21.15.31:0/2922025529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 msgr2=0x55c303628070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x55c303658dd0 tx=0x7efca4009040 comp rx=0 tx=0).stop 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 -- 172.21.15.31:0/2922025529 shutdown_connections 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c3036285b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303628070 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 --2- 172.21.15.31:0/2922025529 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c303650790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 -- 172.21.15.31:0/2922025529 >> 172.21.15.31:0/2922025529 conn(0x55c30362fa20 msgr2=0x55c3036737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 -- 172.21.15.31:0/2922025529 shutdown_connections 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.123+0000 7efcc3011740 1 -- 172.21.15.31:0/2922025529 wait complete. 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.124+0000 7efcc3011740 1 -- start start 2024-09-13T08:42:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.124+0000 7efcc3011740 1 Processor -- start 2024-09-13T08:42:55.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c3036d9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c303629eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c303538330 con 0x55c303673ef0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c303595810 con 0x55c3036748b0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x55c3036857a0 con 0x55c30362ffa0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcc3011740 10 client.0 _ll_register_callbacks cb 0x7efcac08a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40706/0 (socket says 172.21.15.31:40706) 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 -- 172.21.15.31:0/27745820 learned_addr learned my addr 172.21.15.31:0/27745820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbe000640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c303629eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbd600640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c3036d9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 -- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 msgr2=0x55c3036d9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c3036d9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 -- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 msgr2=0x55c303629eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c303629eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbcc00640 1 -- 172.21.15.31:0/27745820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3036d9f50 con 0x55c303673ef0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.125+0000 7efcbd600640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c3036d9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efcbcc00640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efca4004bb0 tx=0x7efca4005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b5ceacd15254fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efc9e000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca402b020 con 0x55c303673ef0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7efca4007420 con 0x55c303673ef0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c30366e5a0 con 0x55c303673ef0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7efca4032050 con 0x55c303673ef0 2024-09-13T08:42:55.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efcc3011740 10 client.7077 fetch_fsmap learned FSMap version 71 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.126+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) -- 0x55c303654e20 con 0x55c303673ef0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55c30366e5a0 con 0x55c303673ef0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 71) ==== 3502+0+0 (secure 0 0 0) 0x7efca4035020 con 0x55c303673ef0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 10 client.7077 fetch_fsmap finished waiting for FSMap version 71 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 10 client.7077 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 20 client.7077 populate_metadata read hostname 'smithi031' 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c30362cdd0 0x55c3036f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 4 client.7077 mds_command: new command op to 6963 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 --> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] -- command(tid 0: {"prefix": "status"}) -- 0x55c30366f120 con 0x55c30362cdd0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.127+0000 7efcbd600640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c30362cdd0 0x55c3036f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcbd600640 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c30362cdd0 0x55c3036f71e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e6505dfbb5a803 server_cookie=e9a739409ca64ff3 in_seq=0 out_seq=0 2024-09-13T08:42:55.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efc9e000640 10 client.7077 ms_handle_connect on v2:172.21.15.45:6800/3014570213 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efc9e000640 1 -- 172.21.15.31:0/27745820 <== mds.0 v2:172.21.15.45:6800/3014570213 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55c30366f120 con 0x55c30362cdd0 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efc9e000640 10 client.7077 handle_command_reply: tid=0 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 client.7077 shutdown 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 msgr2=0x55c303629970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efca4004bb0 tx=0x7efca4005590 comp rx=0 tx=0).stop 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 shutdown_connections 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:6800/3014570213,v1:172.21.15.45:6801/3014570213] conn(0x55c30362cdd0 0x55c3036f71e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c3036748b0 0x55c303629eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c303673ef0 0x55c303629970 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 --2- 172.21.15.31:0/27745820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x55c30362ffa0 0x55c3036d9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efc9e000640 0 client.7077 ms_handle_reset on v2:172.21.15.45:6800/3014570213 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 >> 172.21.15.31:0/27745820 conn(0x55c30362fa20 msgr2=0x55c303628200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 shutdown_connections 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.128+0000 7efcc3011740 1 -- 172.21.15.31:0/27745820 wait complete. 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc3011740 20 client.7077 trim_cache size 0 max 16384 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:42:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 6963, 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.229644231, 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 71, 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 108, 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 108, 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 10.509822767999999 2024-09-13T08:42:55.136 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 msgr2=0x7efc9807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 0x7efc9807ae30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a730 tx=0x7efcb000d040 comp rx=0 tx=0).stop 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 msgr2=0x7efcb815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efcac034040 tx=0x7efcac0039e0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 shutdown_connections 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efc98078970 0x7efc9807ae30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb815be30 0x7efcb81f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efcb80ced20 0x7efcb80b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 --2- 172.21.15.31:0/4241393362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb8073860 0x7efcb815b020 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 >> 172.21.15.31:0/4241393362 conn(0x7efcb80b18f0 msgr2=0x7efcb80b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 shutdown_connections 2024-09-13T08:42:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.129+0000 7efcc0000640 1 -- 172.21.15.31:0/4241393362 wait complete. 2024-09-13T08:42:55.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:42:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 -- 172.21.15.31:0/230107579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 msgr2=0x7fdf0c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 --2- 172.21.15.31:0/230107579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0cf850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009830 tx=0x7fdefc02a5b0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 -- 172.21.15.31:0/230107579 shutdown_connections 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 --2- 172.21.15.31:0/230107579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0cf850 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 -- 172.21.15.31:0/230107579 >> 172.21.15.31:0/230107579 conn(0x7fdf0c0cc910 msgr2=0x7fdf0c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 -- 172.21.15.31:0/230107579 shutdown_connections 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.304+0000 7fdf11600640 1 -- 172.21.15.31:0/230107579 wait complete. 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.306+0000 7fdf11600640 1 Processor -- start 2024-09-13T08:42:55.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.306+0000 7fdf11600640 1 -- start start 2024-09-13T08:42:55.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf0c15fe20 0x7fdf0c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c0cec50 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c0cead0 con 0x7fdf0c15bd70 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf11600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdf0c0cb850 con 0x7fdf0c15fe20 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40708/0 (socket says 172.21.15.31:40708) 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 -- 172.21.15.31:0/3466486883 learned_addr learned my addr 172.21.15.31:0/3466486883 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0b400640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf0c15fe20 0x7fdf0c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0aa00640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 -- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf0c15fe20 msgr2=0x7fdf0c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf0c15fe20 0x7fdf0c155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 -- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 msgr2=0x7fdf0c0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0a000640 1 -- 172.21.15.31:0/3466486883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c0c6490 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.307+0000 7fdf0aa00640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:42:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf0a000640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c490 tx=0x7fdef800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef8016020 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdef8012940 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef8012d90 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0c0c6770 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.308+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0c1f8370 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.309+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf0c0cf850 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.310+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fdef8019020 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.311+0000 7fdf02a00640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 0x7fdeec07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:55.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.312+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fdef809a870 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.312+0000 7fdf0aa00640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 0x7fdeec07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:55.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.312+0000 7fdf0aa00640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 0x7fdeec07ae30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdefc034000 tx=0x7fdefc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:55.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.316+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdef8020030 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.555+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdf0c0cd100 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.555+0000 7fdf02a00640 1 -- 172.21.15.31:0/3466486883 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v71) ==== 53+0+73 (secure 0 0 0) 0x7fdef8062cb0 con 0x7fdf0c15f9e0 2024-09-13T08:42:55.557 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:42:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 msgr2=0x7fdeec07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 0x7fdeec07ae30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdefc034000 tx=0x7fdefc004730 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 msgr2=0x7fdf0c1f8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c490 tx=0x7fdef800c950 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 shutdown_connections 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdeec078970 0x7fdeec07ae30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf0c15fe20 0x7fdf0c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c15f9e0 0x7fdf0c1f8b70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 --2- 172.21.15.31:0/3466486883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c15bd70 0x7fdf0c0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.557+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 >> 172.21.15.31:0/3466486883 conn(0x7fdf0c0cc910 msgr2=0x7fdf0c15e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.558+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 shutdown_connections 2024-09-13T08:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55.558+0000 7fdf11600640 1 -- 172.21.15.31:0/3466486883 wait complete. 2024-09-13T08:42:55.574 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:42:55.575 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:55.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:55.575 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:42:55.612 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42356sec preferred_lft 42356sec 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:42:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:42:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:55.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:55.615 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:42:55.706 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:42:55.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:55.709 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:42:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:42:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:42:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:42:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:42:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:42:55.746 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:42:55.747 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:42:55.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:42:55.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:42:55.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:42:55.747 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:42:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:42:55.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:55.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:55.946 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:42:55.982 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:42:55.982 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:42:55.982 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:55.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:55.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:42:56.079 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:56.079 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:56.114 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:42:56.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:56.114 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:56.189 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:42:56.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:56.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:56.318 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:56.318 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:56.347 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:56.347 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:56.347 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:42:56.347 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:42:56.348 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:42:56.350 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:42:56.350 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:42:56.350 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:42:56.350 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:42:56.350 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:42:56.353 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:42:56.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:56.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:42:56.410 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:42:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:42:56.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:56.511 INFO:teuthology.orchestra.run.smithi031.stdout:client6730 2024-09-13T08:42:56.511 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:42:56.511 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:56.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:42:56.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 -- 172.21.15.31:0/1805917196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 msgr2=0x7fdec00ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:56.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 --2- 172.21.15.31:0/1805917196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec00ce3c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fdec00ca470 tx=0x7fdea802a560 comp rx=0 tx=0).stop 2024-09-13T08:42:56.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 -- 172.21.15.31:0/1805917196 shutdown_connections 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 --2- 172.21.15.31:0/1805917196 >> v2:172.21.15.45:3300/0 conn(0x7fdec00ce900 0x7fdec00cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 --2- 172.21.15.31:0/1805917196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec00ce3c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 -- 172.21.15.31:0/1805917196 >> 172.21.15.31:0/1805917196 conn(0x7fdec00cc910 msgr2=0x7fdec00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 -- 172.21.15.31:0/1805917196 shutdown_connections 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.746+0000 7fdec5200640 1 -- 172.21.15.31:0/1805917196 wait complete. 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.748+0000 7fdec5200640 1 Processor -- start 2024-09-13T08:42:56.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.748+0000 7fdec5200640 1 -- start start 2024-09-13T08:42:56.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec00ce900 0x7fdec015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:56.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:56.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdec015bd70 0x7fdec014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:56.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdec01600f0 con 0x7fdec00cf850 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00cb850 con 0x7fdec00ce900 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdec5200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdec014bbb0 con 0x7fdec015bd70 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40722/0 (socket says 172.21.15.31:40722) 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 -- 172.21.15.31:0/397451441 learned_addr learned my addr 172.21.15.31:0/397451441 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdebde00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec00ce900 0x7fdec015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdebe800640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdec015bd70 0x7fdec014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 -- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdec015bd70 msgr2=0x7fdec014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdec015bd70 0x7fdec014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.749+0000 7fdeb7e00640 1 -- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec00ce900 msgr2=0x7fdec015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb7e00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec00ce900 0x7fdec015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:56.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb7e00640 1 -- 172.21.15.31:0/397451441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec01f5e20 con 0x7fdec00cf850 2024-09-13T08:42:56.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb7e00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdea8000c00 tx=0x7fdea80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:56.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea803f070 con 0x7fdec00cf850 2024-09-13T08:42:56.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdea80048b0 con 0x7fdec00cf850 2024-09-13T08:42:56.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8035030 con 0x7fdec00cf850 2024-09-13T08:42:56.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec015e480 con 0x7fdec00cf850 2024-09-13T08:42:56.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.750+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec015ea40 con 0x7fdec00cf850 2024-09-13T08:42:56.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.752+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec015f210 con 0x7fdec00cf850 2024-09-13T08:42:56.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.753+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fdea8042070 con 0x7fdec00cf850 2024-09-13T08:42:56.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.754+0000 7fdeb6000640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 0x7fde9c07ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:56.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.754+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fdea80bba30 con 0x7fdec00cf850 2024-09-13T08:42:56.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.754+0000 7fdebde00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 0x7fde9c07ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:56.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.754+0000 7fdebde00640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 0x7fde9c07ae30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdec00c6b60 tx=0x7fdeb0009060 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:56.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.758+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdea8083830 con 0x7fdec00cf850 2024-09-13T08:42:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.995+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdec00cf600 con 0x7fdec00cf850 2024-09-13T08:42:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.996+0000 7fdeb6000640 1 -- 172.21.15.31:0/397451441 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v71) ==== 53+0+73 (secure 0 0 0) 0x7fdec00cf600 con 0x7fdec00cf850 2024-09-13T08:42:56.998 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:42:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 msgr2=0x7fde9c07ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 0x7fde9c07ae30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdec00c6b60 tx=0x7fdeb0009060 comp rx=0 tx=0).stop 2024-09-13T08:42:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 msgr2=0x7fdec01f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdea8000c00 tx=0x7fdea80039e0 comp rx=0 tx=0).stop 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 shutdown_connections 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde9c078970 0x7fde9c07ae30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdec015bd70 0x7fdec014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdec00cf850 0x7fdec01f58e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 --2- 172.21.15.31:0/397451441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec00ce900 0x7fdec015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 >> 172.21.15.31:0/397451441 conn(0x7fdec00cc910 msgr2=0x7fdec00b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 shutdown_connections 2024-09-13T08:42:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:56.998+0000 7fdec5200640 1 -- 172.21.15.31:0/397451441 wait complete. 2024-09-13T08:42:57.016 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:42:57.016 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:42:57.016 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:42:57.016 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:42:57.016 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:57.017 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:42:57.017 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:42:57.017 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:42:57.017 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:57.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.017 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42359sec preferred_lft 42359sec 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:42:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:42:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:42:57.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.061 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:42:57.156 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:42:57.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.159 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:42:57.197 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:42:57.197 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:42:57.197 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:42:57.197 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:42:57.197 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:42:57.197 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:42:57.198 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:42:57.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:42:57.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:42:57.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:42:57.198 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:42:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:42:57.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:57.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.396 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:42:57.431 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:42:57.431 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:42:57.432 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:57.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:42:57.528 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:57.528 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:57.563 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:42:57.564 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:42:57.564 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:57.638 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:42:57.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:57.766 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:57.767 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:57.793 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:42:57.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:42:57.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:42:57.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:42:57.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:42:57.795 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:42:57.795 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:42:57.795 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:42:57.795 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:42:57.795 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:42:57.798 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:42:57.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:57.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:42:57.853 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:42:57.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:42:57.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:42:57.944 INFO:teuthology.orchestra.run.smithi045.stdout:client7092 2024-09-13T08:42:57.944 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:42:57.944 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:42:57.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:42:58.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 -- 172.21.15.31:0/3364076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe578072560 msgr2=0x7fe5780cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 --2- 172.21.15.31:0/3364076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe578072560 0x7fe5780cb740 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe560005fd0 tx=0x7fe56002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 -- 172.21.15.31:0/3364076735 shutdown_connections 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 --2- 172.21.15.31:0/3364076735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe5780cbc80 0x7fe5780b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 --2- 172.21.15.31:0/3364076735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe578072560 0x7fe5780cb740 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 -- 172.21.15.31:0/3364076735 >> 172.21.15.31:0/3364076735 conn(0x7fe5780cfbb0 msgr2=0x7fe5780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 -- 172.21.15.31:0/3364076735 shutdown_connections 2024-09-13T08:42:58.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.153+0000 7fe57f200640 1 -- 172.21.15.31:0/3364076735 wait complete. 2024-09-13T08:42:58.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.155+0000 7fe57f200640 1 Processor -- start 2024-09-13T08:42:58.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.155+0000 7fe57f200640 1 -- start start 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe578072560 0x7fe578152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5780cbc80 0x7fe57814f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe57806e740 con 0x7fe57815bfc0 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe57806e5c0 con 0x7fe5780cbc80 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe5780b2f20 con 0x7fe578072560 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40746/0 (socket says 172.21.15.31:40746) 2024-09-13T08:42:58.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.156+0000 7fe57d200640 1 -- 172.21.15.31:0/2038505637 learned_addr learned my addr 172.21.15.31:0/2038505637 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:58.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe56f400640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5780cbc80 0x7fe57814f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe577e00640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe578072560 0x7fe578152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 -- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe578072560 msgr2=0x7fe578152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe578072560 0x7fe578152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 -- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5780cbc80 msgr2=0x7fe57814f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5780cbc80 0x7fe57814f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 -- 172.21.15.31:0/2038505637 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5780c8690 con 0x7fe57815bfc0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57d200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe57000e970 tx=0x7fe57000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57000cc60 con 0x7fe57815bfc0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe57000ce00 con 0x7fe57815bfc0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570005280 con 0x7fe57815bfc0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5780c8970 con 0x7fe57815bfc0 2024-09-13T08:42:58.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.157+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5780c8f30 con 0x7fe57815bfc0 2024-09-13T08:42:58.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.158+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5780b3350 con 0x7fe57815bfc0 2024-09-13T08:42:58.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.160+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe570005420 con 0x7fe57815bfc0 2024-09-13T08:42:58.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.161+0000 7fe574c00640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 0x7fe55407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:58.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.161+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fe57009b620 con 0x7fe57815bfc0 2024-09-13T08:42:58.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.161+0000 7fe577e00640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 0x7fe55407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:58.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.162+0000 7fe577e00640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 0x7fe55407ae30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe568007fd0 tx=0x7fe568009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:58.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.166+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe570063a60 con 0x7fe57815bfc0 2024-09-13T08:42:58.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.401+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fe5780cb310 con 0x7fe57815bfc0 2024-09-13T08:42:58.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.947+0000 7fe574c00640 1 -- 172.21.15.31:0/2038505637 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v72) ==== 136+0+0 (secure 0 0 0) 0x7fe570063360 con 0x7fe57815bfc0 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 msgr2=0x7fe55407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 0x7fe55407ae30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe568007fd0 tx=0x7fe568009040 comp rx=0 tx=0).stop 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 msgr2=0x7fe5781f25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe57000e970 tx=0x7fe57000ed70 comp rx=0 tx=0).stop 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 shutdown_connections 2024-09-13T08:42:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe554078970 0x7fe55407ae30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe57815bfc0 0x7fe5781f25e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5780cbc80 0x7fe57814f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 --2- 172.21.15.31:0/2038505637 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe578072560 0x7fe578152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 >> 172.21.15.31:0/2038505637 conn(0x7fe5780cfbb0 msgr2=0x7fe5780d0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 shutdown_connections 2024-09-13T08:42:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:58.950+0000 7fe57f200640 1 -- 172.21.15.31:0/2038505637 wait complete. 2024-09-13T08:42:58.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:42:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 -- 172.21.15.31:0/991013040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415bb70 msgr2=0x7fcb940cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 --2- 172.21.15.31:0/991013040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415bb70 0x7fcb940cee40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009910 tx=0x7fcb8402a560 comp rx=0 tx=0).stop 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 -- 172.21.15.31:0/991013040 shutdown_connections 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 --2- 172.21.15.31:0/991013040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9415f820 0x7fcb9415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 --2- 172.21.15.31:0/991013040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415bb70 0x7fcb940cee40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 -- 172.21.15.31:0/991013040 >> 172.21.15.31:0/991013040 conn(0x7fcb940cc7a0 msgr2=0x7fcb940ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 -- 172.21.15.31:0/991013040 shutdown_connections 2024-09-13T08:42:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.136+0000 7fcb99800640 1 -- 172.21.15.31:0/991013040 wait complete. 2024-09-13T08:42:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.138+0000 7fcb99800640 1 Processor -- start 2024-09-13T08:42:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.138+0000 7fcb99800640 1 -- start start 2024-09-13T08:42:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9406e670 0x7fcb94148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415f820 0x7fcb9415d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94070040 con 0x7fcb9415bb70 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9406fec0 con 0x7fcb9415f820 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb99800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcb940701c0 con 0x7fcb9406e670 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40752/0 (socket says 172.21.15.31:40752) 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 -- 172.21.15.31:0/1836200363 learned_addr learned my addr 172.21.15.31:0/1836200363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb98e00640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415f820 0x7fcb9415d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93e00640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9406e670 0x7fcb94148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 -- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9406e670 msgr2=0x7fcb94148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9406e670 0x7fcb94148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 -- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415f820 msgr2=0x7fcb9415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415f820 0x7fcb9415d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.139+0000 7fcb93400640 1 -- 172.21.15.31:0/1836200363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9415e3d0 con 0x7fcb9415bb70 2024-09-13T08:42:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb93400640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c450 tx=0x7fcb8000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb800052d0 con 0x7fcb9415bb70 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9415e6b0 con 0x7fcb9415bb70 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb80005470 con 0x7fcb9415bb70 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80012870 con 0x7fcb9415bb70 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.140+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb94156580 con 0x7fcb9415bb70 2024-09-13T08:42:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.141+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb94070750 con 0x7fcb9415bb70 2024-09-13T08:42:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.143+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fcb800035f0 con 0x7fcb9415bb70 2024-09-13T08:42:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.143+0000 7fcb8be00640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 0x7fcb7407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.144+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fcb8009b070 con 0x7fcb9415bb70 2024-09-13T08:42:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.144+0000 7fcb93e00640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 0x7fcb7407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.145+0000 7fcb93e00640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 0x7fcb7407ae60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcb84034000 tx=0x7fcb84004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:59.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.147+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcb80063550 con 0x7fcb9415bb70 2024-09-13T08:42:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.367+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fcb940cbc30 con 0x7fcb740789a0 2024-09-13T08:42:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.368+0000 7fcb8be00640 1 -- 172.21.15.31:0/1836200363 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fcb940cbc30 con 0x7fcb740789a0 2024-09-13T08:42:59.370 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:42:59.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:42:59.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:42:59.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:42:59.371 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:42:59.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 msgr2=0x7fcb7407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 0x7fcb7407ae60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcb84034000 tx=0x7fcb84004730 comp rx=0 tx=0).stop 2024-09-13T08:42:59.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 msgr2=0x7fcb94149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c450 tx=0x7fcb8000c910 comp rx=0 tx=0).stop 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 shutdown_connections 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb740789a0 0x7fcb7407ae60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9415f820 0x7fcb9415d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb9415bb70 0x7fcb94149280 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 --2- 172.21.15.31:0/1836200363 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb9406e670 0x7fcb94148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 >> 172.21.15.31:0/1836200363 conn(0x7fcb940cc7a0 msgr2=0x7fcb940ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 shutdown_connections 2024-09-13T08:42:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.371+0000 7fcb99800640 1 -- 172.21.15.31:0/1836200363 wait complete. 2024-09-13T08:42:59.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 -- 172.21.15.31:0/4049667855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2424004800 msgr2=0x7f2424004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 --2- 172.21.15.31:0/4049667855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2424004800 0x7f2424004c80 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f2428006fb0 tx=0x7f2424005830 comp rx=0 tx=0).stop 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 -- 172.21.15.31:0/4049667855 shutdown_connections 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 --2- 172.21.15.31:0/4049667855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2424004800 0x7f2424004c80 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 -- 172.21.15.31:0/4049667855 >> 172.21.15.31:0/4049667855 conn(0x7f24400d3e90 msgr2=0x7f24400d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 -- 172.21.15.31:0/4049667855 shutdown_connections 2024-09-13T08:42:59.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.551+0000 7f2445200640 1 -- 172.21.15.31:0/4049667855 wait complete. 2024-09-13T08:42:59.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.552+0000 7f2445200640 1 Processor -- start 2024-09-13T08:42:59.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.552+0000 7f2445200640 1 -- start start 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2424004800 0x7f24400cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24401f55e0 0x7f244015c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24400b57c0 con 0x7f244015c5c0 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24400b5640 con 0x7f2424004800 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.553+0000 7f2445200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f24400d21d0 con 0x7f24401f55e0 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40768/0 (socket says 172.21.15.31:40768) 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 -- 172.21.15.31:0/2728289817 learned_addr learned my addr 172.21.15.31:0/2728289817 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243f400640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2424004800 0x7f24400cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f2436a00640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24401f55e0 0x7f244015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 -- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24401f55e0 msgr2=0x7f244015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24401f55e0 0x7f244015c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 -- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2424004800 msgr2=0x7f24400cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2424004800 0x7f24400cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 -- 172.21.15.31:0/2728289817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24400b7db0 con 0x7f244015c5c0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.554+0000 7f243fe00640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f244008e570 tx=0x7f2438001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.555+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438017070 con 0x7f244015c5c0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.555+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2438003ca0 con 0x7f244015c5c0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.555+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244015cb40 con 0x7f244015c5c0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.555+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438004290 con 0x7f244015c5c0 2024-09-13T08:42:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.555+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24400b8300 con 0x7f244015c5c0 2024-09-13T08:42:59.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.556+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24400b3460 con 0x7f244015c5c0 2024-09-13T08:42:59.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.557+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f2438004430 con 0x7f244015c5c0 2024-09-13T08:42:59.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.558+0000 7f2437e00640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 0x7f241c07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:42:59.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.558+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f243809a8a0 con 0x7f244015c5c0 2024-09-13T08:42:59.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.558+0000 7f243f400640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 0x7f241c07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:42:59.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.559+0000 7f243f400640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 0x7f241c07ae60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f24400b9d30 tx=0x7f2430005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:42:59.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.562+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2438062580 con 0x7f244015c5c0 2024-09-13T08:42:59.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.758+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f244006dea0 con 0x7f244015c5c0 2024-09-13T08:42:59.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.759+0000 7f2437e00640 1 -- 172.21.15.31:0/2728289817 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f2438084070 con 0x7f244015c5c0 2024-09-13T08:42:59.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 msgr2=0x7f241c07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 0x7f241c07ae60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f24400b9d30 tx=0x7f2430005f90 comp rx=0 tx=0).stop 2024-09-13T08:42:59.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 msgr2=0x7f24401f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f244008e570 tx=0x7f2438001b80 comp rx=0 tx=0).stop 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 shutdown_connections 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f241c0789a0 0x7f241c07ae60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244015c5c0 0x7f24401f44f0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f24401f55e0 0x7f244015c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 --2- 172.21.15.31:0/2728289817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2424004800 0x7f24400cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 >> 172.21.15.31:0/2728289817 conn(0x7f24400d3e90 msgr2=0x7f24400b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 shutdown_connections 2024-09-13T08:42:59.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:42:59.761+0000 7f2445200640 1 -- 172.21.15.31:0/2728289817 wait complete. 2024-09-13T08:42:59.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:59.781 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9) 2024-09-13T08:42:59.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:59.847 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/fake) 2024-09-13T08:42:59.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:59.934 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:42:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:42:59.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:42:59.971 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9) 2024-09-13T08:43:00.073 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:00.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol5350/.meta 2024-09-13T08:43:00.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol5350 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.335+0000 7f1aa7600640 1 -- 172.21.15.31:0/3950055540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa006fec0 msgr2=0x7f1aa00b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.335+0000 7f1aa7600640 1 --2- 172.21.15.31:0/3950055540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa006fec0 0x7f1aa00b29e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0069dd0 tx=0x7f1a9802a8b0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 -- 172.21.15.31:0/3950055540 shutdown_connections 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 --2- 172.21.15.31:0/3950055540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa006fec0 0x7f1aa00b29e0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 --2- 172.21.15.31:0/3950055540 >> v2:172.21.15.45:3300/0 conn(0x7f1aa015bfc0 0x7f1aa0072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 -- 172.21.15.31:0/3950055540 >> 172.21.15.31:0/3950055540 conn(0x7f1aa00cfbb0 msgr2=0x7f1aa00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 -- 172.21.15.31:0/3950055540 shutdown_connections 2024-09-13T08:43:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.336+0000 7f1aa7600640 1 -- 172.21.15.31:0/3950055540 wait complete. 2024-09-13T08:43:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.337+0000 7f1aa7600640 1 Processor -- start 2024-09-13T08:43:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.337+0000 7f1aa7600640 1 -- start start 2024-09-13T08:43:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 0x7f1aa00c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 0x7f1aa01490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa00cbfd0 con 0x7f1aa015bfc0 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa00cbe50 con 0x7f1aa00c4ae0 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.338+0000 7f1aa7600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa00b2f20 con 0x7f1aa006fa60 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40792/0 (socket says 172.21.15.31:40792) 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 -- 172.21.15.31:0/2029658488 learned_addr learned my addr 172.21.15.31:0/2029658488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1aa4c00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 0x7f1aa00c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1aa5600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 0x7f1aa01490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 -- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 msgr2=0x7f1aa00c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 0x7f1aa00c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 -- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 msgr2=0x7f1aa01490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 0x7f1aa01490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 -- 172.21.15.31:0/2029658488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa0158070 con 0x7f1aa015bfc0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1aa4c00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 0x7f1aa00c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1aa5600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 0x7f1aa01490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9fe00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00c530 tx=0x7f1a8c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.339+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c017070 con 0x7f1aa015bfc0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.340+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa0158310 con 0x7f1aa015bfc0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.340+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a8c0035f0 con 0x7f1aa015bfc0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.340+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c003a40 con 0x7f1aa015bfc0 2024-09-13T08:43:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.340+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa01588d0 con 0x7f1aa015bfc0 2024-09-13T08:43:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.341+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa00b33d0 con 0x7f1aa015bfc0 2024-09-13T08:43:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.342+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f1a8c01a070 con 0x7f1aa015bfc0 2024-09-13T08:43:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.343+0000 7f1a9cc00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 0x7f1a8007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.343+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f1a8c099fe0 con 0x7f1aa015bfc0 2024-09-13T08:43:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.343+0000 7f1aa4c00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 0x7f1a8007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.344+0000 7f1aa4c00640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 0x7f1a8007ae30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009aa0 tx=0x7f1a94005db0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.347+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1a8c062420 con 0x7f1aa015bfc0 2024-09-13T08:43:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.573+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f1aa00cb310 con 0x7f1a80078970 2024-09-13T08:43:00.580 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:00.579+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:00.580 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:00.579+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:00.581 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:00.579+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:00.581 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:00.579+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:00.581 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:00.579+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.626+0000 7f1a9cc00640 1 -- 172.21.15.31:0/2029658488 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7f1aa00cb310 con 0x7f1a80078970 2024-09-13T08:43:00.628 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 msgr2=0x7f1a8007ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 0x7f1a8007ae30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009aa0 tx=0x7f1a94005db0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 msgr2=0x7f1aa0148bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00c530 tx=0x7f1a8c00c9f0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 shutdown_connections 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a80078970 0x7f1a8007ae30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00c4ae0 0x7f1aa01490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1aa015bfc0 0x7f1aa0148bb0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 --2- 172.21.15.31:0/2029658488 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1aa006fa60 0x7f1aa00c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.628+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 >> 172.21.15.31:0/2029658488 conn(0x7f1aa00cfbb0 msgr2=0x7f1aa00b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.629+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 shutdown_connections 2024-09-13T08:43:00.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.629+0000 7f1aa7600640 1 -- 172.21.15.31:0/2029658488 wait complete. 2024-09-13T08:43:00.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-13T08:43:00.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 -- 172.21.15.31:0/1347736631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f685c005210 msgr2=0x7f685c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 --2- 172.21.15.31:0/1347736631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f685c005210 0x7f685c00b9c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f686c001d90 tx=0x7f685c005830 comp rx=0 tx=0).stop 2024-09-13T08:43:00.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 -- 172.21.15.31:0/1347736631 shutdown_connections 2024-09-13T08:43:00.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 --2- 172.21.15.31:0/1347736631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f685c005210 0x7f685c00b9c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 -- 172.21.15.31:0/1347736631 >> 172.21.15.31:0/1347736631 conn(0x7f68740cfb70 msgr2=0x7f68740cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:00.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 -- 172.21.15.31:0/1347736631 shutdown_connections 2024-09-13T08:43:00.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.809+0000 7f687c400640 1 -- 172.21.15.31:0/1347736631 wait complete. 2024-09-13T08:43:00.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.811+0000 7f687c400640 1 Processor -- start 2024-09-13T08:43:00.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.811+0000 7f687c400640 1 -- start start 2024-09-13T08:43:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f685c005210 0x7f68740c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68741500e0 0x7f6874150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f68740729a0 con 0x7f68740ad890 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6874072820 con 0x7f68741500e0 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687c400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f68740cb9c0 con 0x7f685c005210 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f6879000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f6879000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40840/0 (socket says 172.21.15.31:40840) 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f6879000640 1 -- 172.21.15.31:0/1992185036 learned_addr learned my addr 172.21.15.31:0/1992185036 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f687a400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68741500e0 0x7f6874150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.812+0000 7f6879a00640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f685c005210 0x7f68740c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 -- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f685c005210 msgr2=0x7f68740c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f685c005210 0x7f68740c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 -- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68741500e0 msgr2=0x7f6874150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68741500e0 0x7f6874150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 -- 172.21.15.31:0/1992185036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68740c5250 con 0x7f68740ad890 2024-09-13T08:43:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6879000640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f6860011ab0 tx=0x7f6860011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686001e070 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f686000fe70 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68600042b0 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68741f4bb0 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.813+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68741f5170 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.815+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68740b29a0 con 0x7f68740ad890 2024-09-13T08:43:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.816+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f6860021070 con 0x7f68740ad890 2024-09-13T08:43:00.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.817+0000 7f6871600640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 0x7f685407ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:00.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.817+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f686009b510 con 0x7f68740ad890 2024-09-13T08:43:00.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.817+0000 7f6879a00640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 0x7f685407ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:00.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.817+0000 7f6879a00640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 0x7f685407ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f687406e510 tx=0x7f68680011c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:00.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:00.821+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f68600633a0 con 0x7f68740ad890 2024-09-13T08:43:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.044+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f687414dac0 con 0x7f6854078970 2024-09-13T08:43:01.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.049+0000 7f6871600640 1 -- 172.21.15.31:0/1992185036 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f687414dac0 con 0x7f6854078970 2024-09-13T08:43:01.051 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:43:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:43:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subvol5350" 2024-09-13T08:43:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:01.051 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 msgr2=0x7f685407ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 0x7f685407ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f687406e510 tx=0x7f68680011c0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 msgr2=0x7f68740add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f6860011ab0 tx=0x7f6860011f70 comp rx=0 tx=0).stop 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 shutdown_connections 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6854078970 0x7f685407ae30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68741500e0 0x7f6874150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68740ad890 0x7f68740add10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 --2- 172.21.15.31:0/1992185036 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f685c005210 0x7f68740c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 >> 172.21.15.31:0/1992185036 conn(0x7f68740cfb70 msgr2=0x7f68740b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 shutdown_connections 2024-09-13T08:43:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.051+0000 7f687c400640 1 -- 172.21.15.31:0/1992185036 wait complete. 2024-09-13T08:43:01.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol5350 2024-09-13T08:43:01.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.230+0000 7fe268200640 1 -- 172.21.15.31:0/2261093551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26006fa60 msgr2=0x7fe2600cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.230+0000 7fe268200640 1 --2- 172.21.15.31:0/2261093551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26006fa60 0x7fe2600cb740 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe260069dd0 tx=0x7fe24c02a560 comp rx=0 tx=0).stop 2024-09-13T08:43:01.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 -- 172.21.15.31:0/2261093551 shutdown_connections 2024-09-13T08:43:01.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 --2- 172.21.15.31:0/2261093551 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fec0 0x7fe2600cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 --2- 172.21.15.31:0/2261093551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26006fa60 0x7fe2600cb740 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 --2- 172.21.15.31:0/2261093551 >> v2:172.21.15.45:3300/0 conn(0x7fe26015bfc0 0x7fe260072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 -- 172.21.15.31:0/2261093551 >> 172.21.15.31:0/2261093551 conn(0x7fe2600cfbb0 msgr2=0x7fe2600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:01.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 -- 172.21.15.31:0/2261093551 shutdown_connections 2024-09-13T08:43:01.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.231+0000 7fe268200640 1 -- 172.21.15.31:0/2261093551 wait complete. 2024-09-13T08:43:01.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.232+0000 7fe268200640 1 Processor -- start 2024-09-13T08:43:01.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.232+0000 7fe268200640 1 -- start start 2024-09-13T08:43:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fa60 0x7fe2600cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe26014de10 0x7fe2600c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2600b2b60 con 0x7fe26015bfc0 2024-09-13T08:43:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2600b29e0 con 0x7fe26014de10 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.233+0000 7fe268200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe2600b2ce0 con 0x7fe26006fa60 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40862/0 (socket says 172.21.15.31:40862) 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 -- 172.21.15.31:0/1155329634 learned_addr learned my addr 172.21.15.31:0/1155329634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe265800640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fa60 0x7fe2600cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe266200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe26014de10 0x7fe2600c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 -- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fa60 msgr2=0x7fe2600cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fa60 0x7fe2600cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 -- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe26014de10 msgr2=0x7fe2600c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe26014de10 0x7fe2600c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 -- 172.21.15.31:0/1155329634 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26015c6d0 con 0x7fe26015bfc0 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.234+0000 7fe264e00640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:01.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.235+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c03f070 con 0x7fe26015bfc0 2024-09-13T08:43:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.235+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe24c0047a0 con 0x7fe26015bfc0 2024-09-13T08:43:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.235+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2600cf500 con 0x7fe26015bfc0 2024-09-13T08:43:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.235+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c035030 con 0x7fe26015bfc0 2024-09-13T08:43:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.235+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2600c7090 con 0x7fe26015bfc0 2024-09-13T08:43:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.236+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2600b3200 con 0x7fe26015bfc0 2024-09-13T08:43:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.237+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7fe24c042070 con 0x7fe26015bfc0 2024-09-13T08:43:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.238+0000 7fe25d600640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 0x7fe24007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.238+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fe24c0bb880 con 0x7fe26015bfc0 2024-09-13T08:43:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.238+0000 7fe265800640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 0x7fe24007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.239+0000 7fe265800640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 0x7fe24007ae60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe25400bb50 tx=0x7fe25400b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:01.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.242+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe26015c6d0 con 0x7fe26015bfc0 2024-09-13T08:43:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.464+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7fe2600cf7e0 con 0x7fe2400789a0 2024-09-13T08:43:01.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.494+0000 7fe25d600640 1 -- 172.21.15.31:0/1155329634 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+702 (secure 0 0 0) 0x7fe2600cf7e0 con 0x7fe2400789a0 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "atime": "2024-09-13 08:42:59", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_pcent": "undefined", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_quota": "infinite", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_used": 0, 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "created_at": "2024-09-13 08:42:59", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "ctime": "2024-09-13 08:43:00", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "cephfs_data", 2024-09-13T08:43:01.496 INFO:teuthology.orchestra.run.smithi031.stdout: "earmark": "", 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "features": [ 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-clone", 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-autoprotect" 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flavor": 1, 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 0, 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": 16877, 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_addrs": [ 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6789", 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6789", 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6790" 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-09-13 08:42:59", 2024-09-13T08:43:01.497 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9", 2024-09-13T08:43:01.498 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_namespace": "", 2024-09-13T08:43:01.498 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "complete", 2024-09-13T08:43:01.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "subvolume", 2024-09-13T08:43:01.498 INFO:teuthology.orchestra.run.smithi031.stdout: "uid": 0 2024-09-13T08:43:01.498 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 msgr2=0x7fe24007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 0x7fe24007ae60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe25400bb50 tx=0x7fe25400b2e0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 msgr2=0x7fe26014d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c003910 comp rx=0 tx=0).stop 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 shutdown_connections 2024-09-13T08:43:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe2400789a0 0x7fe24007ae60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe26014de10 0x7fe2600c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26015bfc0 0x7fe26014d600 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 --2- 172.21.15.31:0/1155329634 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe26006fa60 0x7fe2600cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 >> 172.21.15.31:0/1155329634 conn(0x7fe2600cfbb0 msgr2=0x7fe2600b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 shutdown_connections 2024-09-13T08:43:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.497+0000 7fe268200640 1 -- 172.21.15.31:0/1155329634 wait complete. 2024-09-13T08:43:01.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize cephfs subvol5350 10485760 2024-09-13T08:43:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 -- 172.21.15.31:0/1450017116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0cabb0 msgr2=0x7f009c165320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 --2- 172.21.15.31:0/1450017116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0cabb0 0x7f009c165320 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0094005fd0 tx=0x7f009402ab30 comp rx=0 tx=0).stop 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 -- 172.21.15.31:0/1450017116 shutdown_connections 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 --2- 172.21.15.31:0/1450017116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c0cabb0 0x7f009c165320 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 -- 172.21.15.31:0/1450017116 >> 172.21.15.31:0/1450017116 conn(0x7f009c0cfbb0 msgr2=0x7f009c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 -- 172.21.15.31:0/1450017116 shutdown_connections 2024-09-13T08:43:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.679+0000 7f00a2800640 1 -- 172.21.15.31:0/1450017116 wait complete. 2024-09-13T08:43:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.681+0000 7f00a2800640 1 Processor -- start 2024-09-13T08:43:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.681+0000 7f00a2800640 1 -- start start 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0cabb0 0x7f009c1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c15ad00 0x7f009c1506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f009c06e740 con 0x7f009c06ee50 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f009c06e5c0 con 0x7f009c0cabb0 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f00a2800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f009c06e8c0 con 0x7f009c15ad00 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f009b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f009b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40886/0 (socket says 172.21.15.31:40886) 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f009b400640 1 -- 172.21.15.31:0/1314866068 learned_addr learned my addr 172.21.15.31:0/1314866068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f0092a00640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0cabb0 0x7f009c1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.682+0000 7f009be00640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c15ad00 0x7f009c1506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 -- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c15ad00 msgr2=0x7f009c1506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c15ad00 0x7f009c1506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 -- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0cabb0 msgr2=0x7f009c1501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0cabb0 0x7f009c1501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 -- 172.21.15.31:0/1314866068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c073950 con 0x7f009c06ee50 2024-09-13T08:43:01.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f009b400640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c370 tx=0x7f008c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c017070 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f008c0035f0 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c003a40 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c14c160 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.683+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c14de00 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.684+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009c074660 con 0x7f009c06ee50 2024-09-13T08:43:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.685+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 101389+0+0 (secure 0 0 0) 0x7f008c01a070 con 0x7f009c06ee50 2024-09-13T08:43:01.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.686+0000 7f0093e00640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 0x7f007807ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:01.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.686+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f008c099da0 con 0x7f009c06ee50 2024-09-13T08:43:01.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.687+0000 7f0092a00640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 0x7f007807ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:01.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.687+0000 7f0092a00640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 0x7f007807ae30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f009c1f7f60 tx=0x7f0084009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:01.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.691+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f009c074660 con 0x7f009c06ee50 2024-09-13T08:43:01.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.919+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "cephfs", "sub_name": "subvol5350", "new_size": "10485760", "target": ["mon-mgr", ""]}) -- 0x7f009c0ca4a0 con 0x7f0078078970 2024-09-13T08:43:01.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.951+0000 7f0093e00640 1 -- 172.21.15.31:0/1314866068 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+128 (secure 0 0 0) 0x7f009c0ca4a0 con 0x7f0078078970 2024-09-13T08:43:01.953 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:43:01.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:43:01.953 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_used": 0 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_quota": 10485760 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_pcent": "0.00" 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:01.954 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:43:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 msgr2=0x7f007807ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 0x7f007807ae30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f009c1f7f60 tx=0x7f0084009040 comp rx=0 tx=0).stop 2024-09-13T08:43:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 msgr2=0x7f009c1477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c370 tx=0x7f008c00c830 comp rx=0 tx=0).stop 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 shutdown_connections 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0078078970 0x7f007807ae30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f009c15ad00 0x7f009c1506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f009c0cabb0 0x7f009c1501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 --2- 172.21.15.31:0/1314866068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f009c06ee50 0x7f009c1477c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 >> 172.21.15.31:0/1314866068 conn(0x7f009c0cfbb0 msgr2=0x7f009c0cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 shutdown_connections 2024-09-13T08:43:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:01.954+0000 7f00a2800640 1 -- 172.21.15.31:0/1314866068 wait complete. 2024-09-13T08:43:01.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol5350 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 -- 172.21.15.31:0/799281735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b5430 msgr2=0x7f51f00d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 --2- 172.21.15.31:0/799281735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b5430 0x7f51f00d24a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009830 tx=0x7f51d802a4a0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 -- 172.21.15.31:0/799281735 shutdown_connections 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 --2- 172.21.15.31:0/799281735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b5430 0x7f51f00d24a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 -- 172.21.15.31:0/799281735 >> 172.21.15.31:0/799281735 conn(0x7f51f00d3e90 msgr2=0x7f51f00d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 -- 172.21.15.31:0/799281735 shutdown_connections 2024-09-13T08:43:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.136+0000 7f51efe00640 1 -- 172.21.15.31:0/799281735 wait complete. 2024-09-13T08:43:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.137+0000 7f51efe00640 1 Processor -- start 2024-09-13T08:43:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.138+0000 7f51efe00640 1 -- start start 2024-09-13T08:43:02.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.138+0000 7f51efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.138+0000 7f51efe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 0x7f51f00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.138+0000 7f51efe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51f0161470 0x7f51f01500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51efe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51f00b2020 con 0x7f51f00b1480 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51efe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51f00d2b50 con 0x7f51f00b5430 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51efe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f51f00b21a0 con 0x7f51f0161470 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40908/0 (socket says 172.21.15.31:40908) 2024-09-13T08:43:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 -- 172.21.15.31:0/161169495 learned_addr learned my addr 172.21.15.31:0/161169495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 -- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51f0161470 msgr2=0x7f51f01500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51f0161470 0x7f51f01500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51e6000640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 0x7f51f00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 -- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 msgr2=0x7f51f00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 0x7f51f00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 -- 172.21.15.31:0/161169495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f0150b60 con 0x7f51f00b1480 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51e6000640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 0x7f51f00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.139+0000 7f51eea00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c7b0 tx=0x7f51e000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.140+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0003b70 con 0x7f51f00b1480 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.140+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51e0003d10 con 0x7f51f00b1480 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.140+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e000aab0 con 0x7f51f00b1480 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.140+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f0150e40 con 0x7f51f00b1480 2024-09-13T08:43:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.140+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f00cd300 con 0x7f51f00b1480 2024-09-13T08:43:02.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.141+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f00b2600 con 0x7f51f00b1480 2024-09-13T08:43:02.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.142+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f51e000ac50 con 0x7f51f00b1480 2024-09-13T08:43:02.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.143+0000 7f51e7400640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 0x7f51cc07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.143+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f51e009ac60 con 0x7f51f00b1480 2024-09-13T08:43:02.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.143+0000 7f51e6000640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 0x7f51cc07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.144+0000 7f51e6000640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 0x7f51cc07aec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f51d8034000 tx=0x7f51d8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:02.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.148+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f51e0062ea0 con 0x7f51f00b1480 2024-09-13T08:43:02.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.370+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f51f015f6b0 con 0x7f51cc078a00 2024-09-13T08:43:02.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.398+0000 7f51e7400640 1 -- 172.21.15.31:0/161169495 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+695 (secure 0 0 0) 0x7f51f015f6b0 con 0x7f51cc078a00 2024-09-13T08:43:02.400 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "atime": "2024-09-13 08:42:59", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_pcent": "0.00", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_quota": 10485760, 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_used": 0, 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "created_at": "2024-09-13 08:42:59", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "ctime": "2024-09-13 08:43:01", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "cephfs_data", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "earmark": "", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "features": [ 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-clone", 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-autoprotect" 2024-09-13T08:43:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flavor": 1, 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 0, 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": 16877, 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_addrs": [ 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6789", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6789", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6790" 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-09-13 08:42:59", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_namespace": "", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "complete", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "subvolume", 2024-09-13T08:43:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "uid": 0 2024-09-13T08:43:02.403 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 msgr2=0x7f51cc07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 0x7f51cc07aec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f51d8034000 tx=0x7f51d8004730 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 msgr2=0x7f51f00c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c7b0 tx=0x7f51e000cc70 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 shutdown_connections 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f51cc078a00 0x7f51cc07aec0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51f0161470 0x7f51f01500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51f00b5430 0x7f51f00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 --2- 172.21.15.31:0/161169495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51f00b1480 0x7f51f00c8a10 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 >> 172.21.15.31:0/161169495 conn(0x7f51f00d3e90 msgr2=0x7f51f00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 shutdown_connections 2024-09-13T08:43:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.401+0000 7f51efe00640 1 -- 172.21.15.31:0/161169495 wait complete. 2024-09-13T08:43:02.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol5350 --mode 777 --uid 1000 --gid 1000 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 -- 172.21.15.31:0/2155718482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3840072560 msgr2=0x7f38400b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 --2- 172.21.15.31:0/2155718482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3840072560 0x7f38400b29a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f38400cb310 tx=0x7f382c02a560 comp rx=0 tx=0).stop 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 -- 172.21.15.31:0/2155718482 shutdown_connections 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 --2- 172.21.15.31:0/2155718482 >> v2:172.21.15.45:3300/0 conn(0x7f38400cb570 0x7f38400cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 --2- 172.21.15.31:0/2155718482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3840072560 0x7f38400b29a0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 --2- 172.21.15.31:0/2155718482 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f384015c040 0x7f38400d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.587+0000 7f3848200640 1 -- 172.21.15.31:0/2155718482 >> 172.21.15.31:0/2155718482 conn(0x7f38400cfb70 msgr2=0x7f38400cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.588+0000 7f3848200640 1 -- 172.21.15.31:0/2155718482 shutdown_connections 2024-09-13T08:43:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.588+0000 7f3848200640 1 -- 172.21.15.31:0/2155718482 wait complete. 2024-09-13T08:43:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.589+0000 7f3848200640 1 Processor -- start 2024-09-13T08:43:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.589+0000 7f3848200640 1 -- start start 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3840072560 0x7f38400b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 0x7f384015dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f384006f810 con 0x7f384015c040 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f384006f690 con 0x7f38400cb570 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3848200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f384006f990 con 0x7f3840072560 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3846200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3846200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40926/0 (socket says 172.21.15.31:40926) 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.590+0000 7f3846200640 1 -- 172.21.15.31:0/3940103615 learned_addr learned my addr 172.21.15.31:0/3940103615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3845800640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3840072560 0x7f38400b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 -- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3840072560 msgr2=0x7f38400b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3844e00640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 0x7f384015dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3840072560 0x7f38400b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 -- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 msgr2=0x7f384015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 0x7f384015dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 -- 172.21.15.31:0/3940103615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38401f8290 con 0x7f384015c040 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3844e00640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 0x7f384015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3846200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9b0 tx=0x7f383800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838004d20 con 0x7f384015c040 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f383800cb20 con 0x7f384015c040 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838005020 con 0x7f384015c040 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.591+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38401f8570 con 0x7f384015c040 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.592+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38401f8b30 con 0x7f384015c040 2024-09-13T08:43:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.593+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384006fef0 con 0x7f384015c040 2024-09-13T08:43:02.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.594+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f38380051c0 con 0x7f384015c040 2024-09-13T08:43:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.595+0000 7f383d600640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 0x7f382007aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.595+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f383809b650 con 0x7f384015c040 2024-09-13T08:43:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.596+0000 7f3845800640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 0x7f382007aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.596+0000 7f3845800640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 0x7f382007aec0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3834005fd0 tx=0x7f3834005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.600+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3838016030 con 0x7f384015c040 2024-09-13T08:43:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.824+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol5350", "mode": "777", "uid": 1000, "gid": 1000, "target": ["mon-mgr", ""]}) -- 0x7f38400b4390 con 0x7f3820078a00 2024-09-13T08:43:02.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.859+0000 7f383d600640 1 -- 172.21.15.31:0/3940103615 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f38400b4390 con 0x7f3820078a00 2024-09-13T08:43:02.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 msgr2=0x7f382007aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 0x7f382007aec0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3834005fd0 tx=0x7f3834005850 comp rx=0 tx=0).stop 2024-09-13T08:43:02.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 msgr2=0x7f38401572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:02.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9b0 tx=0x7f383800ee70 comp rx=0 tx=0).stop 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 shutdown_connections 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3820078a00 0x7f382007aec0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f384015c040 0x7f38401572d0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38400cb570 0x7f384015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 --2- 172.21.15.31:0/3940103615 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3840072560 0x7f38400b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 >> 172.21.15.31:0/3940103615 conn(0x7f38400cfb70 msgr2=0x7f38400cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 shutdown_connections 2024-09-13T08:43:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:02.862+0000 7f3848200640 1 -- 172.21.15.31:0/3940103615 wait complete. 2024-09-13T08:43:02.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol5350 2024-09-13T08:43:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 -- 172.21.15.31:0/4064813636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c74003a50 msgr2=0x7f9c74003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 --2- 172.21.15.31:0/4064813636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c74003a50 0x7f9c74003e70 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009b20 tx=0x7f9c74005830 comp rx=0 tx=0).stop 2024-09-13T08:43:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c8fe00640 1 -- 172.21.15.31:0/4064813636 reap_dead start 2024-09-13T08:43:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 -- 172.21.15.31:0/4064813636 shutdown_connections 2024-09-13T08:43:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 -- 172.21.15.31:0/4064813636 >> 172.21.15.31:0/4064813636 conn(0x7f9c900cfbb0 msgr2=0x7f9c900cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 -- 172.21.15.31:0/4064813636 shutdown_connections 2024-09-13T08:43:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.043+0000 7f9c96400640 1 -- 172.21.15.31:0/4064813636 wait complete. 2024-09-13T08:43:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.045+0000 7f9c96400640 1 Processor -- start 2024-09-13T08:43:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.045+0000 7f9c96400640 1 -- start start 2024-09-13T08:43:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c74003a50 0x7f9c900c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c90147050 0x7f9c900c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c900cbdc0 con 0x7f9c900c8570 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c900cbc40 con 0x7f9c90147050 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c96400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9c900cbf40 con 0x7f9c74003a50 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.046+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40952/0 (socket says 172.21.15.31:40952) 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 -- 172.21.15.31:0/3824423507 learned_addr learned my addr 172.21.15.31:0/3824423507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c86a00640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c90147050 0x7f9c900c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8f400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c74003a50 0x7f9c900c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 -- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c74003a50 msgr2=0x7f9c900c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c74003a50 0x7f9c900c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 -- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c90147050 msgr2=0x7f9c900c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c90147050 0x7f9c900c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 -- 172.21.15.31:0/3824423507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c901474d0 con 0x7f9c900c8570 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c8fe00640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f9c900cfb50 tx=0x7f9c88010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.047+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88003970 con 0x7f9c900c8570 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.048+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c88015070 con 0x7f9c900c8570 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.048+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c900c9240 con 0x7f9c900c8570 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.048+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88004030 con 0x7f9c900c8570 2024-09-13T08:43:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.048+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c900c9800 con 0x7f9c900c8570 2024-09-13T08:43:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.049+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c900cdaf0 con 0x7f9c900c8570 2024-09-13T08:43:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.050+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f9c880041d0 con 0x7f9c900c8570 2024-09-13T08:43:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.051+0000 7f9c87e00640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 0x7f9c6c07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.051+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f9c8809b5d0 con 0x7f9c900c8570 2024-09-13T08:43:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.051+0000 7f9c8f400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 0x7f9c6c07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.052+0000 7f9c8f400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 0x7f9c6c07aef0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9c80035000 tx=0x7f9c8002d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.055+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9c88063300 con 0x7f9c900c8570 2024-09-13T08:43:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.279+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f9c900b5b30 con 0x7f9c6c078a30 2024-09-13T08:43:03.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.307+0000 7f9c87e00640 1 -- 172.21.15.31:0/3824423507 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7f9c900b5b30 con 0x7f9c6c078a30 2024-09-13T08:43:03.309 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 msgr2=0x7f9c6c07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 0x7f9c6c07aef0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9c80035000 tx=0x7f9c8002d040 comp rx=0 tx=0).stop 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 msgr2=0x7f9c900c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f9c900cfb50 tx=0x7f9c88010f40 comp rx=0 tx=0).stop 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 shutdown_connections 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c6c078a30 0x7f9c6c07aef0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c900c8570 0x7f9c900c89f0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c90147050 0x7f9c900c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 --2- 172.21.15.31:0/3824423507 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c74003a50 0x7f9c900c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:03.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 >> 172.21.15.31:0/3824423507 conn(0x7f9c900cfbb0 msgr2=0x7f9c900c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:03.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 shutdown_connections 2024-09-13T08:43:03.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:03.310+0000 7f9c96400640 1 -- 172.21.15.31:0/3824423507 wait complete. 2024-09-13T08:43:03.329 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol5350 with 8 files each 1MB size under directory volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9 2024-09-13T08:43:03.329 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.329 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:03.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:03.365 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187477 s, 55.9 MB/s 2024-09-13T08:43:03.463 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.463 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:03.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:03.498 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161453 s, 64.9 MB/s 2024-09-13T08:43:03.593 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.594 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:03.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:03.630 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:03.722 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:03.722 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:03.722 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01609 s, 65.2 MB/s 2024-09-13T08:43:03.723 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.723 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:03.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:03.759 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176262 s, 59.5 MB/s 2024-09-13T08:43:03.854 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.854 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:03.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:03.890 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:03.983 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:03.984 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:03.984 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175305 s, 59.8 MB/s 2024-09-13T08:43:03.985 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:03.985 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:04.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:04.020 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145055 s, 72.3 MB/s 2024-09-13T08:43:04.113 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:04.113 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:04.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:04.149 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:04.242 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:04.243 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:04.243 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176292 s, 59.5 MB/s 2024-09-13T08:43:04.244 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:43:04.244 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:43:04.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:04.279 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T08:43:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-13T08:43:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-13T08:43:04.371 INFO:teuthology.orchestra.run.smithi031.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148342 s, 70.7 MB/s 2024-09-13T08:43:04.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol5350 subvol_snap26060 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.536+0000 7f8bd6000640 1 -- 172.21.15.31:0/2279629321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd0072560 msgr2=0x7f8bd00cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.536+0000 7f8bd6000640 1 --2- 172.21.15.31:0/2279629321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd0072560 0x7f8bd00cb740 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009830 tx=0x7f8bb802a510 comp rx=0 tx=0).stop 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.536+0000 7f8bd6000640 1 -- 172.21.15.31:0/2279629321 shutdown_connections 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.536+0000 7f8bd6000640 1 --2- 172.21.15.31:0/2279629321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd0072560 0x7f8bd00cb740 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.537+0000 7f8bd6000640 1 -- 172.21.15.31:0/2279629321 >> 172.21.15.31:0/2279629321 conn(0x7f8bd00cfbb0 msgr2=0x7f8bd00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.537+0000 7f8bd6000640 1 -- 172.21.15.31:0/2279629321 shutdown_connections 2024-09-13T08:43:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.537+0000 7f8bd6000640 1 -- 172.21.15.31:0/2279629321 wait complete. 2024-09-13T08:43:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.538+0000 7f8bd6000640 1 Processor -- start 2024-09-13T08:43:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.538+0000 7f8bd6000640 1 -- start start 2024-09-13T08:43:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8bd0072560 0x7f8bd01f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd00cbc80 0x7f8bd01f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd006e740 con 0x7f8bd00cbc80 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd006e5c0 con 0x7f8bd015bfc0 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.539+0000 7f8bd6000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd00b30f0 con 0x7f8bd0072560 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:51350/0 (socket says 172.21.15.31:51350) 2024-09-13T08:43:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 -- 172.21.15.31:0/623226758 learned_addr learned my addr 172.21.15.31:0/623226758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 -- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8bd0072560 msgr2=0x7f8bd01f8380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T08:43:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8bd0072560 0x7f8bd01f8380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.540+0000 7f8bcfe00640 1 -- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd00cbc80 msgr2=0x7f8bd01f88c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T08:43:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.541+0000 7f8bcfe00640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd00cbc80 0x7f8bd01f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.541+0000 7f8bcfe00640 1 -- 172.21.15.31:0/623226758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd01f8f60 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.541+0000 7f8bcfe00640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c7b0 tx=0x7f8bc800cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.541+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8004b60 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.542+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd00c6d20 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.542+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bc8004df0 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.542+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800aa90 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.542+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd00c72e0 con 0x7f8bd015bfc0 2024-09-13T08:43:04.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.543+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd00cb740 con 0x7f8bd015bfc0 2024-09-13T08:43:04.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.544+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f8bc800ac30 con 0x7f8bd015bfc0 2024-09-13T08:43:04.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.545+0000 7f8bc7e00640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 0x7f8bac07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:04.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.545+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f8bc809b130 con 0x7f8bd015bfc0 2024-09-13T08:43:04.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.545+0000 7f8bcf400640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 0x7f8bac07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:04.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.546+0000 7f8bcf400640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 0x7f8bac07aec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd015b4f0 tx=0x7f8bc0007a10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:04.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.548+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8bc8020070 con 0x7f8bd015bfc0 2024-09-13T08:43:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.783+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "subvol_snap26060", "target": ["mon-mgr", ""]}) -- 0x7f8bd00c8b20 con 0x7f8bac078a00 2024-09-13T08:43:04.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.820+0000 7f8bc7e00640 1 -- 172.21.15.31:0/623226758 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8bd00c8b20 con 0x7f8bac078a00 2024-09-13T08:43:04.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 msgr2=0x7f8bac07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 0x7f8bac07aec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd015b4f0 tx=0x7f8bc0007a10 comp rx=0 tx=0).stop 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 msgr2=0x7f8bd01470e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c7b0 tx=0x7f8bc800cc70 comp rx=0 tx=0).stop 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 shutdown_connections 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bac078a00 0x7f8bac07aec0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd015bfc0 0x7f8bd01470e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bd00cbc80 0x7f8bd01f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 --2- 172.21.15.31:0/623226758 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8bd0072560 0x7f8bd01f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:04.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 >> 172.21.15.31:0/623226758 conn(0x7f8bd00cfbb0 msgr2=0x7f8bd00ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:04.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 shutdown_connections 2024-09-13T08:43:04.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:04.822+0000 7f8bd6000640 1 -- 172.21.15.31:0/623226758 wait complete. 2024-09-13T08:43:04.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol5350 subvol_snap26060 subvol_clone87640 2024-09-13T08:43:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 -- 172.21.15.31:0/2527207473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82815bfc0 msgr2=0x7fb828072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2527207473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82815bfc0 0x7fb828072560 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb818005fd0 tx=0x7fb81802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:43:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 -- 172.21.15.31:0/2527207473 shutdown_connections 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2527207473 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb82806fa60 0x7fb8280cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2527207473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82815bfc0 0x7fb828072560 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 -- 172.21.15.31:0/2527207473 >> 172.21.15.31:0/2527207473 conn(0x7fb8280cfbb0 msgr2=0x7fb8280cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 -- 172.21.15.31:0/2527207473 shutdown_connections 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.011+0000 7fb82ca00640 1 -- 172.21.15.31:0/2527207473 wait complete. 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.012+0000 7fb82ca00640 1 Processor -- start 2024-09-13T08:43:05.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.013+0000 7fb82ca00640 1 -- start start 2024-09-13T08:43:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.013+0000 7fb82ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb82ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb82815bfc0 0x7fb828150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb82ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 0x7fb828148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb82ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8280b3270 con 0x7fb82806fa60 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb82ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8280b30f0 con 0x7fb8280c36e0 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb82ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb8280b33f0 con 0x7fb82815bfc0 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37378/0 (socket says 172.21.15.31:37378) 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 -- 172.21.15.31:0/2816180480 learned_addr learned my addr 172.21.15.31:0/2816180480 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 -- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb82815bfc0 msgr2=0x7fb828150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb82815bfc0 0x7fb828150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 -- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 msgr2=0x7fb828148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb826800640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 0x7fb828148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 0x7fb828148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb825e00640 1 -- 172.21.15.31:0/2816180480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82815c760 con 0x7fb82806fa60 2024-09-13T08:43:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.014+0000 7fb826800640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 0x7fb828148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb825e00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb818034040 tx=0x7fb8180038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81803e030 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb818003ca0 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8281f9480 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818035040 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.015+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8281f91a0 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.016+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8280cbf20 con 0x7fb82806fa60 2024-09-13T08:43:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.017+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7fb818041030 con 0x7fb82806fa60 2024-09-13T08:43:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.018+0000 7fb81e000640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 0x7fb80807aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.018+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fb8180bb290 con 0x7fb82806fa60 2024-09-13T08:43:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.018+0000 7fb825400640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 0x7fb80807aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.019+0000 7fb825400640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 0x7fb80807aec0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb814005fd0 tx=0x7fb814005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:05.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.023+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb818083560 con 0x7fb82806fa60 2024-09-13T08:43:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:05.254+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "subvol_snap26060", "target_sub_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fb8280cb310 con 0x7fb808078a00 2024-09-13T08:43:06.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:43:06.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:43:09.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.714+0000 7fb81e000640 1 -- 172.21.15.31:0/2816180480 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb8280cb310 con 0x7fb808078a00 2024-09-13T08:43:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 msgr2=0x7fb80807aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 0x7fb80807aec0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb814005fd0 tx=0x7fb814005850 comp rx=0 tx=0).stop 2024-09-13T08:43:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 msgr2=0x7fb8281f8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb818034040 tx=0x7fb8180038a0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 shutdown_connections 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb808078a00 0x7fb80807aec0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8280c36e0 0x7fb828148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb82815bfc0 0x7fb828150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 --2- 172.21.15.31:0/2816180480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb82806fa60 0x7fb8281f8570 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 >> 172.21.15.31:0/2816180480 conn(0x7fb8280cfbb0 msgr2=0x7fb828073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 shutdown_connections 2024-09-13T08:43:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.718+0000 7fb82ca00640 1 -- 172.21.15.31:0/2816180480 wait complete. 2024-09-13T08:43:09.722 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.720+0000 7fe170c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.722 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.720+0000 7fe170c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.722 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.720+0000 7fe170c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.722 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.720+0000 7fe170c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.722 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.720+0000 7fe170c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:09.742 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.740+0000 7fe172a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.743 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.740+0000 7fe172a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.743 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.740+0000 7fe172a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.743 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.740+0000 7fe172a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.743 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:09.740+0000 7fe172a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 -- 172.21.15.31:0/3050357377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 msgr2=0x7f925c0d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 --2- 172.21.15.31:0/3050357377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c0d28f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9248009830 tx=0x7f924802a530 comp rx=0 tx=0).stop 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 -- 172.21.15.31:0/3050357377 shutdown_connections 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 --2- 172.21.15.31:0/3050357377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c0d28f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 -- 172.21.15.31:0/3050357377 >> 172.21.15.31:0/3050357377 conn(0x7f925c0d1dd0 msgr2=0x7f925c0d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 -- 172.21.15.31:0/3050357377 shutdown_connections 2024-09-13T08:43:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.914+0000 7f9262000640 1 -- 172.21.15.31:0/3050357377 wait complete. 2024-09-13T08:43:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.916+0000 7f9262000640 1 Processor -- start 2024-09-13T08:43:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.916+0000 7f9262000640 1 -- start start 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f925c0ceb50 0x7f925c15e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 0x7f925c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b6070 con 0x7f925c0cf050 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b5ef0 con 0x7f925c0ceb50 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f9262000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0b61f0 con 0x7f925c15bec0 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f925aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f925aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37434/0 (socket says 172.21.15.31:37434) 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.917+0000 7f925aa00640 1 -- 172.21.15.31:0/170954587 learned_addr learned my addr 172.21.15.31:0/170954587 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925b400640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f925c0ceb50 0x7f925c15e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925be00640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 0x7f925c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 -- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 msgr2=0x7f925c06ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 0x7f925c06ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 -- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f925c0ceb50 msgr2=0x7f925c15e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f925c0ceb50 0x7f925c15e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 -- 172.21.15.31:0/170954587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c15d410 con 0x7f925c0cf050 2024-09-13T08:43:09.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925be00640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 0x7f925c06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f925aa00640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f924802ab20 tx=0x7f9248032c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.918+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248040070 con 0x7f925c0cf050 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.919+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9248005310 con 0x7f925c0cf050 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.919+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480047a0 con 0x7f925c0cf050 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.919+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c1f56d0 con 0x7f925c0cf050 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.919+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c15d9b0 con 0x7f925c0cf050 2024-09-13T08:43:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.920+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c0b6710 con 0x7f925c0cf050 2024-09-13T08:43:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.921+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f9248039040 con 0x7f925c0cf050 2024-09-13T08:43:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.922+0000 7f9253400640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 0x7f923c07aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.922+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f92480bbfb0 con 0x7f925c0cf050 2024-09-13T08:43:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.922+0000 7f925b400640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 0x7f923c07aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.923+0000 7f925b400640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 0x7f923c07aec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f924c000fb0 tx=0x7f924c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:09.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:09.926+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9248084200 con 0x7f925c0cf050 2024-09-13T08:43:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.158+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f925c0b6eb0 con 0x7f923c078a00 2024-09-13T08:43:10.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.190+0000 7f9253400640 1 -- 172.21.15.31:0/170954587 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+300 (secure 0 0 0) 0x7f925c0b6eb0 con 0x7f923c078a00 2024-09-13T08:43:10.192 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "0.003%", 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "212/8.0M", 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "3/9" 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:10.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:10.194 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:10.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 msgr2=0x7f923c07aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:10.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 0x7f923c07aec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f924c000fb0 tx=0x7f924c009040 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 msgr2=0x7f925c154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f924802ab20 tx=0x7f9248032c00 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 shutdown_connections 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f923c078a00 0x7f923c07aec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f925c15bec0 0x7f925c06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f925c0cf050 0x7f925c154370 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 --2- 172.21.15.31:0/170954587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f925c0ceb50 0x7f925c15e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 >> 172.21.15.31:0/170954587 conn(0x7f925c0d1dd0 msgr2=0x7f925c0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.193+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 shutdown_connections 2024-09-13T08:43:10.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:10.194+0000 7f9262000640 1 -- 172.21.15.31:0/170954587 wait complete. 2024-09-13T08:43:10.212 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:10.725 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:10.723+0000 7fe153600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:10.725 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:10.723+0000 7fe153600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:10.725 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:10.723+0000 7fe153600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:10.725 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:10.723+0000 7fe153600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:10.725 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:43:10.723+0000 7fe153600640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:43:11.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 -- 172.21.15.31:0/3747486412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 msgr2=0x7f42800ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 --2- 172.21.15.31:0/3747486412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42800ce930 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4270009830 tx=0x7f427002a480 comp rx=0 tx=0).stop 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 -- 172.21.15.31:0/3747486412 shutdown_connections 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 --2- 172.21.15.31:0/3747486412 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800d49d0 0x7f42800cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 --2- 172.21.15.31:0/3747486412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42800ce930 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 -- 172.21.15.31:0/3747486412 >> 172.21.15.31:0/3747486412 conn(0x7f42800d3e20 msgr2=0x7f42800d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 -- 172.21.15.31:0/3747486412 shutdown_connections 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.378+0000 7f427f200640 1 -- 172.21.15.31:0/3747486412 wait complete. 2024-09-13T08:43:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.380+0000 7f427f200640 1 Processor -- start 2024-09-13T08:43:11.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.380+0000 7f427f200640 1 -- start start 2024-09-13T08:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800cf3b0 0x7f42801f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 0x7f42801f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42800d0df0 con 0x7f428015df10 2024-09-13T08:43:11.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42800d0c70 con 0x7f42800d49d0 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f42800d0f70 con 0x7f42800cf3b0 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37478/0 (socket says 172.21.15.31:37478) 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427e800640 1 -- 172.21.15.31:0/1585589713 learned_addr learned my addr 172.21.15.31:0/1585589713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f427de00640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800cf3b0 0x7f42801f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.381+0000 7f4275400640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 0x7f42801f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 -- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800cf3b0 msgr2=0x7f42801f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800cf3b0 0x7f42801f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 -- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 msgr2=0x7f42801f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 0x7f42801f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 -- 172.21.15.31:0/1585589713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4280159ee0 con 0x7f428015df10 2024-09-13T08:43:11.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f4275400640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 0x7f42801f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427e800640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f427800cbf0 tx=0x7f427800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278016020 con 0x7f428015df10 2024-09-13T08:43:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f427800acf0 con 0x7f428015df10 2024-09-13T08:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278012680 con 0x7f428015df10 2024-09-13T08:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42801f5790 con 0x7f428015df10 2024-09-13T08:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.382+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4280159120 con 0x7f428015df10 2024-09-13T08:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.383+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42800d1500 con 0x7f428015df10 2024-09-13T08:43:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.384+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 101440+0+0 (secure 0 0 0) 0x7f4278019020 con 0x7f428015df10 2024-09-13T08:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.385+0000 7f4276a00640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 0x7f425c07aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.385+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f427809afd0 con 0x7f428015df10 2024-09-13T08:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.386+0000 7f427de00640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 0x7f425c07aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.386+0000 7f427de00640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 0x7f425c07aef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4270034000 tx=0x7f4270004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.389+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4278020030 con 0x7f428015df10 2024-09-13T08:43:11.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.621+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f428006d6c0 con 0x7f425c078a30 2024-09-13T08:43:11.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.640+0000 7f4276a00640 1 -- 172.21.15.31:0/1585589713 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+300 (secure 0 0 0) 0x7f428006d6c0 con 0x7f425c078a30 2024-09-13T08:43:11.646 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "0.003%", 2024-09-13T08:43:11.647 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "212/8.0M", 2024-09-13T08:43:11.648 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "3/9" 2024-09-13T08:43:11.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:11.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:11.648 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 msgr2=0x7f425c07aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 0x7f425c07aef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4270034000 tx=0x7f4270004730 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 msgr2=0x7f42801f8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f427800cbf0 tx=0x7f427800a430 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 shutdown_connections 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f425c078a30 0x7f425c07aef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f428015df10 0x7f42801f8390 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42800d49d0 0x7f42801f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 --2- 172.21.15.31:0/1585589713 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f42800cf3b0 0x7f42801f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 >> 172.21.15.31:0/1585589713 conn(0x7f42800d3e20 msgr2=0x7f428015bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 shutdown_connections 2024-09-13T08:43:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:11.643+0000 7f427f200640 1 -- 172.21.15.31:0/1585589713 wait complete. 2024-09-13T08:43:11.661 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:12.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 -- 172.21.15.31:0/1067388912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 msgr2=0x7fe36c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 --2- 172.21.15.31:0/1067388912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe36c003e70 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe378009b50 tx=0x7fe36c005830 comp rx=0 tx=0).stop 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 -- 172.21.15.31:0/1067388912 shutdown_connections 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 --2- 172.21.15.31:0/1067388912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe36c003e70 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 -- 172.21.15.31:0/1067388912 >> 172.21.15.31:0/1067388912 conn(0x7fe3880d3e90 msgr2=0x7fe3880d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 -- 172.21.15.31:0/1067388912 shutdown_connections 2024-09-13T08:43:12.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.838+0000 7fe387e00640 1 -- 172.21.15.31:0/1067388912 wait complete. 2024-09-13T08:43:12.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.839+0000 7fe387e00640 1 Processor -- start 2024-09-13T08:43:12.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 -- start start 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe38815d030 0x7fe3881f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3881f8560 0x7fe3881f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe3880b57c0 con 0x7fe36c003a50 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe3880b5640 con 0x7fe3881f8560 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.840+0000 7fe387e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe3880d21d0 con 0x7fe38815d030 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37508/0 (socket says 172.21.15.31:37508) 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 -- 172.21.15.31:0/2010682026 learned_addr learned my addr 172.21.15.31:0/2010682026 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe37fe00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe38815d030 0x7fe3881f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe387400640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3881f8560 0x7fe3881f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 -- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe38815d030 msgr2=0x7fe3881f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe38815d030 0x7fe3881f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 -- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3881f8560 msgr2=0x7fe3881f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3881f8560 0x7fe3881f89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 -- 172.21.15.31:0/2010682026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38815b6e0 con 0x7fe36c003a50 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.841+0000 7fe386a00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe378009a70 tx=0x7fe3780336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.842+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378040020 con 0x7fe36c003a50 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.842+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe378004310 con 0x7fe36c003a50 2024-09-13T08:43:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.842+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378038040 con 0x7fe36c003a50 2024-09-13T08:43:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.842+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38815b9c0 con 0x7fe36c003a50 2024-09-13T08:43:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.842+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe38815bf80 con 0x7fe36c003a50 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.843+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3880b4820 con 0x7fe36c003a50 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.844+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fe37802b8a0 con 0x7fe36c003a50 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.845+0000 7fe37ea00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 0x7fe36407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.845+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fe3780bb2f0 con 0x7fe36c003a50 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.845+0000 7fe37fe00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 0x7fe36407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.846+0000 7fe37fe00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 0x7fe36407b070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe370001860 tx=0x7fe370000ed0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:12.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:12.849+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe37804a020 con 0x7fe36c003a50 2024-09-13T08:43:13.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.087+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fe38815f890 con 0x7fe364078bb0 2024-09-13T08:43:13.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.112+0000 7fe37ea00640 1 -- 172.21.15.31:0/2010682026 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+300 (secure 0 0 0) 0x7fe38815f890 con 0x7fe364078bb0 2024-09-13T08:43:13.114 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "0.003%", 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "212/8.0M", 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "3/9" 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:13.115 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 msgr2=0x7fe36407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 0x7fe36407b070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe370001860 tx=0x7fe370000ed0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 msgr2=0x7fe3881f1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe378009a70 tx=0x7fe3780336f0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 shutdown_connections 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe364078bb0 0x7fe36407b070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3881f8560 0x7fe3881f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe38815d030 0x7fe3881f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 --2- 172.21.15.31:0/2010682026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe36c003a50 0x7fe3881f1df0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 >> 172.21.15.31:0/2010682026 conn(0x7fe3880d3e90 msgr2=0x7fe38806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 shutdown_connections 2024-09-13T08:43:13.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:13.115+0000 7fe387e00640 1 -- 172.21.15.31:0/2010682026 wait complete. 2024-09-13T08:43:13.132 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:14.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:14.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 -- 172.21.15.31:0/2166944539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 msgr2=0x7ff1000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 --2- 172.21.15.31:0/2166944539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff1000051c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ed0 tx=0x7ff100005830 comp rx=0 tx=0).stop 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff121600640 1 -- 172.21.15.31:0/2166944539 reap_dead start 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 -- 172.21.15.31:0/2166944539 shutdown_connections 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 -- 172.21.15.31:0/2166944539 >> 172.21.15.31:0/2166944539 conn(0x7ff11c0cfbb0 msgr2=0x7ff11c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 -- 172.21.15.31:0/2166944539 shutdown_connections 2024-09-13T08:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.306+0000 7ff123600640 1 -- 172.21.15.31:0/2166944539 wait complete. 2024-09-13T08:43:14.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.308+0000 7ff123600640 1 Processor -- start 2024-09-13T08:43:14.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.308+0000 7ff123600640 1 -- start start 2024-09-13T08:43:14.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c0c6da0 0x7ff11c0c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff11c1f5130 0x7ff11c1f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c0cbe50 con 0x7ff100004d40 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c070040 con 0x7ff11c0c6da0 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff123600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c0cbfd0 con 0x7ff11c1f5130 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff120c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff120c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37512/0 (socket says 172.21.15.31:37512) 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.309+0000 7ff120c00640 1 -- 172.21.15.31:0/2016585766 learned_addr learned my addr 172.21.15.31:0/2016585766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff121600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff11c1f5130 0x7ff11c1f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff113e00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c0c6da0 0x7ff11c0c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 -- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff11c1f5130 msgr2=0x7ff11c1f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:14.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff11c1f5130 0x7ff11c1f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 -- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c0c6da0 msgr2=0x7ff11c0c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c0c6da0 0x7ff11c0c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 -- 172.21.15.31:0/2016585766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c0c8f60 con 0x7ff100004d40 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff120c00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff10c006680 tx=0x7ff10c0017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.310+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c015070 con 0x7ff100004d40 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.311+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff10c011900 con 0x7ff100004d40 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.311+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c1f5b20 con 0x7ff100004d40 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.311+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c011d30 con 0x7ff100004d40 2024-09-13T08:43:14.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.311+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c0c94e0 con 0x7ff100004d40 2024-09-13T08:43:14.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.312+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c0ca370 con 0x7ff100004d40 2024-09-13T08:43:14.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.313+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7ff10c018070 con 0x7ff100004d40 2024-09-13T08:43:14.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.314+0000 7ff119600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 0x7ff0f807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:14.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.315+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7ff10c09bd50 con 0x7ff100004d40 2024-09-13T08:43:14.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.315+0000 7ff113e00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 0x7ff0f807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:14.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.315+0000 7ff113e00640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 0x7ff0f807b070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0b21c0 tx=0x7ff104006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:14.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.319+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff10c0637e0 con 0x7ff100004d40 2024-09-13T08:43:14.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.553+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7ff11c0b5b30 con 0x7ff0f8078bb0 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.573+0000 7ff119600640 1 -- 172.21.15.31:0/2016585766 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+300 (secure 0 0 0) 0x7ff11c0b5b30 con 0x7ff0f8078bb0 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:14.575 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "0.003%", 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "212/8.0M", 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "3/9" 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:14.576 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 msgr2=0x7ff0f807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 0x7ff0f807b070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0b21c0 tx=0x7ff104006040 comp rx=0 tx=0).stop 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 msgr2=0x7ff11c0afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff10c006680 tx=0x7ff10c0017f0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 shutdown_connections 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0f8078bb0 0x7ff0f807b070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff11c1f5130 0x7ff11c1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff11c0c6da0 0x7ff11c0c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 --2- 172.21.15.31:0/2016585766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff100004d40 0x7ff11c0afd50 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:14.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 >> 172.21.15.31:0/2016585766 conn(0x7ff11c0cfbb0 msgr2=0x7ff11c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:14.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 shutdown_connections 2024-09-13T08:43:14.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:14.576+0000 7ff123600640 1 -- 172.21.15.31:0/2016585766 wait complete. 2024-09-13T08:43:14.594 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:15.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:15.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 -- 172.21.15.31:0/2045349603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb600cb380 msgr2=0x7fcb600cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 --2- 172.21.15.31:0/2045349603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb600cb380 0x7fcb600cdcc0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fcb600767f0 tx=0x7fcb5802a820 comp rx=0 tx=0).stop 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 -- 172.21.15.31:0/2045349603 shutdown_connections 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 --2- 172.21.15.31:0/2045349603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb600cb380 0x7fcb600cdcc0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 --2- 172.21.15.31:0/2045349603 >> v2:172.21.15.45:3300/0 conn(0x7fcb60161480 0x7fcb600cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 -- 172.21.15.31:0/2045349603 >> 172.21.15.31:0/2045349603 conn(0x7fcb600cfbb0 msgr2=0x7fcb600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 -- 172.21.15.31:0/2045349603 shutdown_connections 2024-09-13T08:43:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.768+0000 7fcb67400640 1 -- 172.21.15.31:0/2045349603 wait complete. 2024-09-13T08:43:15.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.771+0000 7fcb67400640 1 Processor -- start 2024-09-13T08:43:15.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.771+0000 7fcb67400640 1 -- start start 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600cb380 0x7fcb601f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb600cc190 0x7fcb601506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb600cbdc0 con 0x7fcb60161480 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb600cbc40 con 0x7fcb600cb380 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb67400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcb600cbf40 con 0x7fcb600cc190 2024-09-13T08:43:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54938/0 (socket says 172.21.15.31:54938) 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 -- 172.21.15.31:0/2572307005 learned_addr learned my addr 172.21.15.31:0/2572307005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb64a00640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600cb380 0x7fcb601f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb57e00640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb600cc190 0x7fcb601506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 -- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb600cc190 msgr2=0x7fcb601506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb600cc190 0x7fcb601506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 -- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600cb380 msgr2=0x7fcb601f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600cb380 0x7fcb601f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 -- 172.21.15.31:0/2572307005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601f88c0 con 0x7fcb60161480 2024-09-13T08:43:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.772+0000 7fcb65400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009ca0 tx=0x7fcb58032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.773+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58037040 con 0x7fcb60161480 2024-09-13T08:43:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.773+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb58042070 con 0x7fcb60161480 2024-09-13T08:43:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.773+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb580042a0 con 0x7fcb60161480 2024-09-13T08:43:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.773+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb601f8ba0 con 0x7fcb60161480 2024-09-13T08:43:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.773+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6014d570 con 0x7fcb60161480 2024-09-13T08:43:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.774+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb601f90c0 con 0x7fcb60161480 2024-09-13T08:43:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.776+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fcb58004440 con 0x7fcb60161480 2024-09-13T08:43:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.776+0000 7fcb5d600640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 0x7fcb3c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.776+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fcb580bbae0 con 0x7fcb60161480 2024-09-13T08:43:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.776+0000 7fcb64a00640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 0x7fcb3c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.777+0000 7fcb64a00640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 0x7fcb3c07b070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005fd0 tx=0x7fcb5000b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:15.780+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcb601f90c0 con 0x7fcb60161480 2024-09-13T08:43:16.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.016+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fcb600b29e0 con 0x7fcb3c078bb0 2024-09-13T08:43:16.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.037+0000 7fcb5d600640 1 -- 172.21.15.31:0/2572307005 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fcb5000b7c0 con 0x7fcb3c078bb0 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "25.003%", 2024-09-13T08:43:16.040 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "2.0M/8.0M", 2024-09-13T08:43:16.041 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "6/9" 2024-09-13T08:43:16.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:16.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:16.041 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 msgr2=0x7fcb3c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 0x7fcb3c07b070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005fd0 tx=0x7fcb5000b2e0 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 msgr2=0x7fcb600c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009ca0 tx=0x7fcb58032d10 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 shutdown_connections 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb3c078bb0 0x7fcb3c07b070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb60161480 0x7fcb600c59c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb600cc190 0x7fcb601506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 --2- 172.21.15.31:0/2572307005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600cb380 0x7fcb601f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 >> 172.21.15.31:0/2572307005 conn(0x7fcb600cfbb0 msgr2=0x7fcb60073f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 shutdown_connections 2024-09-13T08:43:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:16.041+0000 7fcb67400640 1 -- 172.21.15.31:0/2572307005 wait complete. 2024-09-13T08:43:16.058 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:17.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 -- 172.21.15.31:0/3895972789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c161480 msgr2=0x7ff17c0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 --2- 172.21.15.31:0/3895972789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c161480 0x7ff17c0cd780 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009830 tx=0x7ff16c02a480 comp rx=0 tx=0).stop 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 -- 172.21.15.31:0/3895972789 shutdown_connections 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 --2- 172.21.15.31:0/3895972789 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c0cc190 0x7ff17c0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 --2- 172.21.15.31:0/3895972789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c161480 0x7ff17c0cd780 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.230+0000 7ff181800640 1 -- 172.21.15.31:0/3895972789 >> 172.21.15.31:0/3895972789 conn(0x7ff17c0cfbb0 msgr2=0x7ff17c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.231+0000 7ff181800640 1 -- 172.21.15.31:0/3895972789 shutdown_connections 2024-09-13T08:43:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.231+0000 7ff181800640 1 -- 172.21.15.31:0/3895972789 wait complete. 2024-09-13T08:43:17.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.232+0000 7ff181800640 1 Processor -- start 2024-09-13T08:43:17.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.232+0000 7ff181800640 1 -- start start 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 0x7ff17c1f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c161480 0x7ff17c0c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0cb500 con 0x7ff17c0cc190 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0cb380 con 0x7ff17c0cbc40 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff181800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff17c0cb680 con 0x7ff17c161480 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff172000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff172000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54958/0 (socket says 172.21.15.31:54958) 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.233+0000 7ff172000640 1 -- 172.21.15.31:0/3645235630 learned_addr learned my addr 172.21.15.31:0/3645235630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff17aa00640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 0x7ff17c1f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff17b400640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c161480 0x7ff17c0c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 -- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c161480 msgr2=0x7ff17c0c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c161480 0x7ff17c0c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 -- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 msgr2=0x7ff17c1f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 0x7ff17c1f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 -- 172.21.15.31:0/3645235630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c1f5a20 con 0x7ff17c0cc190 2024-09-13T08:43:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff17aa00640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 0x7ff17c1f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff172000640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff16400d980 tx=0x7ff16400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.234+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164017070 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.235+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff164009d20 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.235+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c15f770 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.235+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1640106c0 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.235+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c15fd30 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.236+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c1625f0 con 0x7ff17c0cc190 2024-09-13T08:43:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.237+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7ff164010860 con 0x7ff17c0cc190 2024-09-13T08:43:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.238+0000 7ff173400640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 0x7ff15807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.238+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7ff16409b3e0 con 0x7ff17c0cc190 2024-09-13T08:43:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.239+0000 7ff17aa00640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 0x7ff15807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.239+0000 7ff17aa00640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 0x7ff15807b070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff16c034000 tx=0x7ff16c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.243+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff164062da0 con 0x7ff17c0cc190 2024-09-13T08:43:17.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.482+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7ff17c0d0550 con 0x7ff158078bb0 2024-09-13T08:43:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.504+0000 7ff173400640 1 -- 172.21.15.31:0/3645235630 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7ff17c0d0550 con 0x7ff158078bb0 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "25.003%", 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "2.0M/8.0M", 2024-09-13T08:43:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "6/9" 2024-09-13T08:43:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:17.506 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 msgr2=0x7ff15807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 0x7ff15807b070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff16c034000 tx=0x7ff16c004730 comp rx=0 tx=0).stop 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 msgr2=0x7ff17c15ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff16400d980 tx=0x7ff16400de40 comp rx=0 tx=0).stop 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 shutdown_connections 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff158078bb0 0x7ff15807b070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff17c161480 0x7ff17c0c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c0cc190 0x7ff17c15ecb0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 --2- 172.21.15.31:0/3645235630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff17c0cbc40 0x7ff17c1f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:17.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 >> 172.21.15.31:0/3645235630 conn(0x7ff17c0cfbb0 msgr2=0x7ff17c0ad890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:17.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 shutdown_connections 2024-09-13T08:43:17.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:17.506+0000 7ff181800640 1 -- 172.21.15.31:0/3645235630 wait complete. 2024-09-13T08:43:17.523 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:18.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754f600640 1 -- 172.21.15.31:0/2697586517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 msgr2=0x7f752c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754f600640 1 --2- 172.21.15.31:0/2697586517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f752c0051c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7540001d60 tx=0x7f752c005830 comp rx=0 tx=0).stop 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754d600640 1 -- 172.21.15.31:0/2697586517 reap_dead start 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754f600640 1 -- 172.21.15.31:0/2697586517 shutdown_connections 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754f600640 1 -- 172.21.15.31:0/2697586517 >> 172.21.15.31:0/2697586517 conn(0x7f75480cfbb0 msgr2=0x7f75480cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.699+0000 7f754f600640 1 -- 172.21.15.31:0/2697586517 shutdown_connections 2024-09-13T08:43:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.700+0000 7f754f600640 1 -- 172.21.15.31:0/2697586517 wait complete. 2024-09-13T08:43:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.701+0000 7f754f600640 1 Processor -- start 2024-09-13T08:43:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.701+0000 7f754f600640 1 -- start start 2024-09-13T08:43:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 0x7f75481fcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 0x7f75481fd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75480cbdc0 con 0x7f752c004d40 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75480cbc40 con 0x7f75480b1cc0 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754f600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f75480cbf40 con 0x7f75480b1840 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54976/0 (socket says 172.21.15.31:54976) 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.702+0000 7f754cc00640 1 -- 172.21.15.31:0/3182497871 learned_addr learned my addr 172.21.15.31:0/3182497871 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f753fe00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 0x7f75481fcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 -- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 msgr2=0x7f75481fcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754d600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 0x7f75481fd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 0x7f75481fcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 -- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 msgr2=0x7f75481fd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 0x7f75481fd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 -- 172.21.15.31:0/3182497871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75480b2220 con 0x7f752c004d40 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f753fe00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 0x7f75481fcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754d600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 0x7f75481fd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f754cc00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7538011b10 tx=0x7f7538009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.703+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753801e070 con 0x7f752c004d40 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.704+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7538004050 con 0x7f752c004d40 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.704+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75480c6db0 con 0x7f752c004d40 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.704+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753800fea0 con 0x7f752c004d40 2024-09-13T08:43:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.704+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75480c7370 con 0x7f752c004d40 2024-09-13T08:43:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.705+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75480cda00 con 0x7f752c004d40 2024-09-13T08:43:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.706+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f7538021070 con 0x7f752c004d40 2024-09-13T08:43:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.707+0000 7f7545600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 0x7f752407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.707+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f753809ba40 con 0x7f752c004d40 2024-09-13T08:43:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.707+0000 7f753fe00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 0x7f752407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.708+0000 7f753fe00640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 0x7f752407b070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75480b1580 tx=0x7f75300067a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:18.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.711+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f75380044a0 con 0x7f752c004d40 2024-09-13T08:43:18.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.948+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f75480b2500 con 0x7f7524078bb0 2024-09-13T08:43:18.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.969+0000 7f7545600640 1 -- 172.21.15.31:0/3182497871 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f75480b2500 con 0x7f7524078bb0 2024-09-13T08:43:18.970 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "25.003%", 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "2.0M/8.0M", 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "6/9" 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:18.971 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 msgr2=0x7f752407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 0x7f752407b070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75480b1580 tx=0x7f75300067a0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 msgr2=0x7f75480af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7538011b10 tx=0x7f7538009e90 comp rx=0 tx=0).stop 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 shutdown_connections 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7524078bb0 0x7f752407b070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75480b1cc0 0x7f75481fd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75480b1840 0x7f75481fcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 --2- 172.21.15.31:0/3182497871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f752c004d40 0x7f75480af510 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:18.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 >> 172.21.15.31:0/3182497871 conn(0x7f75480cfbb0 msgr2=0x7f75480b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:18.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 shutdown_connections 2024-09-13T08:43:18.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:18.972+0000 7f754f600640 1 -- 172.21.15.31:0/3182497871 wait complete. 2024-09-13T08:43:18.989 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:19.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 -- 172.21.15.31:0/3726663173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 msgr2=0x7f91140ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 --2- 172.21.15.31:0/3726663173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f91140ca020 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f91140b22f0 tx=0x7f911002a4d0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 -- 172.21.15.31:0/3726663173 shutdown_connections 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 --2- 172.21.15.31:0/3726663173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f91140ca020 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 --2- 172.21.15.31:0/3726663173 >> v2:172.21.15.45:3300/0 conn(0x7f91140b5300 0x7f91140cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 --2- 172.21.15.31:0/3726663173 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 0x7f911406fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 -- 172.21.15.31:0/3726663173 >> 172.21.15.31:0/3726663173 conn(0x7f91140b1520 msgr2=0x7f91140b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 -- 172.21.15.31:0/3726663173 shutdown_connections 2024-09-13T08:43:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.165+0000 7f911b200640 1 -- 172.21.15.31:0/3726663173 wait complete. 2024-09-13T08:43:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.167+0000 7f911b200640 1 Processor -- start 2024-09-13T08:43:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.167+0000 7f911b200640 1 -- start start 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 0x7f91141f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 0x7f91140b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91140ca560 con 0x7f91140cc920 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f911415c090 con 0x7f91140b5300 2024-09-13T08:43:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f91140ca6e0 con 0x7f911415bb70 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f9119400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f9119400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54984/0 (socket says 172.21.15.31:54984) 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f9119400640 1 -- 172.21.15.31:0/3129623918 learned_addr learned my addr 172.21.15.31:0/3129623918 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f9119e00640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 0x7f91141f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.168+0000 7f911a800640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 0x7f91140b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 -- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 msgr2=0x7f91140b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 0x7f91140b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 -- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 msgr2=0x7f91141f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 0x7f91141f7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 -- 172.21.15.31:0/3129623918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911414daa0 con 0x7f91140cc920 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119e00640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 0x7f91141f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f9119400640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f910400c970 tx=0x7f910400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9104007c10 con 0x7f91140cc920 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9104018480 con 0x7f91140cc920 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91040188d0 con 0x7f91140cc920 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9114150290 con 0x7f91140cc920 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.169+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9114150850 con 0x7f91140cc920 2024-09-13T08:43:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.170+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91140cabb0 con 0x7f91140cc920 2024-09-13T08:43:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.171+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f9104018a70 con 0x7f91140cc920 2024-09-13T08:43:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.172+0000 7f910a000640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 0x7f90f807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.173+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f910409ba10 con 0x7f91140cc920 2024-09-13T08:43:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.173+0000 7f9119e00640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 0x7f90f807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:20.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.173+0000 7f9119e00640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 0x7f90f807b070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f910c00a2e0 tx=0x7f910c00a270 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.177+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9104063bf0 con 0x7f91140cc920 2024-09-13T08:43:20.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.411+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f91140c9670 con 0x7f90f8078bb0 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.432+0000 7f910a000640 1 -- 172.21.15.31:0/3129623918 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f91140c9670 con 0x7f90f8078bb0 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:20.433 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "7/9" 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:20.434 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:20.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.434+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 msgr2=0x7f90f807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.434+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 0x7f90f807b070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f910c00a2e0 tx=0x7f910c00a270 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.434+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 msgr2=0x7f911414d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.434+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f910400c970 tx=0x7f910400ce30 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 shutdown_connections 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f90f8078bb0 0x7f90f807b070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f911415bb70 0x7f91140b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91140cc920 0x7f911414d560 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 --2- 172.21.15.31:0/3129623918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91140b5300 0x7f91141f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 >> 172.21.15.31:0/3129623918 conn(0x7f91140b1520 msgr2=0x7f91140b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 shutdown_connections 2024-09-13T08:43:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:20.435+0000 7f911b200640 1 -- 172.21.15.31:0/3129623918 wait complete. 2024-09-13T08:43:20.451 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:21.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 -- 172.21.15.31:0/1365420305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d4015ad00 msgr2=0x7f2d4015b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 --2- 172.21.15.31:0/1365420305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d4015ad00 0x7f2d4015b0e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009830 tx=0x7f2d3002a510 comp rx=0 tx=0).stop 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 -- 172.21.15.31:0/1365420305 shutdown_connections 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 --2- 172.21.15.31:0/1365420305 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d400cabb0 0x7f2d400cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 --2- 172.21.15.31:0/1365420305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d4015ad00 0x7f2d4015b0e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 -- 172.21.15.31:0/1365420305 >> 172.21.15.31:0/1365420305 conn(0x7f2d400cfbb0 msgr2=0x7f2d400cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 -- 172.21.15.31:0/1365420305 shutdown_connections 2024-09-13T08:43:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.628+0000 7f2d46600640 1 -- 172.21.15.31:0/1365420305 wait complete. 2024-09-13T08:43:21.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.630+0000 7f2d46600640 1 Processor -- start 2024-09-13T08:43:21.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.630+0000 7f2d46600640 1 -- start start 2024-09-13T08:43:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 0x7f2d400b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d4015b620 0x7f2d401e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4006eee0 con 0x7f2d400cabb0 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4006ed60 con 0x7f2d4015ad00 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.631+0000 7f2d46600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4006f060 con 0x7f2d4015b620 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55012/0 (socket says 172.21.15.31:55012) 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 -- 172.21.15.31:0/1959989066 learned_addr learned my addr 172.21.15.31:0/1959989066 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3fe00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d4015b620 0x7f2d401e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d36a00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 0x7f2d400b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 -- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d4015b620 msgr2=0x7f2d401e40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:21.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d4015b620 0x7f2d401e40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 -- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 msgr2=0x7f2d400b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 0x7f2d400b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 -- 172.21.15.31:0/1959989066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d401f4b60 con 0x7f2d400cabb0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d36a00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 0x7f2d400b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.632+0000 7f2d3f400640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2d3002ab00 tx=0x7f2d30032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.633+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30040070 con 0x7f2d400cabb0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.633+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d30005310 con 0x7f2d400cabb0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.633+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30005c90 con 0x7f2d400cabb0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.633+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d400b3300 con 0x7f2d400cabb0 2024-09-13T08:43:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.633+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d400b2f40 con 0x7f2d400cabb0 2024-09-13T08:43:21.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.634+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d400cdba0 con 0x7f2d400cabb0 2024-09-13T08:43:21.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.635+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f2d30039040 con 0x7f2d400cabb0 2024-09-13T08:43:21.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.636+0000 7f2d37e00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 0x7f2d1c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:21.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.636+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f2d300bc0f0 con 0x7f2d400cabb0 2024-09-13T08:43:21.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.636+0000 7f2d36a00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 0x7f2d1c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:21.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.637+0000 7f2d36a00640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 0x7f2d1c07b070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007a50 tx=0x7f2d28009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:21.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.640+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2d30084290 con 0x7f2d400cabb0 2024-09-13T08:43:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.870+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f2d400b5b30 con 0x7f2d1c078bb0 2024-09-13T08:43:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.892+0000 7f2d37e00640 1 -- 172.21.15.31:0/1959989066 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f2d400b5b30 con 0x7f2d1c078bb0 2024-09-13T08:43:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:21.893 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:21.893 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "7/9" 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:21.894 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 msgr2=0x7f2d1c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 0x7f2d1c07b070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007a50 tx=0x7f2d28009040 comp rx=0 tx=0).stop 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 msgr2=0x7f2d401f55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2d3002ab00 tx=0x7f2d30032c10 comp rx=0 tx=0).stop 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 shutdown_connections 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d1c078bb0 0x7f2d1c07b070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d4015b620 0x7f2d401e40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4015ad00 0x7f2d400b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 --2- 172.21.15.31:0/1959989066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d400cabb0 0x7f2d401f55b0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:21.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 >> 172.21.15.31:0/1959989066 conn(0x7f2d400cfbb0 msgr2=0x7f2d40156640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:21.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 shutdown_connections 2024-09-13T08:43:21.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:21.895+0000 7f2d46600640 1 -- 172.21.15.31:0/1959989066 wait complete. 2024-09-13T08:43:21.911 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:22.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 -- 172.21.15.31:0/3302634810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd858003c30 msgr2=0x7fd858004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 --2- 172.21.15.31:0/3302634810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd858003c30 0x7fd858004030 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd864009b50 tx=0x7fd858005830 comp rx=0 tx=0).stop 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 -- 172.21.15.31:0/3302634810 shutdown_connections 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 --2- 172.21.15.31:0/3302634810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd858003c30 0x7fd858004030 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 -- 172.21.15.31:0/3302634810 >> 172.21.15.31:0/3302634810 conn(0x7fd8700cfbb0 msgr2=0x7fd8700cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 -- 172.21.15.31:0/3302634810 shutdown_connections 2024-09-13T08:43:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.089+0000 7fd878600640 1 -- 172.21.15.31:0/3302634810 wait complete. 2024-09-13T08:43:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.091+0000 7fd878600640 1 Processor -- start 2024-09-13T08:43:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.091+0000 7fd878600640 1 -- start start 2024-09-13T08:43:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd858003c30 0x7fd8700c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8701f6ac0 0x7fd8701418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8700726e0 con 0x7fd8701511f0 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd870072560 con 0x7fd8701f6ac0 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.092+0000 7fd878600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd870072860 con 0x7fd858003c30 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd875200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8701f6ac0 0x7fd8701418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55030/0 (socket says 172.21.15.31:55030) 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 -- 172.21.15.31:0/3511819699 learned_addr learned my addr 172.21.15.31:0/3511819699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd875c00640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd858003c30 0x7fd8700c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 -- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd858003c30 msgr2=0x7fd8700c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd858003c30 0x7fd8700c4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 -- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8701f6ac0 msgr2=0x7fd8701418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8701f6ac0 0x7fd8701418b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.093+0000 7fd876600640 1 -- 172.21.15.31:0/3511819699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8701f3a60 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd876600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd8700b5a90 tx=0x7fd868001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868011870 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd868017070 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8701f2ee0 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868011da0 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.094+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8701f34a0 con 0x7fd8701511f0 2024-09-13T08:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.095+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8700cb970 con 0x7fd8701511f0 2024-09-13T08:43:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.097+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fd86801c460 con 0x7fd8701511f0 2024-09-13T08:43:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.098+0000 7fd86d600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 0x7fd85007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.098+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fd86809b5f0 con 0x7fd8701511f0 2024-09-13T08:43:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.099+0000 7fd875c00640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 0x7fd85007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.099+0000 7fd875c00640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 0x7fd85007b070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd864035000 tx=0x7fd8640046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.103+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd868063140 con 0x7fd8701511f0 2024-09-13T08:43:23.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.335+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fd8700cb310 con 0x7fd850078bb0 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.358+0000 7fd86d600640 1 -- 172.21.15.31:0/3511819699 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fd8700cb310 con 0x7fd850078bb0 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "7/9" 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:23.360 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.360+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 msgr2=0x7fd85007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.360+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 0x7fd85007b070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd864035000 tx=0x7fd8640046c0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.360+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 msgr2=0x7fd870151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd8700b5a90 tx=0x7fd868001b80 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 shutdown_connections 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd850078bb0 0x7fd85007b070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8701511f0 0x7fd870151670 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8701f6ac0 0x7fd8701418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 --2- 172.21.15.31:0/3511819699 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd858003c30 0x7fd8700c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 >> 172.21.15.31:0/3511819699 conn(0x7fd8700cfbb0 msgr2=0x7fd87015d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 shutdown_connections 2024-09-13T08:43:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:23.361+0000 7fd878600640 1 -- 172.21.15.31:0/3511819699 wait complete. 2024-09-13T08:43:23.378 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:24.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 -- 172.21.15.31:0/1588979635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7638003c60 msgr2=0x7f7638004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 --2- 172.21.15.31:0/1588979635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7638003c60 0x7f7638004060 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f76440017d0 tx=0x7f7638005830 comp rx=0 tx=0).stop 2024-09-13T08:43:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7656800640 1 -- 172.21.15.31:0/1588979635 reap_dead start 2024-09-13T08:43:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 -- 172.21.15.31:0/1588979635 shutdown_connections 2024-09-13T08:43:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 -- 172.21.15.31:0/1588979635 >> 172.21.15.31:0/1588979635 conn(0x7f76500cfbb0 msgr2=0x7f76500cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 -- 172.21.15.31:0/1588979635 shutdown_connections 2024-09-13T08:43:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.554+0000 7f7658800640 1 -- 172.21.15.31:0/1588979635 wait complete. 2024-09-13T08:43:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.556+0000 7f7658800640 1 Processor -- start 2024-09-13T08:43:24.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.556+0000 7f7658800640 1 -- start start 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 0x7f76500c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76501f5250 0x7f76500c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f765006e740 con 0x7f7650155f30 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f765006e5c0 con 0x7f7638003c60 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7658800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f76500cb740 con 0x7f76501f5250 2024-09-13T08:43:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55042/0 (socket says 172.21.15.31:55042) 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655400640 1 -- 172.21.15.31:0/1045925013 learned_addr learned my addr 172.21.15.31:0/1045925013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655e00640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 0x7f76500c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7656800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76501f5250 0x7f76500c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655400640 1 -- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76501f5250 msgr2=0x7f76500c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.557+0000 7f7655400640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76501f5250 0x7f76500c5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7655400640 1 -- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 msgr2=0x7f76500c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7655400640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 0x7f76500c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7655400640 1 -- 172.21.15.31:0/1045925013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76501f4a70 con 0x7f7650155f30 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7655e00640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 0x7f76500c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7655400640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7640011a50 tx=0x7f7640011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764001f070 con 0x7f7650155f30 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f764000fdc0 con 0x7f7650155f30 2024-09-13T08:43:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76500c38f0 con 0x7f7650155f30 2024-09-13T08:43:24.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400126b0 con 0x7f7650155f30 2024-09-13T08:43:24.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.558+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76501f4d30 con 0x7f7650155f30 2024-09-13T08:43:24.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.559+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765006fa60 con 0x7f7650155f30 2024-09-13T08:43:24.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.561+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f7640022070 con 0x7f7650155f30 2024-09-13T08:43:24.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.561+0000 7f764a000640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 0x7f763407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:24.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.562+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f764009ba70 con 0x7f7650155f30 2024-09-13T08:43:24.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.562+0000 7f7655e00640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 0x7f763407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.563+0000 7f7655e00640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 0x7f763407b070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7644032000 tx=0x7f7644001360 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.566+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7640063430 con 0x7f7650155f30 2024-09-13T08:43:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.806+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f76500cb310 con 0x7f7634078bb0 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.830+0000 7f764a000640 1 -- 172.21.15.31:0/1045925013 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f76500cb310 con 0x7f7634078bb0 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "8/9" 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:24.832 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 msgr2=0x7f763407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 0x7f763407b070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7644032000 tx=0x7f7644001360 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 msgr2=0x7f76500cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7640011a50 tx=0x7f7640011f10 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 shutdown_connections 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7634078bb0 0x7f763407b070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76501f5250 0x7f76500c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650155f30 0x7f76500cdb80 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.832+0000 7f7658800640 1 --2- 172.21.15.31:0/1045925013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7638003c60 0x7f76500c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.833+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 >> 172.21.15.31:0/1045925013 conn(0x7f76500cfbb0 msgr2=0x7f765015d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.833+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 shutdown_connections 2024-09-13T08:43:24.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:24.833+0000 7f7658800640 1 -- 172.21.15.31:0/1045925013 wait complete. 2024-09-13T08:43:24.849 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:25.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.024+0000 7fa43f200640 1 -- 172.21.15.31:0/2852880476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424003c10 msgr2=0x7fa424004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.024+0000 7fa43f200640 1 --2- 172.21.15.31:0/2852880476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa424003c10 0x7fa424004010 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa430001830 tx=0x7fa424005830 comp rx=0 tx=0).stop 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.024+0000 7fa43e800640 1 -- 172.21.15.31:0/2852880476 reap_dead start 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.025+0000 7fa43f200640 1 -- 172.21.15.31:0/2852880476 shutdown_connections 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.025+0000 7fa43f200640 1 -- 172.21.15.31:0/2852880476 >> 172.21.15.31:0/2852880476 conn(0x7fa4400d3e10 msgr2=0x7fa4400d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:26.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.025+0000 7fa43f200640 1 -- 172.21.15.31:0/2852880476 shutdown_connections 2024-09-13T08:43:26.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.025+0000 7fa43f200640 1 -- 172.21.15.31:0/2852880476 wait complete. 2024-09-13T08:43:26.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.026+0000 7fa43f200640 1 Processor -- start 2024-09-13T08:43:26.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.026+0000 7fa43f200640 1 -- start start 2024-09-13T08:43:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa424003c10 0x7fa4400b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4401f88b0 0x7fa4401f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa440166cb0 con 0x7fa440151c10 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4400b2290 con 0x7fa424003c10 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.027+0000 7fa43f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa440166e30 con 0x7fa4401f88b0 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55634/0 (socket says 172.21.15.31:55634) 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 -- 172.21.15.31:0/348353214 learned_addr learned my addr 172.21.15.31:0/348353214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43de00640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa424003c10 0x7fa4400b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43d400640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4401f88b0 0x7fa4401f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 -- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4401f88b0 msgr2=0x7fa4401f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4401f88b0 0x7fa4401f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 -- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa424003c10 msgr2=0x7fa4400b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa424003c10 0x7fa4400b6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 -- 172.21.15.31:0/348353214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4400b8c00 con 0x7fa440151c10 2024-09-13T08:43:26.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.028+0000 7fa43e800640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa4400d1840 tx=0x7fa43800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.029+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438004b20 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.029+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa438004cc0 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.029+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438003bd0 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.029+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4401f4fe0 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.029+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401f55a0 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.030+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4400b3cc0 con 0x7fa440151c10 2024-09-13T08:43:26.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.031+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fa438004030 con 0x7fa440151c10 2024-09-13T08:43:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.032+0000 7fa436000640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 0x7fa42007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.032+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fa43809b460 con 0x7fa440151c10 2024-09-13T08:43:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.032+0000 7fa43de00640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 0x7fa42007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.033+0000 7fa43de00640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 0x7fa42007b0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa430032000 tx=0x7fa430001360 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:26.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.036+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa438062e60 con 0x7fa440151c10 2024-09-13T08:43:26.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.275+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fa44015bfe0 con 0x7fa420078be0 2024-09-13T08:43:26.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.296+0000 7fa436000640 1 -- 172.21.15.31:0/348353214 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fa44015bfe0 con 0x7fa420078be0 2024-09-13T08:43:26.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:26.297 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "8/9" 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:26.298 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:26.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 msgr2=0x7fa42007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:26.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 0x7fa42007b0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa430032000 tx=0x7fa430001360 comp rx=0 tx=0).stop 2024-09-13T08:43:26.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 msgr2=0x7fa4400b8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa4400d1840 tx=0x7fa43800f3c0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 shutdown_connections 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa420078be0 0x7fa42007b0a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa440151c10 0x7fa4400b8fe0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4401f88b0 0x7fa4401f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 --2- 172.21.15.31:0/348353214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa424003c10 0x7fa4400b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 >> 172.21.15.31:0/348353214 conn(0x7fa4400d3e10 msgr2=0x7fa4400ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 shutdown_connections 2024-09-13T08:43:26.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:26.299+0000 7fa43f200640 1 -- 172.21.15.31:0/348353214 wait complete. 2024-09-13T08:43:26.316 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:27.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 -- 172.21.15.31:0/4643636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 msgr2=0x7f0380003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 --2- 172.21.15.31:0/4643636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f0380003e70 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f038c006f60 tx=0x7f0380005830 comp rx=0 tx=0).stop 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a1200640 1 -- 172.21.15.31:0/4643636 reap_dead start 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 -- 172.21.15.31:0/4643636 shutdown_connections 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 -- 172.21.15.31:0/4643636 >> 172.21.15.31:0/4643636 conn(0x7f039c0cfbb0 msgr2=0x7f039c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 -- 172.21.15.31:0/4643636 shutdown_connections 2024-09-13T08:43:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.492+0000 7f03a3200640 1 -- 172.21.15.31:0/4643636 wait complete. 2024-09-13T08:43:27.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.494+0000 7f03a3200640 1 Processor -- start 2024-09-13T08:43:27.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.494+0000 7f03a3200640 1 -- start start 2024-09-13T08:43:27.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f039c1f45f0 0x7f039c1f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f039c1f4f90 0x7f039c1f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f039c070020 con 0x7f0380003a50 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f039c0cbe90 con 0x7f039c1f4f90 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f039c0ca370 con 0x7f039c1f45f0 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f039be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f039be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55650/0 (socket says 172.21.15.31:55650) 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f039be00640 1 -- 172.21.15.31:0/1068291401 learned_addr learned my addr 172.21.15.31:0/1068291401 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f0393400640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f039c1f45f0 0x7f039c1f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.495+0000 7f03a1200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f039c1f4f90 0x7f039c1f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:27.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 -- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f039c1f45f0 msgr2=0x7f039c1f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:27.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f039c1f45f0 0x7f039c1f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 -- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f039c1f4f90 msgr2=0x7f039c1f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:27.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f039c1f4f90 0x7f039c1f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 -- 172.21.15.31:0/1068291401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c1597d0 con 0x7f0380003a50 2024-09-13T08:43:27.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f039be00640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f038c00c350 tx=0x7f038c0064a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c003a00 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f038c003c90 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c02e6a0 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039c15a520 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.496+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f039c159d20 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.498+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039c0cd780 con 0x7f0380003a50 2024-09-13T08:43:27.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.499+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f038c035020 con 0x7f0380003a50 2024-09-13T08:43:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.500+0000 7f0398c00640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 0x7f037807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.500+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f038c0bb080 con 0x7f0380003a50 2024-09-13T08:43:27.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.500+0000 7f0393400640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 0x7f037807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:27.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.501+0000 7f0393400640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 0x7f037807b0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f039c160c90 tx=0x7f0384005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:27.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.504+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f038c047030 con 0x7f0380003a50 2024-09-13T08:43:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.739+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f039c0b5b30 con 0x7f0378078be0 2024-09-13T08:43:27.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.760+0000 7f0398c00640 1 -- 172.21.15.31:0/1068291401 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f039c0b5b30 con 0x7f0378078be0 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "8/9" 2024-09-13T08:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:27.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 msgr2=0x7f037807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 0x7f037807b0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f039c160c90 tx=0x7f0384005f90 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 msgr2=0x7f039c1567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f038c00c350 tx=0x7f038c0064a0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 shutdown_connections 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0378078be0 0x7f037807b0a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f039c1f4f90 0x7f039c1f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f039c1f45f0 0x7f039c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 --2- 172.21.15.31:0/1068291401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0380003a50 0x7f039c1567b0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 >> 172.21.15.31:0/1068291401 conn(0x7f039c0cfbb0 msgr2=0x7f039c161c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 shutdown_connections 2024-09-13T08:43:27.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:27.762+0000 7f03a3200640 1 -- 172.21.15.31:0/1068291401 wait complete. 2024-09-13T08:43:27.778 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:28.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:28.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 -- 172.21.15.31:0/2662135707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa4072560 msgr2=0x7fdaa40cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 --2- 172.21.15.31:0/2662135707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa4072560 0x7fdaa40cb740 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009830 tx=0x7fda8c02a5c0 comp rx=0 tx=0).stop 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 -- 172.21.15.31:0/2662135707 shutdown_connections 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 --2- 172.21.15.31:0/2662135707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa4072560 0x7fdaa40cb740 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 -- 172.21.15.31:0/2662135707 >> 172.21.15.31:0/2662135707 conn(0x7fdaa40cfbb0 msgr2=0x7fdaa40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 -- 172.21.15.31:0/2662135707 shutdown_connections 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.956+0000 7fdaab200640 1 -- 172.21.15.31:0/2662135707 wait complete. 2024-09-13T08:43:28.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.958+0000 7fdaab200640 1 Processor -- start 2024-09-13T08:43:28.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.958+0000 7fdaab200640 1 -- start start 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4072560 0x7fdaa415edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 0x7fdaa40b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa406e740 con 0x7fdaa415bfc0 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa406e5c0 con 0x7fdaa4072560 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaab200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa40b30f0 con 0x7fdaa40cbc80 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaa9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaa9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55668/0 (socket says 172.21.15.31:55668) 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaa9200640 1 -- 172.21.15.31:0/173958286 learned_addr learned my addr 172.21.15.31:0/173958286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fdaa3e00640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4072560 0x7fdaa415edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.959+0000 7fda9b400640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 0x7fdaa40b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 -- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 msgr2=0x7fdaa40b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:28.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 0x7fdaa40b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:28.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 -- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4072560 msgr2=0x7fdaa415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:28.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4072560 0x7fdaa415edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:28.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 -- 172.21.15.31:0/173958286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa40c6c80 con 0x7fdaa415bfc0 2024-09-13T08:43:28.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fda9b400640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 0x7fdaa40b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:28.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa9200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c7a0 tx=0x7fda9c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c003b70 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda9c003d10 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c00aab0 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa40b22e0 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.960+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa40c4a70 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.962+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa40cb740 con 0x7fdaa415bfc0 2024-09-13T08:43:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.963+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fda9c00ac50 con 0x7fdaa415bfc0 2024-09-13T08:43:28.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.964+0000 7fdaa0c00640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 0x7fda8007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:28.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.964+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fda9c09aea0 con 0x7fdaa415bfc0 2024-09-13T08:43:28.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.964+0000 7fdaa3e00640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 0x7fda8007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:28.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.965+0000 7fdaa3e00640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 0x7fda8007b070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fda94007fd0 tx=0x7fda94001370 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:28.969+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fda9c062ef0 con 0x7fdaa415bfc0 2024-09-13T08:43:29.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.203+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fdaa4153300 con 0x7fda80078bb0 2024-09-13T08:43:29.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.229+0000 7fdaa0c00640 1 -- 172.21.15.31:0/173958286 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fda94001500 con 0x7fda80078bb0 2024-09-13T08:43:29.230 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "50.003%", 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "4.0M/8.0M", 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "8/9" 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:29.231 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:29.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 msgr2=0x7fda8007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:29.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 0x7fda8007b070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fda94007fd0 tx=0x7fda94001370 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 msgr2=0x7fdaa4152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c7a0 tx=0x7fda9c00cc60 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 shutdown_connections 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fda80078bb0 0x7fda8007b070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaa415bfc0 0x7fdaa4152dc0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdaa40cbc80 0x7fdaa40b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 --2- 172.21.15.31:0/173958286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaa4072560 0x7fdaa415edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.232+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 >> 172.21.15.31:0/173958286 conn(0x7fdaa40cfbb0 msgr2=0x7fdaa4157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.233+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 shutdown_connections 2024-09-13T08:43:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:29.233+0000 7fdaab200640 1 -- 172.21.15.31:0/173958286 wait complete. 2024-09-13T08:43:29.249 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:30.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:30.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 -- 172.21.15.31:0/1663472180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe564004d40 msgr2=0x7fe5640051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 --2- 172.21.15.31:0/1663472180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe564004d40 0x7fe5640051a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe578001dc0 tx=0x7fe564005830 comp rx=0 tx=0).stop 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe57f400640 1 -- 172.21.15.31:0/1663472180 reap_dead start 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 -- 172.21.15.31:0/1663472180 shutdown_connections 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 -- 172.21.15.31:0/1663472180 >> 172.21.15.31:0/1663472180 conn(0x7fe5800cfbb0 msgr2=0x7fe5800cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 -- 172.21.15.31:0/1663472180 shutdown_connections 2024-09-13T08:43:30.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.428+0000 7fe585e00640 1 -- 172.21.15.31:0/1663472180 wait complete. 2024-09-13T08:43:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.430+0000 7fe585e00640 1 Processor -- start 2024-09-13T08:43:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.430+0000 7fe585e00640 1 -- start start 2024-09-13T08:43:30.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe564004d40 0x7fe58014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:30.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5801552f0 0x7fe580149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5800cd900 con 0x7fe5801497a0 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5800cd780 con 0x7fe5801552f0 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.431+0000 7fe585e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe5800cda80 con 0x7fe564004d40 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe564004d40 0x7fe58014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55698/0 (socket says 172.21.15.31:55698) 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57e000640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5801552f0 0x7fe580149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 -- 172.21.15.31:0/2126350436 learned_addr learned my addr 172.21.15.31:0/2126350436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 -- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe564004d40 msgr2=0x7fe58014b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe564004d40 0x7fe58014b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 -- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5801552f0 msgr2=0x7fe580149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5801552f0 0x7fe580149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 -- 172.21.15.31:0/2126350436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58014b830 con 0x7fe5801497a0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.432+0000 7fe57f400640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe578001ee0 tx=0x7fe578001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.433+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5780060b0 con 0x7fe5801497a0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.433+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe578006250 con 0x7fe5801497a0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.433+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578006680 con 0x7fe5801497a0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.433+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58014bad0 con 0x7fe5801497a0 2024-09-13T08:43:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.433+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58014c090 con 0x7fe5801497a0 2024-09-13T08:43:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.434+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58015f220 con 0x7fe5801497a0 2024-09-13T08:43:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.435+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fe578033480 con 0x7fe5801497a0 2024-09-13T08:43:30.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.436+0000 7fe577400640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 0x7fe56007b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:30.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.436+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fe5780baa90 con 0x7fe5801497a0 2024-09-13T08:43:30.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.436+0000 7fe57ea00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 0x7fe56007b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:30.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.437+0000 7fe57ea00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 0x7fe56007b070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe570001eb0 tx=0x7fe570006720 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.440+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe5780825e0 con 0x7fe5801497a0 2024-09-13T08:43:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.676+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fe58014c260 con 0x7fe560078bb0 2024-09-13T08:43:30.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.698+0000 7fe577400640 1 -- 172.21.15.31:0/2126350436 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fe570006c70 con 0x7fe560078bb0 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:30.699 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "62.503%", 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "5.0M/8.0M", 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "9/9" 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:30.700 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 msgr2=0x7fe56007b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 0x7fe56007b070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe570001eb0 tx=0x7fe570006720 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 msgr2=0x7fe5801486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe578001ee0 tx=0x7fe578001b10 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 shutdown_connections 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe560078bb0 0x7fe56007b070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5801497a0 0x7fe5801486f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5801552f0 0x7fe580149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 --2- 172.21.15.31:0/2126350436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe564004d40 0x7fe58014b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 >> 172.21.15.31:0/2126350436 conn(0x7fe5800cfbb0 msgr2=0x7fe5800b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 shutdown_connections 2024-09-13T08:43:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:30.701+0000 7fe585e00640 1 -- 172.21.15.31:0/2126350436 wait complete. 2024-09-13T08:43:30.718 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:31.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 -- 172.21.15.31:0/575151622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006fa60 msgr2=0x7f33600cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 --2- 172.21.15.31:0/575151622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006fa60 0x7f33600cb740 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f3360069dd0 tx=0x7f334802a560 comp rx=0 tx=0).stop 2024-09-13T08:43:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 -- 172.21.15.31:0/575151622 shutdown_connections 2024-09-13T08:43:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 --2- 172.21.15.31:0/575151622 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fec0 0x7f33600cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 --2- 172.21.15.31:0/575151622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006fa60 0x7f33600cb740 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 --2- 172.21.15.31:0/575151622 >> v2:172.21.15.45:3300/0 conn(0x7f336015bfc0 0x7f3360072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 -- 172.21.15.31:0/575151622 >> 172.21.15.31:0/575151622 conn(0x7f33600cfbb0 msgr2=0x7f33600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 -- 172.21.15.31:0/575151622 shutdown_connections 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.893+0000 7f3366800640 1 -- 172.21.15.31:0/575151622 wait complete. 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.895+0000 7f3366800640 1 Processor -- start 2024-09-13T08:43:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.895+0000 7f3366800640 1 -- start start 2024-09-13T08:43:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fa60 0x7f336014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 0x7f33600c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33600b2b60 con 0x7f33600c58b0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33600b29e0 con 0x7f336015bfc0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3366800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f33600b2ce0 con 0x7f336006fa60 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55722/0 (socket says 172.21.15.31:55722) 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335fe00640 1 -- 172.21.15.31:0/779684113 learned_addr learned my addr 172.21.15.31:0/779684113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335f400640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fa60 0x7f336014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f3356a00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 0x7f33600c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335fe00640 1 -- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fa60 msgr2=0x7f336014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.896+0000 7f335fe00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fa60 0x7f336014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f335fe00640 1 -- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 msgr2=0x7f33600c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f335fe00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 0x7f33600c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f335fe00640 1 -- 172.21.15.31:0/779684113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33600c7ad0 con 0x7f33600c58b0 2024-09-13T08:43:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3356a00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 0x7f33600c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f335fe00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f335800e980 tx=0x7f335800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358004d20 con 0x7f33600c58b0 2024-09-13T08:43:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f335800cb20 con 0x7f33600c58b0 2024-09-13T08:43:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358005020 con 0x7f33600c58b0 2024-09-13T08:43:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33600c8a20 con 0x7f33600c58b0 2024-09-13T08:43:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.897+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33600c8fe0 con 0x7f33600c58b0 2024-09-13T08:43:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.898+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33600c6260 con 0x7f33600c58b0 2024-09-13T08:43:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.900+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f33580051c0 con 0x7f33600c58b0 2024-09-13T08:43:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.901+0000 7f3357e00640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 0x7f333c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.901+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f335809b8a0 con 0x7f33600c58b0 2024-09-13T08:43:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.901+0000 7f335f400640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 0x7f333c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:31.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.902+0000 7f335f400640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 0x7f333c07b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f335000bb50 tx=0x7f335000b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:31.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:31.905+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f33600c6260 con 0x7f33600c58b0 2024-09-13T08:43:32.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.139+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f33600b3200 con 0x7f333c078be0 2024-09-13T08:43:32.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.160+0000 7f3357e00640 1 -- 172.21.15.31:0/779684113 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f33600b3200 con 0x7f333c078be0 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "62.503%", 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "5.0M/8.0M", 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "9/9" 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:32.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 msgr2=0x7f333c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:32.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 0x7f333c07b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f335000bb50 tx=0x7f335000b2e0 comp rx=0 tx=0).stop 2024-09-13T08:43:32.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 msgr2=0x7f33600c5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f335800e980 tx=0x7f335800ee40 comp rx=0 tx=0).stop 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 shutdown_connections 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f333c078be0 0x7f333c07b0a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33600c58b0 0x7f33600c5d20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f336015bfc0 0x7f33600c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 --2- 172.21.15.31:0/779684113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f336006fa60 0x7f336014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 >> 172.21.15.31:0/779684113 conn(0x7f33600cfbb0 msgr2=0x7f33600b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.162+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 shutdown_connections 2024-09-13T08:43:32.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:32.163+0000 7f3366800640 1 -- 172.21.15.31:0/779684113 wait complete. 2024-09-13T08:43:32.179 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:33.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:33.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 -- 172.21.15.31:0/4133714339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc3c0051c0 msgr2=0x7fbc3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:33.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 --2- 172.21.15.31:0/4133714339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc3c0051c0 0x7fbc3c00b9c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbc54072510 tx=0x7fbc3c005830 comp rx=0 tx=0).stop 2024-09-13T08:43:33.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 -- 172.21.15.31:0/4133714339 shutdown_connections 2024-09-13T08:43:33.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 --2- 172.21.15.31:0/4133714339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc3c0051c0 0x7fbc3c00b9c0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 -- 172.21.15.31:0/4133714339 >> 172.21.15.31:0/4133714339 conn(0x7fbc540cfbb0 msgr2=0x7fbc540cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:33.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 -- 172.21.15.31:0/4133714339 shutdown_connections 2024-09-13T08:43:33.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.361+0000 7fbc5c400640 1 -- 172.21.15.31:0/4133714339 wait complete. 2024-09-13T08:43:33.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.363+0000 7fbc5c400640 1 Processor -- start 2024-09-13T08:43:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.363+0000 7fbc5c400640 1 -- start start 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 0x7fbc540c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 0x7fbc541f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc540726e0 con 0x7fbc541f6220 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54072560 con 0x7fbc54147820 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5c400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbc54072860 con 0x7fbc3c0051c0 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55740/0 (socket says 172.21.15.31:55740) 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5a400640 1 -- 172.21.15.31:0/3539576985 learned_addr learned my addr 172.21.15.31:0/3539576985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc59a00640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 0x7fbc540c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc59000640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 0x7fbc541f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.364+0000 7fbc5a400640 1 -- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 msgr2=0x7fbc540c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5a400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 0x7fbc540c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5a400640 1 -- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 msgr2=0x7fbc541f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5a400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 0x7fbc541f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5a400640 1 -- 172.21.15.31:0/3539576985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc54153360 con 0x7fbc541f6220 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc59000640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 0x7fbc541f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc59a00640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 0x7fbc540c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5a400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c700 tx=0x7fbc4c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c02bb60 con 0x7fbc541f6220 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc4c03a070 con 0x7fbc541f6220 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c006330 con 0x7fbc541f6220 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc54150c50 con 0x7fbc541f6220 2024-09-13T08:43:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.365+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc54153600 con 0x7fbc541f6220 2024-09-13T08:43:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.366+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc540cb9c0 con 0x7fbc541f6220 2024-09-13T08:43:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.367+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fbc4c0064d0 con 0x7fbc541f6220 2024-09-13T08:43:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.368+0000 7fbc51600640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 0x7fbc3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.368+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fbc4c0bb1f0 con 0x7fbc541f6220 2024-09-13T08:43:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.369+0000 7fbc59a00640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 0x7fbc3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:33.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.369+0000 7fbc59a00640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 0x7fbc3407b0a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009bd0 tx=0x7fbc48005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:33.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.372+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbc4c083290 con 0x7fbc541f6220 2024-09-13T08:43:33.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.612+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fbc540cb310 con 0x7fbc34078be0 2024-09-13T08:43:33.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.633+0000 7fbc51600640 1 -- 172.21.15.31:0/3539576985 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fbc540cb310 con 0x7fbc34078be0 2024-09-13T08:43:33.634 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "62.503%", 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "5.0M/8.0M", 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "9/9" 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:33.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:33.636 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 msgr2=0x7fbc3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 0x7fbc3407b0a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009bd0 tx=0x7fbc48005f90 comp rx=0 tx=0).stop 2024-09-13T08:43:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 msgr2=0x7fbc541f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c700 tx=0x7fbc4c003c20 comp rx=0 tx=0).stop 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 shutdown_connections 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc34078be0 0x7fbc3407b0a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc541f6220 0x7fbc541f66b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc54147820 0x7fbc541f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 --2- 172.21.15.31:0/3539576985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc3c0051c0 0x7fbc540c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 >> 172.21.15.31:0/3539576985 conn(0x7fbc540cfbb0 msgr2=0x7fbc54157750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 shutdown_connections 2024-09-13T08:43:33.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:33.636+0000 7fbc5c400640 1 -- 172.21.15.31:0/3539576985 wait complete. 2024-09-13T08:43:33.653 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:34.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.821+0000 7f025d400640 1 -- 172.21.15.31:0/1020874736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f025815bb70 msgr2=0x7f025806fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.821+0000 7f025d400640 1 --2- 172.21.15.31:0/1020874736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f025815bb70 0x7f025806fa60 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f02580c3430 tx=0x7f024802a980 comp rx=0 tx=0).stop 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 -- 172.21.15.31:0/1020874736 shutdown_connections 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 --2- 172.21.15.31:0/1020874736 >> v2:172.21.15.45:3300/0 conn(0x7f02580b5300 0x7f02580cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 --2- 172.21.15.31:0/1020874736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f025815bb70 0x7f025806fa60 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:34.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 -- 172.21.15.31:0/1020874736 >> 172.21.15.31:0/1020874736 conn(0x7f02580b1520 msgr2=0x7f02580b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 -- 172.21.15.31:0/1020874736 shutdown_connections 2024-09-13T08:43:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.822+0000 7f025d400640 1 -- 172.21.15.31:0/1020874736 wait complete. 2024-09-13T08:43:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.823+0000 7f025d400640 1 Processor -- start 2024-09-13T08:43:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.823+0000 7f025d400640 1 -- start start 2024-09-13T08:43:34.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:34.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 0x7f02581f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:34.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f025815bb70 0x7f025815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02580b29a0 con 0x7f02580b5300 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02580b2820 con 0x7f02580cc920 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f025d400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f02580ca730 con 0x7f025815bb70 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.824+0000 7f0257e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41850/0 (socket says 172.21.15.31:41850) 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 -- 172.21.15.31:0/2705821376 learned_addr learned my addr 172.21.15.31:0/2705821376 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 -- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f025815bb70 msgr2=0x7f025815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f025815bb70 0x7f025815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 0x7f02581f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 -- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 msgr2=0x7f02581f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 0x7f02581f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 -- 172.21.15.31:0/2705821376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0258154480 con 0x7f02580b5300 2024-09-13T08:43:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 0x7f02581f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f0257e00640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f80 tx=0x7f0248003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.825+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248034040 con 0x7f02580b5300 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.826+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0248004640 con 0x7f02580b5300 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.826+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248045480 con 0x7f02580b5300 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.826+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02581f4ef0 con 0x7f02580b5300 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.826+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02580c4710 con 0x7f02580b5300 2024-09-13T08:43:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.827+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02581f5410 con 0x7f02580b5300 2024-09-13T08:43:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.828+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f0248037020 con 0x7f02580b5300 2024-09-13T08:43:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.829+0000 7f024fe00640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 0x7f023807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.829+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f02480bc880 con 0x7f02580b5300 2024-09-13T08:43:34.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.829+0000 7f0257400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 0x7f023807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:34.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.830+0000 7f0257400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 0x7f023807b070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f025815ae90 tx=0x7f0244005d30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:34.833+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f02581f5410 con 0x7f02580b5300 2024-09-13T08:43:35.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.066+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f02580c6210 con 0x7f0238078bb0 2024-09-13T08:43:35.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.087+0000 7f024fe00640 1 -- 172.21.15.31:0/2705821376 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f02580c6210 con 0x7f0238078bb0 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "75.003%", 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "6.0M/8.0M", 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "10/9" 2024-09-13T08:43:35.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:35.089 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:35.089 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 msgr2=0x7f023807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 0x7f023807b070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f025815ae90 tx=0x7f0244005d30 comp rx=0 tx=0).stop 2024-09-13T08:43:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 msgr2=0x7f0258153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f80 tx=0x7f0248003f90 comp rx=0 tx=0).stop 2024-09-13T08:43:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 shutdown_connections 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0238078bb0 0x7f023807b070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f025815bb70 0x7f025815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02580cc920 0x7f02581f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 --2- 172.21.15.31:0/2705821376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02580b5300 0x7f0258153f40 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.089+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 >> 172.21.15.31:0/2705821376 conn(0x7f02580b1520 msgr2=0x7f02580b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.090+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 shutdown_connections 2024-09-13T08:43:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:35.090+0000 7f025d400640 1 -- 172.21.15.31:0/2705821376 wait complete. 2024-09-13T08:43:35.106 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:36.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 -- 172.21.15.31:0/3135556602 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 msgr2=0x7fb4300cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 --2- 172.21.15.31:0/3135556602 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300cdcc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb418009830 tx=0x7fb41802a480 comp rx=0 tx=0).stop 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 -- 172.21.15.31:0/3135556602 shutdown_connections 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 --2- 172.21.15.31:0/3135556602 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300cdcc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 --2- 172.21.15.31:0/3135556602 >> v2:172.21.15.45:3300/0 conn(0x7fb430161480 0x7fb4300cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 -- 172.21.15.31:0/3135556602 >> 172.21.15.31:0/3135556602 conn(0x7fb4300cfbb0 msgr2=0x7fb4300cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 -- 172.21.15.31:0/3135556602 shutdown_connections 2024-09-13T08:43:36.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.281+0000 7fb435000640 1 -- 172.21.15.31:0/3135556602 wait complete. 2024-09-13T08:43:36.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.282+0000 7fb435000640 1 Processor -- start 2024-09-13T08:43:36.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.282+0000 7fb435000640 1 -- start start 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4300cbc40 0x7fb4300c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4300cb500 con 0x7fb430161480 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4300cb380 con 0x7fb4300cbc40 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.283+0000 7fb435000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb4300cb680 con 0x7fb4300cc190 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:36.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41866/0 (socket says 172.21.15.31:41866) 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 -- 172.21.15.31:0/388466337 learned_addr learned my addr 172.21.15.31:0/388466337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb425400640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42de00640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4300cbc40 0x7fb4300c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 -- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 msgr2=0x7fb4300c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 -- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4300cbc40 msgr2=0x7fb4300c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4300cbc40 0x7fb4300c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.284+0000 7fb42e800640 1 -- 172.21.15.31:0/388466337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4300b3af0 con 0x7fb430161480 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb42e800640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb42800c770 tx=0x7fb42800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428003da0 con 0x7fb430161480 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb428005070 con 0x7fb430161480 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428005500 con 0x7fb430161480 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4300c9bc0 con 0x7fb430161480 2024-09-13T08:43:36.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.285+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4300b4070 con 0x7fb430161480 2024-09-13T08:43:36.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.286+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4301625f0 con 0x7fb430161480 2024-09-13T08:43:36.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.287+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fb4280056a0 con 0x7fb430161480 2024-09-13T08:43:36.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.288+0000 7fb426a00640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 0x7fb40c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:36.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.288+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fb42809acd0 con 0x7fb430161480 2024-09-13T08:43:36.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.289+0000 7fb42de00640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 0x7fb40c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:36.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.289+0000 7fb42de00640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 0x7fb40c07b070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb4300b5b00 tx=0x7fb420005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:36.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.293+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb428062720 con 0x7fb430161480 2024-09-13T08:43:36.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.536+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fb4300b5b30 con 0x7fb40c078bb0 2024-09-13T08:43:36.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.553+0000 7fb426a00640 1 -- 172.21.15.31:0/388466337 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7fb4300b5b30 con 0x7fb40c078bb0 2024-09-13T08:43:36.554 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "75.003%", 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "6.0M/8.0M", 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "10/9" 2024-09-13T08:43:36.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:36.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:36.556 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 msgr2=0x7fb40c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 0x7fb40c07b070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb4300b5b00 tx=0x7fb420005f90 comp rx=0 tx=0).stop 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 msgr2=0x7fb4301f7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb42800c770 tx=0x7fb42800cc30 comp rx=0 tx=0).stop 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 shutdown_connections 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb40c078bb0 0x7fb40c07b070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb430161480 0x7fb4301f7ff0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4300cc190 0x7fb4300c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 --2- 172.21.15.31:0/388466337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4300cbc40 0x7fb4300c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 >> 172.21.15.31:0/388466337 conn(0x7fb4300cfbb0 msgr2=0x7fb430075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 shutdown_connections 2024-09-13T08:43:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:36.556+0000 7fb435000640 1 -- 172.21.15.31:0/388466337 wait complete. 2024-09-13T08:43:36.573 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:36.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:43:36.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:43:37.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 -- 172.21.15.31:0/1834684653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 msgr2=0x7f50e40ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 --2- 172.21.15.31:0/1834684653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e40ccaf0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005fd0 tx=0x7f50dc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 -- 172.21.15.31:0/1834684653 shutdown_connections 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 --2- 172.21.15.31:0/1834684653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e40ccaf0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 --2- 172.21.15.31:0/1834684653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e415abf0 0x7f50e40ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 -- 172.21.15.31:0/1834684653 >> 172.21.15.31:0/1834684653 conn(0x7f50e40b1520 msgr2=0x7f50e40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 -- 172.21.15.31:0/1834684653 shutdown_connections 2024-09-13T08:43:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.741+0000 7f50ea000640 1 -- 172.21.15.31:0/1834684653 wait complete. 2024-09-13T08:43:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.742+0000 7f50ea000640 1 Processor -- start 2024-09-13T08:43:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.743+0000 7f50ea000640 1 -- start start 2024-09-13T08:43:37.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e406f530 0x7f50e40b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:37.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50e415abf0 0x7f50e41f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50e415e6e0 con 0x7f50e406f970 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50e415e560 con 0x7f50e415abf0 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50ea000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f50e40cdf60 con 0x7f50e406f530 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41876/0 (socket says 172.21.15.31:41876) 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 -- 172.21.15.31:0/1814918083 learned_addr learned my addr 172.21.15.31:0/1814918083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e8c00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e406f530 0x7f50e40b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e9600640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50e415abf0 0x7f50e41f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 -- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e406f530 msgr2=0x7f50e40b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e406f530 0x7f50e40b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:37.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 -- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50e415abf0 msgr2=0x7f50e41f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50e415abf0 0x7f50e41f8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.744+0000 7f50e3e00640 1 -- 172.21.15.31:0/1814918083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e40b52e0 con 0x7f50e406f970 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50e3e00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f50d000c450 tx=0x7f50d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0016020 con 0x7f50e406f970 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50d0012920 con 0x7f50e406f970 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0012d30 con 0x7f50e406f970 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e40b55c0 con 0x7f50e406f970 2024-09-13T08:43:37.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.745+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e40c6390 con 0x7f50e406f970 2024-09-13T08:43:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.746+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e40cd080 con 0x7f50e406f970 2024-09-13T08:43:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.748+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f50d0019020 con 0x7f50e406f970 2024-09-13T08:43:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.748+0000 7f50e0c00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 0x7f50c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.749+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f50d009ade0 con 0x7f50e406f970 2024-09-13T08:43:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.749+0000 7f50e8c00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 0x7f50c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:37.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.749+0000 7f50e8c00640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 0x7f50c407b0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f50d8005fd0 tx=0x7f50d800b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:37.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.753+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f50d0020020 con 0x7f50e406f970 2024-09-13T08:43:37.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:37.990+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f50e40cf350 con 0x7f50c4078be0 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.011+0000 7f50e0c00640 1 -- 172.21.15.31:0/1814918083 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f50e40cf350 con 0x7f50c4078be0 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:38.012 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "75.003%", 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "6.0M/8.0M", 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "10/9" 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:38.013 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 msgr2=0x7f50c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 0x7f50c407b0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f50d8005fd0 tx=0x7f50d800b2e0 comp rx=0 tx=0).stop 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 msgr2=0x7f50e41f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f50d000c450 tx=0x7f50d000c910 comp rx=0 tx=0).stop 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 shutdown_connections 2024-09-13T08:43:38.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f50c4078be0 0x7f50c407b0a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50e415abf0 0x7f50e41f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e406f970 0x7f50e41f3d50 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 --2- 172.21.15.31:0/1814918083 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50e406f530 0x7f50e40b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 >> 172.21.15.31:0/1814918083 conn(0x7f50e40b1520 msgr2=0x7f50e415d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 shutdown_connections 2024-09-13T08:43:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:38.014+0000 7f50ea000640 1 -- 172.21.15.31:0/1814918083 wait complete. 2024-09-13T08:43:38.031 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:39.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.206+0000 7f747de00640 1 -- 172.21.15.31:0/345446510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7478072560 msgr2=0x7f74780cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.206+0000 7f747de00640 1 --2- 172.21.15.31:0/345446510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7478072560 0x7f74780cb740 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7460009830 tx=0x7f746002a510 comp rx=0 tx=0).stop 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.207+0000 7f747de00640 1 -- 172.21.15.31:0/345446510 shutdown_connections 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.207+0000 7f747de00640 1 --2- 172.21.15.31:0/345446510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7478072560 0x7f74780cb740 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.207+0000 7f747de00640 1 -- 172.21.15.31:0/345446510 >> 172.21.15.31:0/345446510 conn(0x7f74780cfbb0 msgr2=0x7f74780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.207+0000 7f747de00640 1 -- 172.21.15.31:0/345446510 shutdown_connections 2024-09-13T08:43:39.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.207+0000 7f747de00640 1 -- 172.21.15.31:0/345446510 wait complete. 2024-09-13T08:43:39.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.208+0000 7f747de00640 1 Processor -- start 2024-09-13T08:43:39.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.208+0000 7f747de00640 1 -- start start 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7478072560 0x7f74780cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f747815bfc0 0x7f74781f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f747806e740 con 0x7f74780cbc80 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f747806e5c0 con 0x7f7478072560 2024-09-13T08:43:39.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.209+0000 7f747de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f74780b30f0 con 0x7f747815bfc0 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41906/0 (socket says 172.21.15.31:41906) 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 -- 172.21.15.31:0/840727848 learned_addr learned my addr 172.21.15.31:0/840727848 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f7477400640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f747815bfc0 0x7f74781f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f7476a00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7478072560 0x7f74780cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 -- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f747815bfc0 msgr2=0x7f74781f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f747815bfc0 0x7f74781f5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 -- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7478072560 msgr2=0x7f74780cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7478072560 0x7f74780cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.210+0000 7f746e000640 1 -- 172.21.15.31:0/840727848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74780cd820 con 0x7f74780cbc80 2024-09-13T08:43:39.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f746e000640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7460009950 tx=0x7f7460032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460040070 con 0x7f74780cbc80 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7460005310 con 0x7f74780cbc80 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600047a0 con 0x7f74780cbc80 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747814ef00 con 0x7f74780cbc80 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.211+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74780cdd70 con 0x7f74780cbc80 2024-09-13T08:43:39.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.212+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74780cb740 con 0x7f74780cbc80 2024-09-13T08:43:39.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.213+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f7460039040 con 0x7f74780cbc80 2024-09-13T08:43:39.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.214+0000 7f746fe00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 0x7f745407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:39.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.214+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f74600bc300 con 0x7f74780cbc80 2024-09-13T08:43:39.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.214+0000 7f7476a00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 0x7f745407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:39.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.215+0000 7f7476a00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 0x7f745407b070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7468000f60 tx=0x7f7468009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:39.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.219+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7460083bb0 con 0x7f74780cbc80 2024-09-13T08:43:39.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.453+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f74780cb310 con 0x7f7454078bb0 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.476+0000 7f746fe00640 1 -- 172.21.15.31:0/840727848 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f74780cb310 con 0x7f7454078bb0 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "in-progress", 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "subvol_snap26060" 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "progress_report": { 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "percentage cloned": "75.003%", 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "amount cloned": "6.0M/8.0M", 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "files cloned": "10/9" 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:39.478 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.478+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 msgr2=0x7f745407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.478+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 0x7f745407b070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7468000f60 tx=0x7f7468009040 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.478+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 msgr2=0x7f747814e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.478+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7460009950 tx=0x7f7460032c10 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 shutdown_connections 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7454078bb0 0x7f745407b070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f747815bfc0 0x7f74781f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74780cbc80 0x7f747814e9c0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 --2- 172.21.15.31:0/840727848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7478072560 0x7f74780cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 >> 172.21.15.31:0/840727848 conn(0x7f74780cfbb0 msgr2=0x7f74781579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 shutdown_connections 2024-09-13T08:43:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:39.479+0000 7f747de00640 1 -- 172.21.15.31:0/840727848 wait complete. 2024-09-13T08:43:39.495 DEBUG:tasks.cephfs.test_volumes:current clone state = in-progress 2024-09-13T08:43:40.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone87640 2024-09-13T08:43:40.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 -- 172.21.15.31:0/4197738968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 msgr2=0x7f847c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 --2- 172.21.15.31:0/4197738968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f847c003e20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8488009b50 tx=0x7f847c005830 comp rx=0 tx=0).stop 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f8496800640 1 -- 172.21.15.31:0/4197738968 reap_dead start 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 -- 172.21.15.31:0/4197738968 shutdown_connections 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 -- 172.21.15.31:0/4197738968 >> 172.21.15.31:0/4197738968 conn(0x7f84980cfbb0 msgr2=0x7f84980cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 -- 172.21.15.31:0/4197738968 shutdown_connections 2024-09-13T08:43:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.672+0000 7f849ca00640 1 -- 172.21.15.31:0/4197738968 wait complete. 2024-09-13T08:43:40.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.674+0000 7f849ca00640 1 Processor -- start 2024-09-13T08:43:40.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.674+0000 7f849ca00640 1 -- start start 2024-09-13T08:43:40.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:40.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84980b5340 0x7f84980c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 0x7f84980c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8498070020 con 0x7f847c003a00 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84980cb590 con 0x7f84980b5340 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.675+0000 7f849ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f84980cdcc0 con 0x7f84980c8290 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41928/0 (socket says 172.21.15.31:41928) 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 -- 172.21.15.31:0/207516330 learned_addr learned my addr 172.21.15.31:0/207516330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f848d400640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84980b5340 0x7f84980c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8496800640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 0x7f84980c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 -- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 msgr2=0x7f84980c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 0x7f84980c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 -- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84980b5340 msgr2=0x7f84980c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:40.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84980b5340 0x7f84980c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 -- 172.21.15.31:0/207516330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8498152c00 con 0x7f847c003a00 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8496800640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 0x7f84980c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.676+0000 7f8495e00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f848802b4f0 tx=0x7f8488033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.677+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488038040 con 0x7f847c003a00 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.677+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8488004020 con 0x7f847c003a00 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.677+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84980c8d60 con 0x7f847c003a00 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.677+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488004450 con 0x7f847c003a00 2024-09-13T08:43:40.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.677+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84980c9320 con 0x7f847c003a00 2024-09-13T08:43:40.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.678+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84980cd780 con 0x7f847c003a00 2024-09-13T08:43:40.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.679+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f8488045020 con 0x7f847c003a00 2024-09-13T08:43:40.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.680+0000 7f848ea00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 0x7f847407b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:40.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.680+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f84880bb6f0 con 0x7f847c003a00 2024-09-13T08:43:40.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.680+0000 7f848d400640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 0x7f847407b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:40.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.681+0000 7f848d400640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 0x7f847407b010 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f84981f81d0 tx=0x7f8480005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:40.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.684+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f84880831d0 con 0x7f847c003a00 2024-09-13T08:43:40.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:40.922+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f8498073560 con 0x7f8474078b50 2024-09-13T08:43:44.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.633+0000 7f848ea00640 1 -- 172.21.15.31:0/207516330 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f8498073560 con 0x7f8474078b50 2024-09-13T08:43:44.635 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "complete" 2024-09-13T08:43:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:44.636 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 msgr2=0x7f847407b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 0x7f847407b010 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f84981f81d0 tx=0x7f8480005f90 comp rx=0 tx=0).stop 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 msgr2=0x7f84981526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f848802b4f0 tx=0x7f8488033670 comp rx=0 tx=0).stop 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 shutdown_connections 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8474078b50 0x7f847407b010 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f84980c8290 0x7f84980c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84980b5340 0x7f84980c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 --2- 172.21.15.31:0/207516330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f847c003a00 0x7f84981526c0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 >> 172.21.15.31:0/207516330 conn(0x7f84980cfbb0 msgr2=0x7f84980af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 shutdown_connections 2024-09-13T08:43:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.637+0000 7f849ca00640 1 -- 172.21.15.31:0/207516330 wait complete. 2024-09-13T08:43:44.657 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-13T08:43:44.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:44.658 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone87640/.meta) 2024-09-13T08:43:44.719 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:43:44.719 INFO:teuthology.orchestra.run.smithi031.stdout:version = 2 2024-09-13T08:43:44.720 INFO:teuthology.orchestra.run.smithi031.stdout:type = clone 2024-09-13T08:43:44.720 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1 2024-09-13T08:43:44.720 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:43:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:43:44.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol5350 2024-09-13T08:43:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 -- 172.21.15.31:0/3806870619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f007415bfc0 msgr2=0x7f0074072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 --2- 172.21.15.31:0/3806870619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f007415bfc0 0x7f0074072560 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f0068009830 tx=0x7f006802a510 comp rx=0 tx=0).stop 2024-09-13T08:43:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 -- 172.21.15.31:0/3806870619 shutdown_connections 2024-09-13T08:43:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 --2- 172.21.15.31:0/3806870619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f007415bfc0 0x7f0074072560 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 -- 172.21.15.31:0/3806870619 >> 172.21.15.31:0/3806870619 conn(0x7f00740cfbb0 msgr2=0x7f00740cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 -- 172.21.15.31:0/3806870619 shutdown_connections 2024-09-13T08:43:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.887+0000 7f007c400640 1 -- 172.21.15.31:0/3806870619 wait complete. 2024-09-13T08:43:44.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.889+0000 7f007c400640 1 Processor -- start 2024-09-13T08:43:44.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.889+0000 7f007c400640 1 -- start start 2024-09-13T08:43:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f007406fa60 0x7f0074154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f007415bfc0 0x7f00741524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00740cbfd0 con 0x7f00741529f0 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00740cbe50 con 0x7f007406fa60 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007c400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f00740b30f0 con 0x7f007415bfc0 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46140/0 (socket says 172.21.15.31:46140) 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f007a400640 1 -- 172.21.15.31:0/1196227236 learned_addr learned my addr 172.21.15.31:0/1196227236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f0079a00640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f007406fa60 0x7f0074154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.890+0000 7f0079000640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f007415bfc0 0x7f00741524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 -- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f007415bfc0 msgr2=0x7f00741524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f007415bfc0 0x7f00741524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 -- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f007406fa60 msgr2=0x7f0074154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f007406fa60 0x7f0074154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 -- 172.21.15.31:0/1196227236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00740c7690 con 0x7f00741529f0 2024-09-13T08:43:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007a400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f006c00c780 tx=0x7f006c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c017070 con 0x7f00741529f0 2024-09-13T08:43:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f006c00ce30 con 0x7f00741529f0 2024-09-13T08:43:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0074153e40 con 0x7f00741529f0 2024-09-13T08:43:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c0052b0 con 0x7f00741529f0 2024-09-13T08:43:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.891+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00740c7c30 con 0x7f00741529f0 2024-09-13T08:43:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.892+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00740c7e30 con 0x7f00741529f0 2024-09-13T08:43:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.893+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f006c01a070 con 0x7f00741529f0 2024-09-13T08:43:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.894+0000 7f0071600640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 0x7f005407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.894+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f006c09a540 con 0x7f00741529f0 2024-09-13T08:43:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.895+0000 7f0079a00640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 0x7f005407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.895+0000 7f0079a00640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 0x7f005407b070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0068034000 tx=0x7f0068004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:44.899+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f00740c7e30 con 0x7f00741529f0 2024-09-13T08:43:45.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.126+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f00740b5280 con 0x7f0054078bb0 2024-09-13T08:43:45.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.142+0000 7f0071600640 1 -- 172.21.15.31:0/1196227236 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7f00740b5280 con 0x7f0054078bb0 2024-09-13T08:43:45.143 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9 2024-09-13T08:43:45.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 msgr2=0x7f005407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 0x7f005407b070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0068034000 tx=0x7f0068004730 comp rx=0 tx=0).stop 2024-09-13T08:43:45.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 msgr2=0x7f00741564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f006c00c780 tx=0x7f006c00cc40 comp rx=0 tx=0).stop 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 shutdown_connections 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0054078bb0 0x7f005407b070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00741529f0 0x7f00741564c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f007415bfc0 0x7f00741524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 --2- 172.21.15.31:0/1196227236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f007406fa60 0x7f0074154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 >> 172.21.15.31:0/1196227236 conn(0x7f00740cfbb0 msgr2=0x7f00741530a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 shutdown_connections 2024-09-13T08:43:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.145+0000 7f007c400640 1 -- 172.21.15.31:0/1196227236 wait complete. 2024-09-13T08:43:45.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone87640 2024-09-13T08:43:45.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 -- 172.21.15.31:0/2339315697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340b2820 msgr2=0x7fab3406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 --2- 172.21.15.31:0/2339315697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340b2820 0x7fab3406fc80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fab340d0a70 tx=0x7fab2002a4d0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 -- 172.21.15.31:0/2339315697 shutdown_connections 2024-09-13T08:43:45.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 --2- 172.21.15.31:0/2339315697 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab340cc3e0 0x7fab340cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 --2- 172.21.15.31:0/2339315697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340b2820 0x7fab3406fc80 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 --2- 172.21.15.31:0/2339315697 >> v2:172.21.15.45:3300/0 conn(0x7fab3415bc00 0x7fab34073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 -- 172.21.15.31:0/2339315697 >> 172.21.15.31:0/2339315697 conn(0x7fab340b1520 msgr2=0x7fab340b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 -- 172.21.15.31:0/2339315697 shutdown_connections 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.325+0000 7fab3a600640 1 -- 172.21.15.31:0/2339315697 wait complete. 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.327+0000 7fab3a600640 1 Processor -- start 2024-09-13T08:43:45.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.327+0000 7fab3a600640 1 -- start start 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab340b2820 0x7fab3415d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab3415bc00 0x7fab341f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab340caad0 con 0x7fab340cc3e0 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab340ca950 con 0x7fab340b2820 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab3a600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab340cac50 con 0x7fab3415bc00 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46156/0 (socket says 172.21.15.31:46156) 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 -- 172.21.15.31:0/1365638931 learned_addr learned my addr 172.21.15.31:0/1365638931 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab39200640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab340b2820 0x7fab3415d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab39c00640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab3415bc00 0x7fab341f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 -- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab3415bc00 msgr2=0x7fab341f4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab3415bc00 0x7fab341f4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 -- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab340b2820 msgr2=0x7fab3415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab340b2820 0x7fab3415d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 -- 172.21.15.31:0/1365638931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab341f5350 con 0x7fab340cc3e0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.328+0000 7fab33e00640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fab20009850 tx=0x7fab20003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.329+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2003f070 con 0x7fab340cc3e0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.329+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab200047a0 con 0x7fab340cc3e0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.329+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20035030 con 0x7fab340cc3e0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.329+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3415d7d0 con 0x7fab340cc3e0 2024-09-13T08:43:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.329+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3415dd90 con 0x7fab340cc3e0 2024-09-13T08:43:45.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.330+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab340cc830 con 0x7fab340cc3e0 2024-09-13T08:43:45.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.331+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fab20042070 con 0x7fab340cc3e0 2024-09-13T08:43:45.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.332+0000 7fab30c00640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 0x7fab1407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.332+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fab200bbe30 con 0x7fab340cc3e0 2024-09-13T08:43:45.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.332+0000 7fab39200640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 0x7fab1407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.333+0000 7fab39200640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 0x7fab1407b070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab2800d4b0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:45.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.336+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab341f5350 con 0x7fab340cc3e0 2024-09-13T08:43:45.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.558+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fab3415b040 con 0x7fab14078bb0 2024-09-13T08:43:45.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.566+0000 7fab30c00640 1 -- 172.21.15.31:0/1365638931 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+73 (secure 0 0 0) 0x7fab3415b040 con 0x7fab14078bb0 2024-09-13T08:43:45.567 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 msgr2=0x7fab1407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 0x7fab1407b070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab2800d4b0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 msgr2=0x7fab3415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fab20009850 tx=0x7fab20003910 comp rx=0 tx=0).stop 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 shutdown_connections 2024-09-13T08:43:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab14078bb0 0x7fab1407b070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab3415bc00 0x7fab341f4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab340cc3e0 0x7fab3415e870 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 --2- 172.21.15.31:0/1365638931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab340b2820 0x7fab3415d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 >> 172.21.15.31:0/1365638931 conn(0x7fab340b1520 msgr2=0x7fab3406eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 shutdown_connections 2024-09-13T08:43:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.569+0000 7fab3a600640 1 -- 172.21.15.31:0/1365638931 wait complete. 2024-09-13T08:43:45.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:45.586 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:45.625 INFO:teuthology.orchestra.run.smithi031.stdout:9 2024-09-13T08:43:45.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:45.625 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1) 2024-09-13T08:43:45.703 INFO:teuthology.orchestra.run.smithi031.stdout:9 2024-09-13T08:43:45.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone87640 2024-09-13T08:43:45.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.906+0000 7fc5d0800640 1 -- 172.21.15.31:0/2829453464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c806fa60 msgr2=0x7fc5c80cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.906+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2829453464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c806fa60 0x7fc5c80cb740 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009830 tx=0x7fc5b802a510 comp rx=0 tx=0).stop 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 -- 172.21.15.31:0/2829453464 shutdown_connections 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2829453464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c806fa60 0x7fc5c80cb740 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2829453464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c815bfc0 0x7fc5c8072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 -- 172.21.15.31:0/2829453464 >> 172.21.15.31:0/2829453464 conn(0x7fc5c80cfbb0 msgr2=0x7fc5c80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 -- 172.21.15.31:0/2829453464 shutdown_connections 2024-09-13T08:43:45.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.907+0000 7fc5d0800640 1 -- 172.21.15.31:0/2829453464 wait complete. 2024-09-13T08:43:45.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.908+0000 7fc5d0800640 1 Processor -- start 2024-09-13T08:43:45.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.908+0000 7fc5d0800640 1 -- start start 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c806fa60 0x7fc5c8142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c815bfc0 0x7fc5c80c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c80b3270 con 0x7fc5c8149720 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c80b30f0 con 0x7fc5c815bfc0 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.909+0000 7fc5d0800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c80b33f0 con 0x7fc5c806fa60 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46172/0 (socket says 172.21.15.31:46172) 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 -- 172.21.15.31:0/2650658365 learned_addr learned my addr 172.21.15.31:0/2650658365 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5cd400640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c815bfc0 0x7fc5c80c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5cde00640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c806fa60 0x7fc5c8142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 -- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c806fa60 msgr2=0x7fc5c8142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c806fa60 0x7fc5c8142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 -- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c815bfc0 msgr2=0x7fc5c80c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c815bfc0 0x7fc5c80c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:45.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.910+0000 7fc5ce800640 1 -- 172.21.15.31:0/2650658365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c80c9e30 con 0x7fc5c8149720 2024-09-13T08:43:45.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5ce800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400cbc0 tx=0x7fc5c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:45.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4016020 con 0x7fc5c8149720 2024-09-13T08:43:45.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5c400ae30 con 0x7fc5c8149720 2024-09-13T08:43:45.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4012660 con 0x7fc5c8149720 2024-09-13T08:43:45.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c81437d0 con 0x7fc5c8149720 2024-09-13T08:43:45.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.911+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c8143d90 con 0x7fc5c8149720 2024-09-13T08:43:45.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.912+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c80ca000 con 0x7fc5c8149720 2024-09-13T08:43:45.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.913+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fc5c4019020 con 0x7fc5c8149720 2024-09-13T08:43:45.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.914+0000 7fc5c2000640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 0x7fc5ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:45.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.914+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7fc5c409b4a0 con 0x7fc5c8149720 2024-09-13T08:43:45.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.915+0000 7fc5cde00640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 0x7fc5ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:45.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.915+0000 7fc5cde00640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 0x7fc5ac07b0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0095f0 tx=0x7fc5bc005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:45.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:45.918+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc5c80ca000 con 0x7fc5c8149720 2024-09-13T08:43:46.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.142+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7fc5c8143580 con 0x7fc5ac078be0 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.152+0000 7fc5c2000640 1 -- 172.21.15.31:0/2650658365 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+741 (secure 0 0 0) 0x7fc5c8143580 con 0x7fc5ac078be0 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "atime": "2024-09-13 08:42:59", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_pcent": "80.00", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_quota": 10485760, 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "bytes_used": 8388608, 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "created_at": "2024-09-13 08:43:09", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "ctime": "2024-09-13 08:43:39", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "cephfs_data", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "earmark": "", 2024-09-13T08:43:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "features": [ 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-clone", 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-autoprotect", 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot-retention" 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flavor": 2, 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 1000, 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": 16895, 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_addrs": [ 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6789", 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6789", 2024-09-13T08:43:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.45:6790" 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "mtime": "2024-09-13 08:43:04", 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1", 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_namespace": "", 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "complete", 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "clone", 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "uid": 1000 2024-09-13T08:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 msgr2=0x7fc5ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 0x7fc5ac07b0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0095f0 tx=0x7fc5bc005f90 comp rx=0 tx=0).stop 2024-09-13T08:43:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 msgr2=0x7fc5c80c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400cbc0 tx=0x7fc5c400a430 comp rx=0 tx=0).stop 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 shutdown_connections 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc5ac078be0 0x7fc5ac07b0a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5c8149720 0x7fc5c80c6f00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c815bfc0 0x7fc5c80c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 --2- 172.21.15.31:0/2650658365 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5c806fa60 0x7fc5c8142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 >> 172.21.15.31:0/2650658365 conn(0x7fc5c80cfbb0 msgr2=0x7fc5c8073ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 shutdown_connections 2024-09-13T08:43:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:46.154+0000 7fc5d0800640 1 -- 172.21.15.31:0/2650658365 wait complete. 2024-09-13T08:43:46.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.172 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.208 INFO:teuthology.orchestra.run.smithi031.stderr:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060: ceph.quota.max_bytes: No such attribute 2024-09-13T08:43:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:43:46.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.209 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.286 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:43:46.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.287 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.363 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.440 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.2 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.5 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.4 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.7 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.6 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.0 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.3 2024-09-13T08:43:46.441 INFO:teuthology.orchestra.run.smithi031.stdout:volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.1 2024-09-13T08:43:46.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.442 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.517 INFO:teuthology.orchestra.run.smithi031.stdout:41ff 2024-09-13T08:43:46.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.518 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/) 2024-09-13T08:43:46.593 INFO:teuthology.orchestra.run.smithi031.stdout:41ff 2024-09-13T08:43:46.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.594 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.669 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:46.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.671 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/) 2024-09-13T08:43:46.745 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:46.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.747 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.822 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:46.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.823 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/) 2024-09-13T08:43:46.897 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:46.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.899 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060) 2024-09-13T08:43:46.974 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:46.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:46.975 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/) 2024-09-13T08:43:47.050 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:47.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.051 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.2) 2024-09-13T08:43:47.126 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:47.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.127 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.2) 2024-09-13T08:43:47.204 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:47.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.206 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.2) 2024-09-13T08:43:47.282 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:47.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.283 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.2) 2024-09-13T08:43:47.359 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:47.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.360 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.2) 2024-09-13T08:43:47.434 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:47.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.435 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.2) 2024-09-13T08:43:47.510 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:47.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.512 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.2) 2024-09-13T08:43:47.586 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:47.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.587 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.2) 2024-09-13T08:43:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:47.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.663 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.5) 2024-09-13T08:43:47.738 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:47.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.739 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.5) 2024-09-13T08:43:47.816 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:47.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.817 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.5) 2024-09-13T08:43:47.893 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:47.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.894 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.5) 2024-09-13T08:43:47.969 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:47.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:47.971 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.5) 2024-09-13T08:43:48.045 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:48.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.046 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.5) 2024-09-13T08:43:48.121 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:48.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.123 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.5) 2024-09-13T08:43:48.198 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:48.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.199 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.5) 2024-09-13T08:43:48.273 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:48.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.274 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.4) 2024-09-13T08:43:48.349 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:48.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.351 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.4) 2024-09-13T08:43:48.426 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:48.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.428 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.4) 2024-09-13T08:43:48.502 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:48.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.503 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.4) 2024-09-13T08:43:48.578 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:48.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.579 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.4) 2024-09-13T08:43:48.655 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:48.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.656 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.4) 2024-09-13T08:43:48.730 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:48.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.731 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.4) 2024-09-13T08:43:48.806 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:48.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.808 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.4) 2024-09-13T08:43:48.882 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:48.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.884 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.7) 2024-09-13T08:43:48.959 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:48.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:48.960 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.7) 2024-09-13T08:43:49.036 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:49.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.037 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.7) 2024-09-13T08:43:49.111 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:49.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.112 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.7) 2024-09-13T08:43:49.188 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:49.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.189 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.7) 2024-09-13T08:43:49.265 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:49.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.266 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.7) 2024-09-13T08:43:49.340 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:49.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.342 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.7) 2024-09-13T08:43:49.416 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:49.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.417 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.7) 2024-09-13T08:43:49.492 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:49.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.493 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.6) 2024-09-13T08:43:49.569 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:49.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.570 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.6) 2024-09-13T08:43:49.646 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:49.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.647 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.6) 2024-09-13T08:43:49.722 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:49.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.723 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.6) 2024-09-13T08:43:49.799 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:49.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.800 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.6) 2024-09-13T08:43:49.875 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:49.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.876 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.6) 2024-09-13T08:43:49.952 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:49.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:49.953 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.6) 2024-09-13T08:43:50.028 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:50.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.030 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.6) 2024-09-13T08:43:50.104 INFO:teuthology.orchestra.run.smithi031.stdout:1726216984 2024-09-13T08:43:50.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.105 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.0) 2024-09-13T08:43:50.181 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:50.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.183 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.0) 2024-09-13T08:43:50.259 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:50.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.260 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.0) 2024-09-13T08:43:50.334 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:50.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.336 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.0) 2024-09-13T08:43:50.410 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:50.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.412 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.0) 2024-09-13T08:43:50.487 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:50.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.488 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.0) 2024-09-13T08:43:50.564 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:50.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.565 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.0) 2024-09-13T08:43:50.639 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:50.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.641 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.0) 2024-09-13T08:43:50.716 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:50.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.717 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.3) 2024-09-13T08:43:50.791 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:50.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.792 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.3) 2024-09-13T08:43:50.868 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:50.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.870 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.3) 2024-09-13T08:43:50.945 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:50.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:50.946 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.3) 2024-09-13T08:43:51.022 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:51.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.023 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.3) 2024-09-13T08:43:51.098 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:51.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.099 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.3) 2024-09-13T08:43:51.175 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:51.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.176 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.3) 2024-09-13T08:43:51.251 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:51.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.252 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.3) 2024-09-13T08:43:51.328 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:51.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.329 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.1) 2024-09-13T08:43:51.406 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:51.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.408 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.1) 2024-09-13T08:43:51.483 INFO:teuthology.orchestra.run.smithi031.stdout:81a4 2024-09-13T08:43:51.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.484 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.1) 2024-09-13T08:43:51.559 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:51.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.560 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.1) 2024-09-13T08:43:51.634 INFO:teuthology.orchestra.run.smithi031.stdout:1000 2024-09-13T08:43:51.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.636 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.1) 2024-09-13T08:43:51.712 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:51.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.713 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.1) 2024-09-13T08:43:51.787 INFO:teuthology.orchestra.run.smithi031.stdout:1333 2024-09-13T08:43:51.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.789 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9/.snap/subvol_snap26060/subvolume_file.1) 2024-09-13T08:43:51.864 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:51.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:51.865 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone87640/cee67cba-f19e-451e-b2e3-0544cb9961e1/subvolume_file.1) 2024-09-13T08:43:51.941 INFO:teuthology.orchestra.run.smithi031.stdout:1726216983 2024-09-13T08:43:51.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol5350 fake subvol_clone85481 2024-09-13T08:43:52.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 -- 172.21.15.31:0/4167183439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b0072560 msgr2=0x7f75b00cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 --2- 172.21.15.31:0/4167183439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b0072560 0x7f75b00cb740 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7598009830 tx=0x7f759802a5b0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 -- 172.21.15.31:0/4167183439 shutdown_connections 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 --2- 172.21.15.31:0/4167183439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b0072560 0x7f75b00cb740 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 -- 172.21.15.31:0/4167183439 >> 172.21.15.31:0/4167183439 conn(0x7f75b00cfbb0 msgr2=0x7f75b00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 -- 172.21.15.31:0/4167183439 shutdown_connections 2024-09-13T08:43:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.147+0000 7f75b5400640 1 -- 172.21.15.31:0/4167183439 wait complete. 2024-09-13T08:43:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.149+0000 7f75b5400640 1 Processor -- start 2024-09-13T08:43:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.149+0000 7f75b5400640 1 -- start start 2024-09-13T08:43:52.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75b0072560 0x7f75b00c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b015bfc0 0x7f75b0151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75b006e740 con 0x7f75b00cbc80 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75b006e5c0 con 0x7f75b015bfc0 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75b5400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f75b00b30f0 con 0x7f75b0072560 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75a5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75a5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46198/0 (socket says 172.21.15.31:46198) 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75a5e00640 1 -- 172.21.15.31:0/1583386423 learned_addr learned my addr 172.21.15.31:0/1583386423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75af200640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b015bfc0 0x7f75b0151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.150+0000 7f75ae800640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75b0072560 0x7f75b00c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 -- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75b0072560 msgr2=0x7f75b00c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75b0072560 0x7f75b00c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 -- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b015bfc0 msgr2=0x7f75b0151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b015bfc0 0x7f75b0151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 -- 172.21.15.31:0/1583386423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b00c3c20 con 0x7f75b00cbc80 2024-09-13T08:43:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a5e00640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c00 tx=0x7f7598032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598040070 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7598003af0 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b00b2500 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598045460 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.151+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b00c56e0 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.152+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b00cb740 con 0x7f75b00cbc80 2024-09-13T08:43:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.153+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f7598039040 con 0x7f75b00cbc80 2024-09-13T08:43:52.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.154+0000 7f75a7400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 0x7f758c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.155+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5883+0+0 (secure 0 0 0) 0x7f75980bc610 con 0x7f75b00cbc80 2024-09-13T08:43:52.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.155+0000 7f75ae800640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 0x7f758c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.155+0000 7f75ae800640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 0x7f758c07b070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75a00015e0 tx=0x7f75a0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.159+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7598084700 con 0x7f75b00cbc80 2024-09-13T08:43:52.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.398+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "fake", "target_sub_name": "subvol_clone85481", "target": ["mon-mgr", ""]}) -- 0x7f75b00cb310 con 0x7f758c078bb0 2024-09-13T08:43:52.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.483+0000 7f75a7400640 1 -- 172.21.15.31:0/1583386423 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f75b00cb310 con 0x7f758c078bb0 2024-09-13T08:43:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 msgr2=0x7f758c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 0x7f758c07b070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75a00015e0 tx=0x7f75a0009040 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 msgr2=0x7f75b00b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c00 tx=0x7f7598032d50 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 shutdown_connections 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f758c078bb0 0x7f758c07b070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b015bfc0 0x7f75b0151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b00cbc80 0x7f75b00b1fc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 --2- 172.21.15.31:0/1583386423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f75b0072560 0x7f75b00c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 >> 172.21.15.31:0/1583386423 conn(0x7f75b00cfbb0 msgr2=0x7f75b01532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 shutdown_connections 2024-09-13T08:43:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.486+0000 7f75b5400640 1 -- 172.21.15.31:0/1583386423 wait complete. 2024-09-13T08:43:52.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone85481 2024-09-13T08:43:52.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 -- 172.21.15.31:0/816089113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 msgr2=0x7fbc4c0d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 --2- 172.21.15.31:0/816089113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c0d21d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0b6a80 tx=0x7fbc3402a440 comp rx=0 tx=0).stop 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 -- 172.21.15.31:0/816089113 shutdown_connections 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 --2- 172.21.15.31:0/816089113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c0d21d0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 --2- 172.21.15.31:0/816089113 >> v2:172.21.15.45:3300/0 conn(0x7fbc4c161470 0x7fbc4c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 -- 172.21.15.31:0/816089113 >> 172.21.15.31:0/816089113 conn(0x7fbc4c0d3e90 msgr2=0x7fbc4c0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 -- 172.21.15.31:0/816089113 shutdown_connections 2024-09-13T08:43:52.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.668+0000 7fbc50a00640 1 -- 172.21.15.31:0/816089113 wait complete. 2024-09-13T08:43:52.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.670+0000 7fbc50a00640 1 Processor -- start 2024-09-13T08:43:52.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.670+0000 7fbc50a00640 1 -- start start 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc4c0d2710 0x7fbc4c0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4c161470 0x7fbc4c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c0b2610 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c0b2490 con 0x7fbc4c161470 2024-09-13T08:43:52.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc50a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c0b2790 con 0x7fbc4c0d2710 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46214/0 (socket says 172.21.15.31:46214) 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 -- 172.21.15.31:0/3462110127 learned_addr learned my addr 172.21.15.31:0/3462110127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4b200640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4c161470 0x7fbc4c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc43e00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc4c0d2710 0x7fbc4c0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 -- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc4c0d2710 msgr2=0x7fbc4c0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc4c0d2710 0x7fbc4c0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 -- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4c161470 msgr2=0x7fbc4c15d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4c161470 0x7fbc4c15d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:52.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.671+0000 7fbc4a800640 1 -- 172.21.15.31:0/3462110127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0cb5a0 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc4a800640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00d6d0 tx=0x7fbc3c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:52.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c017070 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc3c005330 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c009820 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c0cd5b0 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.672+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c0cd850 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.673+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c0c77d0 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.674+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fbc3c009bd0 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.675+0000 7fbc42a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 0x7fbc2807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:52.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.675+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7fbc3c09b320 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.676+0000 7fbc43e00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 0x7fbc2807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:52.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.676+0000 7fbc43e00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 0x7fbc2807b070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc34033000 tx=0x7fbc34004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.680+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbc3c062cc0 con 0x7fbc4c0d1d90 2024-09-13T08:43:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:52.916+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone85481", "target": ["mon-mgr", ""]}) -- 0x7fbc4c0d21d0 con 0x7fbc28078bb0 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.648+0000 7fbc42a00640 1 -- 172.21.15.31:0/3462110127 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+152 (secure 0 0 0) 0x7fbc4c0d21d0 con 0x7fbc28078bb0 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "pending", 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "cephfs", 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "subvol5350", 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "fake" 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:54.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:54.651 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 msgr2=0x7fbc2807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 0x7fbc2807b070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc34033000 tx=0x7fbc34004730 comp rx=0 tx=0).stop 2024-09-13T08:43:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 msgr2=0x7fbc4c1f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00d6d0 tx=0x7fbc3c00db90 comp rx=0 tx=0).stop 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 shutdown_connections 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc28078bb0 0x7fbc2807b070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4c161470 0x7fbc4c15d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc4c0d2710 0x7fbc4c0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 --2- 172.21.15.31:0/3462110127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc4c0d1d90 0x7fbc4c1f54c0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 >> 172.21.15.31:0/3462110127 conn(0x7fbc4c0d3e90 msgr2=0x7fbc4c0b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 shutdown_connections 2024-09-13T08:43:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:54.652+0000 7fbc50a00640 1 -- 172.21.15.31:0/3462110127 wait complete. 2024-09-13T08:43:54.675 DEBUG:tasks.cephfs.test_volumes:current clone state = pending 2024-09-13T08:43:55.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone85481 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 -- 172.21.15.31:0/3982628016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65940039e0 msgr2=0x7f6594003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 --2- 172.21.15.31:0/3982628016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65940039e0 0x7f6594003dc0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009b50 tx=0x7f6594005830 comp rx=0 tx=0).stop 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65af400640 1 -- 172.21.15.31:0/3982628016 reap_dead start 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 -- 172.21.15.31:0/3982628016 shutdown_connections 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 -- 172.21.15.31:0/3982628016 >> 172.21.15.31:0/3982628016 conn(0x7f65b00cfbb0 msgr2=0x7f65b00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 -- 172.21.15.31:0/3982628016 shutdown_connections 2024-09-13T08:43:55.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.852+0000 7f65b5600640 1 -- 172.21.15.31:0/3982628016 wait complete. 2024-09-13T08:43:55.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.854+0000 7f65b5600640 1 Processor -- start 2024-09-13T08:43:55.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.854+0000 7f65b5600640 1 -- start start 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.854+0000 7f65b5600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65940039e0 0x7f65b00c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65b5600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65b5600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b0072460 0x7f65b01595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65b5600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65b00cbdc0 con 0x7f65b014e3d0 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65b5600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65b00cbc40 con 0x7f65b0072460 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65b5600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65b00cbf40 con 0x7f65940039e0 2024-09-13T08:43:55.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33802/0 (socket says 172.21.15.31:33802) 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 -- 172.21.15.31:0/3557505793 learned_addr learned my addr 172.21.15.31:0/3557505793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65aea00640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65940039e0 0x7f65b00c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65af400640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b0072460 0x7f65b01595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 -- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65940039e0 msgr2=0x7f65b00c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65940039e0 0x7f65b00c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 -- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b0072460 msgr2=0x7f65b01595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b0072460 0x7f65b01595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 -- 172.21.15.31:0/3557505793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b01f87e0 con 0x7f65b014e3d0 2024-09-13T08:43:55.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.855+0000 7f65a6000640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6598001860 tx=0x7f65980039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.856+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598015070 con 0x7f65b014e3d0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.856+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6598011840 con 0x7f65b014e3d0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.856+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b0158690 con 0x7f65b014e3d0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.856+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598011c90 con 0x7f65b014e3d0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.856+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b01f7d00 con 0x7f65b014e3d0 2024-09-13T08:43:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.857+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b00cda00 con 0x7f65b014e3d0 2024-09-13T08:43:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.858+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f6598018070 con 0x7f65b014e3d0 2024-09-13T08:43:55.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.859+0000 7f65a7400640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 0x7f658c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:55.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.859+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f659809bdd0 con 0x7f65b014e3d0 2024-09-13T08:43:55.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.859+0000 7f65aea00640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 0x7f658c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:55.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.860+0000 7f65aea00640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 0x7f658c07b0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f65a0035000 tx=0x7f65a002d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:55.864+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6598063cb0 con 0x7f65b014e3d0 2024-09-13T08:43:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:56.097+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone85481", "target": ["mon-mgr", ""]}) -- 0x7f65b00b5b30 con 0x7f658c078be0 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.633+0000 7f65a7400640 1 -- 172.21.15.31:0/3557505793 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f65b00b5b30 con 0x7f658c078be0 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "complete" 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:43:59.636 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.636+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 msgr2=0x7f658c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.636+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 0x7f658c07b0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f65a0035000 tx=0x7f65a002d040 comp rx=0 tx=0).stop 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.636+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 msgr2=0x7f65b01590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.636+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6598001860 tx=0x7f65980039c0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 shutdown_connections 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f658c078be0 0x7f658c07b0a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b0072460 0x7f65b01595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65b014e3d0 0x7f65b01590b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 --2- 172.21.15.31:0/3557505793 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65940039e0 0x7f65b00c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 >> 172.21.15.31:0/3557505793 conn(0x7f65b00cfbb0 msgr2=0x7f65b0073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:59.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 shutdown_connections 2024-09-13T08:43:59.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.637+0000 7f65b5600640 1 -- 172.21.15.31:0/3557505793 wait complete. 2024-09-13T08:43:59.657 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-13T08:43:59.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:43:59.658 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone85481/.meta) 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:version = 2 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:type = clone 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol_clone85481/92fc6db1-7b84-404c-a4f3-95387627b7a8 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:43:59.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:43:59.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info cephfs subvol5350 subvol_snap26060 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 -- 172.21.15.31:0/674294509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e8004000 msgr2=0x7fe9e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 --2- 172.21.15.31:0/674294509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e8004000 0x7fe9e8004800 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0009b50 tx=0x7fe9e8005830 comp rx=0 tx=0).stop 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea09a00640 1 -- 172.21.15.31:0/674294509 reap_dead start 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 -- 172.21.15.31:0/674294509 shutdown_connections 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 -- 172.21.15.31:0/674294509 >> 172.21.15.31:0/674294509 conn(0x7fea040b1520 msgr2=0x7fea040b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 -- 172.21.15.31:0/674294509 shutdown_connections 2024-09-13T08:43:59.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.887+0000 7fea0a400640 1 -- 172.21.15.31:0/674294509 wait complete. 2024-09-13T08:43:59.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.889+0000 7fea0a400640 1 Processor -- start 2024-09-13T08:43:59.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.889+0000 7fea0a400640 1 -- start start 2024-09-13T08:43:59.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9e8004000 0x7fea040c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 0x7fea0415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fea040ccd30 con 0x7fea040c7c60 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea040ccbb0 con 0x7fea041f3080 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea0a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fea0406fc80 con 0x7fe9e8004000 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea03e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea03e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33818/0 (socket says 172.21.15.31:33818) 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.890+0000 7fea03e00640 1 -- 172.21.15.31:0/2515204812 learned_addr learned my addr 172.21.15.31:0/2515204812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea09000640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9e8004000 0x7fea040c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea09a00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 0x7fea0415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:59.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 -- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9e8004000 msgr2=0x7fea040c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9e8004000 0x7fea040c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 -- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 msgr2=0x7fea0415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 0x7fea0415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 -- 172.21.15.31:0/2515204812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0415ef30 con 0x7fea040c7c60 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea09a00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 0x7fea0415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea03e00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0036040 tx=0x7fe9f0033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0038040 con 0x7fea040c7c60 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9f0004030 con 0x7fea040c7c60 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0004480 con 0x7fea040c7c60 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.891+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea040c6c40 con 0x7fea040c7c60 2024-09-13T08:43:59.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.892+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea040c8280 con 0x7fea040c7c60 2024-09-13T08:43:59.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.893+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea040701b0 con 0x7fea040c7c60 2024-09-13T08:43:59.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.894+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fe9f0033860 con 0x7fea040c7c60 2024-09-13T08:43:59.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.895+0000 7fea00c00640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 0x7fe9e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:43:59.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.895+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7fe9f00bb4a0 con 0x7fea040c7c60 2024-09-13T08:43:59.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.895+0000 7fea09000640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 0x7fe9e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:43:59.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.895+0000 7fea09000640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 0x7fe9e407b0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009cd0 tx=0x7fe9f4005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:43:59.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:43:59.899+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe9f0083400 con 0x7fea040c7c60 2024-09-13T08:44:00.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.130+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "subvol_snap26060", "target": ["mon-mgr", ""]}) -- 0x7fea040c6a80 con 0x7fe9e4078be0 2024-09-13T08:44:00.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.152+0000 7fea00c00640 1 -- 172.21.15.31:0/2515204812 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+115 (secure 0 0 0) 0x7fe9f4009d00 con 0x7fe9e4078be0 2024-09-13T08:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout: "created_at": "2024-09-13 08:43:04.815882", 2024-09-13T08:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "cephfs_data", 2024-09-13T08:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout: "has_pending_clones": "no" 2024-09-13T08:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:44:00.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 msgr2=0x7fe9e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 0x7fe9e407b0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009cd0 tx=0x7fe9f4005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:00.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 msgr2=0x7fea0415c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0036040 tx=0x7fe9f0033670 comp rx=0 tx=0).stop 2024-09-13T08:44:00.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 shutdown_connections 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe9e4078be0 0x7fe9e407b0a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea041f3080 0x7fea0415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea040c7c60 0x7fea0415c1c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 --2- 172.21.15.31:0/2515204812 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9e8004000 0x7fea040c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 >> 172.21.15.31:0/2515204812 conn(0x7fea040b1520 msgr2=0x7fea040b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 shutdown_connections 2024-09-13T08:44:00.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.154+0000 7fea0a400640 1 -- 172.21.15.31:0/2515204812 wait complete. 2024-09-13T08:44:00.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs subvol5350 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.335+0000 7f117da00640 1 -- 172.21.15.31:0/3241466047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cc190 msgr2=0x7f11780caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.335+0000 7f117da00640 1 --2- 172.21.15.31:0/3241466047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cc190 0x7f11780caa80 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f1170005fd0 tx=0x7f117002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.336+0000 7f117da00640 1 -- 172.21.15.31:0/3241466047 shutdown_connections 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.336+0000 7f117da00640 1 --2- 172.21.15.31:0/3241466047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cc190 0x7f11780caa80 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.336+0000 7f117da00640 1 -- 172.21.15.31:0/3241466047 >> 172.21.15.31:0/3241466047 conn(0x7f11780cfbb0 msgr2=0x7f11780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.336+0000 7f117da00640 1 -- 172.21.15.31:0/3241466047 shutdown_connections 2024-09-13T08:44:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.336+0000 7f117da00640 1 -- 172.21.15.31:0/3241466047 wait complete. 2024-09-13T08:44:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.337+0000 7f117da00640 1 Processor -- start 2024-09-13T08:44:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.337+0000 7f117da00640 1 -- start start 2024-09-13T08:44:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11780cc190 0x7f11781f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1178161480 0x7f11780b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f11780cbfd0 con 0x7f11780cde90 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11780cbe50 con 0x7f11780cc190 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.338+0000 7f117da00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f11780cafc0 con 0x7f1178161480 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33832/0 (socket says 172.21.15.31:33832) 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 -- 172.21.15.31:0/1108181393 learned_addr learned my addr 172.21.15.31:0/1108181393 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f1176a00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11780cc190 0x7f11781f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f1177400640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1178161480 0x7f11780b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 -- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1178161480 msgr2=0x7f11780b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1178161480 0x7f11780b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 -- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11780cc190 msgr2=0x7f11781f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11780cc190 0x7f11781f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.339+0000 7f116e000640 1 -- 172.21.15.31:0/1108181393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11780b3af0 con 0x7f11780cde90 2024-09-13T08:44:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f116e000640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f116000c460 tx=0x7f116000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160016020 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1160012920 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11780b3d90 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160012d50 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.340+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11780b4350 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.341+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11780cb470 con 0x7f11780cde90 2024-09-13T08:44:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.342+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f1160019020 con 0x7f11780cde90 2024-09-13T08:44:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.343+0000 7f116f400640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 0x7f115407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.343+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f116009aa20 con 0x7f11780cde90 2024-09-13T08:44:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.344+0000 7f1176a00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 0x7f115407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.344+0000 7f1176a00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 0x7f115407b070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f11680015e0 tx=0x7f1168009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.347+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1160020070 con 0x7f11780cde90 2024-09-13T08:44:00.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.574+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f11780b5b30 con 0x7f1154078bb0 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.590+0000 7f116f400640 1 -- 172.21.15.31:0/1108181393 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+87 (secure 0 0 0) 0x7f11780b5b30 con 0x7f1154078bb0 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fake" 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subvol_snap26060" 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:00.592 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:44:00.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 msgr2=0x7f115407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 0x7f115407b070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f11680015e0 tx=0x7f1168009040 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 msgr2=0x7f11780b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f116000c460 tx=0x7f116000c920 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 shutdown_connections 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1154078bb0 0x7f115407b070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1178161480 0x7f11780b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11780cde90 0x7f11780b4e10 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 --2- 172.21.15.31:0/1108181393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11780cc190 0x7f11781f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 >> 172.21.15.31:0/1108181393 conn(0x7f11780cfbb0 msgr2=0x7f1178160430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 shutdown_connections 2024-09-13T08:44:00.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.592+0000 7f117da00640 1 -- 172.21.15.31:0/1108181393 wait complete. 2024-09-13T08:44:00.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol5350 subvol_snap26060 2024-09-13T08:44:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 -- 172.21.15.31:0/3102441182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c06fec0 msgr2=0x7fa78c0cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 --2- 172.21.15.31:0/3102441182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c06fec0 0x7fa78c0cbc80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0694f0 tx=0x7fa78402aa30 comp rx=0 tx=0).stop 2024-09-13T08:44:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 -- 172.21.15.31:0/3102441182 shutdown_connections 2024-09-13T08:44:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 --2- 172.21.15.31:0/3102441182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c06fec0 0x7fa78c0cbc80 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 --2- 172.21.15.31:0/3102441182 >> v2:172.21.15.45:3300/0 conn(0x7fa78c06fa60 0x7fa78c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 --2- 172.21.15.31:0/3102441182 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c15bfc0 0x7fa78c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 -- 172.21.15.31:0/3102441182 >> 172.21.15.31:0/3102441182 conn(0x7fa78c0cfbb0 msgr2=0x7fa78c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 -- 172.21.15.31:0/3102441182 shutdown_connections 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.775+0000 7fa792000640 1 -- 172.21.15.31:0/3102441182 wait complete. 2024-09-13T08:44:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.777+0000 7fa792000640 1 Processor -- start 2024-09-13T08:44:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.777+0000 7fa792000640 1 -- start start 2024-09-13T08:44:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 0x7fa78c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 0x7fa78c1f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0b2b60 con 0x7fa78c15bfc0 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0b29e0 con 0x7fa78c06fa60 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa792000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa78c0b2ce0 con 0x7fa78c14ed60 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33844/0 (socket says 172.21.15.31:33844) 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 -- 172.21.15.31:0/3445833314 learned_addr learned my addr 172.21.15.31:0/3445833314 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa78b400640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 0x7fa78c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa78be00640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 0x7fa78c1f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 -- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 msgr2=0x7fa78c1f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 0x7fa78c1f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 -- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 msgr2=0x7fa78c0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 0x7fa78c0c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:00.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa783e00640 1 -- 172.21.15.31:0/3445833314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c14f260 con 0x7fa78c15bfc0 2024-09-13T08:44:00.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa78b400640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 0x7fa78c0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:00.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.778+0000 7fa78be00640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 0x7fa78c1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:00.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa783e00640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa77400e9f0 tx=0x7fa77400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77400cc60 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa77400ce00 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774004280 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78c14f540 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.779+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78c14fb00 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.780+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c1f2c90 con 0x7fa78c15bfc0 2024-09-13T08:44:00.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.781+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7fa7740035f0 con 0x7fa78c15bfc0 2024-09-13T08:44:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.782+0000 7fa783400640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 0x7fa76807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.782+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7fa77409bba0 con 0x7fa78c15bfc0 2024-09-13T08:44:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.783+0000 7fa78b400640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 0x7fa76807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.783+0000 7fa78b400640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 0x7fa76807b070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa78c15b880 tx=0x7fa77c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:00.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:00.787+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa78c1f2c90 con 0x7fa78c15bfc0 2024-09-13T08:44:01.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.020+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "subvol_snap26060", "target": ["mon-mgr", ""]}) -- 0x7fa78c1f71e0 con 0x7fa768078bb0 2024-09-13T08:44:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.055+0000 7fa783400640 1 -- 172.21.15.31:0/3445833314 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa77c011be0 con 0x7fa768078bb0 2024-09-13T08:44:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 msgr2=0x7fa76807b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 0x7fa76807b070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa78c15b880 tx=0x7fa77c005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 msgr2=0x7fa78c1e4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa77400e9f0 tx=0x7fa77400eeb0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 shutdown_connections 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa768078bb0 0x7fa76807b070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa78c14ed60 0x7fa78c1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa78c15bfc0 0x7fa78c1e4440 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.057+0000 7fa792000640 1 --2- 172.21.15.31:0/3445833314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa78c06fa60 0x7fa78c0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.058+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 >> 172.21.15.31:0/3445833314 conn(0x7fa78c0cfbb0 msgr2=0x7fa78c0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.058+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 shutdown_connections 2024-09-13T08:44:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.058+0000 7fa792000640 1 -- 172.21.15.31:0/3445833314 wait complete. 2024-09-13T08:44:01.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol5350 fake 2024-09-13T08:44:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 -- 172.21.15.31:0/3895779109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 msgr2=0x7f4cc80ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3895779109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80ca8b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80cfb50 tx=0x7f4cc002aa30 comp rx=0 tx=0).stop 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 -- 172.21.15.31:0/3895779109 shutdown_connections 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3895779109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80ca8b0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3895779109 >> v2:172.21.15.45:3300/0 conn(0x7f4cc80cde90 0x7f4cc80ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 -- 172.21.15.31:0/3895779109 >> 172.21.15.31:0/3895779109 conn(0x7f4cc80cfbb0 msgr2=0x7f4cc80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 -- 172.21.15.31:0/3895779109 shutdown_connections 2024-09-13T08:44:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.240+0000 7f4ccf000640 1 -- 172.21.15.31:0/3895779109 wait complete. 2024-09-13T08:44:01.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.242+0000 7f4ccf000640 1 Processor -- start 2024-09-13T08:44:01.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.242+0000 7f4ccf000640 1 -- start start 2024-09-13T08:44:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4cc80cde90 0x7f4cc8073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc8161480 0x7f4cc80c3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80cbfd0 con 0x7f4cc80cc190 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80cbe50 con 0x7f4cc8161480 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccf000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc80cadf0 con 0x7f4cc80cde90 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4cc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4cc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33856/0 (socket says 172.21.15.31:33856) 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4cc7e00640 1 -- 172.21.15.31:0/3365938423 learned_addr learned my addr 172.21.15.31:0/3365938423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4cbf400640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4cc80cde90 0x7f4cc8073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.243+0000 7f4ccd000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc8161480 0x7f4cc80c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 -- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4cc80cde90 msgr2=0x7f4cc8073da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4cc80cde90 0x7f4cc8073da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 -- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc8161480 msgr2=0x7f4cc80c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc8161480 0x7f4cc80c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 -- 172.21.15.31:0/3365938423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc80b24c0 con 0x7f4cc80cc190 2024-09-13T08:44:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc7e00640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c3a0 tx=0x7f4cb800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8017070 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cb80053c0 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc8072d00 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003850 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.244+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc80735a0 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.245+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc80cb2a0 con 0x7f4cc80cc190 2024-09-13T08:44:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.247+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f4cb801a070 con 0x7f4cc80cc190 2024-09-13T08:44:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.248+0000 7f4cc4c00640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 0x7f4ca407b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.248+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f4cb809a2a0 con 0x7f4cc80cc190 2024-09-13T08:44:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.248+0000 7f4cbf400640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 0x7f4ca407b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.248+0000 7f4cbf400640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 0x7f4ca407b070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80af550 tx=0x7f4cb000b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:01.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.252+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4cb8061d40 con 0x7f4cc80cc190 2024-09-13T08:44:01.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.480+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol5350", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7f4cc80b5b30 con 0x7f4ca4078bb0 2024-09-13T08:44:01.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.514+0000 7f4cc4c00640 1 -- 172.21.15.31:0/3365938423 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4cc80b5b30 con 0x7f4ca4078bb0 2024-09-13T08:44:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 msgr2=0x7f4ca407b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 0x7f4ca407b070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80af550 tx=0x7f4cb000b390 comp rx=0 tx=0).stop 2024-09-13T08:44:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 msgr2=0x7f4cc80b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c3a0 tx=0x7f4cb800c860 comp rx=0 tx=0).stop 2024-09-13T08:44:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 shutdown_connections 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ca4078bb0 0x7f4ca407b070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc8161480 0x7f4cc80c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4cc80cde90 0x7f4cc8073da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 --2- 172.21.15.31:0/3365938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cc80cc190 0x7f4cc80b1f80 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 >> 172.21.15.31:0/3365938423 conn(0x7f4cc80cfbb0 msgr2=0x7f4cc8072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 shutdown_connections 2024-09-13T08:44:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.517+0000 7f4ccf000640 1 -- 172.21.15.31:0/3365938423 wait complete. 2024-09-13T08:44:01.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:01.535 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol5350/.meta) 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol5350/f65791a0-4dd6-4855-8400-c9b686592ee9 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:44:01.592 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:01.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol5350 2024-09-13T08:44:01.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 -- 172.21.15.31:0/139756794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f015bfc0 msgr2=0x7f47f0072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 --2- 172.21.15.31:0/139756794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f015bfc0 0x7f47f0072560 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009830 tx=0x7f47e002a520 comp rx=0 tx=0).stop 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 -- 172.21.15.31:0/139756794 shutdown_connections 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 --2- 172.21.15.31:0/139756794 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f006fec0 0x7f47f00b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 --2- 172.21.15.31:0/139756794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f015bfc0 0x7f47f0072560 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 -- 172.21.15.31:0/139756794 >> 172.21.15.31:0/139756794 conn(0x7f47f00cfbb0 msgr2=0x7f47f00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 -- 172.21.15.31:0/139756794 shutdown_connections 2024-09-13T08:44:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.757+0000 7f47f5800640 1 -- 172.21.15.31:0/139756794 wait complete. 2024-09-13T08:44:01.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.759+0000 7f47f5800640 1 Processor -- start 2024-09-13T08:44:01.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.759+0000 7f47f5800640 1 -- start start 2024-09-13T08:44:01.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47f015bfc0 0x7f47f015e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f015c560 0x7f47f015c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47f00cbfd0 con 0x7f47f006fa60 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47f00cbe50 con 0x7f47f015bfc0 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47f5800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f47f00b2f20 con 0x7f47f015c560 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33874/0 (socket says 172.21.15.31:33874) 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47eea00640 1 -- 172.21.15.31:0/1773140651 learned_addr learned my addr 172.21.15.31:0/1773140651 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47e6000640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47f015bfc0 0x7f47f015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.760+0000 7f47ef400640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f015c560 0x7f47f015c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 -- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f015c560 msgr2=0x7f47f015c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f015c560 0x7f47f015c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 -- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47f015bfc0 msgr2=0x7f47f015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47f015bfc0 0x7f47f015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 -- 172.21.15.31:0/1773140651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f01f84c0 con 0x7f47f006fa60 2024-09-13T08:44:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47eea00640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f47e0000c00 tx=0x7f47e0033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e0005c90 con 0x7f47f006fa60 2024-09-13T08:44:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47e0031e50 con 0x7f47f006fa60 2024-09-13T08:44:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.761+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f01f87a0 con 0x7f47f006fa60 2024-09-13T08:44:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.762+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e00332f0 con 0x7f47f006fa60 2024-09-13T08:44:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.762+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f01f8d60 con 0x7f47f006fa60 2024-09-13T08:44:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.763+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f00b33d0 con 0x7f47f006fa60 2024-09-13T08:44:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.764+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f47e0033490 con 0x7f47f006fa60 2024-09-13T08:44:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.766+0000 7f47e7400640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 0x7f47cc07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.766+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f47e00bc2f0 con 0x7f47f006fa60 2024-09-13T08:44:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.766+0000 7f47e6000640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 0x7f47cc07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.766+0000 7f47e6000640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 0x7f47cc07b070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b26a0 tx=0x7f47d8007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.770+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f47e003f030 con 0x7f47f006fa60 2024-09-13T08:44:01.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:01.991+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol5350", "target": ["mon-mgr", ""]}) -- 0x7f47f00cb310 con 0x7f47cc078bb0 2024-09-13T08:44:02.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.036+0000 7f47e7400640 1 -- 172.21.15.31:0/1773140651 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f47f00cb310 con 0x7f47cc078bb0 2024-09-13T08:44:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 msgr2=0x7f47cc07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 0x7f47cc07b070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b26a0 tx=0x7f47d8007e90 comp rx=0 tx=0).stop 2024-09-13T08:44:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 msgr2=0x7f47f015ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f47e0000c00 tx=0x7f47e0033c90 comp rx=0 tx=0).stop 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 shutdown_connections 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47cc078bb0 0x7f47cc07b070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47f015c560 0x7f47f015c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47f015bfc0 0x7f47f015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 --2- 172.21.15.31:0/1773140651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f006fa60 0x7f47f015ef50 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.038+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 >> 172.21.15.31:0/1773140651 conn(0x7f47f00cfbb0 msgr2=0x7f47f0152fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.039+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 shutdown_connections 2024-09-13T08:44:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.039+0000 7f47f5800640 1 -- 172.21.15.31:0/1773140651 wait complete. 2024-09-13T08:44:02.047 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.041+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.047 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.041+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.048 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.041+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.048 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.041+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.048 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.041+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone87640 2024-09-13T08:44:02.060 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.058+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.060 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.058+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.061 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.058+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.061 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.058+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.061 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:02.058+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:02.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.226+0000 7f8534600640 1 -- 172.21.15.31:0/348887923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c161480 msgr2=0x7f852c0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.226+0000 7f8534600640 1 --2- 172.21.15.31:0/348887923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c161480 0x7f852c0cd780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8520009830 tx=0x7f852002a480 comp rx=0 tx=0).stop 2024-09-13T08:44:02.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 -- 172.21.15.31:0/348887923 shutdown_connections 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 --2- 172.21.15.31:0/348887923 >> v2:172.21.15.45:3300/0 conn(0x7f852c0cb380 0x7f852c0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 --2- 172.21.15.31:0/348887923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c161480 0x7f852c0cd780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 -- 172.21.15.31:0/348887923 >> 172.21.15.31:0/348887923 conn(0x7f852c0cfbb0 msgr2=0x7f852c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 -- 172.21.15.31:0/348887923 shutdown_connections 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.227+0000 7f8534600640 1 -- 172.21.15.31:0/348887923 wait complete. 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.228+0000 7f8534600640 1 Processor -- start 2024-09-13T08:44:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.228+0000 7f8534600640 1 -- start start 2024-09-13T08:44:02.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 0x7f852c1f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f852c161480 0x7f852c0c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0cbe00 con 0x7f852c0cc190 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0cbc80 con 0x7f852c161480 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.229+0000 7f8534600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f852c0cbf80 con 0x7f852c0cb380 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33924/0 (socket says 172.21.15.31:33924) 2024-09-13T08:44:02.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 -- 172.21.15.31:0/4058521687 learned_addr learned my addr 172.21.15.31:0/4058521687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531c00640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 0x7f852c1f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8532600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f852c161480 0x7f852c0c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 -- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 msgr2=0x7f852c1f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 0x7f852c1f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 -- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f852c161480 msgr2=0x7f852c0c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f852c161480 0x7f852c0c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 -- 172.21.15.31:0/4058521687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c1f5660 con 0x7f852c0cc190 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531c00640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 0x7f852c1f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.230+0000 7f8531200640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f851800c450 tx=0x7f851800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.231+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518016020 con 0x7f852c0cc190 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.231+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8518012920 con 0x7f852c0cc190 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.231+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c1f5940 con 0x7f852c0cc190 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.231+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518012d30 con 0x7f852c0cc190 2024-09-13T08:44:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.231+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c0c40d0 con 0x7f852c0cc190 2024-09-13T08:44:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.232+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852c162630 con 0x7f852c0cc190 2024-09-13T08:44:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.233+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f8518019020 con 0x7f852c0cc190 2024-09-13T08:44:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.234+0000 7f8529600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 0x7f850c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.234+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f851809af60 con 0x7f852c0cc190 2024-09-13T08:44:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.234+0000 7f8531c00640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 0x7f850c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.235+0000 7f8531c00640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 0x7f850c07b070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8520034000 tx=0x7f8520004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.238+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8518020020 con 0x7f852c0cc190 2024-09-13T08:44:02.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.462+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone87640", "target": ["mon-mgr", ""]}) -- 0x7f852c0b5b30 con 0x7f850c078bb0 2024-09-13T08:44:02.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.486+0000 7f8529600640 1 -- 172.21.15.31:0/4058521687 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f852c0b5b30 con 0x7f850c078bb0 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 msgr2=0x7f850c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 0x7f850c07b070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8520034000 tx=0x7f8520004730 comp rx=0 tx=0).stop 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 msgr2=0x7f852c0c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f851800c450 tx=0x7f851800c910 comp rx=0 tx=0).stop 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 shutdown_connections 2024-09-13T08:44:02.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f850c078bb0 0x7f850c07b070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f852c161480 0x7f852c0c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f852c0cc190 0x7f852c0c6ef0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 --2- 172.21.15.31:0/4058521687 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f852c0cb380 0x7f852c1f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 >> 172.21.15.31:0/4058521687 conn(0x7f852c0cfbb0 msgr2=0x7f852c0afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 shutdown_connections 2024-09-13T08:44:02.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.489+0000 7f8534600640 1 -- 172.21.15.31:0/4058521687 wait complete. 2024-09-13T08:44:02.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone85481 2024-09-13T08:44:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 -- 172.21.15.31:0/2590860651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 msgr2=0x7f35900b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 --2- 172.21.15.31:0/2590860651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900b1070 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f359008cdd0 tx=0x7f358802a820 comp rx=0 tx=0).stop 2024-09-13T08:44:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 -- 172.21.15.31:0/2590860651 shutdown_connections 2024-09-13T08:44:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 --2- 172.21.15.31:0/2590860651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900b1070 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 --2- 172.21.15.31:0/2590860651 >> v2:172.21.15.45:3300/0 conn(0x7f35901613f0 0x7f35900b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 -- 172.21.15.31:0/2590860651 >> 172.21.15.31:0/2590860651 conn(0x7f35900d3e10 msgr2=0x7f35900d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 -- 172.21.15.31:0/2590860651 shutdown_connections 2024-09-13T08:44:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.674+0000 7f3594a00640 1 -- 172.21.15.31:0/2590860651 wait complete. 2024-09-13T08:44:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.675+0000 7f3594a00640 1 Processor -- start 2024-09-13T08:44:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.676+0000 7f3594a00640 1 -- start start 2024-09-13T08:44:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.676+0000 7f3594a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3594a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 0x7f35900cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3594a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35901617d0 0x7f35901552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3594a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35900b1940 con 0x7f35900d4930 2024-09-13T08:44:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3594a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f35900b17c0 con 0x7f35901613f0 2024-09-13T08:44:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3594a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f35900b1ac0 con 0x7f35901617d0 2024-09-13T08:44:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33952/0 (socket says 172.21.15.31:33952) 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 -- 172.21.15.31:0/2016202063 learned_addr learned my addr 172.21.15.31:0/2016202063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3587e00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 0x7f35900cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358f200640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35901617d0 0x7f35901552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 -- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35901617d0 msgr2=0x7f35901552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35901617d0 0x7f35901552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 -- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 msgr2=0x7f35900cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 0x7f35900cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f358e800640 1 -- 172.21.15.31:0/2016202063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3590155800 con 0x7f35900d4930 2024-09-13T08:44:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.677+0000 7f3587e00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 0x7f35900cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f358e800640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f358000d950 tx=0x7f358000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580017070 con 0x7f35900d4930 2024-09-13T08:44:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3580009d20 con 0x7f35900d4930 2024-09-13T08:44:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35900b6dc0 con 0x7f35900d4930 2024-09-13T08:44:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35800106c0 con 0x7f35900d4930 2024-09-13T08:44:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.678+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35900b7380 con 0x7f35900d4930 2024-09-13T08:44:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.679+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35900b1070 con 0x7f35900d4930 2024-09-13T08:44:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.680+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f3580010860 con 0x7f35900d4930 2024-09-13T08:44:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.682+0000 7f3586a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 0x7f356c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.682+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f358009b420 con 0x7f35900d4930 2024-09-13T08:44:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.682+0000 7f3587e00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 0x7f356c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.683+0000 7f3587e00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 0x7f356c07b070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35901f76b0 tx=0x7f3578009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.686+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3580062dc0 con 0x7f35900d4930 2024-09-13T08:44:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.911+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone85481", "target": ["mon-mgr", ""]}) -- 0x7f35900cf780 con 0x7f356c078bb0 2024-09-13T08:44:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.930+0000 7f3586a00640 1 -- 172.21.15.31:0/2016202063 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f35900cf780 con 0x7f356c078bb0 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.932+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 msgr2=0x7f356c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.932+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 0x7f356c07b070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35901f76b0 tx=0x7f3578009040 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.932+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 msgr2=0x7f35900caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.932+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f358000d950 tx=0x7f358000de10 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 shutdown_connections 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f356c078bb0 0x7f356c07b070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35901617d0 0x7f35901552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35901613f0 0x7f35900cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 --2- 172.21.15.31:0/2016202063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35900d4930 0x7f35900caee0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 >> 172.21.15.31:0/2016202063 conn(0x7f35900d3e10 msgr2=0x7f35900cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 shutdown_connections 2024-09-13T08:44:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:02.933+0000 7f3594a00640 1 -- 172.21.15.31:0/2016202063 wait complete. 2024-09-13T08:44:02.952 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:44:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:44:03.040 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:44:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:03.049 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:44:03.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:03.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:03.218 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:44:03.218 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:03.219 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:03.219 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:03.257 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:03.257 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:03.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 -- 172.21.15.31:0/3774396556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 msgr2=0x7f6844004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 --2- 172.21.15.31:0/3774396556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f6844004c80 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f68480016c0 tx=0x7f6844005830 comp rx=0 tx=0).stop 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 -- 172.21.15.31:0/3774396556 shutdown_connections 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 --2- 172.21.15.31:0/3774396556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f6844004c80 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 -- 172.21.15.31:0/3774396556 >> 172.21.15.31:0/3774396556 conn(0x7f68600b18f0 msgr2=0x7f68600b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.535+0000 7f6865c00640 1 -- 172.21.15.31:0/3774396556 shutdown_connections 2024-09-13T08:44:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.536+0000 7f6865c00640 1 -- 172.21.15.31:0/3774396556 wait complete. 2024-09-13T08:44:03.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.537+0000 7f6865c00640 1 Processor -- start 2024-09-13T08:44:03.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.537+0000 7f6865c00640 1 -- start start 2024-09-13T08:44:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68600ada70 0x7f68600ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 0x7f68600ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f686006e9f0 con 0x7f6844004800 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f686006e870 con 0x7f68600ada70 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6865c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f686006f380 con 0x7f68600cbc30 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f685ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f685ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33972/0 (socket says 172.21.15.31:33972) 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f685ea00640 1 -- 172.21.15.31:0/3267155438 learned_addr learned my addr 172.21.15.31:0/3267155438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f6856000640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68600ada70 0x7f68600ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.538+0000 7f685f400640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 0x7f68600ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 -- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 msgr2=0x7f68600ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 0x7f68600ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 -- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68600ada70 msgr2=0x7f68600ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:03.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68600ada70 0x7f68600ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 -- 172.21.15.31:0/3267155438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686014ed50 con 0x7f6844004800 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685f400640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 0x7f68600ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f685ea00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6850011ae0 tx=0x7f6850009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685000fcb0 con 0x7f6844004800 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f685000fe50 con 0x7f6844004800 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850004280 con 0x7f6844004800 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.539+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68600aeb80 con 0x7f6844004800 2024-09-13T08:44:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.540+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68600ad270 con 0x7f6844004800 2024-09-13T08:44:03.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.541+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68600cc660 con 0x7f6844004800 2024-09-13T08:44:03.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.542+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f68500035f0 con 0x7f6844004800 2024-09-13T08:44:03.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.543+0000 7f6857400640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 0x7f683c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:03.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.543+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f685009ba10 con 0x7f6844004800 2024-09-13T08:44:03.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.543+0000 7f6856000640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 0x7f683c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:03.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.544+0000 7f6856000640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 0x7f683c07b070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6848032000 tx=0x7f684802c040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.547+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f68500634f0 con 0x7f6844004800 2024-09-13T08:44:03.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.754+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f68600cd860 con 0x7f6844004800 2024-09-13T08:44:03.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.754+0000 7f6857400640 1 -- 172.21.15.31:0/3267155438 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) ==== 80+0+0 (secure 0 0 0) 0x7f6850067f20 con 0x7f6844004800 2024-09-13T08:44:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:44:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 msgr2=0x7f683c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 0x7f683c07b070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6848032000 tx=0x7f684802c040 comp rx=0 tx=0).stop 2024-09-13T08:44:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 msgr2=0x7f686014e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6850011ae0 tx=0x7f6850009f10 comp rx=0 tx=0).stop 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 shutdown_connections 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f683c078bb0 0x7f683c07b070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68600cbc30 0x7f68600ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68600ada70 0x7f68600ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 --2- 172.21.15.31:0/3267155438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844004800 0x7f686014e810 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 >> 172.21.15.31:0/3267155438 conn(0x7f68600b18f0 msgr2=0x7f68601600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 shutdown_connections 2024-09-13T08:44:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:03.757+0000 7f6865c00640 1 -- 172.21.15.31:0/3267155438 wait complete. 2024-09-13T08:44:03.774 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:44:03.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:03.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:03.874 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:44:03.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:03.874 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:44:03.914 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:03.915 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:03.915 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:03.952 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:03.953 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:04.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:44:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 -- 172.21.15.31:0/279649069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 msgr2=0x7f4b400cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 --2- 172.21.15.31:0/279649069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400cdb80 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4b30005fd0 tx=0x7f4b3002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 -- 172.21.15.31:0/279649069 shutdown_connections 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 --2- 172.21.15.31:0/279649069 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 0x7f4b400ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 --2- 172.21.15.31:0/279649069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400cdb80 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 -- 172.21.15.31:0/279649069 >> 172.21.15.31:0/279649069 conn(0x7f4b400b18f0 msgr2=0x7f4b400b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.191+0000 7f4b46600640 1 -- 172.21.15.31:0/279649069 shutdown_connections 2024-09-13T08:44:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.192+0000 7f4b46600640 1 -- 172.21.15.31:0/279649069 wait complete. 2024-09-13T08:44:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.193+0000 7f4b46600640 1 Processor -- start 2024-09-13T08:44:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.193+0000 7f4b46600640 1 -- start start 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b400ce600 0x7f4b4014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 0x7f4b400b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b400733e0 con 0x7f4b4015be30 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4b40160160 con 0x7f4b400ce600 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b46600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4b401602e0 con 0x7f4b400ced20 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.194+0000 7f4b3fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b400ce600 0x7f4b4014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33996/0 (socket says 172.21.15.31:33996) 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 -- 172.21.15.31:0/2946509568 learned_addr learned my addr 172.21.15.31:0/2946509568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b36a00640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 0x7f4b400b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 -- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 msgr2=0x7f4b400b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 0x7f4b400b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 -- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b400ce600 msgr2=0x7f4b4014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b400ce600 0x7f4b4014f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 -- 172.21.15.31:0/2946509568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b400b3540 con 0x7f4b4015be30 2024-09-13T08:44:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b3fe00640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c7d0 tx=0x7f4b3800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38003b70 con 0x7f4b4015be30 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b38003d10 con 0x7f4b4015be30 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.195+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800aab0 con 0x7f4b4015be30 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.196+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b400b3820 con 0x7f4b4015be30 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.196+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b40153e80 con 0x7f4b4015be30 2024-09-13T08:44:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.197+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b40160800 con 0x7f4b4015be30 2024-09-13T08:44:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.198+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 101593+0+0 (secure 0 0 0) 0x7f4b3800ac50 con 0x7f4b4015be30 2024-09-13T08:44:04.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.199+0000 7f4b37e00640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 0x7f4b1c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.200+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f4b3809ae30 con 0x7f4b4015be30 2024-09-13T08:44:04.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.200+0000 7f4b3f400640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 0x7f4b1c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.200+0000 7f4b3f400640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 0x7f4b1c07b070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4b30033000 tx=0x7f4b30004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:04.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.204+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4b380627d0 con 0x7f4b4015be30 2024-09-13T08:44:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.237+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f4b3805ea40 con 0x7f4b4015be30 2024-09-13T08:44:04.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.417+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4b4006f9c0 con 0x7f4b4015be30 2024-09-13T08:44:04.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.417+0000 7f4b37e00640 1 -- 172.21.15.31:0/2946509568 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) ==== 80+0+0 (secure 0 0 0) 0x7f4b38072230 con 0x7f4b4015be30 2024-09-13T08:44:04.419 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 msgr2=0x7f4b1c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 0x7f4b1c07b070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4b30033000 tx=0x7f4b30004730 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 msgr2=0x7f4b400b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c7d0 tx=0x7f4b3800cc90 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 shutdown_connections 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4b1c078bb0 0x7f4b1c07b070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b4015be30 0x7f4b400b0020 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4b400ced20 0x7f4b400b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 --2- 172.21.15.31:0/2946509568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b400ce600 0x7f4b4014f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.419+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 >> 172.21.15.31:0/2946509568 conn(0x7f4b400b18f0 msgr2=0x7f4b400cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:04.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.420+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 shutdown_connections 2024-09-13T08:44:04.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.420+0000 7f4b46600640 1 -- 172.21.15.31:0/2946509568 wait complete. 2024-09-13T08:44:04.437 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:44:04.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:04.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:04.504 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:44:04.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:04.504 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:44:04.541 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:04.541 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:04.541 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:04.576 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:04.577 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:04.577 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:04.577 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:04.616 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:04.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2135197157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d280ca5c0 msgr2=0x7f6d280ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2135197157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d280ca5c0 0x7f6d280ce600 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6d280cd8d0 tx=0x7f6d1002a560 comp rx=0 tx=0).stop 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2135197157 shutdown_connections 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2135197157 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280b1910 0x7f6d280b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2135197157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d280ca5c0 0x7f6d280ce600 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2135197157 >> v2:172.21.15.45:3300/0 conn(0x7f6d2815dfc0 0x7f6d280cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2135197157 >> 172.21.15.31:0/2135197157 conn(0x7f6d280cc910 msgr2=0x7f6d280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2135197157 shutdown_connections 2024-09-13T08:44:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.775+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2135197157 wait complete. 2024-09-13T08:44:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.777+0000 7f6d2cc00640 1 Processor -- start 2024-09-13T08:44:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.777+0000 7f6d2cc00640 1 -- start start 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d280b1910 0x7f6d2809d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280ca5c0 0x7f6d280c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d28073900 con 0x7f6d2815dfc0 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d28073780 con 0x7f6d280b1910 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d2cc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6d28073a80 con 0x7f6d280ca5c0 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d26800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d26800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34014/0 (socket says 172.21.15.31:34014) 2024-09-13T08:44:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d25e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d280b1910 0x7f6d2809d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d26800640 1 -- 172.21.15.31:0/2009694873 learned_addr learned my addr 172.21.15.31:0/2009694873 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.778+0000 7f6d1d400640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280ca5c0 0x7f6d280c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 -- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280ca5c0 msgr2=0x7f6d280c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280ca5c0 0x7f6d280c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 -- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d280b1910 msgr2=0x7f6d2809d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d280b1910 0x7f6d2809d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 -- 172.21.15.31:0/2009694873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d281f5db0 con 0x7f6d2815dfc0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d26800640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e980 tx=0x7f6d2000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20004d20 con 0x7f6d2815dfc0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d2000cb20 con 0x7f6d2815dfc0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20005020 con 0x7f6d2815dfc0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d281f5ad0 con 0x7f6d2815dfc0 2024-09-13T08:44:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.779+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d280c8b80 con 0x7f6d2815dfc0 2024-09-13T08:44:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.780+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d280c86d0 con 0x7f6d2815dfc0 2024-09-13T08:44:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.781+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f6d200051c0 con 0x7f6d2815dfc0 2024-09-13T08:44:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.782+0000 7f6d1ea00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 0x7f6d0407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.782+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f6d2009bb60 con 0x7f6d2815dfc0 2024-09-13T08:44:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.782+0000 7f6d25e00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 0x7f6d0407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.783+0000 7f6d25e00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 0x7f6d0407b190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800bb50 tx=0x7f6d1800b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:04.787+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6d20016030 con 0x7f6d2815dfc0 2024-09-13T08:44:05.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.026+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d28073f80 con 0x7f6d2815dfc0 2024-09-13T08:44:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.027+0000 7f6d1ea00640 1 -- 172.21.15.31:0/2009694873 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5222 (secure 0 0 0) 0x7f6d20063300 con 0x7f6d2815dfc0 2024-09-13T08:44:05.029 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:05.029 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-13T08:42:58:947114+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:44:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 msgr2=0x7f6d0407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 0x7f6d0407b190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800bb50 tx=0x7f6d1800b2e0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 msgr2=0x7f6d281f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e980 tx=0x7f6d2000ee40 comp rx=0 tx=0).stop 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 shutdown_connections 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6d04078cd0 0x7f6d0407b190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d2815dfc0 0x7f6d281f4ef0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6d280ca5c0 0x7f6d280c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 --2- 172.21.15.31:0/2009694873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d280b1910 0x7f6d2809d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 >> 172.21.15.31:0/2009694873 conn(0x7f6d280cc910 msgr2=0x7f6d2806f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 shutdown_connections 2024-09-13T08:44:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.030+0000 7f6d2cc00640 1 -- 172.21.15.31:0/2009694873 wait complete. 2024-09-13T08:44:05.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 -- 172.21.15.31:0/879844430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 msgr2=0x7f30ac0b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 --2- 172.21.15.31:0/879844430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac0b0f80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f30a4005fd0 tx=0x7f30a402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 -- 172.21.15.31:0/879844430 shutdown_connections 2024-09-13T08:44:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 --2- 172.21.15.31:0/879844430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac0b0f80 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 --2- 172.21.15.31:0/879844430 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac0b47f0 0x7f30ac0d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 -- 172.21.15.31:0/879844430 >> 172.21.15.31:0/879844430 conn(0x7f30ac08d8b0 msgr2=0x7f30ac09d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 -- 172.21.15.31:0/879844430 shutdown_connections 2024-09-13T08:44:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.207+0000 7f30ab200640 1 -- 172.21.15.31:0/879844430 wait complete. 2024-09-13T08:44:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.208+0000 7f30ab200640 1 Processor -- start 2024-09-13T08:44:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 -- start start 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30ac0b47f0 0x7f30ac0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac15dfa0 0x7f30ac14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac0b1930 con 0x7f30ac0d4110 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac0b17b0 con 0x7f30ac0b47f0 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.209+0000 7f30ab200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f30ac0b1ab0 con 0x7f30ac15dfa0 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49584/0 (socket says 172.21.15.31:49584) 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 -- 172.21.15.31:0/1026547499 learned_addr learned my addr 172.21.15.31:0/1026547499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a9e00640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30ac0b47f0 0x7f30ac0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30aa800640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac15dfa0 0x7f30ac14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 -- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac15dfa0 msgr2=0x7f30ac14d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac15dfa0 0x7f30ac14d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 -- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30ac0b47f0 msgr2=0x7f30ac0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30ac0b47f0 0x7f30ac0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 -- 172.21.15.31:0/1026547499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac1e7b30 con 0x7f30ac0d4110 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.210+0000 7f30a1400640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f309400e9f0 tx=0x7f309400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.211+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309400cd80 con 0x7f30ac0d4110 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.211+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3094010430 con 0x7f30ac0d4110 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.211+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094010840 con 0x7f30ac0d4110 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.211+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac1e7e10 con 0x7f30ac0d4110 2024-09-13T08:44:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.211+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac14d170 con 0x7f30ac0d4110 2024-09-13T08:44:05.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.212+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ac0b0ef0 con 0x7f30ac0d4110 2024-09-13T08:44:05.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.213+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f30940109e0 con 0x7f30ac0d4110 2024-09-13T08:44:05.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.214+0000 7f30a2a00640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 0x7f308807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.214+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5883+0+0 (secure 0 0 0) 0x7f309409be10 con 0x7f30ac0d4110 2024-09-13T08:44:05.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.214+0000 7f30a9e00640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 0x7f308807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.215+0000 7f30a9e00640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 0x7f308807b190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f309c001e20 tx=0x7f309c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:05.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.218+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f30ac0b0ef0 con 0x7f30ac0d4110 2024-09-13T08:44:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.461+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30ac0b1fd0 con 0x7f30ac0d4110 2024-09-13T08:44:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.461+0000 7f30a2a00640 1 -- 172.21.15.31:0/1026547499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5222 (secure 0 0 0) 0x7f3094063670 con 0x7f30ac0d4110 2024-09-13T08:44:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:05.463 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-13T08:42:58:947114+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:44:05.464 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-13T08:44:05.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 msgr2=0x7f308807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 0x7f308807b190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f309c001e20 tx=0x7f309c009040 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 msgr2=0x7f30ac1f5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f309400e9f0 tx=0x7f309400eeb0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 shutdown_connections 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3088078cd0 0x7f308807b190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f30ac15dfa0 0x7f30ac14d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30ac0d4110 0x7f30ac1f5b60 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 --2- 172.21.15.31:0/1026547499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30ac0b47f0 0x7f30ac0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 >> 172.21.15.31:0/1026547499 conn(0x7f30ac08d8b0 msgr2=0x7f30ac075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 shutdown_connections 2024-09-13T08:44:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.464+0000 7f30ab200640 1 -- 172.21.15.31:0/1026547499 wait complete. 2024-09-13T08:44:05.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 -- 172.21.15.31:0/3564231690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c15bd70 msgr2=0x7fea2c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 --2- 172.21.15.31:0/3564231690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c15bd70 0x7fea2c0cf850 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0c6f10 tx=0x7fea1c02aac0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 -- 172.21.15.31:0/3564231690 shutdown_connections 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 --2- 172.21.15.31:0/3564231690 >> v2:172.21.15.45:3300/0 conn(0x7fea2c15f9e0 0x7fea2c15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 --2- 172.21.15.31:0/3564231690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c15bd70 0x7fea2c0cf850 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 -- 172.21.15.31:0/3564231690 >> 172.21.15.31:0/3564231690 conn(0x7fea2c0cc910 msgr2=0x7fea2c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 -- 172.21.15.31:0/3564231690 shutdown_connections 2024-09-13T08:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.641+0000 7fea33000640 1 -- 172.21.15.31:0/3564231690 wait complete. 2024-09-13T08:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.643+0000 7fea33000640 1 Processor -- start 2024-09-13T08:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.643+0000 7fea33000640 1 -- start start 2024-09-13T08:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea2c15bd70 0x7fea2c09c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 0x7fea2c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c0cb9d0 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c0cb850 con 0x7fea2c15f9e0 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea33000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fea2c0cc360 con 0x7fea2c15bd70 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49598/0 (socket says 172.21.15.31:49598) 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 -- 172.21.15.31:0/1678481186 learned_addr learned my addr 172.21.15.31:0/1678481186 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2b400640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea2c15bd70 0x7fea2c09c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea31000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 0x7fea2c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 -- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea2c15bd70 msgr2=0x7fea2c09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea2c15bd70 0x7fea2c09c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 -- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 msgr2=0x7fea2c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 0x7fea2c0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea2be00640 1 -- 172.21.15.31:0/1678481186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c06e590 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.644+0000 7fea31000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 0x7fea2c0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea2be00640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000f80 tx=0x7fea1c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c044070 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea1c004250 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c004680 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c06e870 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.645+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c06ee30 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.646+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea2c0cf850 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.647+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fea1c036400 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.648+0000 7fea23e00640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 0x7fea0c07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.648+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5939+0+0 (secure 0 0 0) 0x7fea1c0bb940 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.649+0000 7fea2b400640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 0x7fea0c07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.649+0000 7fea2b400640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 0x7fea0c07b190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea2c06f620 tx=0x7fea1800b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.653+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fea1c032030 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.854+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea2c0b03a0 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.855+0000 7fea23e00640 1 -- 172.21.15.31:0/1678481186 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+17092 (secure 0 0 0) 0x7fea1c004820 con 0x7fea2c0ce3c0 2024-09-13T08:44:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:05.857 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:05.241213+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:46.132965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:47.310441+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.c","mappings":[{"from":4,"to":3}]},{"pgid":"17.5","mappings":[{"from":4,"to":5}]},{"pgid":"17.6","mappings":[{"from":6,"to":5}]},{"pgid":"17.f","mappings":[{"from":6,"to":1}]},{"pgid":"17.25","mappings":[{"from":6,"to":5}]},{"pgid":"17.2e","mappings":[{"from":6,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:05.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.857+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 msgr2=0x7fea0c07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.857+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 0x7fea0c07b190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea2c06f620 tx=0x7fea1800b2e0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.857+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 msgr2=0x7fea2c1541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.857+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000f80 tx=0x7fea1c003f10 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 shutdown_connections 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fea0c078cd0 0x7fea0c07b190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea2c15f9e0 0x7fea2c0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fea2c15bd70 0x7fea2c09c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 --2- 172.21.15.31:0/1678481186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea2c0ce3c0 0x7fea2c1541b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 >> 172.21.15.31:0/1678481186 conn(0x7fea2c0cc910 msgr2=0x7fea2c0af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 shutdown_connections 2024-09-13T08:44:05.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:05.858+0000 7fea33000640 1 -- 172.21.15.31:0/1678481186 wait complete. 2024-09-13T08:44:05.876 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:44:05.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 -- 172.21.15.31:0/2969519978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 msgr2=0x7f1ce00ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/2969519978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce00ce3c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00cfe60 tx=0x7f1ccc02a660 comp rx=0 tx=0).stop 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 -- 172.21.15.31:0/2969519978 shutdown_connections 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/2969519978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce00ce3c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 -- 172.21.15.31:0/2969519978 >> 172.21.15.31:0/2969519978 conn(0x7f1ce00cc910 msgr2=0x7f1ce00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 -- 172.21.15.31:0/2969519978 shutdown_connections 2024-09-13T08:44:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.037+0000 7f1ce5a00640 1 -- 172.21.15.31:0/2969519978 wait complete. 2024-09-13T08:44:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.038+0000 7f1ce5a00640 1 Processor -- start 2024-09-13T08:44:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.038+0000 7f1ce5a00640 1 -- start start 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 0x7f1ce0152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 0x7f1ce009b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01600f0 con 0x7f1ce00cf850 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce00cb850 con 0x7f1ce015bd70 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.039+0000 7f1ce5a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce009bad0 con 0x7f1ce00ce900 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49616/0 (socket says 172.21.15.31:49616) 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 -- 172.21.15.31:0/494065852 learned_addr learned my addr 172.21.15.31:0/494065852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cdea00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 0x7f1ce0152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 -- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 msgr2=0x7f1ce0152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cdf400640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 0x7f1ce009b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 0x7f1ce0152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 -- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 msgr2=0x7f1ce009b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 0x7f1ce009b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 -- 172.21.15.31:0/494065852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce015e8a0 con 0x7f1ce00cf850 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cdea00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 0x7f1ce0152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cdf400640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 0x7f1ce009b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.040+0000 7f1cde000640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009850 tx=0x7f1ccc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.041+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc03f070 con 0x7f1ce00cf850 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.041+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ccc004890 con 0x7f1ce00cf850 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.041+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce00b2df0 con 0x7f1ce00cf850 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.041+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc035030 con 0x7f1ce00cf850 2024-09-13T08:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.041+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce00b33b0 con 0x7f1ce00cf850 2024-09-13T08:44:06.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.042+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce00cc3b0 con 0x7f1ce00cf850 2024-09-13T08:44:06.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.043+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f1ccc042070 con 0x7f1ce00cf850 2024-09-13T08:44:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.044+0000 7f1cd6a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 0x7f1cc007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.044+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5939+0+0 (secure 0 0 0) 0x7f1ccc0bc280 con 0x7f1ce00cf850 2024-09-13T08:44:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.044+0000 7f1cdea00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 0x7f1cc007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.045+0000 7f1cdea00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 0x7f1cc007b190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000f60 tx=0x7f1cd0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.048+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1ccc084080 con 0x7f1ce00cf850 2024-09-13T08:44:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.289+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ce00ae920 con 0x7f1ce00cf850 2024-09-13T08:44:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.290+0000 7f1cd6a00640 1 -- 172.21.15.31:0/494065852 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5222 (secure 0 0 0) 0x7f1ccc083980 con 0x7f1ce00cf850 2024-09-13T08:44:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:06.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-13T08:42:58:947114+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:44:06.293 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 msgr2=0x7f1cc007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 0x7f1cc007b190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000f60 tx=0x7f1cd0009040 comp rx=0 tx=0).stop 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 msgr2=0x7f1ce0151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009850 tx=0x7f1ccc0039e0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 shutdown_connections 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1cc0078cd0 0x7f1cc007b190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ce015bd70 0x7f1ce009b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce00cf850 0x7f1ce0151ee0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 --2- 172.21.15.31:0/494065852 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1ce00ce900 0x7f1ce0152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 >> 172.21.15.31:0/494065852 conn(0x7f1ce00cc910 msgr2=0x7f1ce00af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 shutdown_connections 2024-09-13T08:44:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.293+0000 7f1ce5a00640 1 -- 172.21.15.31:0/494065852 wait complete. 2024-09-13T08:44:06.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 -- 172.21.15.31:0/1525566285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa69015dfa0 msgr2=0x7fa6900d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 --2- 172.21.15.31:0/1525566285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa69015dfa0 0x7fa6900d2500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa684005fd0 tx=0x7fa68402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 -- 172.21.15.31:0/1525566285 shutdown_connections 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 --2- 172.21.15.31:0/1525566285 >> v2:172.21.15.45:3300/0 conn(0x7fa6900d42e0 0x7fa6900b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 --2- 172.21.15.31:0/1525566285 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa69015dfa0 0x7fa6900d2500 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 -- 172.21.15.31:0/1525566285 >> 172.21.15.31:0/1525566285 conn(0x7fa69008d8b0 msgr2=0x7fa69009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 -- 172.21.15.31:0/1525566285 shutdown_connections 2024-09-13T08:44:06.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.470+0000 7fa696400640 1 -- 172.21.15.31:0/1525566285 wait complete. 2024-09-13T08:44:06.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.472+0000 7fa696400640 1 Processor -- start 2024-09-13T08:44:06.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.472+0000 7fa696400640 1 -- start start 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6900b47f0 0x7fa6900d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6900d42e0 0x7fa6900cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa6900b1900 con 0x7fa69015dfa0 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6900b1780 con 0x7fa6900b47f0 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa696400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa6900b1a80 con 0x7fa6900d42e0 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa695a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa695a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49624/0 (socket says 172.21.15.31:49624) 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa695a00640 1 -- 172.21.15.31:0/1709599371 learned_addr learned my addr 172.21.15.31:0/1709599371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa695000640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6900b47f0 0x7fa6900d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.473+0000 7fa68fe00640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6900d42e0 0x7fa6900cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 -- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6900d42e0 msgr2=0x7fa6900cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6900d42e0 0x7fa6900cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 -- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6900b47f0 msgr2=0x7fa6900d0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6900b47f0 0x7fa6900d0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 -- 172.21.15.31:0/1709599371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6901f8e40 con 0x7fa69015dfa0 2024-09-13T08:44:06.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa695a00640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa68800efd0 tx=0x7fa68800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688003b70 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa688003e00 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688004280 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.474+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6901f59b0 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.475+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6900d0040 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.476+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6900b1f30 con 0x7fa69015dfa0 2024-09-13T08:44:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.477+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fa688004420 con 0x7fa69015dfa0 2024-09-13T08:44:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.477+0000 7fa68cc00640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 0x7fa67007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.478+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5939+0+0 (secure 0 0 0) 0x7fa68809c260 con 0x7fa69015dfa0 2024-09-13T08:44:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.478+0000 7fa695000640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 0x7fa67007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.478+0000 7fa695000640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 0x7fa67007b190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa684033000 tx=0x7fa684004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.482+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa6901f8e40 con 0x7fa69015dfa0 2024-09-13T08:44:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.717+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6900d4cf0 con 0x7fa69015dfa0 2024-09-13T08:44:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.718+0000 7fa68cc00640 1 -- 172.21.15.31:0/1709599371 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5222 (secure 0 0 0) 0x7fa688063900 con 0x7fa69015dfa0 2024-09-13T08:44:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:06.720 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-13T08:42:58:947114+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:42:48.347422+0000","modified":"2024-09-13T08:42:51.365032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6963":{"gid":6963,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6801/3014570213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3014570213},{"type":"v1","addr":"172.21.15.45:6801","nonce":3014570213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6963,"qdb_cluster":[6963]},"id":8}]} 2024-09-13T08:44:06.720 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.720+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 msgr2=0x7fa67007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.720+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 0x7fa67007b190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa684033000 tx=0x7fa684004730 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.720+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 msgr2=0x7fa6901f5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.720+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa68800efd0 tx=0x7fa68800c6a0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 shutdown_connections 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa670078cd0 0x7fa67007b190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa69015dfa0 0x7fa6901f5470 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6900d42e0 0x7fa6900cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 --2- 172.21.15.31:0/1709599371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6900b47f0 0x7fa6900d0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 >> 172.21.15.31:0/1709599371 conn(0x7fa69008d8b0 msgr2=0x7fa6900ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 shutdown_connections 2024-09-13T08:44:06.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.721+0000 7fa696400640 1 -- 172.21.15.31:0/1709599371 wait complete. 2024-09-13T08:44:06.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:06.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 -- 172.21.15.31:0/2114053958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 msgr2=0x7f02e8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 --2- 172.21.15.31:0/2114053958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f02e8003e70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009b80 tx=0x7f02e8005830 comp rx=0 tx=0).stop 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f0303e00640 1 -- 172.21.15.31:0/2114053958 reap_dead start 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 -- 172.21.15.31:0/2114053958 shutdown_connections 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 -- 172.21.15.31:0/2114053958 >> 172.21.15.31:0/2114053958 conn(0x7f03040cc910 msgr2=0x7f03040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 -- 172.21.15.31:0/2114053958 shutdown_connections 2024-09-13T08:44:06.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.897+0000 7f030a000640 1 -- 172.21.15.31:0/2114053958 wait complete. 2024-09-13T08:44:06.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.899+0000 7f030a000640 1 Processor -- start 2024-09-13T08:44:06.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.899+0000 7f030a000640 1 -- start start 2024-09-13T08:44:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 0x7f03041f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03041f8b00 0x7f03041f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03040cec50 con 0x7f02e8003a50 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03040cead0 con 0x7f03041f8b00 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f030a000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03040cb850 con 0x7f030414ff00 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f0303400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f0303400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49646/0 (socket says 172.21.15.31:49646) 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f0303400640 1 -- 172.21.15.31:0/1585262580 learned_addr learned my addr 172.21.15.31:0/1585262580 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f0303e00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03041f8b00 0x7f03041f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.900+0000 7f02faa00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 0x7f03041f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 -- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 msgr2=0x7f03041f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 0x7f03041f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 -- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03041f8b00 msgr2=0x7f03041f8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03041f8b00 0x7f03041f8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:06.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 -- 172.21.15.31:0/1585262580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0304150670 con 0x7f02e8003a50 2024-09-13T08:44:06.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f02faa00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 0x7f03041f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f0303400640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ca0 tx=0x7f02f4033b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4038040 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02f402b850 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4004020 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0304150950 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.901+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03041f85e0 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.902+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03040cf850 con 0x7f02e8003a50 2024-09-13T08:44:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.904+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f02f4045020 con 0x7f02e8003a50 2024-09-13T08:44:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.904+0000 7f02fbe00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 0x7f02e007b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.905+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5939+0+0 (secure 0 0 0) 0x7f02f40bbc40 con 0x7f02e8003a50 2024-09-13T08:44:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.905+0000 7f02faa00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 0x7f02e007b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:06.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.905+0000 7f02faa00640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 0x7f02e007b1c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02ec001010 tx=0x7f02ec006720 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:06.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:06.909+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f02f402ba20 con 0x7f02e8003a50 2024-09-13T08:44:07.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:44:07.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:44:07.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.104+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f030406f6c0 con 0x7f02e8003a50 2024-09-13T08:44:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.105+0000 7f02fbe00640 1 -- 172.21.15.31:0/1585262580 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+17092 (secure 0 0 0) 0x7f02f40834d0 con 0x7f02e8003a50 2024-09-13T08:44:07.107 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:07.108 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:05.241213+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:42:46.132965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T08:42:47.310441+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.c","mappings":[{"from":4,"to":3}]},{"pgid":"17.5","mappings":[{"from":4,"to":5}]},{"pgid":"17.6","mappings":[{"from":6,"to":5}]},{"pgid":"17.f","mappings":[{"from":6,"to":1}]},{"pgid":"17.25","mappings":[{"from":6,"to":5}]},{"pgid":"17.2e","mappings":[{"from":6,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 msgr2=0x7f02e007b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 0x7f02e007b1c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02ec001010 tx=0x7f02ec006720 comp rx=0 tx=0).stop 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 msgr2=0x7f0304153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ca0 tx=0x7f02f4033b30 comp rx=0 tx=0).stop 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 shutdown_connections 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f02e0078d00 0x7f02e007b1c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03041f8b00 0x7f03041f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f030414ff00 0x7f03041f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 --2- 172.21.15.31:0/1585262580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02e8003a50 0x7f0304153a30 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 >> 172.21.15.31:0/1585262580 conn(0x7f03040cc910 msgr2=0x7f03040add00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 shutdown_connections 2024-09-13T08:44:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.108+0000 7f030a000640 1 -- 172.21.15.31:0/1585262580 wait complete. 2024-09-13T08:44:07.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:44:07.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 -- 172.21.15.31:0/3601591431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab34004d40 msgr2=0x7fab340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 --2- 172.21.15.31:0/3601591431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab34004d40 0x7fab340051c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fab4800c640 tx=0x7fab34005830 comp rx=0 tx=0).stop 2024-09-13T08:44:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab4f400640 1 -- 172.21.15.31:0/3601591431 reap_dead start 2024-09-13T08:44:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 -- 172.21.15.31:0/3601591431 shutdown_connections 2024-09-13T08:44:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 -- 172.21.15.31:0/3601591431 >> 172.21.15.31:0/3601591431 conn(0x7fab500b18f0 msgr2=0x7fab500b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 -- 172.21.15.31:0/3601591431 shutdown_connections 2024-09-13T08:44:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.327+0000 7fab55e00640 1 -- 172.21.15.31:0/3601591431 wait complete. 2024-09-13T08:44:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.329+0000 7fab55e00640 1 Processor -- start 2024-09-13T08:44:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.329+0000 7fab55e00640 1 -- start start 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab34004d40 0x7fab501f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab500cb780 0x7fab500cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab500749f0 con 0x7fab5015c490 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab50074870 con 0x7fab500cb780 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab55e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fab500ce0c0 con 0x7fab34004d40 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab4f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab4f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49664/0 (socket says 172.21.15.31:49664) 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab4f400640 1 -- 172.21.15.31:0/2688307262 learned_addr learned my addr 172.21.15.31:0/2688307262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab4ea00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab34004d40 0x7fab501f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.330+0000 7fab4e000640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab500cb780 0x7fab500cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 -- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab34004d40 msgr2=0x7fab501f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab34004d40 0x7fab501f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 -- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab500cb780 msgr2=0x7fab500cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab500cb780 0x7fab500cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 -- 172.21.15.31:0/2688307262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab500c9d50 con 0x7fab5015c490 2024-09-13T08:44:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab4f400640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fab4800c730 tx=0x7fab48004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4802bc00 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab48006070 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab480064c0 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab500caca0 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.331+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab500cb260 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.333+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab500cdb80 con 0x7fab5015c490 2024-09-13T08:44:07.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.333+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fab48006660 con 0x7fab5015c490 2024-09-13T08:44:07.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.334+0000 7fab47400640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 0x7fab3007b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.335+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5939+0+0 (secure 0 0 0) 0x7fab480bb5d0 con 0x7fab5015c490 2024-09-13T08:44:07.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.335+0000 7fab4ea00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 0x7fab3007b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.335+0000 7fab4ea00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 0x7fab3007b1c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab40009df0 tx=0x7fab40005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.339+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fab48082c60 con 0x7fab5015c490 2024-09-13T08:44:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.571+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fab500cb430 con 0x7fab5015c490 2024-09-13T08:44:07.574 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:44:07.572+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:44:07.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.574+0000 7fab47400640 1 -- 172.21.15.31:0/2688307262 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) ==== 161+0+0 (secure 0 0 0) 0x7fab48087690 con 0x7fab5015c490 2024-09-13T08:44:07.576 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:44:07.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.577+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 msgr2=0x7fab3007b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.577+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 0x7fab3007b1c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab40009df0 tx=0x7fab40005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:07.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.577+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 msgr2=0x7fab5015c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.577+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fab4800c730 tx=0x7fab48004bb0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.577+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 shutdown_connections 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fab30078d00 0x7fab3007b1c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab5015c490 0x7fab5015c940 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab500cb780 0x7fab500cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 --2- 172.21.15.31:0/2688307262 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fab34004d40 0x7fab501f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 >> 172.21.15.31:0/2688307262 conn(0x7fab500b18f0 msgr2=0x7fab500737d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 shutdown_connections 2024-09-13T08:44:07.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.578+0000 7fab55e00640 1 -- 172.21.15.31:0/2688307262 wait complete. 2024-09-13T08:44:07.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:44:07.649 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:44:07.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.759+0000 7f8f92400640 1 -- 172.21.15.31:0/1711810936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c15c040 msgr2=0x7f8f8c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.759+0000 7f8f92400640 1 --2- 172.21.15.31:0/1711810936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0d0690 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c005fd0 tx=0x7f8f7c02a540 comp rx=0 tx=0).stop 2024-09-13T08:44:07.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 -- 172.21.15.31:0/1711810936 shutdown_connections 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 --2- 172.21.15.31:0/1711810936 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c072560 0x7f8f8c0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 --2- 172.21.15.31:0/1711810936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0d0690 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 -- 172.21.15.31:0/1711810936 >> 172.21.15.31:0/1711810936 conn(0x7f8f8c0cfb70 msgr2=0x7f8f8c0cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 -- 172.21.15.31:0/1711810936 shutdown_connections 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.760+0000 7f8f92400640 1 -- 172.21.15.31:0/1711810936 wait complete. 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.761+0000 7f8f92400640 1 Processor -- start 2024-09-13T08:44:07.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.761+0000 7f8f92400640 1 -- start start 2024-09-13T08:44:07.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 0x7f8f8c1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c06ff20 con 0x7f8f8c072560 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c06fda0 con 0x7f8f8c15c040 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f92400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8c0700a0 con 0x7f8f8c0cb570 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f8b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f8b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49682/0 (socket says 172.21.15.31:49682) 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.762+0000 7f8f8b400640 1 -- 172.21.15.31:0/1120133623 learned_addr learned my addr 172.21.15.31:0/1120133623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8be00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f82a00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 0x7f8f8c1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 -- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 msgr2=0x7f8f8c1f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 0x7f8f8c1f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 -- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 msgr2=0x7f8f8c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:07.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 -- 172.21.15.31:0/1120133623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8c0b0b10 con 0x7f8f8c072560 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8be00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f82a00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 0x7f8f8c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f8b400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c034040 tx=0x7f8f7c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c03e030 con 0x7f8f8c072560 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f7c0047a0 con 0x7f8f8c072560 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.763+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c035040 con 0x7f8f8c072560 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.764+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8c0c57b0 con 0x7f8f8c072560 2024-09-13T08:44:07.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.764+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8c0c4150 con 0x7f8f8c072560 2024-09-13T08:44:07.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.765+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8c0cbd10 con 0x7f8f8c072560 2024-09-13T08:44:07.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.766+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f8f7c041030 con 0x7f8f8c072560 2024-09-13T08:44:07.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.767+0000 7f8f83e00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 0x7f8f6807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:07.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.767+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5997+0+0 (secure 0 0 0) 0x7f8f7c0bbb90 con 0x7f8f8c072560 2024-09-13T08:44:07.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.767+0000 7f8f82a00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 0x7f8f6807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:07.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.768+0000 7f8f82a00640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 0x7f8f6807b190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0a95a0 tx=0x7f8f7400a5e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:07.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:07.771+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8f7c083910 con 0x7f8f8c072560 2024-09-13T08:44:08.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.009+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8f8c06f8e0 con 0x7f8f8c072560 2024-09-13T08:44:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.576+0000 7f8f83e00640 1 -- 172.21.15.31:0/1120133623 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) ==== 104+0+0 (secure 0 0 0) 0x7f8f7c083210 con 0x7f8f8c072560 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 msgr2=0x7f8f6807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 0x7f8f6807b190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0a95a0 tx=0x7f8f7400a5e0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 msgr2=0x7f8f8c1583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c034040 tx=0x7f8f7c0039e0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 shutdown_connections 2024-09-13T08:44:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8f68078cd0 0x7f8f6807b190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f8c15c040 0x7f8f8c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f8c0cb570 0x7f8f8c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 --2- 172.21.15.31:0/1120133623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f8c072560 0x7f8f8c1583b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 >> 172.21.15.31:0/1120133623 conn(0x7f8f8c0cfb70 msgr2=0x7f8f8c15e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 shutdown_connections 2024-09-13T08:44:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.579+0000 7f8f92400640 1 -- 172.21.15.31:0/1120133623 wait complete. 2024-09-13T08:44:08.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:44:08.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 -- 172.21.15.31:0/2585871580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa20004800 msgr2=0x7ffa20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 --2- 172.21.15.31:0/2585871580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa20004800 0x7ffa20004c80 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009b50 tx=0x7ffa20005830 comp rx=0 tx=0).stop 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa3be00640 1 -- 172.21.15.31:0/2585871580 reap_dead start 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 -- 172.21.15.31:0/2585871580 shutdown_connections 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 -- 172.21.15.31:0/2585871580 >> 172.21.15.31:0/2585871580 conn(0x7ffa3c0cfbb0 msgr2=0x7ffa3c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 -- 172.21.15.31:0/2585871580 shutdown_connections 2024-09-13T08:44:08.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.769+0000 7ffa42600640 1 -- 172.21.15.31:0/2585871580 wait complete. 2024-09-13T08:44:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.771+0000 7ffa42600640 1 Processor -- start 2024-09-13T08:44:08.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.771+0000 7ffa42600640 1 -- start start 2024-09-13T08:44:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa20004800 0x7ffa3c150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:08.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa3c15e160 0x7ffa3c15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c070020 con 0x7ffa3c160350 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c0cbe90 con 0x7ffa20004800 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa42600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c0ca370 con 0x7ffa3c15e160 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa32a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa32a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49694/0 (socket says 172.21.15.31:49694) 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa32a00640 1 -- 172.21.15.31:0/2253185154 learned_addr learned my addr 172.21.15.31:0/2253185154 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa3be00640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa3c15e160 0x7ffa3c15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.772+0000 7ffa3b400640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa20004800 0x7ffa3c150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 -- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa3c15e160 msgr2=0x7ffa3c15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa3c15e160 0x7ffa3c15e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 -- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa20004800 msgr2=0x7ffa3c150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:08.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa20004800 0x7ffa3c150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:08.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 -- 172.21.15.31:0/2253185154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c0c9f40 con 0x7ffa3c160350 2024-09-13T08:44:08.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa32a00640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ffa24036040 tx=0x7ffa24033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24043070 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa24004320 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24038040 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c0c93c0 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.773+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c0c9980 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.774+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c0cd780 con 0x7ffa3c160350 2024-09-13T08:44:08.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.776+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7ffa24046070 con 0x7ffa3c160350 2024-09-13T08:44:08.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.776+0000 7ffa33e00640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 0x7ffa1807b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:08.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.777+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5997+0+0 (secure 0 0 0) 0x7ffa240bb5e0 con 0x7ffa3c160350 2024-09-13T08:44:08.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.777+0000 7ffa3b400640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 0x7ffa1807b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:08.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.777+0000 7ffa3b400640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 0x7ffa1807b1c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c006b90 tx=0x7ffa2c0067a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:08.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.781+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffa24082d60 con 0x7ffa3c160350 2024-09-13T08:44:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:08.980+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffa3c0b5b30 con 0x7ffa3c160350 2024-09-13T08:44:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.790+0000 7ffa33e00640 1 -- 172.21.15.31:0/2253185154 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) ==== 181+0+0 (secure 0 0 0) 0x7ffa24087790 con 0x7ffa3c160350 2024-09-13T08:44:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:44:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.794+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 msgr2=0x7ffa1807b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.794+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 0x7ffa1807b1c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c006b90 tx=0x7ffa2c0067a0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.794+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 msgr2=0x7ffa3c15dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.794+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ffa24036040 tx=0x7ffa24033670 comp rx=0 tx=0).stop 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 shutdown_connections 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffa18078d00 0x7ffa1807b1c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffa3c15e160 0x7ffa3c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa3c160350 0x7ffa3c15dc20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 --2- 172.21.15.31:0/2253185154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa20004800 0x7ffa3c150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 >> 172.21.15.31:0/2253185154 conn(0x7ffa3c0cfbb0 msgr2=0x7ffa3c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 shutdown_connections 2024-09-13T08:44:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.795+0000 7ffa42600640 1 -- 172.21.15.31:0/2253185154 wait complete. 2024-09-13T08:44:09.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:44:09.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 -- 172.21.15.31:0/2022449923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb3415aca0 msgr2=0x7fdb340cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/2022449923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb3415aca0 0x7fdb340cf9a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009900 tx=0x7fdb2402a600 comp rx=0 tx=0).stop 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 -- 172.21.15.31:0/2022449923 shutdown_connections 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/2022449923 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340ceab0 0x7fdb340ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/2022449923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb3415aca0 0x7fdb340cf9a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 -- 172.21.15.31:0/2022449923 >> 172.21.15.31:0/2022449923 conn(0x7fdb340d0cd0 msgr2=0x7fdb340d10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.987+0000 7fdb3aa00640 1 -- 172.21.15.31:0/2022449923 shutdown_connections 2024-09-13T08:44:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.988+0000 7fdb3aa00640 1 -- 172.21.15.31:0/2022449923 wait complete. 2024-09-13T08:44:09.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.989+0000 7fdb3aa00640 1 Processor -- start 2024-09-13T08:44:09.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.989+0000 7fdb3aa00640 1 -- start start 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3415aca0 0x7fdb34149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340afe10 0x7fdb34157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3406f6d0 con 0x7fdb340ceab0 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb34073fc0 con 0x7fdb3415aca0 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb3aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdb3406f850 con 0x7fdb340afe10 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.990+0000 7fdb33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49714/0 (socket says 172.21.15.31:49714) 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 -- 172.21.15.31:0/768058936 learned_addr learned my addr 172.21.15.31:0/768058936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb2b400640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3415aca0 0x7fdb34149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb38a00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340afe10 0x7fdb34157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 -- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340afe10 msgr2=0x7fdb34157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:09.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340afe10 0x7fdb34157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 -- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3415aca0 msgr2=0x7fdb34149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3415aca0 0x7fdb34149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 -- 172.21.15.31:0/768058936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3414a2a0 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.991+0000 7fdb33e00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdb24000c00 tx=0x7fdb24003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.992+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2403e030 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.992+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb24003e60 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.992+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24035040 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.992+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb34157160 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.992+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb34157420 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.993+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb3406fd40 con 0x7fdb340ceab0 2024-09-13T08:44:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.994+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fdb24041030 con 0x7fdb340ceab0 2024-09-13T08:44:09.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.995+0000 7fdb30c00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 0x7fdb1007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:09.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.996+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5502+0+0 (secure 0 0 0) 0x7fdb240bbb50 con 0x7fdb340ceab0 2024-09-13T08:44:09.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.996+0000 7fdb2b400640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 0x7fdb1007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:09.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:09.996+0000 7fdb2b400640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 0x7fdb1007b190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406d420 tx=0x7fdb1c00a2d0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:10.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.000+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdb240834c0 con 0x7fdb340ceab0 2024-09-13T08:44:10.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.195+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdb34074880 con 0x7fdb340ceab0 2024-09-13T08:44:10.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.798+0000 7fdb30c00640 1 -- 172.21.15.31:0/768058936 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v115) ==== 169+0+0 (secure 0 0 0) 0x7fdb24087ef0 con 0x7fdb340ceab0 2024-09-13T08:44:10.800 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 msgr2=0x7fdb1007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 0x7fdb1007b190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406d420 tx=0x7fdb1c00a2d0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 msgr2=0x7fdb34159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdb24000c00 tx=0x7fdb24003970 comp rx=0 tx=0).stop 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 shutdown_connections 2024-09-13T08:44:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb10078cd0 0x7fdb1007b190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb340afe10 0x7fdb34157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3415aca0 0x7fdb34149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 --2- 172.21.15.31:0/768058936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb340ceab0 0x7fdb34159e00 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.803+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 >> 172.21.15.31:0/768058936 conn(0x7fdb340d0cd0 msgr2=0x7fdb340b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.804+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 shutdown_connections 2024-09-13T08:44:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.804+0000 7fdb3aa00640 1 -- 172.21.15.31:0/768058936 wait complete. 2024-09-13T08:44:10.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:44:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 -- 172.21.15.31:0/1819687430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7c4003a50 msgr2=0x7fc7c4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/1819687430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7c4003a50 0x7fc7c4003e70 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009b50 tx=0x7fc7c4005830 comp rx=0 tx=0).stop 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7df400640 1 -- 172.21.15.31:0/1819687430 reap_dead start 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 -- 172.21.15.31:0/1819687430 shutdown_connections 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 -- 172.21.15.31:0/1819687430 >> 172.21.15.31:0/1819687430 conn(0x7fc7e00b18f0 msgr2=0x7fc7e00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 -- 172.21.15.31:0/1819687430 shutdown_connections 2024-09-13T08:44:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.990+0000 7fc7e5c00640 1 -- 172.21.15.31:0/1819687430 wait complete. 2024-09-13T08:44:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.992+0000 7fc7e5c00640 1 Processor -- start 2024-09-13T08:44:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.992+0000 7fc7e5c00640 1 -- start start 2024-09-13T08:44:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c4003a50 0x7fc7e014a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7e014aa10 0x7fc7e009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e00b3d40 con 0x7fc7e01f8910 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e00b3bc0 con 0x7fc7c4003a50 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7e5c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e00b3ec0 con 0x7fc7e014aa10 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49718/0 (socket says 172.21.15.31:49718) 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7df400640 1 -- 172.21.15.31:0/2852269159 learned_addr learned my addr 172.21.15.31:0/2852269159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7d6000640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7e014aa10 0x7fc7e009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.993+0000 7fc7dea00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c4003a50 0x7fc7e014a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 -- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7e014aa10 msgr2=0x7fc7e009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7e014aa10 0x7fc7e009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 -- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c4003a50 msgr2=0x7fc7e014a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c4003a50 0x7fc7e014a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 -- 172.21.15.31:0/2852269159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e01f9260 con 0x7fc7e01f8910 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7df400640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005fd0 tx=0x7fc7d8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d800fd10 con 0x7fc7e01f8910 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7d8004b20 con 0x7fc7e01f8910 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e01f9540 con 0x7fc7e01f8910 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.994+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8003970 con 0x7fc7e01f8910 2024-09-13T08:44:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.995+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e009af50 con 0x7fc7e01f8910 2024-09-13T08:44:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.996+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7e01f9780 con 0x7fc7e01f8910 2024-09-13T08:44:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.997+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fc7d8020020 con 0x7fc7e01f8910 2024-09-13T08:44:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.998+0000 7fc7d7400640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 0x7fc7bc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.998+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7fc7d809b5c0 con 0x7fc7e01f8910 2024-09-13T08:44:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.998+0000 7fc7dea00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 0x7fc7bc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:10.999+0000 7fc7dea00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 0x7fc7bc07b190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0035000 tx=0x7fc7d002d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.002+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc7e01f9780 con 0x7fc7e01f8910 2024-09-13T08:44:11.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.200+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fc7e006e660 con 0x7fc7e01f8910 2024-09-13T08:44:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.203+0000 7fc7d7400640 1 -- 172.21.15.31:0/2852269159 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7fc7d8063110 con 0x7fc7e01f8910 2024-09-13T08:44:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 msgr2=0x7fc7bc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 0x7fc7bc07b190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0035000 tx=0x7fc7d002d040 comp rx=0 tx=0).stop 2024-09-13T08:44:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 msgr2=0x7fc7e0152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005fd0 tx=0x7fc7d8011f90 comp rx=0 tx=0).stop 2024-09-13T08:44:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 shutdown_connections 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc7bc078cd0 0x7fc7bc07b190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7e01f8910 0x7fc7e0152ec0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc7e014aa10 0x7fc7e009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 --2- 172.21.15.31:0/2852269159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c4003a50 0x7fc7e014a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 >> 172.21.15.31:0/2852269159 conn(0x7fc7e00b18f0 msgr2=0x7fc7e015feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 shutdown_connections 2024-09-13T08:44:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.206+0000 7fc7e5c00640 1 -- 172.21.15.31:0/2852269159 wait complete. 2024-09-13T08:44:11.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.382+0000 7fcc2d800640 1 -- 172.21.15.31:0/2781280587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 msgr2=0x7fcc280b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.382+0000 7fcc2d800640 1 --2- 172.21.15.31:0/2781280587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b1da0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcc20005fd0 tx=0x7fcc2002aa90 comp rx=0 tx=0).stop 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 -- 172.21.15.31:0/2781280587 shutdown_connections 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 --2- 172.21.15.31:0/2781280587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b1da0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 --2- 172.21.15.31:0/2781280587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 0x7fcc280cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 -- 172.21.15.31:0/2781280587 >> 172.21.15.31:0/2781280587 conn(0x7fcc280cc910 msgr2=0x7fcc280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:11.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 -- 172.21.15.31:0/2781280587 shutdown_connections 2024-09-13T08:44:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.383+0000 7fcc2d800640 1 -- 172.21.15.31:0/2781280587 wait complete. 2024-09-13T08:44:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.384+0000 7fcc2d800640 1 Processor -- start 2024-09-13T08:44:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.384+0000 7fcc2d800640 1 -- start start 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 0x7fcc280c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 0x7fcc280c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28073b10 con 0x7fcc280b1910 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28073990 con 0x7fcc280ca5c0 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.385+0000 7fcc2d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcc28073c90 con 0x7fcc2815dfc0 2024-09-13T08:44:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49740/0 (socket says 172.21.15.31:49740) 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 -- 172.21.15.31:0/3892481211 learned_addr learned my addr 172.21.15.31:0/3892481211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc27400640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 0x7fcc280c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26000640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 0x7fcc280c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 -- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 msgr2=0x7fcc280c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 0x7fcc280c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 -- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 msgr2=0x7fcc280c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 0x7fcc280c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 -- 172.21.15.31:0/3892481211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc280b2d60 con 0x7fcc280b1910 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26000640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 0x7fcc280c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.386+0000 7fcc26a00640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d740 tx=0x7fcc1800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.387+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18017070 con 0x7fcc280b1910 2024-09-13T08:44:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.387+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc18009d20 con 0x7fcc280b1910 2024-09-13T08:44:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.387+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc280b3000 con 0x7fcc280b1910 2024-09-13T08:44:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.387+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc180106c0 con 0x7fcc280b1910 2024-09-13T08:44:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.387+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc280b35c0 con 0x7fcc280b1910 2024-09-13T08:44:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.388+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc280b1da0 con 0x7fcc280b1910 2024-09-13T08:44:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.390+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fcc1801a070 con 0x7fcc280b1910 2024-09-13T08:44:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.391+0000 7fcc1ea00640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 0x7fcc0807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.391+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7fcc1809b080 con 0x7fcc280b1910 2024-09-13T08:44:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.395+0000 7fcc26000640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 0x7fcc0807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.398+0000 7fcc26000640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 0x7fcc0807b190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc14007fd0 tx=0x7fcc14009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:11.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.403+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcc18062b90 con 0x7fcc280b1910 2024-09-13T08:44:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.669+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7fcc280af5d0 con 0x7fcc280b1910 2024-09-13T08:44:11.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.804+0000 7fcc1ea00640 1 -- 172.21.15.31:0/3892481211 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v383) ==== 149+0+0 (secure 0 0 0) 0x7fcc180675c0 con 0x7fcc280b1910 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 msgr2=0x7fcc0807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 0x7fcc0807b190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc14007fd0 tx=0x7fcc14009040 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 msgr2=0x7fcc280b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d740 tx=0x7fcc1800dc00 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 shutdown_connections 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcc08078cd0 0x7fcc0807b190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcc2815dfc0 0x7fcc280c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc280ca5c0 0x7fcc280c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.806+0000 7fcc2d800640 1 --2- 172.21.15.31:0/3892481211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc280b1910 0x7fcc280b4860 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.807+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 >> 172.21.15.31:0/3892481211 conn(0x7fcc280cc910 msgr2=0x7fcc2815d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.807+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 shutdown_connections 2024-09-13T08:44:11.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.807+0000 7fcc2d800640 1 -- 172.21.15.31:0/3892481211 wait complete. 2024-09-13T08:44:11.825 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-13T08:44:11.825 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-13T08:44:11.825 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:44:11.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.985+0000 7f19ea600640 1 -- 172.21.15.31:0/2006955916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cc3e0 msgr2=0x7f19e40cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.985+0000 7f19ea600640 1 --2- 172.21.15.31:0/2006955916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cc3e0 0x7f19e40cc800 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f19e40b2640 tx=0x7f19dc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 -- 172.21.15.31:0/2006955916 shutdown_connections 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 --2- 172.21.15.31:0/2006955916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cc3e0 0x7f19e40cc800 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 --2- 172.21.15.31:0/2006955916 >> v2:172.21.15.45:3300/0 conn(0x7f19e40caa00 0x7f19e4161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 -- 172.21.15.31:0/2006955916 >> 172.21.15.31:0/2006955916 conn(0x7f19e406cb00 msgr2=0x7f19e406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 -- 172.21.15.31:0/2006955916 shutdown_connections 2024-09-13T08:44:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.986+0000 7f19ea600640 1 -- 172.21.15.31:0/2006955916 wait complete. 2024-09-13T08:44:11.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.987+0000 7f19ea600640 1 Processor -- start 2024-09-13T08:44:11.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.987+0000 7f19ea600640 1 -- start start 2024-09-13T08:44:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19e40caa00 0x7f19e40b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e40cc3e0 0x7f19e40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19e40ce3c0 con 0x7f19e40cf2f0 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19e40ce240 con 0x7f19e40cc3e0 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.988+0000 7f19ea600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f19e40ce540 con 0x7f19e40caa00 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49746/0 (socket says 172.21.15.31:49746) 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 -- 172.21.15.31:0/3538316839 learned_addr learned my addr 172.21.15.31:0/3538316839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9200640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19e40caa00 0x7f19e40b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e3e00640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e40cc3e0 0x7f19e40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 -- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19e40caa00 msgr2=0x7f19e40b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19e40caa00 0x7f19e40b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 -- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e40cc3e0 msgr2=0x7f19e40b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e40cc3e0 0x7f19e40b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 -- 172.21.15.31:0/3538316839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e40c98a0 con 0x7f19e40cf2f0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.989+0000 7f19e9c00640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ca0 tx=0x7f19dc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.990+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc040070 con 0x7f19e40cf2f0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.990+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19dc004130 con 0x7f19e40cf2f0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.990+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc004560 con 0x7f19e40cf2f0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.990+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e40b2b10 con 0x7f19e40cf2f0 2024-09-13T08:44:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.990+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e40b30d0 con 0x7f19e40cf2f0 2024-09-13T08:44:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.991+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e40cea20 con 0x7f19e40cf2f0 2024-09-13T08:44:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.992+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f19dc039040 con 0x7f19e40cf2f0 2024-09-13T08:44:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.993+0000 7f19e0c00640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 0x7f19c407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.993+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7f19dc0bc570 con 0x7f19e40cf2f0 2024-09-13T08:44:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.993+0000 7f19e9200640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 0x7f19c407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.994+0000 7f19e9200640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 0x7f19c407b190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f19d400bbe0 tx=0x7f19d400b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:11.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:11.997+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f19dc0847a0 con 0x7f19e40cf2f0 2024-09-13T08:44:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.193+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19e40707a0 con 0x7f19e40cf2f0 2024-09-13T08:44:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.194+0000 7f19e0c00640 1 -- 172.21.15.31:0/3538316839 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+12994 (secure 0 0 0) 0x7f19dc0840a0 con 0x7f19e40cf2f0 2024-09-13T08:44:12.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:12.196 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":115,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:10.792018+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/3014570213":"2024-09-14T08:44:07.573453+0000","172.21.15.45:6800/3014570213":"2024-09-14T08:44:07.573453+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 msgr2=0x7f19c407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 0x7f19c407b190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f19d400bbe0 tx=0x7f19d400b390 comp rx=0 tx=0).stop 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 msgr2=0x7f19e40c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ca0 tx=0x7f19dc003970 comp rx=0 tx=0).stop 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 shutdown_connections 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f19c4078cd0 0x7f19c407b190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19e40cf2f0 0x7f19e40c9360 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e40cc3e0 0x7f19e40b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 --2- 172.21.15.31:0/3538316839 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19e40caa00 0x7f19e40b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 >> 172.21.15.31:0/3538316839 conn(0x7f19e406cb00 msgr2=0x7f19e406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 shutdown_connections 2024-09-13T08:44:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.197+0000 7f19ea600640 1 -- 172.21.15.31:0/3538316839 wait complete. 2024-09-13T08:44:12.215 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:44:12.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:44:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 -- 172.21.15.31:0/3396235555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c15abd0 msgr2=0x7fe91c0ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 --2- 172.21.15.31:0/3396235555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c15abd0 0x7fe91c0ca2e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe90c005fd0 tx=0x7fe90c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 -- 172.21.15.31:0/3396235555 shutdown_connections 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 --2- 172.21.15.31:0/3396235555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c072a50 0x7fe91c06f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 --2- 172.21.15.31:0/3396235555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c15abd0 0x7fe91c0ca2e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 -- 172.21.15.31:0/3396235555 >> 172.21.15.31:0/3396235555 conn(0x7fe91c0cf9b0 msgr2=0x7fe91c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 -- 172.21.15.31:0/3396235555 shutdown_connections 2024-09-13T08:44:12.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.379+0000 7fe91be00640 1 -- 172.21.15.31:0/3396235555 wait complete. 2024-09-13T08:44:12.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.381+0000 7fe91be00640 1 Processor -- start 2024-09-13T08:44:12.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.381+0000 7fe91be00640 1 -- start start 2024-09-13T08:44:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 0x7fe91c1f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 0x7fe91c155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c06d210 con 0x7fe91c072a50 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c0cf380 con 0x7fe91c06fa90 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe91c06d390 con 0x7fe91c15abd0 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49752/0 (socket says 172.21.15.31:49752) 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91a000640 1 -- 172.21.15.31:0/3642813753 learned_addr learned my addr 172.21.15.31:0/3642813753 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91b400640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 0x7fe91c155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91aa00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 0x7fe91c1f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.382+0000 7fe91a000640 1 -- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 msgr2=0x7fe91c155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91a000640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 0x7fe91c155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91a000640 1 -- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 msgr2=0x7fe91c1f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91a000640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 0x7fe91c1f7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91a000640 1 -- 172.21.15.31:0/3642813753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c1f8310 con 0x7fe91c072a50 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91b400640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 0x7fe91c155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91aa00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 0x7fe91c1f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91a000640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9f0 tx=0x7fe90800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90800cd80 con 0x7fe91c072a50 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91c1f85f0 con 0x7fe91c072a50 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe908010430 con 0x7fe91c072a50 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908010840 con 0x7fe91c072a50 2024-09-13T08:44:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.383+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91c155e40 con 0x7fe91c072a50 2024-09-13T08:44:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.384+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91c1f8b10 con 0x7fe91c072a50 2024-09-13T08:44:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.385+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fe9080109e0 con 0x7fe91c072a50 2024-09-13T08:44:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.386+0000 7fe912a00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 0x7fe8fc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.387+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7fe90809bb30 con 0x7fe91c072a50 2024-09-13T08:44:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.387+0000 7fe91aa00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 0x7fe8fc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.387+0000 7fe91aa00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 0x7fe8fc07b190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe90c033000 tx=0x7fe90c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.391+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe91c1f8b10 con 0x7fe91c072a50 2024-09-13T08:44:12.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.587+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe91c06d8e0 con 0x7fe91c072a50 2024-09-13T08:44:12.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.587+0000 7fe912a00640 1 -- 172.21.15.31:0/3642813753 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7fe908063d60 con 0x7fe91c072a50 2024-09-13T08:44:12.590 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:44:12.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 msgr2=0x7fe8fc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 0x7fe8fc07b190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe90c033000 tx=0x7fe90c004730 comp rx=0 tx=0).stop 2024-09-13T08:44:12.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 msgr2=0x7fe91c0c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9f0 tx=0x7fe90800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 shutdown_connections 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe8fc078cd0 0x7fe8fc07b190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe91c15abd0 0x7fe91c155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe91c072a50 0x7fe91c0c9640 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 --2- 172.21.15.31:0/3642813753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe91c06fa90 0x7fe91c1f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 >> 172.21.15.31:0/3642813753 conn(0x7fe91c0cf9b0 msgr2=0x7fe91c0b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 shutdown_connections 2024-09-13T08:44:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.590+0000 7fe91be00640 1 -- 172.21.15.31:0/3642813753 wait complete. 2024-09-13T08:44:12.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 -- 172.21.15.31:0/4188276905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 msgr2=0x7f1b940ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 --2- 172.21.15.31:0/4188276905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b940ceea0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1b940cba80 tx=0x7f1b8402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 -- 172.21.15.31:0/4188276905 shutdown_connections 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 --2- 172.21.15.31:0/4188276905 >> v2:172.21.15.45:3300/0 conn(0x7f1b9406cb00 0x7f1b9406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 --2- 172.21.15.31:0/4188276905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b940ceea0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 -- 172.21.15.31:0/4188276905 >> 172.21.15.31:0/4188276905 conn(0x7f1b940cda20 msgr2=0x7f1b940cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.768+0000 7f1b99c00640 1 -- 172.21.15.31:0/4188276905 shutdown_connections 2024-09-13T08:44:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.769+0000 7f1b99c00640 1 -- 172.21.15.31:0/4188276905 wait complete. 2024-09-13T08:44:12.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.770+0000 7f1b99c00640 1 Processor -- start 2024-09-13T08:44:12.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.770+0000 7f1b99c00640 1 -- start start 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 0x7f1b94159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 0x7f1b941f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94073f60 con 0x7f1b9415bc00 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94073de0 con 0x7f1b9406cb00 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b99c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1b940740e0 con 0x7f1b940cb5f0 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49776/0 (socket says 172.21.15.31:49776) 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.771+0000 7f1b93400640 1 -- 172.21.15.31:0/2099649749 learned_addr learned my addr 172.21.15.31:0/2099649749 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:12.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 -- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 msgr2=0x7f1b941f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b92a00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 0x7f1b94159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b8a000640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 0x7f1b941f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 0x7f1b941f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 -- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 msgr2=0x7f1b94159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 0x7f1b94159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 -- 172.21.15.31:0/2099649749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941f5b00 con 0x7f1b9415bc00 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b92a00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 0x7f1b94159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b8a000640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 0x7f1b941f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b93400640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00d9e0 tx=0x7f1b8c00dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c017070 con 0x7f1b9415bc00 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.772+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b8c00be30 con 0x7f1b9415bc00 2024-09-13T08:44:12.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.773+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c003bb0 con 0x7f1b9415bc00 2024-09-13T08:44:12.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.773+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941f5de0 con 0x7f1b9415bc00 2024-09-13T08:44:12.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.773+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941f8820 con 0x7f1b9415bc00 2024-09-13T08:44:12.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.774+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94074590 con 0x7f1b9415bc00 2024-09-13T08:44:12.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.775+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f1b8c010460 con 0x7f1b9415bc00 2024-09-13T08:44:12.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.776+0000 7f1b8b400640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 0x7f1b7007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:12.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.776+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7f1b8c09b390 con 0x7f1b9415bc00 2024-09-13T08:44:12.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.776+0000 7f1b92a00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 0x7f1b7007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:12.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.776+0000 7f1b92a00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 0x7f1b7007b190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b84033000 tx=0x7f1b84004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:12.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:12.780+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1b8c063580 con 0x7f1b9415bc00 2024-09-13T08:44:13.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.049+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7f1b940b6cb0 con 0x7f1b9415bc00 2024-09-13T08:44:13.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.811+0000 7f1b8b400640 1 -- 172.21.15.31:0/2099649749 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v385) ==== 151+0+0 (secure 0 0 0) 0x7f1b8c062e80 con 0x7f1b9415bc00 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.814+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 msgr2=0x7f1b7007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.814+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 0x7f1b7007b190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b84033000 tx=0x7f1b84004730 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.814+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 msgr2=0x7f1b941f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.814+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00d9e0 tx=0x7f1b8c00dea0 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 shutdown_connections 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1b70078cd0 0x7f1b7007b190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b9415bc00 0x7f1b941f51b0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1b940cb5f0 0x7f1b941f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 --2- 172.21.15.31:0/2099649749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9406cb00 0x7f1b94159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:13.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 >> 172.21.15.31:0/2099649749 conn(0x7f1b940cda20 msgr2=0x7f1b940ae3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 shutdown_connections 2024-09-13T08:44:13.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.815+0000 7f1b99c00640 1 -- 172.21.15.31:0/2099649749 wait complete. 2024-09-13T08:44:13.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 -- 172.21.15.31:0/802571186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8180ca5c0 msgr2=0x7fe8180cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 --2- 172.21.15.31:0/802571186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8180ca5c0 0x7fe8180cc300 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe804009830 tx=0x7fe80402a510 comp rx=0 tx=0).stop 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 -- 172.21.15.31:0/802571186 shutdown_connections 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 --2- 172.21.15.31:0/802571186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8180ca5c0 0x7fe8180cc300 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 -- 172.21.15.31:0/802571186 >> 172.21.15.31:0/802571186 conn(0x7fe8180cc910 msgr2=0x7fe8180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 -- 172.21.15.31:0/802571186 shutdown_connections 2024-09-13T08:44:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:13.999+0000 7fe81f000640 1 -- 172.21.15.31:0/802571186 wait complete. 2024-09-13T08:44:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.001+0000 7fe81f000640 1 Processor -- start 2024-09-13T08:44:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.001+0000 7fe81f000640 1 -- start start 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8180b1910 0x7fe818152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8180ca5c0 0x7fe8180c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe8180b1d60 con 0x7fe81815dfc0 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe8180740a0 con 0x7fe8180b1910 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe8180b1ee0 con 0x7fe8180ca5c0 2024-09-13T08:44:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49792/0 (socket says 172.21.15.31:49792) 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.002+0000 7fe81d000640 1 -- 172.21.15.31:0/3922109629 learned_addr learned my addr 172.21.15.31:0/3922109629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe817400640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8180ca5c0 0x7fe8180c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe817e00640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8180b1910 0x7fe818152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 -- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8180ca5c0 msgr2=0x7fe8180c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8180ca5c0 0x7fe8180c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 -- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8180b1910 msgr2=0x7fe818152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8180b1910 0x7fe818152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 -- 172.21.15.31:0/3922109629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8180ca160 con 0x7fe81815dfc0 2024-09-13T08:44:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe81d000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe81000cbc0 tx=0x7fe81000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810016020 con 0x7fe81815dfc0 2024-09-13T08:44:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe81000ae30 con 0x7fe81815dfc0 2024-09-13T08:44:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.003+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810012660 con 0x7fe81815dfc0 2024-09-13T08:44:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.004+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8180c7970 con 0x7fe81815dfc0 2024-09-13T08:44:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.004+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8180c8210 con 0x7fe81815dfc0 2024-09-13T08:44:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.005+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8180c7d60 con 0x7fe81815dfc0 2024-09-13T08:44:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.006+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fe810019020 con 0x7fe81815dfc0 2024-09-13T08:44:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.007+0000 7fe80fe00640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 0x7fe7f807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.007+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7fe81009b230 con 0x7fe81815dfc0 2024-09-13T08:44:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.007+0000 7fe817e00640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 0x7fe7f807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.008+0000 7fe817e00640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 0x7fe7f807b190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe808006fd0 tx=0x7fe808009060 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:14.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.011+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe810020030 con 0x7fe81815dfc0 2024-09-13T08:44:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.207+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fe8180cc300 con 0x7fe81815dfc0 2024-09-13T08:44:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.208+0000 7fe80fe00640 1 -- 172.21.15.31:0/3922109629 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v39) ==== 59+0+1998 (secure 0 0 0) 0x7fe8180cbe40 con 0x7fe81815dfc0 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:44:14.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:44:14.211 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:44:14.212 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:44:14.212 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:44:14.212 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:44:14.212 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 msgr2=0x7fe7f807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 0x7fe7f807b190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe808006fd0 tx=0x7fe808009060 comp rx=0 tx=0).stop 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 msgr2=0x7fe818151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe81000cbc0 tx=0x7fe81000a430 comp rx=0 tx=0).stop 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 shutdown_connections 2024-09-13T08:44:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7f8078cd0 0x7fe7f807b190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81815dfc0 0x7fe818151a60 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe8180ca5c0 0x7fe8180c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 --2- 172.21.15.31:0/3922109629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8180b1910 0x7fe818152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 >> 172.21.15.31:0/3922109629 conn(0x7fe8180cc910 msgr2=0x7fe8180cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 shutdown_connections 2024-09-13T08:44:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.210+0000 7fe81f000640 1 -- 172.21.15.31:0/3922109629 wait complete. 2024-09-13T08:44:14.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:44:14.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 -- 172.21.15.31:0/3511344285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cde90 msgr2=0x7fba000ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 --2- 172.21.15.31:0/3511344285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cde90 0x7fba000ca370 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8005fd0 tx=0x7fb9e802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 -- 172.21.15.31:0/3511344285 shutdown_connections 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 --2- 172.21.15.31:0/3511344285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cde90 0x7fba000ca370 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 -- 172.21.15.31:0/3511344285 >> 172.21.15.31:0/3511344285 conn(0x7fba000cfbb0 msgr2=0x7fba000cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 -- 172.21.15.31:0/3511344285 shutdown_connections 2024-09-13T08:44:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.389+0000 7fba05a00640 1 -- 172.21.15.31:0/3511344285 wait complete. 2024-09-13T08:44:14.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.391+0000 7fba05a00640 1 Processor -- start 2024-09-13T08:44:14.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.391+0000 7fba05a00640 1 -- start start 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 0x7fba0015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 0x7fba001f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba000cbfd0 con 0x7fba000cc190 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba000cbe50 con 0x7fba00161480 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fba05a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fba000cafc0 con 0x7fba000cde90 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.392+0000 7fb9fea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49812/0 (socket says 172.21.15.31:49812) 2024-09-13T08:44:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 -- 172.21.15.31:0/1632669555 learned_addr learned my addr 172.21.15.31:0/1632669555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9f6000640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 0x7fba0015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9ff400640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 0x7fba001f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 -- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 msgr2=0x7fba0015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 0x7fba0015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 -- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 msgr2=0x7fba001f8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 0x7fba001f8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 -- 172.21.15.31:0/1632669555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0015c0b0 con 0x7fba000cc190 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9f6000640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 0x7fba0015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9ff400640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 0x7fba001f8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9fea00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c400 tx=0x7fb9f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:14.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.393+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0017070 con 0x7fba000cc190 2024-09-13T08:44:14.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.394+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9f00053c0 con 0x7fba000cc190 2024-09-13T08:44:14.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.394+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0015c390 con 0x7fba000cc190 2024-09-13T08:44:14.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.394+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0003850 con 0x7fba000cc190 2024-09-13T08:44:14.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.394+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba000b0190 con 0x7fba000cc190 2024-09-13T08:44:14.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.395+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba00074860 con 0x7fba000cc190 2024-09-13T08:44:14.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.396+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fb9f001a070 con 0x7fba000cc190 2024-09-13T08:44:14.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.397+0000 7fb9f7400640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 0x7fb9dc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:14.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.398+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7fb9f009a290 con 0x7fba000cc190 2024-09-13T08:44:14.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.398+0000 7fb9f6000640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 0x7fb9dc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:14.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.399+0000 7fb9f6000640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 0x7fb9dc07b190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8033000 tx=0x7fb9e8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:14.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.402+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fba00074860 con 0x7fba000cc190 2024-09-13T08:44:14.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.596+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fba000cf090 con 0x7fba000cc190 2024-09-13T08:44:14.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.600+0000 7fb9f7400640 1 -- 172.21.15.31:0/1632669555 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7fba000cf090 con 0x7fba000cc190 2024-09-13T08:44:14.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 msgr2=0x7fb9dc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 0x7fb9dc07b190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8033000 tx=0x7fb9e8004730 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 msgr2=0x7fba00159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c400 tx=0x7fb9f000c8c0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 shutdown_connections 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb9dc078cd0 0x7fb9dc07b190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba00161480 0x7fba001f8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fba000cde90 0x7fba0015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 --2- 172.21.15.31:0/1632669555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba000cc190 0x7fba00159550 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 >> 172.21.15.31:0/1632669555 conn(0x7fba000cfbb0 msgr2=0x7fba00074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 shutdown_connections 2024-09-13T08:44:14.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:14.602+0000 7fba05a00640 1 -- 172.21.15.31:0/1632669555 wait complete. 2024-09-13T08:44:14.620 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:44:14.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:14.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:14.808 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:44:14.808 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:14.808 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:14.845 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:14.845 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:14.845 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:14.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:14.883 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:14.883 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:14.920 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:14.920 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:14.920 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:14.961 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:14.961 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:44:14.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94bae00640 1 -- 172.21.15.31:0/1321066413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9498004800 msgr2=0x7f9498004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94bae00640 1 --2- 172.21.15.31:0/1321066413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9498004800 0x7f9498004c80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c340 tx=0x7f9498005830 comp rx=0 tx=0).stop 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94b8e00640 1 -- 172.21.15.31:0/1321066413 reap_dead start 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94bae00640 1 -- 172.21.15.31:0/1321066413 shutdown_connections 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94bae00640 1 -- 172.21.15.31:0/1321066413 >> 172.21.15.31:0/1321066413 conn(0x7f94b40cc910 msgr2=0x7f94b40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.121+0000 7f94bae00640 1 -- 172.21.15.31:0/1321066413 shutdown_connections 2024-09-13T08:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.122+0000 7f94bae00640 1 -- 172.21.15.31:0/1321066413 wait complete. 2024-09-13T08:44:15.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.123+0000 7f94bae00640 1 Processor -- start 2024-09-13T08:44:15.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.123+0000 7f94bae00640 1 -- start start 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9498004800 0x7f94b4156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 0x7f94b40c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94b40cc6c0 con 0x7f94b4153230 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94b40cc540 con 0x7f9498004800 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94bae00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f94b4073780 con 0x7f94b40b2cc0 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94b8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94b8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59876/0 (socket says 172.21.15.31:59876) 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.124+0000 7f94b8e00640 1 -- 172.21.15.31:0/2891940166 learned_addr learned my addr 172.21.15.31:0/2891940166 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b3e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9498004800 0x7f94b4156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94ab400640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 0x7f94b40c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 -- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 msgr2=0x7f94b40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 0x7f94b40c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 -- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9498004800 msgr2=0x7f94b4156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9498004800 0x7f94b4156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 -- 172.21.15.31:0/2891940166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b40b5c30 con 0x7f94b4153230 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94ab400640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 0x7f94b40c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b8e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f94ac010c80 tx=0x7f94ac00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac004d20 con 0x7f94b4153230 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94ac015030 con 0x7f94b4153230 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b40c8200 con 0x7f94b4153230 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.125+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac005280 con 0x7f94b4153230 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.126+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b40b5950 con 0x7f94b4153230 2024-09-13T08:44:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.127+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b4073ce0 con 0x7f94b4153230 2024-09-13T08:44:15.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.128+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f94ac005420 con 0x7f94b4153230 2024-09-13T08:44:15.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.129+0000 7f94b0c00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 0x7f949007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.129+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7f94ac09bac0 con 0x7f94b4153230 2024-09-13T08:44:15.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.129+0000 7f94b3e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 0x7f949007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.130+0000 7f94b3e00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 0x7f949007b190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006fd0 tx=0x7f94a4006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:15.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.133+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f94b40b5c30 con 0x7f94b4153230 2024-09-13T08:44:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.365+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94b40af5d0 con 0x7f94b4153230 2024-09-13T08:44:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.365+0000 7f94b0c00640 1 -- 172.21.15.31:0/2891940166 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+3908 (secure 0 0 0) 0x7f94ac063610 con 0x7f94b4153230 2024-09-13T08:44:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:15.368 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":74,"btime":"2024-09-13T08:44:08:575833+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6077,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/530940617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":530940617},{"type":"v1","addr":"172.21.15.31:6835","nonce":530940617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6086,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2355808317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2355808317},{"type":"v1","addr":"172.21.15.45:6835","nonce":2355808317}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/3971064995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3971064995},{"type":"v1","addr":"172.21.15.45:6801","nonce":3971064995}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74},{"gid":6951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1483642750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1483642750},{"type":"v1","addr":"172.21.15.31:6833","nonce":1483642750}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[]} 2024-09-13T08:44:15.368 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 74 2024-09-13T08:44:15.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 msgr2=0x7f949007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 0x7f949007b190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006fd0 tx=0x7f94a4006b10 comp rx=0 tx=0).stop 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 msgr2=0x7f94b4072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f94ac010c80 tx=0x7f94ac00e720 comp rx=0 tx=0).stop 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 shutdown_connections 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9490078cd0 0x7f949007b190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4153230 0x7f94b4072670 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f94b40b2cc0 0x7f94b40c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 --2- 172.21.15.31:0/2891940166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9498004800 0x7f94b4156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 >> 172.21.15.31:0/2891940166 conn(0x7f94b40cc910 msgr2=0x7f94b415d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:15.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 shutdown_connections 2024-09-13T08:44:15.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.368+0000 7f94bae00640 1 -- 172.21.15.31:0/2891940166 wait complete. 2024-09-13T08:44:15.387 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:44:15.387 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:44:15.388 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:44:15.388 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:15.388 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:44:15.388 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:44:15.388 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:44:15.388 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:15.388 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:44:15.388 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:44:15.389 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:44:15.389 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:15.389 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:44:15.389 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:44:15.389 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:44:15.389 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:15.402 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:44:15.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:44:15.403 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:44:15.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:44:15.406 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:44:15.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:44:15.407 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:44:15.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:44:15.446 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:44:15.447 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:15.449 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:44:15.449 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:15.453 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:44:15.453 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:15.455 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:44:15.456 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:15.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:44:15.581 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:44:15.584 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:44:15.587 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:44:15.587 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:44:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 -- 172.21.15.31:0/3815843451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e88151e00 msgr2=0x7f9e8815b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 --2- 172.21.15.31:0/3815843451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e88151e00 0x7f9e8815b3d0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f9e88158f10 tx=0x7f9e7402a530 comp rx=0 tx=0).stop 2024-09-13T08:44:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 -- 172.21.15.31:0/3815843451 shutdown_connections 2024-09-13T08:44:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 --2- 172.21.15.31:0/3815843451 >> v2:172.21.15.45:3300/0 conn(0x7f9e8815b910 0x7f9e88073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 --2- 172.21.15.31:0/3815843451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e88151e00 0x7f9e8815b3d0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 -- 172.21.15.31:0/3815843451 >> 172.21.15.31:0/3815843451 conn(0x7f9e880b22b0 msgr2=0x7f9e880b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 -- 172.21.15.31:0/3815843451 shutdown_connections 2024-09-13T08:44:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.626+0000 7f9e8f400640 1 -- 172.21.15.31:0/3815843451 wait complete. 2024-09-13T08:44:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.628+0000 7f9e8f400640 1 Processor -- start 2024-09-13T08:44:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.628+0000 7f9e8f400640 1 -- start start 2024-09-13T08:44:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e88151a20 0x7f9e881502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e88151e00 0x7f9e881398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8806f090 con 0x7f9e8815b910 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e88073970 con 0x7f9e88151a20 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8f400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9e8806f210 con 0x7f9e88151e00 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59928/0 (socket says 172.21.15.31:59928) 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8d400640 1 -- 172.21.15.31:0/2563708735 learned_addr learned my addr 172.21.15.31:0/2563708735 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e8ca00640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e88151a20 0x7f9e881502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.629+0000 7f9e87e00640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e88151e00 0x7f9e881398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 -- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e88151e00 msgr2=0x7f9e881398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e88151e00 0x7f9e881398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 -- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e88151a20 msgr2=0x7f9e881502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e88151a20 0x7f9e881502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 -- 172.21.15.31:0/2563708735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8813d040 con 0x7f9e8815b910 2024-09-13T08:44:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8d400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000eb40 tx=0x7f9e8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80004b20 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e80004ce0 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80005280 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8814f220 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.630+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e8814f7e0 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.632+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e8815b3d0 con 0x7f9e8815b910 2024-09-13T08:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.634+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f9e80005420 con 0x7f9e8815b910 2024-09-13T08:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.634+0000 7f9e84c00640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 0x7f9e6807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.634+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4943+0+0 (secure 0 0 0) 0x7f9e8009bb70 con 0x7f9e8815b910 2024-09-13T08:44:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.634+0000 7f9e8ca00640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 0x7f9e6807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.635+0000 7f9e8ca00640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 0x7f9e6807b190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c001720 tx=0x7f9e7c007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.638+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9e80063680 con 0x7f9e8815b910 2024-09-13T08:44:15.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:15.850+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9e880b56a0 con 0x7f9e8815b910 2024-09-13T08:44:16.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.841+0000 7f9e84c00640 1 -- 172.21.15.31:0/2563708735 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v120) ==== 97+0+0 (secure 0 0 0) 0x7f9e800680b0 con 0x7f9e8815b910 2024-09-13T08:44:16.842 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 msgr2=0x7f9e6807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 0x7f9e6807b190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c001720 tx=0x7f9e7c007e90 comp rx=0 tx=0).stop 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 msgr2=0x7f9e8813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000eb40 tx=0x7f9e8000c2d0 comp rx=0 tx=0).stop 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 shutdown_connections 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e68078cd0 0x7f9e6807b190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:16.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e8815b910 0x7f9e8813cb00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e88151e00 0x7f9e881398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 --2- 172.21.15.31:0/2563708735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e88151a20 0x7f9e881502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 >> 172.21.15.31:0/2563708735 conn(0x7f9e880b22b0 msgr2=0x7f9e88070500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 shutdown_connections 2024-09-13T08:44:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:16.844+0000 7f9e8f400640 1 -- 172.21.15.31:0/2563708735 wait complete. 2024-09-13T08:44:16.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:44:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 -- 172.21.15.31:0/2166697290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c15f9e0 msgr2=0x7f3e1c15fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 --2- 172.21.15.31:0/2166697290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c15fe20 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f3e08005fd0 tx=0x7f3e0802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 -- 172.21.15.31:0/2166697290 shutdown_connections 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 --2- 172.21.15.31:0/2166697290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c15fe20 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 --2- 172.21.15.31:0/2166697290 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 0x7f3e1c0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 -- 172.21.15.31:0/2166697290 >> 172.21.15.31:0/2166697290 conn(0x7f3e1c0cc910 msgr2=0x7f3e1c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 -- 172.21.15.31:0/2166697290 shutdown_connections 2024-09-13T08:44:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.033+0000 7f3e23e00640 1 -- 172.21.15.31:0/2166697290 wait complete. 2024-09-13T08:44:17.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.035+0000 7f3e23e00640 1 Processor -- start 2024-09-13T08:44:17.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.035+0000 7f3e23e00640 1 -- start start 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 0x7f3e1c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c0cb9d0 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c0cb850 con 0x7f3e1c15f9e0 2024-09-13T08:44:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e23e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c0cc360 con 0x7f3e1c15bd70 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e21400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e21400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59958/0 (socket says 172.21.15.31:59958) 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e21400640 1 -- 172.21.15.31:0/3431457263 learned_addr learned my addr 172.21.15.31:0/3431457263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e20a00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 0x7f3e1c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.036+0000 7f3e21e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 -- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 msgr2=0x7f3e1c15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 0x7f3e1c15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 -- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 msgr2=0x7f3e1c153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 -- 172.21.15.31:0/3431457263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c0c9180 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e21400640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d950 tx=0x7f3e1000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10017070 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e10009d20 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e100106c0 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.037+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c0c9450 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.038+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c0c9c90 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.039+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c0cf850 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.040+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f3e10010860 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.041+0000 7f3e19600640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 0x7f3dfc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.041+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 4857+0+0 (secure 0 0 0) 0x7f3e1009b2a0 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.041+0000 7f3e20a00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 0x7f3dfc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.042+0000 7f3e20a00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 0x7f3dfc07b190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e08033000 tx=0x7f3e08004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.045+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3e1c0c9450 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.317+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3e1c0ce8d0 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.317+0000 7f3e19600640 1 -- 172.21.15.31:0/3431457263 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f3e10062eb0 con 0x7f3e1c0ce3c0 2024-09-13T08:44:17.318 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:44:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:17.321 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.322 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:44:17.323 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.327 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.328 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:17.329 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.331 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:44:17.333 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:44:17.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 msgr2=0x7f3dfc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 0x7f3dfc07b190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e08033000 tx=0x7f3e08004730 comp rx=0 tx=0).stop 2024-09-13T08:44:17.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 msgr2=0x7f3e1c0ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d950 tx=0x7f3e1000de10 comp rx=0 tx=0).stop 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 shutdown_connections 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3dfc078cd0 0x7f3dfc07b190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c15f9e0 0x7f3e1c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3e1c15bd70 0x7f3e1c15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 --2- 172.21.15.31:0/3431457263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e1c0ce3c0 0x7f3e1c0ae0b0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 >> 172.21.15.31:0/3431457263 conn(0x7f3e1c0cc910 msgr2=0x7f3e1c0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 shutdown_connections 2024-09-13T08:44:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.320+0000 7f3e23e00640 1 -- 172.21.15.31:0/3431457263 wait complete. 2024-09-13T08:44:17.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:44:17.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.497+0000 7f435aa00640 1 -- 172.21.15.31:0/1178682937 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f43540cbc50 msgr2=0x7f43540cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.497+0000 7f435aa00640 1 --2- 172.21.15.31:0/1178682937 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f43540cbc50 0x7f43540cc0a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f434c009be0 tx=0x7f434c02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.497+0000 7f435aa00640 1 -- 172.21.15.31:0/1178682937 shutdown_connections 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.497+0000 7f435aa00640 1 --2- 172.21.15.31:0/1178682937 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f43540cbc50 0x7f43540cc0a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.497+0000 7f435aa00640 1 -- 172.21.15.31:0/1178682937 >> 172.21.15.31:0/1178682937 conn(0x7f43540cc910 msgr2=0x7f43540cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.498+0000 7f435aa00640 1 -- 172.21.15.31:0/1178682937 shutdown_connections 2024-09-13T08:44:17.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.498+0000 7f435aa00640 1 -- 172.21.15.31:0/1178682937 wait complete. 2024-09-13T08:44:17.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.499+0000 7f435aa00640 1 Processor -- start 2024-09-13T08:44:17.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.499+0000 7f435aa00640 1 -- start start 2024-09-13T08:44:17.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43540cf850 0x7f43540b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 0x7f43541f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f43541600f0 con 0x7f43540cbc50 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f43540cead0 con 0x7f43540cf850 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.500+0000 7f435aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f43541f53b0 con 0x7f435415bd70 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59974/0 (socket says 172.21.15.31:59974) 2024-09-13T08:44:17.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 -- 172.21.15.31:0/4008197499 learned_addr learned my addr 172.21.15.31:0/4008197499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f434be00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43540cf850 0x7f43540b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4358a00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 0x7f43541f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 -- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 msgr2=0x7f43541f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 0x7f43541f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 -- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43540cf850 msgr2=0x7f43540b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43540cf850 0x7f43540b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 -- 172.21.15.31:0/4008197499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43541f5690 con 0x7f43540cbc50 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4358a00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 0x7f43541f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4353e00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f434400c660 tx=0x7f434400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.501+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344016020 con 0x7f43540cbc50 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.502+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f434400a950 con 0x7f43540cbc50 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.502+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434400ce60 con 0x7f43540cbc50 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.502+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435409bbe0 con 0x7f43540cbc50 2024-09-13T08:44:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.502+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43540c6bc0 con 0x7f43540cbc50 2024-09-13T08:44:17.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.503+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43540cb930 con 0x7f43540cbc50 2024-09-13T08:44:17.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.504+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f4344019020 con 0x7f43540cbc50 2024-09-13T08:44:17.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.505+0000 7f4350c00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 0x7f433007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.505+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 4857+0+0 (secure 0 0 0) 0x7f434409b110 con 0x7f43540cbc50 2024-09-13T08:44:17.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.505+0000 7f434be00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 0x7f433007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.506+0000 7f434be00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 0x7f433007b190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f433c007a70 tx=0x7f433c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.509+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4344020020 con 0x7f43540cbc50 2024-09-13T08:44:17.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.706+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f43541f5860 con 0x7f43540cbc50 2024-09-13T08:44:17.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.707+0000 7f4350c00640 1 -- 172.21.15.31:0/4008197499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v120) ==== 84+0+65 (secure 0 0 0) 0x7f4344063230 con 0x7f43540cbc50 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:17.708 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:44:17.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 msgr2=0x7f433007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 0x7f433007b190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f433c007a70 tx=0x7f433c009040 comp rx=0 tx=0).stop 2024-09-13T08:44:17.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 msgr2=0x7f43540c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f434400c660 tx=0x7f434400cb20 comp rx=0 tx=0).stop 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 shutdown_connections 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4330078cd0 0x7f433007b190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f435415bd70 0x7f43541f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43540cf850 0x7f43540b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 --2- 172.21.15.31:0/4008197499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43540cbc50 0x7f43540c6680 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 >> 172.21.15.31:0/4008197499 conn(0x7f43540cc910 msgr2=0x7f43540ae480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 shutdown_connections 2024-09-13T08:44:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.709+0000 7f435aa00640 1 -- 172.21.15.31:0/4008197499 wait complete. 2024-09-13T08:44:17.726 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:44:17.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:44:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 -- 172.21.15.31:0/8558709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a300b5300 msgr2=0x7f5a300cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 --2- 172.21.15.31:0/8558709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a300b5300 0x7f5a300cc3e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009830 tx=0x7f5a1c02a480 comp rx=0 tx=0).stop 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 -- 172.21.15.31:0/8558709 shutdown_connections 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 --2- 172.21.15.31:0/8558709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a300b5300 0x7f5a300cc3e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 --2- 172.21.15.31:0/8558709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a3015bb70 0x7f5a3006fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 -- 172.21.15.31:0/8558709 >> 172.21.15.31:0/8558709 conn(0x7f5a300b1520 msgr2=0x7f5a300b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 -- 172.21.15.31:0/8558709 shutdown_connections 2024-09-13T08:44:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.891+0000 7f5a35800640 1 -- 172.21.15.31:0/8558709 wait complete. 2024-09-13T08:44:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.893+0000 7f5a35800640 1 Processor -- start 2024-09-13T08:44:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.893+0000 7f5a35800640 1 -- start start 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a300cc920 0x7f5a3015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3015bb70 0x7f5a300c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a300b29a0 con 0x7f5a300b5300 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a300b2820 con 0x7f5a3015bb70 2024-09-13T08:44:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a35800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5a300ca730 con 0x7f5a300cc920 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a2fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a2fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59978/0 (socket says 172.21.15.31:59978) 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a2fe00640 1 -- 172.21.15.31:0/2825902102 learned_addr learned my addr 172.21.15.31:0/2825902102 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a34e00640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3015bb70 0x7f5a300c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.894+0000 7f5a2f400640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a300cc920 0x7f5a3015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 -- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a300cc920 msgr2=0x7f5a3015e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a300cc920 0x7f5a3015e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 -- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3015bb70 msgr2=0x7f5a300c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3015bb70 0x7f5a300c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 -- 172.21.15.31:0/2825902102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3015c200 con 0x7f5a300b5300 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a2fe00640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000e790 tx=0x7f5a2000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2000eea0 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a2000cdb0 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20010690 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3015c470 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.895+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3015ca30 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.896+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a300cabb0 con 0x7f5a300b5300 2024-09-13T08:44:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.898+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f5a20010830 con 0x7f5a300b5300 2024-09-13T08:44:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.898+0000 7f5a27e00640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 0x7f5a1007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.899+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 4857+0+0 (secure 0 0 0) 0x7f5a2009b8b0 con 0x7f5a300b5300 2024-09-13T08:44:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.899+0000 7f5a2f400640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 0x7f5a1007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:17.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.900+0000 7f5a2f400640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 0x7f5a1007b190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c034000 tx=0x7f5a1c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:17.903+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5a20063410 con 0x7f5a300b5300 2024-09-13T08:44:18.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.096+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5a3015e9c0 con 0x7f5a300b5300 2024-09-13T08:44:18.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.854+0000 7f5a27e00640 1 -- 172.21.15.31:0/2825902102 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v121) ==== 138+0+0 (secure 0 0 0) 0x7f5a20067e40 con 0x7f5a300b5300 2024-09-13T08:44:18.856 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 msgr2=0x7f5a1007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 0x7f5a1007b190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c034000 tx=0x7f5a1c004730 comp rx=0 tx=0).stop 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 msgr2=0x7f5a300c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000e790 tx=0x7f5a2000ec50 comp rx=0 tx=0).stop 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 shutdown_connections 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a10078cd0 0x7f5a1007b190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a3015bb70 0x7f5a300c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a300cc920 0x7f5a3015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 --2- 172.21.15.31:0/2825902102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a300b5300 0x7f5a300c9970 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.859+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 >> 172.21.15.31:0/2825902102 conn(0x7f5a300b1520 msgr2=0x7f5a300ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.860+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 shutdown_connections 2024-09-13T08:44:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:18.860+0000 7f5a35800640 1 -- 172.21.15.31:0/2825902102 wait complete. 2024-09-13T08:44:18.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:44:19.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 -- 172.21.15.31:0/35359963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 msgr2=0x7f65c80cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 --2- 172.21.15.31:0/35359963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c80cfdd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009830 tx=0x7f65b802a5c0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 -- 172.21.15.31:0/35359963 shutdown_connections 2024-09-13T08:44:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 --2- 172.21.15.31:0/35359963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c80cfdd0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 -- 172.21.15.31:0/35359963 >> 172.21.15.31:0/35359963 conn(0x7f65c80d1b90 msgr2=0x7f65c80d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:19.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.056+0000 7f65c7e00640 1 -- 172.21.15.31:0/35359963 shutdown_connections 2024-09-13T08:44:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.057+0000 7f65c7e00640 1 -- 172.21.15.31:0/35359963 wait complete. 2024-09-13T08:44:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.058+0000 7f65c7e00640 1 Processor -- start 2024-09-13T08:44:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.058+0000 7f65c7e00640 1 -- start start 2024-09-13T08:44:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65c80d0a20 0x7f65c815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c809c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65c806e840 con 0x7f65c80b53f0 2024-09-13T08:44:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65c806e6c0 con 0x7f65c815e160 2024-09-13T08:44:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f65c806e9c0 con 0x7f65c80d0a20 2024-09-13T08:44:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59986/0 (socket says 172.21.15.31:59986) 2024-09-13T08:44:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c6a00640 1 -- 172.21.15.31:0/934094308 learned_addr learned my addr 172.21.15.31:0/934094308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.059+0000 7f65c7400640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c809c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65be000640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65c80d0a20 0x7f65c815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 -- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65c80d0a20 msgr2=0x7f65c815c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65c80d0a20 0x7f65c815c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 -- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 msgr2=0x7f65c809c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c809c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 -- 172.21.15.31:0/934094308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c81f5670 con 0x7f65c80b53f0 2024-09-13T08:44:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.060+0000 7f65c6a00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f65b8000c00 tx=0x7f65b8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.061+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8031d60 con 0x7f65c80b53f0 2024-09-13T08:44:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.061+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65b8033e10 con 0x7f65c80b53f0 2024-09-13T08:44:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.061+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80332b0 con 0x7f65c80b53f0 2024-09-13T08:44:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.061+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c815c9a0 con 0x7f65c80b53f0 2024-09-13T08:44:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.061+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c809f620 con 0x7f65c80b53f0 2024-09-13T08:44:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.062+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c806dda0 con 0x7f65c80b53f0 2024-09-13T08:44:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.064+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f65b8033450 con 0x7f65c80b53f0 2024-09-13T08:44:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.065+0000 7f65bf400640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 0x7f65a407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.065+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5227+0+0 (secure 0 0 0) 0x7f65b80bc860 con 0x7f65c80b53f0 2024-09-13T08:44:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.065+0000 7f65be000640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 0x7f65a407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.065+0000 7f65be000640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 0x7f65a407b190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f65c81f22a0 tx=0x7f65b0007cc0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.069+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f65b8051020 con 0x7f65c80b53f0 2024-09-13T08:44:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.269+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f65c809c050 con 0x7f65c80b53f0 2024-09-13T08:44:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.862+0000 7f65bf400640 1 -- 172.21.15.31:0/934094308 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v122) ==== 170+0+0 (secure 0 0 0) 0x7f65c81f5670 con 0x7f65c80b53f0 2024-09-13T08:44:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:44:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.865+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 msgr2=0x7f65a407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.865+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 0x7f65a407b190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f65c81f22a0 tx=0x7f65b0007cc0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.865+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 msgr2=0x7f65c81f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.865+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f65b8000c00 tx=0x7f65b8033c20 comp rx=0 tx=0).stop 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 shutdown_connections 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f65a4078cd0 0x7f65a407b190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65c815e160 0x7f65c809c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65c80d0a20 0x7f65c815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 --2- 172.21.15.31:0/934094308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65c80b53f0 0x7f65c81f5130 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 >> 172.21.15.31:0/934094308 conn(0x7f65c80d1b90 msgr2=0x7f65c80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 shutdown_connections 2024-09-13T08:44:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:19.866+0000 7f65c7e00640 1 -- 172.21.15.31:0/934094308 wait complete. 2024-09-13T08:44:19.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:44:20.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 -- 172.21.15.31:0/1992191785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8004d40 msgr2=0x7f25b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 --2- 172.21.15.31:0/1992191785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8004d40 0x7f25b80051c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f25c800c670 tx=0x7f25b8005830 comp rx=0 tx=0).stop 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d6600640 1 -- 172.21.15.31:0/1992191785 reap_dead start 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 -- 172.21.15.31:0/1992191785 shutdown_connections 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 -- 172.21.15.31:0/1992191785 >> 172.21.15.31:0/1992191785 conn(0x7f25d00cfbb0 msgr2=0x7f25d00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 -- 172.21.15.31:0/1992191785 shutdown_connections 2024-09-13T08:44:20.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.066+0000 7f25d8600640 1 -- 172.21.15.31:0/1992191785 wait complete. 2024-09-13T08:44:20.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.068+0000 7f25d8600640 1 Processor -- start 2024-09-13T08:44:20.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.068+0000 7f25d8600640 1 -- start start 2024-09-13T08:44:20.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f25b8004d40 0x7f25d014b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:20.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d0157020 0x7f25d01486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:20.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:20.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25d00cf870 con 0x7f25d015cea0 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25d00cdcc0 con 0x7f25d0157020 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d8600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f25d0149210 con 0x7f25b8004d40 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59998/0 (socket says 172.21.15.31:59998) 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d6600640 1 -- 172.21.15.31:0/2673925376 learned_addr learned my addr 172.21.15.31:0/2673925376 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d5200640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d0157020 0x7f25d01486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.069+0000 7f25d5c00640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f25b8004d40 0x7f25d014b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 -- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f25b8004d40 msgr2=0x7f25d014b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f25b8004d40 0x7f25d014b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 -- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d0157020 msgr2=0x7f25d01486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:20.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d0157020 0x7f25d01486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 -- 172.21.15.31:0/2673925376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d00ad6f0 con 0x7f25d015cea0 2024-09-13T08:44:20.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d6600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f25c800c790 tx=0x7f25c8001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:20.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8039070 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25c8006680 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d014b540 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.070+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8002030 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.071+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d014baa0 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.072+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d0148f70 con 0x7f25d015cea0 2024-09-13T08:44:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.073+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f25c803c020 con 0x7f25d015cea0 2024-09-13T08:44:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.074+0000 7f25cd600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 0x7f25b007b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.074+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5609+0+0 (secure 0 0 0) 0x7f25c80bb150 con 0x7f25d015cea0 2024-09-13T08:44:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.074+0000 7f25d5c00640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 0x7f25b007b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.075+0000 7f25d5c00640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 0x7f25b007b1c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009c50 tx=0x7f25c4005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.078+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f25d014b540 con 0x7f25d015cea0 2024-09-13T08:44:20.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.309+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f25d00cd780 con 0x7f25d015cea0 2024-09-13T08:44:20.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.864+0000 7f25cd600640 1 -- 172.21.15.31:0/2673925376 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v77) ==== 174+0+0 (secure 0 0 0) 0x7f25c8083390 con 0x7f25d015cea0 2024-09-13T08:44:20.865 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 18 and data pool 19 2024-09-13T08:44:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.867+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 msgr2=0x7f25b007b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.867+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 0x7f25b007b1c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009c50 tx=0x7f25c4005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.867+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 msgr2=0x7f25d015d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.867+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f25c800c790 tx=0x7f25c8001aa0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 shutdown_connections 2024-09-13T08:44:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f25b0078d00 0x7f25b007b1c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25d015cea0 0x7f25d015d320 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d0157020 0x7f25d01486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 --2- 172.21.15.31:0/2673925376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f25b8004d40 0x7f25d014b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 >> 172.21.15.31:0/2673925376 conn(0x7f25d00cfbb0 msgr2=0x7f25d00b4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 shutdown_connections 2024-09-13T08:44:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:20.868+0000 7f25d8600640 1 -- 172.21.15.31:0/2673925376 wait complete. 2024-09-13T08:44:20.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 -- 172.21.15.31:0/3267012260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 msgr2=0x7f9c280cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 --2- 172.21.15.31:0/3267012260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280cf850 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9c18005fd0 tx=0x7f9c1802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 -- 172.21.15.31:0/3267012260 shutdown_connections 2024-09-13T08:44:21.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 --2- 172.21.15.31:0/3267012260 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c2815f9e0 0x7f9c2815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 --2- 172.21.15.31:0/3267012260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280cf850 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 -- 172.21.15.31:0/3267012260 >> 172.21.15.31:0/3267012260 conn(0x7f9c280cc910 msgr2=0x7f9c280cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 -- 172.21.15.31:0/3267012260 shutdown_connections 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.076+0000 7f9c2de00640 1 -- 172.21.15.31:0/3267012260 wait complete. 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.077+0000 7f9c2de00640 1 Processor -- start 2024-09-13T08:44:21.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.078+0000 7f9c2de00640 1 -- start start 2024-09-13T08:44:21.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.078+0000 7f9c2de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 0x7f9c2814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c2de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c2de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 0x7f9c280b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c2de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c280cb9d0 con 0x7f9c2815bd70 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c2de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c280cb850 con 0x7f9c2815f9e0 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c2de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9c280cc360 con 0x7f9c280ce3c0 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60016/0 (socket says 172.21.15.31:60016) 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 -- 172.21.15.31:0/569426331 learned_addr learned my addr 172.21.15.31:0/569426331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 -- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 msgr2=0x7f9c2814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c27400640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 0x7f9c280b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c26a00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 0x7f9c2814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 0x7f9c2814e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 -- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 msgr2=0x7f9c280b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 0x7f9c280b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c1e000640 1 -- 172.21.15.31:0/569426331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c28158f20 con 0x7f9c2815bd70 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c26a00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 0x7f9c2814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.079+0000 7f9c27400640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 0x7f9c280b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c1e000640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009fd0 tx=0x7f9c1000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000ccc0 con 0x7f9c2815bd70 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c1000ee70 con 0x7f9c2815bd70 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000ce60 con 0x7f9c2815bd70 2024-09-13T08:44:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c280b5730 con 0x7f9c2815bd70 2024-09-13T08:44:21.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.080+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c280b5cf0 con 0x7f9c2815bd70 2024-09-13T08:44:21.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.081+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c280cf850 con 0x7f9c2815bd70 2024-09-13T08:44:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.082+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f9c10004280 con 0x7f9c2815bd70 2024-09-13T08:44:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.084+0000 7f9c1fe00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 0x7f9c0407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.084+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5709+0+0 (secure 0 0 0) 0x7f9c1009bbb0 con 0x7f9c2815bd70 2024-09-13T08:44:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.084+0000 7f9c26a00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 0x7f9c0407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.085+0000 7f9c26a00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 0x7f9c0407b190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c18033000 tx=0x7f9c18004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.088+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9c10063a10 con 0x7f9c2815bd70 2024-09-13T08:44:21.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.285+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c28069a50 con 0x7f9c2815bd70 2024-09-13T08:44:21.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.286+0000 7f9c1fe00640 1 -- 172.21.15.31:0/569426331 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+16652 (secure 0 0 0) 0x7f9c10063310 con 0x7f9c2815bd70 2024-09-13T08:44:21.288 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:21.288 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":123,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:20.310461+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:18.106733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:19.279918+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 msgr2=0x7f9c0407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 0x7f9c0407b190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c18033000 tx=0x7f9c18004730 comp rx=0 tx=0).stop 2024-09-13T08:44:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 msgr2=0x7f9c280b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009fd0 tx=0x7f9c1000ec20 comp rx=0 tx=0).stop 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 shutdown_connections 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c04078cd0 0x7f9c0407b190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c2815f9e0 0x7f9c280b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c2815bd70 0x7f9c280b2cc0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 --2- 172.21.15.31:0/569426331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c280ce3c0 0x7f9c2814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 >> 172.21.15.31:0/569426331 conn(0x7f9c280cc910 msgr2=0x7f9c280ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 shutdown_connections 2024-09-13T08:44:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.289+0000 7f9c2de00640 1 -- 172.21.15.31:0/569426331 wait complete. 2024-09-13T08:44:21.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 -- 172.21.15.31:0/2546002568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 msgr2=0x7f89dc1624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 --2- 172.21.15.31:0/2546002568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc1624a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f89dc08d7e0 tx=0x7f89d402aa30 comp rx=0 tx=0).stop 2024-09-13T08:44:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 -- 172.21.15.31:0/2546002568 shutdown_connections 2024-09-13T08:44:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 --2- 172.21.15.31:0/2546002568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc1624a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 --2- 172.21.15.31:0/2546002568 >> v2:172.21.15.45:3300/0 conn(0x7f89dc0b4b60 0x7f89dc0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 -- 172.21.15.31:0/2546002568 >> 172.21.15.31:0/2546002568 conn(0x7f89dc09d190 msgr2=0x7f89dc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 -- 172.21.15.31:0/2546002568 shutdown_connections 2024-09-13T08:44:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.466+0000 7f89e2400640 1 -- 172.21.15.31:0/2546002568 wait complete. 2024-09-13T08:44:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.468+0000 7f89e2400640 1 Processor -- start 2024-09-13T08:44:21.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.468+0000 7f89e2400640 1 -- start start 2024-09-13T08:44:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89dc0b4b60 0x7f89dc15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89dc0d27b0 0x7f89dc0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc162e20 con 0x7f89dc1620c0 2024-09-13T08:44:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc162ca0 con 0x7f89dc0b4b60 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e2400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f89dc0b3440 con 0x7f89dc0d27b0 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60030/0 (socket says 172.21.15.31:60030) 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 -- 172.21.15.31:0/3008721417 learned_addr learned my addr 172.21.15.31:0/3008721417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89d3e00640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89dc0d27b0 0x7f89dc0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1000640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89dc0b4b60 0x7f89dc15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 -- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89dc0d27b0 msgr2=0x7f89dc0cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89dc0d27b0 0x7f89dc0cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 -- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89dc0b4b60 msgr2=0x7f89dc15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89dc0b4b60 0x7f89dc15d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.469+0000 7f89e1a00640 1 -- 172.21.15.31:0/3008721417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc0cbfa0 con 0x7f89dc1620c0 2024-09-13T08:44:21.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89e1a00640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f89d402b020 tx=0x7f89d4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4033050 con 0x7f89dc1620c0 2024-09-13T08:44:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89d40045f0 con 0x7f89dc1620c0 2024-09-13T08:44:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4038450 con 0x7f89dc1620c0 2024-09-13T08:44:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc15d9d0 con 0x7f89dc1620c0 2024-09-13T08:44:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.470+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc0cbcc0 con 0x7f89dc1620c0 2024-09-13T08:44:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.471+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc0d2b90 con 0x7f89dc1620c0 2024-09-13T08:44:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.472+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f89d4036070 con 0x7f89dc1620c0 2024-09-13T08:44:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.473+0000 7f89d9600640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 0x7f89b807b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.473+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5709+0+0 (secure 0 0 0) 0x7f89d40bc6d0 con 0x7f89dc1620c0 2024-09-13T08:44:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.473+0000 7f89e1000640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 0x7f89b807b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.474+0000 7f89e1000640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 0x7f89b807b1c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06a4a0 tx=0x7f89cc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.478+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f89d4084620 con 0x7f89dc1620c0 2024-09-13T08:44:21.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.674+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89dc161760 con 0x7f89dc1620c0 2024-09-13T08:44:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.675+0000 7f89d9600640 1 -- 172.21.15.31:0/3008721417 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+16652 (secure 0 0 0) 0x7f89d4083f20 con 0x7f89dc1620c0 2024-09-13T08:44:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:21.676 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":123,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:20.310461+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:18.106733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:19.279918+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.677+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 msgr2=0x7f89b807b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.677+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 0x7f89b807b1c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06a4a0 tx=0x7f89cc009040 comp rx=0 tx=0).stop 2024-09-13T08:44:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 msgr2=0x7f89dc0b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f89d402b020 tx=0x7f89d4003f90 comp rx=0 tx=0).stop 2024-09-13T08:44:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 shutdown_connections 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89b8078d00 0x7f89b807b1c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89dc1620c0 0x7f89dc0b1ad0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f89dc0d27b0 0x7f89dc0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 --2- 172.21.15.31:0/3008721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89dc0b4b60 0x7f89dc15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 >> 172.21.15.31:0/3008721417 conn(0x7f89dc09d190 msgr2=0x7f89dc0b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 shutdown_connections 2024-09-13T08:44:21.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.678+0000 7f89e2400640 1 -- 172.21.15.31:0/3008721417 wait complete. 2024-09-13T08:44:21.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:44:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 -- 172.21.15.31:0/2157471960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa60815d5e0 msgr2=0x7fa6080ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 --2- 172.21.15.31:0/2157471960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa60815d5e0 0x7fa6080ce660 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa6080cc600 tx=0x7fa5f802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 -- 172.21.15.31:0/2157471960 shutdown_connections 2024-09-13T08:44:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 --2- 172.21.15.31:0/2157471960 >> v2:172.21.15.45:3300/0 conn(0x7fa6080cd780 0x7fa608072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 --2- 172.21.15.31:0/2157471960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa60815d5e0 0x7fa6080ce660 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 -- 172.21.15.31:0/2157471960 >> 172.21.15.31:0/2157471960 conn(0x7fa6080cfbb0 msgr2=0x7fa6080cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:21.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 -- 172.21.15.31:0/2157471960 shutdown_connections 2024-09-13T08:44:21.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.856+0000 7fa60d000640 1 -- 172.21.15.31:0/2157471960 wait complete. 2024-09-13T08:44:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.858+0000 7fa60d000640 1 Processor -- start 2024-09-13T08:44:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.858+0000 7fa60d000640 1 -- start start 2024-09-13T08:44:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6080ca200 0x7fa60814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60815d5e0 0x7fa6081f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa6080ca690 con 0x7fa6080cd780 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa608070860 con 0x7fa60815d5e0 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa60d000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa6080ca810 con 0x7fa6080ca200 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa5fd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa5fd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60040/0 (socket says 172.21.15.31:60040) 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa5fd400640 1 -- 172.21.15.31:0/4046020972 learned_addr learned my addr 172.21.15.31:0/4046020972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa605e00640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6080ca200 0x7fa60814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.859+0000 7fa606800640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60815d5e0 0x7fa6081f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 -- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6080ca200 msgr2=0x7fa60814d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6080ca200 0x7fa60814d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 -- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60815d5e0 msgr2=0x7fa6081f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60815d5e0 0x7fa6081f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 -- 172.21.15.31:0/4046020972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa608152330 con 0x7fa6080cd780 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fd400640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d770 tx=0x7fa5f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0017070 con 0x7fa6080cd780 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5f0009d20 con 0x7fa6080cd780 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0005280 con 0x7fa6080cd780 2024-09-13T08:44:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.860+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa608152610 con 0x7fa6080cd780 2024-09-13T08:44:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.861+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa608158020 con 0x7fa6080cd780 2024-09-13T08:44:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.862+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa608072560 con 0x7fa6080cd780 2024-09-13T08:44:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.863+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fa5f0005420 con 0x7fa6080cd780 2024-09-13T08:44:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.864+0000 7fa5fea00640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 0x7fa5e407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.864+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5709+0+0 (secure 0 0 0) 0x7fa5f009b3d0 con 0x7fa6080cd780 2024-09-13T08:44:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.865+0000 7fa605e00640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 0x7fa5e407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.865+0000 7fa605e00640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 0x7fa5e407b190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8033000 tx=0x7fa5f8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:21.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:21.869+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa5f0062be0 con 0x7fa6080cd780 2024-09-13T08:44:22.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.106+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa6080d01a0 con 0x7fa6080cd780 2024-09-13T08:44:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.891+0000 7fa5fea00640 1 -- 172.21.15.31:0/4046020972 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) ==== 118+0+0 (secure 0 0 0) 0x7fa5f0067610 con 0x7fa6080cd780 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 msgr2=0x7fa5e407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 0x7fa5e407b190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8033000 tx=0x7fa5f8004730 comp rx=0 tx=0).stop 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 msgr2=0x7fa6081f7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d770 tx=0x7fa5f000dc30 comp rx=0 tx=0).stop 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 shutdown_connections 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5e4078cd0 0x7fa5e407b190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60815d5e0 0x7fa6081f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6080cd780 0x7fa6081f7d30 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 --2- 172.21.15.31:0/4046020972 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa6080ca200 0x7fa60814d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 >> 172.21.15.31:0/4046020972 conn(0x7fa6080cfbb0 msgr2=0x7fa6080b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 shutdown_connections 2024-09-13T08:44:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:22.894+0000 7fa60d000640 1 -- 172.21.15.31:0/4046020972 wait complete. 2024-09-13T08:44:22.913 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:44:22.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:44:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 -- 172.21.15.31:0/3118883286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26015bfc0 msgr2=0x7fb260072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 --2- 172.21.15.31:0/3118883286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26015bfc0 0x7fb260072560 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb250009830 tx=0x7fb25002a4a0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 -- 172.21.15.31:0/3118883286 shutdown_connections 2024-09-13T08:44:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 --2- 172.21.15.31:0/3118883286 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26006fec0 0x7fb2600b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 --2- 172.21.15.31:0/3118883286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26015bfc0 0x7fb260072560 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 -- 172.21.15.31:0/3118883286 >> 172.21.15.31:0/3118883286 conn(0x7fb2600cfbb0 msgr2=0x7fb2600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 -- 172.21.15.31:0/3118883286 shutdown_connections 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.083+0000 7fb266c00640 1 -- 172.21.15.31:0/3118883286 wait complete. 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.085+0000 7fb266c00640 1 Processor -- start 2024-09-13T08:44:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.085+0000 7fb266c00640 1 -- start start 2024-09-13T08:44:23.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26006fa60 0x7fb2600c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:23.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 0x7fb2600c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:23.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2600cbfd0 con 0x7fb2600c9040 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2600cbe50 con 0x7fb26006fa60 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb266c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb2600b2f20 con 0x7fb26015bfc0 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb264c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb264c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60042/0 (socket says 172.21.15.31:60042) 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb264c00640 1 -- 172.21.15.31:0/251360409 learned_addr learned my addr 172.21.15.31:0/251360409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb25fe00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26006fa60 0x7fb2600c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb257e00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 0x7fb2600c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.086+0000 7fb264c00640 1 -- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 msgr2=0x7fb2600c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb264c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 0x7fb2600c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb264c00640 1 -- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26006fa60 msgr2=0x7fb2600c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb264c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26006fa60 0x7fb2600c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb264c00640 1 -- 172.21.15.31:0/251360409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2600c75f0 con 0x7fb2600c9040 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb257e00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 0x7fb2600c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb264c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb25800eb10 tx=0x7fb25800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258004b20 con 0x7fb2600c9040 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb258004ce0 con 0x7fb2600c9040 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2600c78d0 con 0x7fb2600c9040 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258005280 con 0x7fb2600c9040 2024-09-13T08:44:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.087+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2600c85e0 con 0x7fb2600c9040 2024-09-13T08:44:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.088+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2600b33d0 con 0x7fb2600c9040 2024-09-13T08:44:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.090+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fb258005420 con 0x7fb2600c9040 2024-09-13T08:44:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.090+0000 7fb25cc00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 0x7fb23c07b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.091+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7fb25809bea0 con 0x7fb2600c9040 2024-09-13T08:44:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.091+0000 7fb25fe00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 0x7fb23c07b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.091+0000 7fb25fe00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 0x7fb23c07b1c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb250034000 tx=0x7fb250033040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.095+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb258063d40 con 0x7fb2600c9040 2024-09-13T08:44:23.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.327+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb2600cb310 con 0x7fb2600c9040 2024-09-13T08:44:23.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.897+0000 7fb25cc00640 1 -- 172.21.15.31:0/251360409 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v81) ==== 122+0+0 (secure 0 0 0) 0x7fb25809d0c0 con 0x7fb2600c9040 2024-09-13T08:44:23.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 msgr2=0x7fb23c07b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:23.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 0x7fb23c07b1c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb250034000 tx=0x7fb250033040 comp rx=0 tx=0).stop 2024-09-13T08:44:23.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 msgr2=0x7fb26015e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb25800eb10 tx=0x7fb25800c2d0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 shutdown_connections 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb23c078d00 0x7fb23c07b1c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2600c9040 0x7fb26015e6c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb26015bfc0 0x7fb2600c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 --2- 172.21.15.31:0/251360409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb26006fa60 0x7fb2600c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.900+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 >> 172.21.15.31:0/251360409 conn(0x7fb2600cfbb0 msgr2=0x7fb2600b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.901+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 shutdown_connections 2024-09-13T08:44:23.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:23.901+0000 7fb266c00640 1 -- 172.21.15.31:0/251360409 wait complete. 2024-09-13T08:44:23.918 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:44:23.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 -- 172.21.15.31:0/1744032821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f519c003a00 msgr2=0x7f519c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 --2- 172.21.15.31:0/1744032821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f519c003a00 0x7f519c003e20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f51a800c260 tx=0x7f519c005830 comp rx=0 tx=0).stop 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51b6800640 1 -- 172.21.15.31:0/1744032821 reap_dead start 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 -- 172.21.15.31:0/1744032821 shutdown_connections 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 -- 172.21.15.31:0/1744032821 >> 172.21.15.31:0/1744032821 conn(0x7f51b80cc910 msgr2=0x7f51b80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 -- 172.21.15.31:0/1744032821 shutdown_connections 2024-09-13T08:44:24.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.079+0000 7f51bcc00640 1 -- 172.21.15.31:0/1744032821 wait complete. 2024-09-13T08:44:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.081+0000 7f51bcc00640 1 Processor -- start 2024-09-13T08:44:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.081+0000 7f51bcc00640 1 -- start start 2024-09-13T08:44:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c003a00 0x7f51b8070070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51b815ad80 0x7f51b815a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81600f0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51b80cb850 con 0x7f519c003a00 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51bcc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f51b80b3b50 con 0x7f51b815ad80 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60060/0 (socket says 172.21.15.31:60060) 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51b6800640 1 -- 172.21.15.31:0/2007441419 learned_addr learned my addr 172.21.15.31:0/2007441419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51b5e00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c003a00 0x7f51b8070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.082+0000 7f51afe00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51b815ad80 0x7f51b815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 -- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51b815ad80 msgr2=0x7f51b815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51b815ad80 0x7f51b815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 -- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c003a00 msgr2=0x7f51b8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c003a00 0x7f51b8070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 -- 172.21.15.31:0/2007441419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b809d830 con 0x7f51b80b3fe0 2024-09-13T08:44:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51b6800640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f51b80ca470 tx=0x7f51b0004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0004b20 con 0x7f51b80b3fe0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51b0004cc0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0010c30 con 0x7f51b80b3fe0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b809d2b0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.083+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b80b37e0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.084+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b80ce3c0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.086+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f51b001c020 con 0x7f51b80b3fe0 2024-09-13T08:44:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.086+0000 7f51ae000640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 0x7f519407b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.087+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f51b009b5e0 con 0x7f51b80b3fe0 2024-09-13T08:44:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.087+0000 7f51b5e00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 0x7f519407b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.088+0000 7f51b5e00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 0x7f519407b1c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f51a8033000 tx=0x7f51a8038040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.091+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f51b0062f20 con 0x7f51b80b3fe0 2024-09-13T08:44:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.323+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51b80b3d00 con 0x7f51b80b3fe0 2024-09-13T08:44:24.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.324+0000 7f51ae000640 1 -- 172.21.15.31:0/2007441419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5225 (secure 0 0 0) 0x7f51b0067950 con 0x7f51b80b3fe0 2024-09-13T08:44:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:24.325 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"btime":"2024-09-13T08:44:23:892227+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:24.325 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2024-09-13T08:44:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 msgr2=0x7f519407b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 0x7f519407b1c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f51a8033000 tx=0x7f51a8038040 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 msgr2=0x7f51b80b4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f51b80ca470 tx=0x7f51b0004700 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 shutdown_connections 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5194078d00 0x7f519407b1c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51b80b3fe0 0x7f51b80b4460 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f51b815ad80 0x7f51b815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 --2- 172.21.15.31:0/2007441419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c003a00 0x7f51b8070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.326+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 >> 172.21.15.31:0/2007441419 conn(0x7f51b80cc910 msgr2=0x7f51b80cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.327+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 shutdown_connections 2024-09-13T08:44:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.327+0000 7f51bcc00640 1 -- 172.21.15.31:0/2007441419 wait complete. 2024-09-13T08:44:24.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:24.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 -- 172.21.15.31:0/2752061981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 msgr2=0x7ff4f80cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2752061981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f80cc300 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0005fd0 tx=0x7ff4f002ab40 comp rx=0 tx=0).stop 2024-09-13T08:44:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 -- 172.21.15.31:0/2752061981 shutdown_connections 2024-09-13T08:44:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2752061981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f80cc300 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 -- 172.21.15.31:0/2752061981 >> 172.21.15.31:0/2752061981 conn(0x7ff4f80cc910 msgr2=0x7ff4f80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.503+0000 7ff4fd200640 1 -- 172.21.15.31:0/2752061981 shutdown_connections 2024-09-13T08:44:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.504+0000 7ff4fd200640 1 -- 172.21.15.31:0/2752061981 wait complete. 2024-09-13T08:44:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.505+0000 7ff4fd200640 1 Processor -- start 2024-09-13T08:44:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.505+0000 7ff4fd200640 1 -- start start 2024-09-13T08:44:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 0x7ff4f809d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4f815dfc0 0x7ff4f80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8073c20 con 0x7ff4f80b2020 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8073aa0 con 0x7ff4f815dfc0 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4fd200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f8073da0 con 0x7ff4f80ca5c0 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4f5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4f5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60080/0 (socket says 172.21.15.31:60080) 2024-09-13T08:44:24.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4f5e00640 1 -- 172.21.15.31:0/2787762749 learned_addr learned my addr 172.21.15.31:0/2787762749 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:24.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4f6800640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4f815dfc0 0x7ff4f80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.506+0000 7ff4ed400640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 0x7ff4f809d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 -- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 msgr2=0x7ff4f809d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 0x7ff4f809d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 -- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4f815dfc0 msgr2=0x7ff4f80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4f815dfc0 0x7ff4f80c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 -- 172.21.15.31:0/2787762749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f81f6350 con 0x7ff4f80b2020 2024-09-13T08:44:24.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4ed400640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 0x7ff4f809d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:24.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4f5e00640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c400 tx=0x7ff4e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8017070 con 0x7ff4f80b2020 2024-09-13T08:44:24.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4e80053c0 con 0x7ff4f80b2020 2024-09-13T08:44:24.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8003850 con 0x7ff4f80b2020 2024-09-13T08:44:24.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.507+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f81f5590 con 0x7ff4f80b2020 2024-09-13T08:44:24.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.508+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f81f5b50 con 0x7ff4f80b2020 2024-09-13T08:44:24.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.509+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f80ce0b0 con 0x7ff4f80b2020 2024-09-13T08:44:24.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.510+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7ff4e801a070 con 0x7ff4f80b2020 2024-09-13T08:44:24.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.511+0000 7ff4eea00640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 0x7ff4d407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.511+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7ff4e809a530 con 0x7ff4f80b2020 2024-09-13T08:44:24.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.511+0000 7ff4ed400640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 0x7ff4d407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.512+0000 7ff4ed400640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 0x7ff4d407b190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0007fd0 tx=0x7ff4e0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.515+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff4f80ce0b0 con 0x7ff4f80b2020 2024-09-13T08:44:24.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.712+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4f80cc300 con 0x7ff4f80b2020 2024-09-13T08:44:24.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.713+0000 7ff4eea00640 1 -- 172.21.15.31:0/2787762749 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+16639 (secure 0 0 0) 0x7ff4f80cbe40 con 0x7ff4f80b2020 2024-09-13T08:44:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:24.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":124,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:21.878267+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:18.106733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:19.279918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 msgr2=0x7ff4d407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 0x7ff4d407b190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0007fd0 tx=0x7ff4e0009040 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 msgr2=0x7ff4f81573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c400 tx=0x7ff4e800c8c0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 shutdown_connections 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4d4078cd0 0x7ff4d407b190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4f815dfc0 0x7ff4f80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4f80ca5c0 0x7ff4f809d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 --2- 172.21.15.31:0/2787762749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f80b2020 0x7ff4f81573a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 >> 172.21.15.31:0/2787762749 conn(0x7ff4f80cc910 msgr2=0x7ff4f80ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 shutdown_connections 2024-09-13T08:44:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.716+0000 7ff4fd200640 1 -- 172.21.15.31:0/2787762749 wait complete. 2024-09-13T08:44:24.734 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:44:24.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:44:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.895+0000 7f81a9e00640 1 -- 172.21.15.31:0/3899810116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d44f0 msgr2=0x7f81a40b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.895+0000 7f81a9e00640 1 --2- 172.21.15.31:0/3899810116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d44f0 0x7f81a40b7150 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f81a408d310 tx=0x7f819c02ab50 comp rx=0 tx=0).stop 2024-09-13T08:44:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 -- 172.21.15.31:0/3899810116 shutdown_connections 2024-09-13T08:44:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 --2- 172.21.15.31:0/3899810116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d44f0 0x7f81a40b7150 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 --2- 172.21.15.31:0/3899810116 >> v2:172.21.15.45:3300/0 conn(0x7f81a40cf2d0 0x7f81a40b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 -- 172.21.15.31:0/3899810116 >> 172.21.15.31:0/3899810116 conn(0x7f81a409d1c0 msgr2=0x7f81a40d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:24.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 -- 172.21.15.31:0/3899810116 shutdown_connections 2024-09-13T08:44:24.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.896+0000 7f81a9e00640 1 -- 172.21.15.31:0/3899810116 wait complete. 2024-09-13T08:44:24.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.897+0000 7f81a9e00640 1 Processor -- start 2024-09-13T08:44:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.897+0000 7f81a9e00640 1 -- start start 2024-09-13T08:44:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.898+0000 7f81a9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 0x7f81a409fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.898+0000 7f81a9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a9e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f81a40d44f0 0x7f81a41f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a9e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f81a41628f0 con 0x7f81a40d20c0 2024-09-13T08:44:24.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f81a4162770 con 0x7f81a40cf2d0 2024-09-13T08:44:24.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a9e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f81a4162a70 con 0x7f81a40d44f0 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 0x7f81a409fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 0x7f81a409fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:59704/0 (socket says 172.21.15.31:59704) 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46754/0 (socket says 172.21.15.31:46754) 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a8a00640 1 -- 172.21.15.31:0/2003202683 learned_addr learned my addr 172.21.15.31:0/2003202683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a9400640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f81a40d44f0 0x7f81a41f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 -- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f81a40d44f0 msgr2=0x7f81a41f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f81a40d44f0 0x7f81a41f8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 -- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 msgr2=0x7f81a409fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 0x7f81a409fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.899+0000 7f81a3e00640 1 -- 172.21.15.31:0/2003202683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40a06e0 con 0x7f81a40d20c0 2024-09-13T08:44:24.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.900+0000 7f81a3e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f81a40cf700 tx=0x7f819000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.902+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190017070 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.902+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8190004110 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.902+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a41f4ca0 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.902+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81900099d0 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.902+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a41f5260 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.903+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a40b6740 con 0x7f81a40d20c0 2024-09-13T08:44:24.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.904+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f8190009b70 con 0x7f81a40d20c0 2024-09-13T08:44:24.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.905+0000 7f81a0c00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 0x7f818407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:24.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.905+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f819009b550 con 0x7f81a40d20c0 2024-09-13T08:44:24.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.906+0000 7f81a8a00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 0x7f818407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:24.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.906+0000 7f81a8a00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 0x7f818407b190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8194001880 tx=0x7f8194009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:24.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:24.910+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f81a40b6740 con 0x7f81a40d20c0 2024-09-13T08:44:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.101+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81a4162060 con 0x7f8184078cd0 2024-09-13T08:44:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.106+0000 7f81a0c00640 1 -- 172.21.15.31:0/2003202683 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7f81a4162060 con 0x7f8184078cd0 2024-09-13T08:44:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 msgr2=0x7f818407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 0x7f818407b190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8194001880 tx=0x7f8194009040 comp rx=0 tx=0).stop 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 msgr2=0x7f81a41f86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f81a40cf700 tx=0x7f819000dbb0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 shutdown_connections 2024-09-13T08:44:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8184078cd0 0x7f818407b190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f81a40d44f0 0x7f81a41f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81a40d20c0 0x7f81a41f86b0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 --2- 172.21.15.31:0/2003202683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81a40cf2d0 0x7f81a409fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 >> 172.21.15.31:0/2003202683 conn(0x7f81a409d1c0 msgr2=0x7f81a40d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 shutdown_connections 2024-09-13T08:44:25.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.110+0000 7f81a9e00640 1 -- 172.21.15.31:0/2003202683 wait complete. 2024-09-13T08:44:25.127 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-13T08:44:24.991249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279360,"kb_used_data":13160,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754695360,"statfs":{"total":773094113280,"available":772808048640,"internally_reserved":0,"allocated":13475840,"data_stored":6576116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001136"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887316+0000","last_change":"2024-09-13T08:44:20.903156+0000","last_active":"2024-09-13T08:44:21.887316+0000","last_peered":"2024-09-13T08:44:21.887316+0000","last_clean":"2024-09-13T08:44:21.887316+0000","last_became_active":"2024-09-13T08:44:20.902838+0000","last_became_peered":"2024-09-13T08:44:20.902838+0000","last_unstale":"2024-09-13T08:44:21.887316+0000","last_undegraded":"2024-09-13T08:44:21.887316+0000","last_fullsized":"2024-09-13T08:44:21.887316+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:51.360605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888526+0000","last_change":"2024-09-13T08:44:20.889599+0000","last_active":"2024-09-13T08:44:21.888526+0000","last_peered":"2024-09-13T08:44:21.888526+0000","last_clean":"2024-09-13T08:44:21.888526+0000","last_became_active":"2024-09-13T08:44:20.888732+0000","last_became_peered":"2024-09-13T08:44:20.888732+0000","last_unstale":"2024-09-13T08:44:21.888526+0000","last_undegraded":"2024-09-13T08:44:21.888526+0000","last_fullsized":"2024-09-13T08:44:21.888526+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:10:12.086913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904192+0000","last_change":"2024-09-13T08:44:20.896815+0000","last_active":"2024-09-13T08:44:20.904192+0000","last_peered":"2024-09-13T08:44:20.904192+0000","last_clean":"2024-09-13T08:44:20.904192+0000","last_became_active":"2024-09-13T08:44:20.896450+0000","last_became_peered":"2024-09-13T08:44:20.896450+0000","last_unstale":"2024-09-13T08:44:20.904192+0000","last_undegraded":"2024-09-13T08:44:20.904192+0000","last_fullsized":"2024-09-13T08:44:20.904192+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:07.821014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904129+0000","last_change":"2024-09-13T08:44:20.902510+0000","last_active":"2024-09-13T08:44:20.904129+0000","last_peered":"2024-09-13T08:44:20.904129+0000","last_clean":"2024-09-13T08:44:20.904129+0000","last_became_active":"2024-09-13T08:44:20.902193+0000","last_became_peered":"2024-09-13T08:44:20.902193+0000","last_unstale":"2024-09-13T08:44:20.904129+0000","last_undegraded":"2024-09-13T08:44:20.904129+0000","last_fullsized":"2024-09-13T08:44:20.904129+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:13:33.493726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896280+0000","last_change":"2024-09-13T08:44:20.895185+0000","last_active":"2024-09-13T08:44:20.896280+0000","last_peered":"2024-09-13T08:44:20.896280+0000","last_clean":"2024-09-13T08:44:20.896280+0000","last_became_active":"2024-09-13T08:44:20.894862+0000","last_became_peered":"2024-09-13T08:44:20.894862+0000","last_unstale":"2024-09-13T08:44:20.896280+0000","last_undegraded":"2024-09-13T08:44:20.896280+0000","last_fullsized":"2024-09-13T08:44:20.896280+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:24.011812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885673+0000","last_change":"2024-09-13T08:44:20.890323+0000","last_active":"2024-09-13T08:44:21.885673+0000","last_peered":"2024-09-13T08:44:21.885673+0000","last_clean":"2024-09-13T08:44:21.885673+0000","last_became_active":"2024-09-13T08:44:20.889689+0000","last_became_peered":"2024-09-13T08:44:20.889689+0000","last_unstale":"2024-09-13T08:44:21.885673+0000","last_undegraded":"2024-09-13T08:44:21.885673+0000","last_fullsized":"2024-09-13T08:44:21.885673+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:59.911049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907584+0000","last_change":"2024-09-13T08:44:20.905422+0000","last_active":"2024-09-13T08:44:20.907584+0000","last_peered":"2024-09-13T08:44:20.907584+0000","last_clean":"2024-09-13T08:44:20.907584+0000","last_became_active":"2024-09-13T08:44:20.904119+0000","last_became_peered":"2024-09-13T08:44:20.904119+0000","last_unstale":"2024-09-13T08:44:20.907584+0000","last_undegraded":"2024-09-13T08:44:20.907584+0000","last_fullsized":"2024-09-13T08:44:20.907584+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:45.164786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888024+0000","last_change":"2024-09-13T08:44:20.887357+0000","last_active":"2024-09-13T08:44:20.888024+0000","last_peered":"2024-09-13T08:44:20.888024+0000","last_clean":"2024-09-13T08:44:20.888024+0000","last_became_active":"2024-09-13T08:44:20.886582+0000","last_became_peered":"2024-09-13T08:44:20.886582+0000","last_unstale":"2024-09-13T08:44:20.888024+0000","last_undegraded":"2024-09-13T08:44:20.888024+0000","last_fullsized":"2024-09-13T08:44:20.888024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:18.105656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904115+0000","last_change":"2024-09-13T08:44:20.897282+0000","last_active":"2024-09-13T08:44:20.904115+0000","last_peered":"2024-09-13T08:44:20.904115+0000","last_clean":"2024-09-13T08:44:20.904115+0000","last_became_active":"2024-09-13T08:44:20.896799+0000","last_became_peered":"2024-09-13T08:44:20.896799+0000","last_unstale":"2024-09-13T08:44:20.904115+0000","last_undegraded":"2024-09-13T08:44:20.904115+0000","last_fullsized":"2024-09-13T08:44:20.904115+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:10.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888567+0000","last_change":"2024-09-13T08:44:20.902329+0000","last_active":"2024-09-13T08:44:21.888567+0000","last_peered":"2024-09-13T08:44:21.888567+0000","last_clean":"2024-09-13T08:44:21.888567+0000","last_became_active":"2024-09-13T08:44:20.902041+0000","last_became_peered":"2024-09-13T08:44:20.902041+0000","last_unstale":"2024-09-13T08:44:21.888567+0000","last_undegraded":"2024-09-13T08:44:21.888567+0000","last_fullsized":"2024-09-13T08:44:21.888567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:28.417134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896390+0000","last_change":"2024-09-13T08:44:20.888254+0000","last_active":"2024-09-13T08:44:20.896390+0000","last_peered":"2024-09-13T08:44:20.896390+0000","last_clean":"2024-09-13T08:44:20.896390+0000","last_became_active":"2024-09-13T08:44:20.887463+0000","last_became_peered":"2024-09-13T08:44:20.887463+0000","last_unstale":"2024-09-13T08:44:20.896390+0000","last_undegraded":"2024-09-13T08:44:20.896390+0000","last_fullsized":"2024-09-13T08:44:20.896390+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:51.903261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886109+0000","last_change":"2024-09-13T08:44:20.906982+0000","last_active":"2024-09-13T08:44:21.886109+0000","last_peered":"2024-09-13T08:44:21.886109+0000","last_clean":"2024-09-13T08:44:21.886109+0000","last_became_active":"2024-09-13T08:44:20.906698+0000","last_became_peered":"2024-09-13T08:44:20.906698+0000","last_unstale":"2024-09-13T08:44:21.886109+0000","last_undegraded":"2024-09-13T08:44:21.886109+0000","last_fullsized":"2024-09-13T08:44:21.886109+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:30:49.485090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907281+0000","last_change":"2024-09-13T08:44:20.906332+0000","last_active":"2024-09-13T08:44:20.907281+0000","last_peered":"2024-09-13T08:44:20.907281+0000","last_clean":"2024-09-13T08:44:20.907281+0000","last_became_active":"2024-09-13T08:44:20.906063+0000","last_became_peered":"2024-09-13T08:44:20.906063+0000","last_unstale":"2024-09-13T08:44:20.907281+0000","last_undegraded":"2024-09-13T08:44:20.907281+0000","last_fullsized":"2024-09-13T08:44:20.907281+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:14.962867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.880735+0000","last_change":"2024-09-13T08:44:20.874577+0000","last_active":"2024-09-13T08:44:20.880735+0000","last_peered":"2024-09-13T08:44:20.880735+0000","last_clean":"2024-09-13T08:44:20.880735+0000","last_became_active":"2024-09-13T08:44:20.874291+0000","last_became_peered":"2024-09-13T08:44:20.874291+0000","last_unstale":"2024-09-13T08:44:20.880735+0000","last_undegraded":"2024-09-13T08:44:20.880735+0000","last_fullsized":"2024-09-13T08:44:20.880735+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:24.813960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888448+0000","last_change":"2024-09-13T08:44:20.888641+0000","last_active":"2024-09-13T08:44:21.888448+0000","last_peered":"2024-09-13T08:44:21.888448+0000","last_clean":"2024-09-13T08:44:21.888448+0000","last_became_active":"2024-09-13T08:44:20.888374+0000","last_became_peered":"2024-09-13T08:44:20.888374+0000","last_unstale":"2024-09-13T08:44:21.888448+0000","last_undegraded":"2024-09-13T08:44:21.888448+0000","last_fullsized":"2024-09-13T08:44:21.888448+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:29.445866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903039+0000","last_change":"2024-09-13T08:44:20.901320+0000","last_active":"2024-09-13T08:44:20.903039+0000","last_peered":"2024-09-13T08:44:20.903039+0000","last_clean":"2024-09-13T08:44:20.903039+0000","last_became_active":"2024-09-13T08:44:20.901007+0000","last_became_peered":"2024-09-13T08:44:20.901007+0000","last_unstale":"2024-09-13T08:44:20.903039+0000","last_undegraded":"2024-09-13T08:44:20.903039+0000","last_fullsized":"2024-09-13T08:44:20.903039+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:03.428994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904136+0000","last_change":"2024-09-13T08:44:20.897209+0000","last_active":"2024-09-13T08:44:20.904136+0000","last_peered":"2024-09-13T08:44:20.904136+0000","last_clean":"2024-09-13T08:44:20.904136+0000","last_became_active":"2024-09-13T08:44:20.896473+0000","last_became_peered":"2024-09-13T08:44:20.896473+0000","last_unstale":"2024-09-13T08:44:20.904136+0000","last_undegraded":"2024-09-13T08:44:20.904136+0000","last_fullsized":"2024-09-13T08:44:20.904136+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:27.945015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888239+0000","last_change":"2024-09-13T08:44:20.911434+0000","last_active":"2024-09-13T08:44:21.888239+0000","last_peered":"2024-09-13T08:44:21.888239+0000","last_clean":"2024-09-13T08:44:21.888239+0000","last_became_active":"2024-09-13T08:44:20.911149+0000","last_became_peered":"2024-09-13T08:44:20.911149+0000","last_unstale":"2024-09-13T08:44:21.888239+0000","last_undegraded":"2024-09-13T08:44:21.888239+0000","last_fullsized":"2024-09-13T08:44:21.888239+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:44.154166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889097+0000","last_change":"2024-09-13T08:44:20.888958+0000","last_active":"2024-09-13T08:44:21.889097+0000","last_peered":"2024-09-13T08:44:21.889097+0000","last_clean":"2024-09-13T08:44:21.889097+0000","last_became_active":"2024-09-13T08:44:20.888575+0000","last_became_peered":"2024-09-13T08:44:20.888575+0000","last_unstale":"2024-09-13T08:44:21.889097+0000","last_undegraded":"2024-09-13T08:44:21.889097+0000","last_fullsized":"2024-09-13T08:44:21.889097+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:59.324890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887973+0000","last_change":"2024-09-13T08:44:20.881894+0000","last_active":"2024-09-13T08:44:21.887973+0000","last_peered":"2024-09-13T08:44:21.887973+0000","last_clean":"2024-09-13T08:44:21.887973+0000","last_became_active":"2024-09-13T08:44:20.881644+0000","last_became_peered":"2024-09-13T08:44:20.881644+0000","last_unstale":"2024-09-13T08:44:21.887973+0000","last_undegraded":"2024-09-13T08:44:21.887973+0000","last_fullsized":"2024-09-13T08:44:21.887973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:13.214418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885968+0000","last_change":"2024-09-13T08:44:20.888360+0000","last_active":"2024-09-13T08:44:21.885968+0000","last_peered":"2024-09-13T08:44:21.885968+0000","last_clean":"2024-09-13T08:44:21.885968+0000","last_became_active":"2024-09-13T08:44:20.887708+0000","last_became_peered":"2024-09-13T08:44:20.887708+0000","last_unstale":"2024-09-13T08:44:21.885968+0000","last_undegraded":"2024-09-13T08:44:21.885968+0000","last_fullsized":"2024-09-13T08:44:21.885968+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:35.020541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889239+0000","last_change":"2024-09-13T08:44:20.894756+0000","last_active":"2024-09-13T08:44:21.889239+0000","last_peered":"2024-09-13T08:44:21.889239+0000","last_clean":"2024-09-13T08:44:21.889239+0000","last_became_active":"2024-09-13T08:44:20.893097+0000","last_became_peered":"2024-09-13T08:44:20.893097+0000","last_unstale":"2024-09-13T08:44:21.889239+0000","last_undegraded":"2024-09-13T08:44:21.889239+0000","last_fullsized":"2024-09-13T08:44:21.889239+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:30:17.355606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900325+0000","last_change":"2024-09-13T08:44:20.896015+0000","last_active":"2024-09-13T08:44:20.900325+0000","last_peered":"2024-09-13T08:44:20.900325+0000","last_clean":"2024-09-13T08:44:20.900325+0000","last_became_active":"2024-09-13T08:44:20.895283+0000","last_became_peered":"2024-09-13T08:44:20.895283+0000","last_unstale":"2024-09-13T08:44:20.900325+0000","last_undegraded":"2024-09-13T08:44:20.900325+0000","last_fullsized":"2024-09-13T08:44:20.900325+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:22:11.724821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887020+0000","last_change":"2024-09-13T08:44:20.878267+0000","last_active":"2024-09-13T08:44:21.887020+0000","last_peered":"2024-09-13T08:44:21.887020+0000","last_clean":"2024-09-13T08:44:21.887020+0000","last_became_active":"2024-09-13T08:44:20.877981+0000","last_became_peered":"2024-09-13T08:44:20.877981+0000","last_unstale":"2024-09-13T08:44:21.887020+0000","last_undegraded":"2024-09-13T08:44:21.887020+0000","last_fullsized":"2024-09-13T08:44:21.887020+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:04.578179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.901272+0000","last_change":"2024-09-13T08:44:20.899645+0000","last_active":"2024-09-13T08:44:20.901272+0000","last_peered":"2024-09-13T08:44:20.901272+0000","last_clean":"2024-09-13T08:44:20.901272+0000","last_became_active":"2024-09-13T08:44:20.899314+0000","last_became_peered":"2024-09-13T08:44:20.899314+0000","last_unstale":"2024-09-13T08:44:20.901272+0000","last_undegraded":"2024-09-13T08:44:20.901272+0000","last_fullsized":"2024-09-13T08:44:20.901272+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:41:46.604033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888774+0000","last_change":"2024-09-13T08:44:20.892572+0000","last_active":"2024-09-13T08:44:21.888774+0000","last_peered":"2024-09-13T08:44:21.888774+0000","last_clean":"2024-09-13T08:44:21.888774+0000","last_became_active":"2024-09-13T08:44:20.892205+0000","last_became_peered":"2024-09-13T08:44:20.892205+0000","last_unstale":"2024-09-13T08:44:21.888774+0000","last_undegraded":"2024-09-13T08:44:21.888774+0000","last_fullsized":"2024-09-13T08:44:21.888774+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:40:36.918718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.885561+0000","last_change":"2024-09-13T08:44:20.879961+0000","last_active":"2024-09-13T08:44:20.885561+0000","last_peered":"2024-09-13T08:44:20.885561+0000","last_clean":"2024-09-13T08:44:20.885561+0000","last_became_active":"2024-09-13T08:44:20.879014+0000","last_became_peered":"2024-09-13T08:44:20.879014+0000","last_unstale":"2024-09-13T08:44:20.885561+0000","last_undegraded":"2024-09-13T08:44:20.885561+0000","last_fullsized":"2024-09-13T08:44:20.885561+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:34.230969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909309+0000","last_change":"2024-09-13T08:44:20.899245+0000","last_active":"2024-09-13T08:44:20.909309+0000","last_peered":"2024-09-13T08:44:20.909309+0000","last_clean":"2024-09-13T08:44:20.909309+0000","last_became_active":"2024-09-13T08:44:20.898987+0000","last_became_peered":"2024-09-13T08:44:20.898987+0000","last_unstale":"2024-09-13T08:44:20.909309+0000","last_undegraded":"2024-09-13T08:44:20.909309+0000","last_fullsized":"2024-09-13T08:44:20.909309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:21.285653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887541+0000","last_change":"2024-09-13T08:44:20.894293+0000","last_active":"2024-09-13T08:44:21.887541+0000","last_peered":"2024-09-13T08:44:21.887541+0000","last_clean":"2024-09-13T08:44:21.887541+0000","last_became_active":"2024-09-13T08:44:20.894007+0000","last_became_peered":"2024-09-13T08:44:20.894007+0000","last_unstale":"2024-09-13T08:44:21.887541+0000","last_undegraded":"2024-09-13T08:44:21.887541+0000","last_fullsized":"2024-09-13T08:44:21.887541+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:50.547234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889075+0000","last_change":"2024-09-13T08:44:20.897325+0000","last_active":"2024-09-13T08:44:21.889075+0000","last_peered":"2024-09-13T08:44:21.889075+0000","last_clean":"2024-09-13T08:44:21.889075+0000","last_became_active":"2024-09-13T08:44:20.897017+0000","last_became_peered":"2024-09-13T08:44:20.897017+0000","last_unstale":"2024-09-13T08:44:21.889075+0000","last_undegraded":"2024-09-13T08:44:21.889075+0000","last_fullsized":"2024-09-13T08:44:21.889075+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:40.785046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900297+0000","last_change":"2024-09-13T08:44:20.895543+0000","last_active":"2024-09-13T08:44:20.900297+0000","last_peered":"2024-09-13T08:44:20.900297+0000","last_clean":"2024-09-13T08:44:20.900297+0000","last_became_active":"2024-09-13T08:44:20.894846+0000","last_became_peered":"2024-09-13T08:44:20.894846+0000","last_unstale":"2024-09-13T08:44:20.900297+0000","last_undegraded":"2024-09-13T08:44:20.900297+0000","last_fullsized":"2024-09-13T08:44:20.900297+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:24.570405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900425+0000","last_change":"2024-09-13T08:44:20.891157+0000","last_active":"2024-09-13T08:44:20.900425+0000","last_peered":"2024-09-13T08:44:20.900425+0000","last_clean":"2024-09-13T08:44:20.900425+0000","last_became_active":"2024-09-13T08:44:20.890886+0000","last_became_peered":"2024-09-13T08:44:20.890886+0000","last_unstale":"2024-09-13T08:44:20.900425+0000","last_undegraded":"2024-09-13T08:44:20.900425+0000","last_fullsized":"2024-09-13T08:44:20.900425+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:43:50.804092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903177+0000","last_change":"2024-09-13T08:44:20.901491+0000","last_active":"2024-09-13T08:44:20.903177+0000","last_peered":"2024-09-13T08:44:20.903177+0000","last_clean":"2024-09-13T08:44:20.903177+0000","last_became_active":"2024-09-13T08:44:20.901136+0000","last_became_peered":"2024-09-13T08:44:20.901136+0000","last_unstale":"2024-09-13T08:44:20.903177+0000","last_undegraded":"2024-09-13T08:44:20.903177+0000","last_fullsized":"2024-09-13T08:44:20.903177+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:18.157926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"123'4","reported_seq":17,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909670+0000","last_change":"2024-09-13T08:44:19.885646+0000","last_active":"2024-09-13T08:44:20.909670+0000","last_peered":"2024-09-13T08:44:20.909670+0000","last_clean":"2024-09-13T08:44:20.909670+0000","last_became_active":"2024-09-13T08:44:19.885357+0000","last_became_peered":"2024-09-13T08:44:19.885357+0000","last_unstale":"2024-09-13T08:44:20.909670+0000","last_undegraded":"2024-09-13T08:44:20.909670+0000","last_fullsized":"2024-09-13T08:44:20.909670+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:28.140984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904090+0000","last_change":"2024-09-13T08:44:20.897017+0000","last_active":"2024-09-13T08:44:20.904090+0000","last_peered":"2024-09-13T08:44:20.904090+0000","last_clean":"2024-09-13T08:44:20.904090+0000","last_became_active":"2024-09-13T08:44:20.896739+0000","last_became_peered":"2024-09-13T08:44:20.896739+0000","last_unstale":"2024-09-13T08:44:20.904090+0000","last_undegraded":"2024-09-13T08:44:20.904090+0000","last_fullsized":"2024-09-13T08:44:20.904090+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:07.781047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"123'1","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909165+0000","last_change":"2024-09-13T08:44:19.891217+0000","last_active":"2024-09-13T08:44:20.909165+0000","last_peered":"2024-09-13T08:44:20.909165+0000","last_clean":"2024-09-13T08:44:20.909165+0000","last_became_active":"2024-09-13T08:44:19.890974+0000","last_became_peered":"2024-09-13T08:44:19.890974+0000","last_unstale":"2024-09-13T08:44:20.909165+0000","last_undegraded":"2024-09-13T08:44:20.909165+0000","last_fullsized":"2024-09-13T08:44:20.909165+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:24.155271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910017+0000","last_change":"2024-09-13T08:44:20.903157+0000","last_active":"2024-09-13T08:44:20.910017+0000","last_peered":"2024-09-13T08:44:20.910017+0000","last_clean":"2024-09-13T08:44:20.910017+0000","last_became_active":"2024-09-13T08:44:20.902892+0000","last_became_peered":"2024-09-13T08:44:20.902892+0000","last_unstale":"2024-09-13T08:44:20.910017+0000","last_undegraded":"2024-09-13T08:44:20.910017+0000","last_fullsized":"2024-09-13T08:44:20.910017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:47.460535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.899809+0000","last_change":"2024-09-13T08:44:19.883574+0000","last_active":"2024-09-13T08:44:20.899809+0000","last_peered":"2024-09-13T08:44:20.899809+0000","last_clean":"2024-09-13T08:44:20.899809+0000","last_became_active":"2024-09-13T08:44:19.883258+0000","last_became_peered":"2024-09-13T08:44:19.883258+0000","last_unstale":"2024-09-13T08:44:20.899809+0000","last_undegraded":"2024-09-13T08:44:20.899809+0000","last_fullsized":"2024-09-13T08:44:20.899809+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:30:48.474033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886338+0000","last_change":"2024-09-13T08:44:20.889023+0000","last_active":"2024-09-13T08:44:21.886338+0000","last_peered":"2024-09-13T08:44:21.886338+0000","last_clean":"2024-09-13T08:44:21.886338+0000","last_became_active":"2024-09-13T08:44:20.886126+0000","last_became_peered":"2024-09-13T08:44:20.886126+0000","last_unstale":"2024-09-13T08:44:21.886338+0000","last_undegraded":"2024-09-13T08:44:21.886338+0000","last_fullsized":"2024-09-13T08:44:21.886338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:41:46.604033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888211+0000","last_change":"2024-09-13T08:44:19.884249+0000","last_active":"2024-09-13T08:44:21.888211+0000","last_peered":"2024-09-13T08:44:21.888211+0000","last_clean":"2024-09-13T08:44:21.888211+0000","last_became_active":"2024-09-13T08:44:19.883916+0000","last_became_peered":"2024-09-13T08:44:19.883916+0000","last_unstale":"2024-09-13T08:44:21.888211+0000","last_undegraded":"2024-09-13T08:44:21.888211+0000","last_fullsized":"2024-09-13T08:44:21.888211+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:01.249969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888145+0000","last_change":"2024-09-13T08:44:20.888079+0000","last_active":"2024-09-13T08:44:21.888145+0000","last_peered":"2024-09-13T08:44:21.888145+0000","last_clean":"2024-09-13T08:44:21.888145+0000","last_became_active":"2024-09-13T08:44:20.886327+0000","last_became_peered":"2024-09-13T08:44:20.886327+0000","last_unstale":"2024-09-13T08:44:21.888145+0000","last_undegraded":"2024-09-13T08:44:21.888145+0000","last_fullsized":"2024-09-13T08:44:21.888145+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:35.470332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887693+0000","last_change":"2024-09-13T08:44:19.878895+0000","last_active":"2024-09-13T08:44:21.887693+0000","last_peered":"2024-09-13T08:44:21.887693+0000","last_clean":"2024-09-13T08:44:21.887693+0000","last_became_active":"2024-09-13T08:44:19.876269+0000","last_became_peered":"2024-09-13T08:44:19.876269+0000","last_unstale":"2024-09-13T08:44:21.887693+0000","last_undegraded":"2024-09-13T08:44:21.887693+0000","last_fullsized":"2024-09-13T08:44:21.887693+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:30.961922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886172+0000","last_change":"2024-09-13T08:44:20.880296+0000","last_active":"2024-09-13T08:44:21.886172+0000","last_peered":"2024-09-13T08:44:21.886172+0000","last_clean":"2024-09-13T08:44:21.886172+0000","last_became_active":"2024-09-13T08:44:20.880037+0000","last_became_peered":"2024-09-13T08:44:20.880037+0000","last_unstale":"2024-09-13T08:44:21.886172+0000","last_undegraded":"2024-09-13T08:44:21.886172+0000","last_fullsized":"2024-09-13T08:44:21.886172+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:47.908231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888221+0000","last_change":"2024-09-13T08:44:19.881761+0000","last_active":"2024-09-13T08:44:21.888221+0000","last_peered":"2024-09-13T08:44:21.888221+0000","last_clean":"2024-09-13T08:44:21.888221+0000","last_became_active":"2024-09-13T08:44:19.881425+0000","last_became_peered":"2024-09-13T08:44:19.881425+0000","last_unstale":"2024-09-13T08:44:21.888221+0000","last_undegraded":"2024-09-13T08:44:21.888221+0000","last_fullsized":"2024-09-13T08:44:21.888221+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:40:50.583674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885677+0000","last_change":"2024-09-13T08:44:20.889069+0000","last_active":"2024-09-13T08:44:21.885677+0000","last_peered":"2024-09-13T08:44:21.885677+0000","last_clean":"2024-09-13T08:44:21.885677+0000","last_became_active":"2024-09-13T08:44:20.888806+0000","last_became_peered":"2024-09-13T08:44:20.888806+0000","last_unstale":"2024-09-13T08:44:21.885677+0000","last_undegraded":"2024-09-13T08:44:21.885677+0000","last_fullsized":"2024-09-13T08:44:21.885677+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:47.892851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888408+0000","last_change":"2024-09-13T08:44:19.893478+0000","last_active":"2024-09-13T08:44:20.888408+0000","last_peered":"2024-09-13T08:44:20.888408+0000","last_clean":"2024-09-13T08:44:20.888408+0000","last_became_active":"2024-09-13T08:44:19.893152+0000","last_became_peered":"2024-09-13T08:44:19.893152+0000","last_unstale":"2024-09-13T08:44:20.888408+0000","last_undegraded":"2024-09-13T08:44:20.888408+0000","last_fullsized":"2024-09-13T08:44:20.888408+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:34.210418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887137+0000","last_change":"2024-09-13T08:44:20.886922+0000","last_active":"2024-09-13T08:44:21.887137+0000","last_peered":"2024-09-13T08:44:21.887137+0000","last_clean":"2024-09-13T08:44:21.887137+0000","last_became_active":"2024-09-13T08:44:20.885056+0000","last_became_peered":"2024-09-13T08:44:20.885056+0000","last_unstale":"2024-09-13T08:44:21.887137+0000","last_undegraded":"2024-09-13T08:44:21.887137+0000","last_fullsized":"2024-09-13T08:44:21.887137+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:56.485333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887002+0000","last_change":"2024-09-13T08:44:19.886829+0000","last_active":"2024-09-13T08:44:21.887002+0000","last_peered":"2024-09-13T08:44:21.887002+0000","last_clean":"2024-09-13T08:44:21.887002+0000","last_became_active":"2024-09-13T08:44:19.886539+0000","last_became_peered":"2024-09-13T08:44:19.886539+0000","last_unstale":"2024-09-13T08:44:21.887002+0000","last_undegraded":"2024-09-13T08:44:21.887002+0000","last_fullsized":"2024-09-13T08:44:21.887002+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:07.028864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889088+0000","last_change":"2024-09-13T08:44:20.885538+0000","last_active":"2024-09-13T08:44:21.889088+0000","last_peered":"2024-09-13T08:44:21.889088+0000","last_clean":"2024-09-13T08:44:21.889088+0000","last_became_active":"2024-09-13T08:44:20.883566+0000","last_became_peered":"2024-09-13T08:44:20.883566+0000","last_unstale":"2024-09-13T08:44:21.889088+0000","last_undegraded":"2024-09-13T08:44:21.889088+0000","last_fullsized":"2024-09-13T08:44:21.889088+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:55.531704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886093+0000","last_change":"2024-09-13T08:44:19.884118+0000","last_active":"2024-09-13T08:44:21.886093+0000","last_peered":"2024-09-13T08:44:21.886093+0000","last_clean":"2024-09-13T08:44:21.886093+0000","last_became_active":"2024-09-13T08:44:19.883829+0000","last_became_peered":"2024-09-13T08:44:19.883829+0000","last_unstale":"2024-09-13T08:44:21.886093+0000","last_undegraded":"2024-09-13T08:44:21.886093+0000","last_fullsized":"2024-09-13T08:44:21.886093+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:38.305159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.895032+0000","last_change":"2024-09-13T08:44:20.893149+0000","last_active":"2024-09-13T08:44:20.895032+0000","last_peered":"2024-09-13T08:44:20.895032+0000","last_clean":"2024-09-13T08:44:20.895032+0000","last_became_active":"2024-09-13T08:44:20.892542+0000","last_became_peered":"2024-09-13T08:44:20.892542+0000","last_unstale":"2024-09-13T08:44:20.895032+0000","last_undegraded":"2024-09-13T08:44:20.895032+0000","last_fullsized":"2024-09-13T08:44:20.895032+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:59.576955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888366+0000","last_change":"2024-09-13T08:44:19.883397+0000","last_active":"2024-09-13T08:44:21.888366+0000","last_peered":"2024-09-13T08:44:21.888366+0000","last_clean":"2024-09-13T08:44:21.888366+0000","last_became_active":"2024-09-13T08:44:19.883072+0000","last_became_peered":"2024-09-13T08:44:19.883072+0000","last_unstale":"2024-09-13T08:44:21.888366+0000","last_undegraded":"2024-09-13T08:44:21.888366+0000","last_fullsized":"2024-09-13T08:44:21.888366+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:03:42.127573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886313+0000","last_change":"2024-09-13T08:44:20.906998+0000","last_active":"2024-09-13T08:44:21.886313+0000","last_peered":"2024-09-13T08:44:21.886313+0000","last_clean":"2024-09-13T08:44:21.886313+0000","last_became_active":"2024-09-13T08:44:20.906684+0000","last_became_peered":"2024-09-13T08:44:20.906684+0000","last_unstale":"2024-09-13T08:44:21.886313+0000","last_undegraded":"2024-09-13T08:44:21.886313+0000","last_fullsized":"2024-09-13T08:44:21.886313+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:51.903261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"123'3","reported_seq":16,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.905208+0000","last_change":"2024-09-13T08:44:19.885085+0000","last_active":"2024-09-13T08:44:20.905208+0000","last_peered":"2024-09-13T08:44:20.905208+0000","last_clean":"2024-09-13T08:44:20.905208+0000","last_became_active":"2024-09-13T08:44:19.884807+0000","last_became_peered":"2024-09-13T08:44:19.884807+0000","last_unstale":"2024-09-13T08:44:20.905208+0000","last_undegraded":"2024-09-13T08:44:20.905208+0000","last_fullsized":"2024-09-13T08:44:20.905208+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:31.266566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909358+0000","last_change":"2024-09-13T08:44:20.907482+0000","last_active":"2024-09-13T08:44:20.909358+0000","last_peered":"2024-09-13T08:44:20.909358+0000","last_clean":"2024-09-13T08:44:20.909358+0000","last_became_active":"2024-09-13T08:44:20.907191+0000","last_became_peered":"2024-09-13T08:44:20.907191+0000","last_unstale":"2024-09-13T08:44:20.909358+0000","last_undegraded":"2024-09-13T08:44:20.909358+0000","last_fullsized":"2024-09-13T08:44:20.909358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:50:18.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"123'6","reported_seq":19,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.916378+0000","last_change":"2024-09-13T08:44:19.884211+0000","last_active":"2024-09-13T08:44:20.916378+0000","last_peered":"2024-09-13T08:44:20.916378+0000","last_clean":"2024-09-13T08:44:20.916378+0000","last_became_active":"2024-09-13T08:44:19.883901+0000","last_became_peered":"2024-09-13T08:44:19.883901+0000","last_unstale":"2024-09-13T08:44:20.916378+0000","last_undegraded":"2024-09-13T08:44:20.916378+0000","last_fullsized":"2024-09-13T08:44:20.916378+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:06.442119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888350+0000","last_change":"2024-09-13T08:44:20.887134+0000","last_active":"2024-09-13T08:44:20.888350+0000","last_peered":"2024-09-13T08:44:20.888350+0000","last_clean":"2024-09-13T08:44:20.888350+0000","last_became_active":"2024-09-13T08:44:20.886817+0000","last_became_peered":"2024-09-13T08:44:20.886817+0000","last_unstale":"2024-09-13T08:44:20.888350+0000","last_undegraded":"2024-09-13T08:44:20.888350+0000","last_fullsized":"2024-09-13T08:44:20.888350+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:17.100946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.871656+0000","last_change":"2024-09-13T08:44:19.890493+0000","last_active":"2024-09-13T08:44:20.871656+0000","last_peered":"2024-09-13T08:44:20.871656+0000","last_clean":"2024-09-13T08:44:20.871656+0000","last_became_active":"2024-09-13T08:44:19.890164+0000","last_became_peered":"2024-09-13T08:44:19.890164+0000","last_unstale":"2024-09-13T08:44:20.871656+0000","last_undegraded":"2024-09-13T08:44:20.871656+0000","last_fullsized":"2024-09-13T08:44:20.871656+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:29.960427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.902385+0000","last_change":"2024-09-13T08:44:20.899989+0000","last_active":"2024-09-13T08:44:20.902385+0000","last_peered":"2024-09-13T08:44:20.902385+0000","last_clean":"2024-09-13T08:44:20.902385+0000","last_became_active":"2024-09-13T08:44:20.899656+0000","last_became_peered":"2024-09-13T08:44:20.899656+0000","last_unstale":"2024-09-13T08:44:20.902385+0000","last_undegraded":"2024-09-13T08:44:20.902385+0000","last_fullsized":"2024-09-13T08:44:20.902385+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:26.915057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887051+0000","last_change":"2024-09-13T08:44:19.891406+0000","last_active":"2024-09-13T08:44:20.887051+0000","last_peered":"2024-09-13T08:44:20.887051+0000","last_clean":"2024-09-13T08:44:20.887051+0000","last_became_active":"2024-09-13T08:44:19.891080+0000","last_became_peered":"2024-09-13T08:44:19.891080+0000","last_unstale":"2024-09-13T08:44:20.887051+0000","last_undegraded":"2024-09-13T08:44:20.887051+0000","last_fullsized":"2024-09-13T08:44:20.887051+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:56.783762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885672+0000","last_change":"2024-09-13T08:44:20.899331+0000","last_active":"2024-09-13T08:44:21.885672+0000","last_peered":"2024-09-13T08:44:21.885672+0000","last_clean":"2024-09-13T08:44:21.885672+0000","last_became_active":"2024-09-13T08:44:20.898575+0000","last_became_peered":"2024-09-13T08:44:20.898575+0000","last_unstale":"2024-09-13T08:44:21.885672+0000","last_undegraded":"2024-09-13T08:44:21.885672+0000","last_fullsized":"2024-09-13T08:44:21.885672+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:33.518058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"123'8","reported_seq":23,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885528+0000","last_change":"2024-09-13T08:44:19.874633+0000","last_active":"2024-09-13T08:44:21.885528+0000","last_peered":"2024-09-13T08:44:21.885528+0000","last_clean":"2024-09-13T08:44:21.885528+0000","last_became_active":"2024-09-13T08:44:19.874280+0000","last_became_peered":"2024-09-13T08:44:19.874280+0000","last_unstale":"2024-09-13T08:44:21.885528+0000","last_undegraded":"2024-09-13T08:44:21.885528+0000","last_fullsized":"2024-09-13T08:44:21.885528+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:54.258397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896363+0000","last_change":"2024-09-13T08:44:20.894103+0000","last_active":"2024-09-13T08:44:20.896363+0000","last_peered":"2024-09-13T08:44:20.896363+0000","last_clean":"2024-09-13T08:44:20.896363+0000","last_became_active":"2024-09-13T08:44:20.893474+0000","last_became_peered":"2024-09-13T08:44:20.893474+0000","last_unstale":"2024-09-13T08:44:20.896363+0000","last_undegraded":"2024-09-13T08:44:20.896363+0000","last_fullsized":"2024-09-13T08:44:20.896363+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:49.037689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886178+0000","last_change":"2024-09-13T08:44:19.883066+0000","last_active":"2024-09-13T08:44:21.886178+0000","last_peered":"2024-09-13T08:44:21.886178+0000","last_clean":"2024-09-13T08:44:21.886178+0000","last_became_active":"2024-09-13T08:44:19.882775+0000","last_became_peered":"2024-09-13T08:44:19.882775+0000","last_unstale":"2024-09-13T08:44:21.886178+0000","last_undegraded":"2024-09-13T08:44:21.886178+0000","last_fullsized":"2024-09-13T08:44:21.886178+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:45.641338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896319+0000","last_change":"2024-09-13T08:44:20.893529+0000","last_active":"2024-09-13T08:44:20.896319+0000","last_peered":"2024-09-13T08:44:20.896319+0000","last_clean":"2024-09-13T08:44:20.896319+0000","last_became_active":"2024-09-13T08:44:20.892947+0000","last_became_peered":"2024-09-13T08:44:20.892947+0000","last_unstale":"2024-09-13T08:44:20.896319+0000","last_undegraded":"2024-09-13T08:44:20.896319+0000","last_fullsized":"2024-09-13T08:44:20.896319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:04.307017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.906158+0000","last_change":"2024-09-13T08:44:19.887455+0000","last_active":"2024-09-13T08:44:20.906158+0000","last_peered":"2024-09-13T08:44:20.906158+0000","last_clean":"2024-09-13T08:44:20.906158+0000","last_became_active":"2024-09-13T08:44:19.887033+0000","last_became_peered":"2024-09-13T08:44:19.887033+0000","last_unstale":"2024-09-13T08:44:20.906158+0000","last_undegraded":"2024-09-13T08:44:20.906158+0000","last_fullsized":"2024-09-13T08:44:20.906158+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.897173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888211+0000","last_change":"2024-09-13T08:44:20.890908+0000","last_active":"2024-09-13T08:44:21.888211+0000","last_peered":"2024-09-13T08:44:21.888211+0000","last_clean":"2024-09-13T08:44:21.888211+0000","last_became_active":"2024-09-13T08:44:20.890623+0000","last_became_peered":"2024-09-13T08:44:20.890623+0000","last_unstale":"2024-09-13T08:44:21.888211+0000","last_undegraded":"2024-09-13T08:44:21.888211+0000","last_fullsized":"2024-09-13T08:44:21.888211+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:48.645162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"123'1","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910207+0000","last_change":"2024-09-13T08:44:19.887311+0000","last_active":"2024-09-13T08:44:20.910207+0000","last_peered":"2024-09-13T08:44:20.910207+0000","last_clean":"2024-09-13T08:44:20.910207+0000","last_became_active":"2024-09-13T08:44:19.886919+0000","last_became_peered":"2024-09-13T08:44:19.886919+0000","last_unstale":"2024-09-13T08:44:20.910207+0000","last_undegraded":"2024-09-13T08:44:20.910207+0000","last_fullsized":"2024-09-13T08:44:20.910207+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:40.118017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888609+0000","last_change":"2024-09-13T08:44:20.896009+0000","last_active":"2024-09-13T08:44:21.888609+0000","last_peered":"2024-09-13T08:44:21.888609+0000","last_clean":"2024-09-13T08:44:21.888609+0000","last_became_active":"2024-09-13T08:44:20.895657+0000","last_became_peered":"2024-09-13T08:44:20.895657+0000","last_unstale":"2024-09-13T08:44:21.888609+0000","last_undegraded":"2024-09-13T08:44:21.888609+0000","last_fullsized":"2024-09-13T08:44:21.888609+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:34:38.269551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.882094+0000","last_change":"2024-09-13T08:44:19.879449+0000","last_active":"2024-09-13T08:44:20.882094+0000","last_peered":"2024-09-13T08:44:20.882094+0000","last_clean":"2024-09-13T08:44:20.882094+0000","last_became_active":"2024-09-13T08:44:19.879141+0000","last_became_peered":"2024-09-13T08:44:19.879141+0000","last_unstale":"2024-09-13T08:44:20.882094+0000","last_undegraded":"2024-09-13T08:44:20.882094+0000","last_fullsized":"2024-09-13T08:44:20.882094+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:33.347395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889310+0000","last_change":"2024-09-13T08:44:20.889962+0000","last_active":"2024-09-13T08:44:21.889310+0000","last_peered":"2024-09-13T08:44:21.889310+0000","last_clean":"2024-09-13T08:44:21.889310+0000","last_became_active":"2024-09-13T08:44:20.889062+0000","last_became_peered":"2024-09-13T08:44:20.889062+0000","last_unstale":"2024-09-13T08:44:21.889310+0000","last_undegraded":"2024-09-13T08:44:21.889310+0000","last_fullsized":"2024-09-13T08:44:21.889310+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:15.964693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910776+0000","last_change":"2024-09-13T08:44:19.886650+0000","last_active":"2024-09-13T08:44:20.910776+0000","last_peered":"2024-09-13T08:44:20.910776+0000","last_clean":"2024-09-13T08:44:20.910776+0000","last_became_active":"2024-09-13T08:44:19.886380+0000","last_became_peered":"2024-09-13T08:44:19.886380+0000","last_unstale":"2024-09-13T08:44:20.910776+0000","last_undegraded":"2024-09-13T08:44:20.910776+0000","last_fullsized":"2024-09-13T08:44:20.910776+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:20.938256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907218+0000","last_change":"2024-09-13T08:44:20.905415+0000","last_active":"2024-09-13T08:44:20.907218+0000","last_peered":"2024-09-13T08:44:20.907218+0000","last_clean":"2024-09-13T08:44:20.907218+0000","last_became_active":"2024-09-13T08:44:20.903103+0000","last_became_peered":"2024-09-13T08:44:20.903103+0000","last_unstale":"2024-09-13T08:44:20.907218+0000","last_undegraded":"2024-09-13T08:44:20.907218+0000","last_fullsized":"2024-09-13T08:44:20.907218+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:15.991963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888272+0000","last_change":"2024-09-13T08:44:19.888753+0000","last_active":"2024-09-13T08:44:21.888272+0000","last_peered":"2024-09-13T08:44:21.888272+0000","last_clean":"2024-09-13T08:44:21.888272+0000","last_became_active":"2024-09-13T08:44:19.888377+0000","last_became_peered":"2024-09-13T08:44:19.888377+0000","last_unstale":"2024-09-13T08:44:21.888272+0000","last_undegraded":"2024-09-13T08:44:21.888272+0000","last_fullsized":"2024-09-13T08:44:21.888272+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:33.508644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.880848+0000","last_change":"2024-09-13T08:44:20.874498+0000","last_active":"2024-09-13T08:44:20.880848+0000","last_peered":"2024-09-13T08:44:20.880848+0000","last_clean":"2024-09-13T08:44:20.880848+0000","last_became_active":"2024-09-13T08:44:20.874200+0000","last_became_peered":"2024-09-13T08:44:20.874200+0000","last_unstale":"2024-09-13T08:44:20.880848+0000","last_undegraded":"2024-09-13T08:44:20.880848+0000","last_fullsized":"2024-09-13T08:44:20.880848+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:17.188650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.869586+0000","last_change":"2024-09-13T08:44:19.889479+0000","last_active":"2024-09-13T08:44:20.869586+0000","last_peered":"2024-09-13T08:44:20.869586+0000","last_clean":"2024-09-13T08:44:20.869586+0000","last_became_active":"2024-09-13T08:44:19.889113+0000","last_became_peered":"2024-09-13T08:44:19.889113+0000","last_unstale":"2024-09-13T08:44:20.869586+0000","last_undegraded":"2024-09-13T08:44:20.869586+0000","last_fullsized":"2024-09-13T08:44:20.869586+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:21.723397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885675+0000","last_change":"2024-09-13T08:44:20.875542+0000","last_active":"2024-09-13T08:44:21.885675+0000","last_peered":"2024-09-13T08:44:21.885675+0000","last_clean":"2024-09-13T08:44:21.885675+0000","last_became_active":"2024-09-13T08:44:20.875284+0000","last_became_peered":"2024-09-13T08:44:20.875284+0000","last_unstale":"2024-09-13T08:44:21.885675+0000","last_undegraded":"2024-09-13T08:44:21.885675+0000","last_fullsized":"2024-09-13T08:44:21.885675+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:06.833193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.886621+0000","last_change":"2024-09-13T08:44:19.888095+0000","last_active":"2024-09-13T08:44:20.886621+0000","last_peered":"2024-09-13T08:44:20.886621+0000","last_clean":"2024-09-13T08:44:20.886621+0000","last_became_active":"2024-09-13T08:44:19.887798+0000","last_became_peered":"2024-09-13T08:44:19.887798+0000","last_unstale":"2024-09-13T08:44:20.886621+0000","last_undegraded":"2024-09-13T08:44:20.886621+0000","last_fullsized":"2024-09-13T08:44:20.886621+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:41.762806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.908065+0000","last_change":"2024-09-13T08:44:20.906487+0000","last_active":"2024-09-13T08:44:20.908065+0000","last_peered":"2024-09-13T08:44:20.908065+0000","last_clean":"2024-09-13T08:44:20.908065+0000","last_became_active":"2024-09-13T08:44:20.905996+0000","last_became_peered":"2024-09-13T08:44:20.905996+0000","last_unstale":"2024-09-13T08:44:20.908065+0000","last_undegraded":"2024-09-13T08:44:20.908065+0000","last_fullsized":"2024-09-13T08:44:20.908065+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:44:05.542999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888057+0000","last_change":"2024-09-13T08:44:19.889047+0000","last_active":"2024-09-13T08:44:21.888057+0000","last_peered":"2024-09-13T08:44:21.888057+0000","last_clean":"2024-09-13T08:44:21.888057+0000","last_became_active":"2024-09-13T08:44:19.888682+0000","last_became_peered":"2024-09-13T08:44:19.888682+0000","last_unstale":"2024-09-13T08:44:21.888057+0000","last_undegraded":"2024-09-13T08:44:21.888057+0000","last_fullsized":"2024-09-13T08:44:21.888057+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:59.958964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888848+0000","last_change":"2024-09-13T08:44:20.890851+0000","last_active":"2024-09-13T08:44:21.888848+0000","last_peered":"2024-09-13T08:44:21.888848+0000","last_clean":"2024-09-13T08:44:21.888848+0000","last_became_active":"2024-09-13T08:44:20.890468+0000","last_became_peered":"2024-09-13T08:44:20.890468+0000","last_unstale":"2024-09-13T08:44:21.888848+0000","last_undegraded":"2024-09-13T08:44:21.888848+0000","last_fullsized":"2024-09-13T08:44:21.888848+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:33:35.035688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.882957+0000","last_change":"2024-09-13T08:44:19.891057+0000","last_active":"2024-09-13T08:44:20.882957+0000","last_peered":"2024-09-13T08:44:20.882957+0000","last_clean":"2024-09-13T08:44:20.882957+0000","last_became_active":"2024-09-13T08:44:19.890715+0000","last_became_peered":"2024-09-13T08:44:19.890715+0000","last_unstale":"2024-09-13T08:44:20.882957+0000","last_undegraded":"2024-09-13T08:44:20.882957+0000","last_fullsized":"2024-09-13T08:44:20.882957+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:42.556834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887713+0000","last_change":"2024-09-13T08:44:20.885289+0000","last_active":"2024-09-13T08:44:20.887713+0000","last_peered":"2024-09-13T08:44:20.887713+0000","last_clean":"2024-09-13T08:44:20.887713+0000","last_became_active":"2024-09-13T08:44:20.884955+0000","last_became_peered":"2024-09-13T08:44:20.884955+0000","last_unstale":"2024-09-13T08:44:20.887713+0000","last_undegraded":"2024-09-13T08:44:20.887713+0000","last_fullsized":"2024-09-13T08:44:20.887713+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:34:55.796847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887748+0000","last_change":"2024-09-13T08:44:19.885326+0000","last_active":"2024-09-13T08:44:21.887748+0000","last_peered":"2024-09-13T08:44:21.887748+0000","last_clean":"2024-09-13T08:44:21.887748+0000","last_became_active":"2024-09-13T08:44:19.885013+0000","last_became_peered":"2024-09-13T08:44:19.885013+0000","last_unstale":"2024-09-13T08:44:21.887748+0000","last_undegraded":"2024-09-13T08:44:21.887748+0000","last_fullsized":"2024-09-13T08:44:21.887748+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:38.273975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904170+0000","last_change":"2024-09-13T08:44:20.895971+0000","last_active":"2024-09-13T08:44:20.904170+0000","last_peered":"2024-09-13T08:44:20.904170+0000","last_clean":"2024-09-13T08:44:20.904170+0000","last_became_active":"2024-09-13T08:44:20.895684+0000","last_became_peered":"2024-09-13T08:44:20.895684+0000","last_unstale":"2024-09-13T08:44:20.904170+0000","last_undegraded":"2024-09-13T08:44:20.904170+0000","last_fullsized":"2024-09-13T08:44:20.904170+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:27.008517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.869523+0000","last_change":"2024-09-13T08:44:19.891325+0000","last_active":"2024-09-13T08:44:20.869523+0000","last_peered":"2024-09-13T08:44:20.869523+0000","last_clean":"2024-09-13T08:44:20.869523+0000","last_became_active":"2024-09-13T08:44:19.890989+0000","last_became_peered":"2024-09-13T08:44:19.890989+0000","last_unstale":"2024-09-13T08:44:20.869523+0000","last_undegraded":"2024-09-13T08:44:20.869523+0000","last_fullsized":"2024-09-13T08:44:20.869523+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:33.713455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885901+0000","last_change":"2024-09-13T08:44:20.890009+0000","last_active":"2024-09-13T08:44:21.885901+0000","last_peered":"2024-09-13T08:44:21.885901+0000","last_clean":"2024-09-13T08:44:21.885901+0000","last_became_active":"2024-09-13T08:44:20.889349+0000","last_became_peered":"2024-09-13T08:44:20.889349+0000","last_unstale":"2024-09-13T08:44:21.885901+0000","last_undegraded":"2024-09-13T08:44:21.885901+0000","last_fullsized":"2024-09-13T08:44:21.885901+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:57.794819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887804+0000","last_change":"2024-09-13T08:44:19.881019+0000","last_active":"2024-09-13T08:44:21.887804+0000","last_peered":"2024-09-13T08:44:21.887804+0000","last_clean":"2024-09-13T08:44:21.887804+0000","last_became_active":"2024-09-13T08:44:19.880556+0000","last_became_peered":"2024-09-13T08:44:19.880556+0000","last_unstale":"2024-09-13T08:44:21.887804+0000","last_undegraded":"2024-09-13T08:44:21.887804+0000","last_fullsized":"2024-09-13T08:44:21.887804+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:20.654141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886376+0000","last_change":"2024-09-13T08:44:20.902297+0000","last_active":"2024-09-13T08:44:21.886376+0000","last_peered":"2024-09-13T08:44:21.886376+0000","last_clean":"2024-09-13T08:44:21.886376+0000","last_became_active":"2024-09-13T08:44:20.902005+0000","last_became_peered":"2024-09-13T08:44:20.902005+0000","last_unstale":"2024-09-13T08:44:21.886376+0000","last_undegraded":"2024-09-13T08:44:21.886376+0000","last_fullsized":"2024-09-13T08:44:21.886376+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:13:33.493726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885830+0000","last_change":"2024-09-13T08:44:19.878615+0000","last_active":"2024-09-13T08:44:21.885830+0000","last_peered":"2024-09-13T08:44:21.885830+0000","last_clean":"2024-09-13T08:44:21.885830+0000","last_became_active":"2024-09-13T08:44:19.878324+0000","last_became_peered":"2024-09-13T08:44:19.878324+0000","last_unstale":"2024-09-13T08:44:21.885830+0000","last_undegraded":"2024-09-13T08:44:21.885830+0000","last_fullsized":"2024-09-13T08:44:21.885830+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:36.450192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887405+0000","last_change":"2024-09-13T08:44:20.884536+0000","last_active":"2024-09-13T08:44:21.887405+0000","last_peered":"2024-09-13T08:44:21.887405+0000","last_clean":"2024-09-13T08:44:21.887405+0000","last_became_active":"2024-09-13T08:44:20.880278+0000","last_became_peered":"2024-09-13T08:44:20.880278+0000","last_unstale":"2024-09-13T08:44:21.887405+0000","last_undegraded":"2024-09-13T08:44:21.887405+0000","last_fullsized":"2024-09-13T08:44:21.887405+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:23:39.869307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887305+0000","last_change":"2024-09-13T08:44:19.891311+0000","last_active":"2024-09-13T08:44:20.887305+0000","last_peered":"2024-09-13T08:44:20.887305+0000","last_clean":"2024-09-13T08:44:20.887305+0000","last_became_active":"2024-09-13T08:44:19.891028+0000","last_became_peered":"2024-09-13T08:44:19.891028+0000","last_unstale":"2024-09-13T08:44:20.887305+0000","last_undegraded":"2024-09-13T08:44:20.887305+0000","last_fullsized":"2024-09-13T08:44:20.887305+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.009484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.902381+0000","last_change":"2024-09-13T08:44:20.900334+0000","last_active":"2024-09-13T08:44:20.902381+0000","last_peered":"2024-09-13T08:44:20.902381+0000","last_clean":"2024-09-13T08:44:20.902381+0000","last_became_active":"2024-09-13T08:44:20.899057+0000","last_became_peered":"2024-09-13T08:44:20.899057+0000","last_unstale":"2024-09-13T08:44:20.902381+0000","last_undegraded":"2024-09-13T08:44:20.902381+0000","last_fullsized":"2024-09-13T08:44:20.902381+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:40.187667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.870823+0000","last_change":"2024-09-13T08:44:19.891328+0000","last_active":"2024-09-13T08:44:20.870823+0000","last_peered":"2024-09-13T08:44:20.870823+0000","last_clean":"2024-09-13T08:44:20.870823+0000","last_became_active":"2024-09-13T08:44:19.891013+0000","last_became_peered":"2024-09-13T08:44:19.891013+0000","last_unstale":"2024-09-13T08:44:20.870823+0000","last_undegraded":"2024-09-13T08:44:20.870823+0000","last_fullsized":"2024-09-13T08:44:20.870823+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:20.836057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903476+0000","last_change":"2024-09-13T08:44:20.896143+0000","last_active":"2024-09-13T08:44:20.903476+0000","last_peered":"2024-09-13T08:44:20.903476+0000","last_clean":"2024-09-13T08:44:20.903476+0000","last_became_active":"2024-09-13T08:44:20.895880+0000","last_became_peered":"2024-09-13T08:44:20.895880+0000","last_unstale":"2024-09-13T08:44:20.903476+0000","last_undegraded":"2024-09-13T08:44:20.903476+0000","last_fullsized":"2024-09-13T08:44:20.903476+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:36.845832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886550+0000","last_change":"2024-09-13T08:44:19.879464+0000","last_active":"2024-09-13T08:44:21.886550+0000","last_peered":"2024-09-13T08:44:21.886550+0000","last_clean":"2024-09-13T08:44:21.886550+0000","last_became_active":"2024-09-13T08:44:19.878980+0000","last_became_peered":"2024-09-13T08:44:19.878980+0000","last_unstale":"2024-09-13T08:44:21.886550+0000","last_undegraded":"2024-09-13T08:44:21.886550+0000","last_fullsized":"2024-09-13T08:44:21.886550+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:25.767263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":322,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.879903+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:44:20.879903+0000","last_peered":"2024-09-13T08:44:20.879903+0000","last_clean":"2024-09-13T08:44:20.879903+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:44:20.879903+0000","last_undegraded":"2024-09-13T08:44:20.879903+0000","last_fullsized":"2024-09-13T08:44:20.879903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509532,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36852,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334988,"statfs":{"total":96636764160,"available":96599027712,"internally_reserved":0,"allocated":1531904,"data_stored":673854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1527808,"data_stored":673820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37436,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334404,"statfs":{"total":96636764160,"available":96598429696,"internally_reserved":0,"allocated":2134016,"data_stored":1264930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542235,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1527808,"data_stored":673820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509532,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1531904,"data_stored":673890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509532,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1536000,"data_stored":674492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509533,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2138112,"data_stored":1265822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1548288,"data_stored":675488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:44:25.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 -- 172.21.15.31:0/4133435908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0d0e80 msgr2=0x7f6dec0b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 --2- 172.21.15.31:0/4133435908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0d0e80 0x7f6dec0b3ba0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6dec071550 tx=0x7f6dd402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 -- 172.21.15.31:0/4133435908 shutdown_connections 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 --2- 172.21.15.31:0/4133435908 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0b40e0 0x7f6dec0caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 --2- 172.21.15.31:0/4133435908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0d0e80 0x7f6dec0b3ba0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 --2- 172.21.15.31:0/4133435908 >> v2:172.21.15.45:3300/0 conn(0x7f6dec15e2d0 0x7f6dec0ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 -- 172.21.15.31:0/4133435908 >> 172.21.15.31:0/4133435908 conn(0x7f6dec0d0360 msgr2=0x7f6dec0d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:25.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 -- 172.21.15.31:0/4133435908 shutdown_connections 2024-09-13T08:44:25.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.327+0000 7f6debe00640 1 -- 172.21.15.31:0/4133435908 wait complete. 2024-09-13T08:44:25.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.329+0000 7f6debe00640 1 Processor -- start 2024-09-13T08:44:25.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.329+0000 7f6debe00640 1 -- start start 2024-09-13T08:44:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec0b40e0 0x7f6dec15cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0d0e80 0x7f6dec15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec0ce830 con 0x7f6dec15e2d0 2024-09-13T08:44:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec0ce6b0 con 0x7f6dec0b40e0 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6debe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec0cca20 con 0x7f6dec0d0e80 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6deb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6deb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46768/0 (socket says 172.21.15.31:46768) 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6deb400640 1 -- 172.21.15.31:0/431956587 learned_addr learned my addr 172.21.15.31:0/431956587 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6de2000640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0d0e80 0x7f6dec15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.330+0000 7f6deaa00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec0b40e0 0x7f6dec15cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 -- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0d0e80 msgr2=0x7f6dec15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0d0e80 0x7f6dec15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 -- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec0b40e0 msgr2=0x7f6dec15cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec0b40e0 0x7f6dec15cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 -- 172.21.15.31:0/431956587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec0af620 con 0x7f6dec15e2d0 2024-09-13T08:44:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6deb400640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6de400ca30 tx=0x7f6de400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4004d20 con 0x7f6dec15e2d0 2024-09-13T08:44:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6de4007a70 con 0x7f6dec15e2d0 2024-09-13T08:44:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4005020 con 0x7f6dec15e2d0 2024-09-13T08:44:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec15c6f0 con 0x7f6dec15e2d0 2024-09-13T08:44:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.331+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec15cc90 con 0x7f6dec15e2d0 2024-09-13T08:44:25.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.333+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec070bf0 con 0x7f6dec15e2d0 2024-09-13T08:44:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.334+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f6de40051c0 con 0x7f6dec15e2d0 2024-09-13T08:44:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.334+0000 7f6de3400640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 0x7f6dc807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.335+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f6de409be80 con 0x7f6dec15e2d0 2024-09-13T08:44:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.335+0000 7f6deaa00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 0x7f6dc807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.335+0000 7f6deaa00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 0x7f6dc807b190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0b5eb0 tx=0x7f6ddc00a2d0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:25.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.339+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6dec070bf0 con 0x7f6dec15e2d0 2024-09-13T08:44:25.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.531+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6dec09e3d0 con 0x7f6dc8078cd0 2024-09-13T08:44:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.535+0000 7f6de3400640 1 -- 172.21.15.31:0/431956587 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7f6dec09e3d0 con 0x7f6dc8078cd0 2024-09-13T08:44:25.539 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:25.540 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 msgr2=0x7f6dc807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 0x7f6dc807b190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0b5eb0 tx=0x7f6ddc00a2d0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 msgr2=0x7f6dec15a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6de400ca30 tx=0x7f6de400ce30 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 shutdown_connections 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8078cd0 0x7f6dc807b190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec15e2d0 0x7f6dec15a660 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0d0e80 0x7f6dec15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 --2- 172.21.15.31:0/431956587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec0b40e0 0x7f6dec15cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.539+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 >> 172.21.15.31:0/431956587 conn(0x7f6dec0d0360 msgr2=0x7f6dec0b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.540+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 shutdown_connections 2024-09-13T08:44:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.540+0000 7f6debe00640 1 -- 172.21.15.31:0/431956587 wait complete. 2024-09-13T08:44:25.557 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-13T08:44:24.991249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279360,"kb_used_data":13160,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754695360,"statfs":{"total":773094113280,"available":772808048640,"internally_reserved":0,"allocated":13475840,"data_stored":6576116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001136"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887316+0000","last_change":"2024-09-13T08:44:20.903156+0000","last_active":"2024-09-13T08:44:21.887316+0000","last_peered":"2024-09-13T08:44:21.887316+0000","last_clean":"2024-09-13T08:44:21.887316+0000","last_became_active":"2024-09-13T08:44:20.902838+0000","last_became_peered":"2024-09-13T08:44:20.902838+0000","last_unstale":"2024-09-13T08:44:21.887316+0000","last_undegraded":"2024-09-13T08:44:21.887316+0000","last_fullsized":"2024-09-13T08:44:21.887316+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:51.360605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888526+0000","last_change":"2024-09-13T08:44:20.889599+0000","last_active":"2024-09-13T08:44:21.888526+0000","last_peered":"2024-09-13T08:44:21.888526+0000","last_clean":"2024-09-13T08:44:21.888526+0000","last_became_active":"2024-09-13T08:44:20.888732+0000","last_became_peered":"2024-09-13T08:44:20.888732+0000","last_unstale":"2024-09-13T08:44:21.888526+0000","last_undegraded":"2024-09-13T08:44:21.888526+0000","last_fullsized":"2024-09-13T08:44:21.888526+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:10:12.086913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904192+0000","last_change":"2024-09-13T08:44:20.896815+0000","last_active":"2024-09-13T08:44:20.904192+0000","last_peered":"2024-09-13T08:44:20.904192+0000","last_clean":"2024-09-13T08:44:20.904192+0000","last_became_active":"2024-09-13T08:44:20.896450+0000","last_became_peered":"2024-09-13T08:44:20.896450+0000","last_unstale":"2024-09-13T08:44:20.904192+0000","last_undegraded":"2024-09-13T08:44:20.904192+0000","last_fullsized":"2024-09-13T08:44:20.904192+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:07.821014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904129+0000","last_change":"2024-09-13T08:44:20.902510+0000","last_active":"2024-09-13T08:44:20.904129+0000","last_peered":"2024-09-13T08:44:20.904129+0000","last_clean":"2024-09-13T08:44:20.904129+0000","last_became_active":"2024-09-13T08:44:20.902193+0000","last_became_peered":"2024-09-13T08:44:20.902193+0000","last_unstale":"2024-09-13T08:44:20.904129+0000","last_undegraded":"2024-09-13T08:44:20.904129+0000","last_fullsized":"2024-09-13T08:44:20.904129+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:13:33.493726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896280+0000","last_change":"2024-09-13T08:44:20.895185+0000","last_active":"2024-09-13T08:44:20.896280+0000","last_peered":"2024-09-13T08:44:20.896280+0000","last_clean":"2024-09-13T08:44:20.896280+0000","last_became_active":"2024-09-13T08:44:20.894862+0000","last_became_peered":"2024-09-13T08:44:20.894862+0000","last_unstale":"2024-09-13T08:44:20.896280+0000","last_undegraded":"2024-09-13T08:44:20.896280+0000","last_fullsized":"2024-09-13T08:44:20.896280+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:24.011812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885673+0000","last_change":"2024-09-13T08:44:20.890323+0000","last_active":"2024-09-13T08:44:21.885673+0000","last_peered":"2024-09-13T08:44:21.885673+0000","last_clean":"2024-09-13T08:44:21.885673+0000","last_became_active":"2024-09-13T08:44:20.889689+0000","last_became_peered":"2024-09-13T08:44:20.889689+0000","last_unstale":"2024-09-13T08:44:21.885673+0000","last_undegraded":"2024-09-13T08:44:21.885673+0000","last_fullsized":"2024-09-13T08:44:21.885673+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:59.911049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907584+0000","last_change":"2024-09-13T08:44:20.905422+0000","last_active":"2024-09-13T08:44:20.907584+0000","last_peered":"2024-09-13T08:44:20.907584+0000","last_clean":"2024-09-13T08:44:20.907584+0000","last_became_active":"2024-09-13T08:44:20.904119+0000","last_became_peered":"2024-09-13T08:44:20.904119+0000","last_unstale":"2024-09-13T08:44:20.907584+0000","last_undegraded":"2024-09-13T08:44:20.907584+0000","last_fullsized":"2024-09-13T08:44:20.907584+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:45.164786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888024+0000","last_change":"2024-09-13T08:44:20.887357+0000","last_active":"2024-09-13T08:44:20.888024+0000","last_peered":"2024-09-13T08:44:20.888024+0000","last_clean":"2024-09-13T08:44:20.888024+0000","last_became_active":"2024-09-13T08:44:20.886582+0000","last_became_peered":"2024-09-13T08:44:20.886582+0000","last_unstale":"2024-09-13T08:44:20.888024+0000","last_undegraded":"2024-09-13T08:44:20.888024+0000","last_fullsized":"2024-09-13T08:44:20.888024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:18.105656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904115+0000","last_change":"2024-09-13T08:44:20.897282+0000","last_active":"2024-09-13T08:44:20.904115+0000","last_peered":"2024-09-13T08:44:20.904115+0000","last_clean":"2024-09-13T08:44:20.904115+0000","last_became_active":"2024-09-13T08:44:20.896799+0000","last_became_peered":"2024-09-13T08:44:20.896799+0000","last_unstale":"2024-09-13T08:44:20.904115+0000","last_undegraded":"2024-09-13T08:44:20.904115+0000","last_fullsized":"2024-09-13T08:44:20.904115+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:10.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888567+0000","last_change":"2024-09-13T08:44:20.902329+0000","last_active":"2024-09-13T08:44:21.888567+0000","last_peered":"2024-09-13T08:44:21.888567+0000","last_clean":"2024-09-13T08:44:21.888567+0000","last_became_active":"2024-09-13T08:44:20.902041+0000","last_became_peered":"2024-09-13T08:44:20.902041+0000","last_unstale":"2024-09-13T08:44:21.888567+0000","last_undegraded":"2024-09-13T08:44:21.888567+0000","last_fullsized":"2024-09-13T08:44:21.888567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:28.417134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896390+0000","last_change":"2024-09-13T08:44:20.888254+0000","last_active":"2024-09-13T08:44:20.896390+0000","last_peered":"2024-09-13T08:44:20.896390+0000","last_clean":"2024-09-13T08:44:20.896390+0000","last_became_active":"2024-09-13T08:44:20.887463+0000","last_became_peered":"2024-09-13T08:44:20.887463+0000","last_unstale":"2024-09-13T08:44:20.896390+0000","last_undegraded":"2024-09-13T08:44:20.896390+0000","last_fullsized":"2024-09-13T08:44:20.896390+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:51.903261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886109+0000","last_change":"2024-09-13T08:44:20.906982+0000","last_active":"2024-09-13T08:44:21.886109+0000","last_peered":"2024-09-13T08:44:21.886109+0000","last_clean":"2024-09-13T08:44:21.886109+0000","last_became_active":"2024-09-13T08:44:20.906698+0000","last_became_peered":"2024-09-13T08:44:20.906698+0000","last_unstale":"2024-09-13T08:44:21.886109+0000","last_undegraded":"2024-09-13T08:44:21.886109+0000","last_fullsized":"2024-09-13T08:44:21.886109+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:30:49.485090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907281+0000","last_change":"2024-09-13T08:44:20.906332+0000","last_active":"2024-09-13T08:44:20.907281+0000","last_peered":"2024-09-13T08:44:20.907281+0000","last_clean":"2024-09-13T08:44:20.907281+0000","last_became_active":"2024-09-13T08:44:20.906063+0000","last_became_peered":"2024-09-13T08:44:20.906063+0000","last_unstale":"2024-09-13T08:44:20.907281+0000","last_undegraded":"2024-09-13T08:44:20.907281+0000","last_fullsized":"2024-09-13T08:44:20.907281+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:14.962867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.880735+0000","last_change":"2024-09-13T08:44:20.874577+0000","last_active":"2024-09-13T08:44:20.880735+0000","last_peered":"2024-09-13T08:44:20.880735+0000","last_clean":"2024-09-13T08:44:20.880735+0000","last_became_active":"2024-09-13T08:44:20.874291+0000","last_became_peered":"2024-09-13T08:44:20.874291+0000","last_unstale":"2024-09-13T08:44:20.880735+0000","last_undegraded":"2024-09-13T08:44:20.880735+0000","last_fullsized":"2024-09-13T08:44:20.880735+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:24.813960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888448+0000","last_change":"2024-09-13T08:44:20.888641+0000","last_active":"2024-09-13T08:44:21.888448+0000","last_peered":"2024-09-13T08:44:21.888448+0000","last_clean":"2024-09-13T08:44:21.888448+0000","last_became_active":"2024-09-13T08:44:20.888374+0000","last_became_peered":"2024-09-13T08:44:20.888374+0000","last_unstale":"2024-09-13T08:44:21.888448+0000","last_undegraded":"2024-09-13T08:44:21.888448+0000","last_fullsized":"2024-09-13T08:44:21.888448+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:29.445866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903039+0000","last_change":"2024-09-13T08:44:20.901320+0000","last_active":"2024-09-13T08:44:20.903039+0000","last_peered":"2024-09-13T08:44:20.903039+0000","last_clean":"2024-09-13T08:44:20.903039+0000","last_became_active":"2024-09-13T08:44:20.901007+0000","last_became_peered":"2024-09-13T08:44:20.901007+0000","last_unstale":"2024-09-13T08:44:20.903039+0000","last_undegraded":"2024-09-13T08:44:20.903039+0000","last_fullsized":"2024-09-13T08:44:20.903039+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:03.428994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904136+0000","last_change":"2024-09-13T08:44:20.897209+0000","last_active":"2024-09-13T08:44:20.904136+0000","last_peered":"2024-09-13T08:44:20.904136+0000","last_clean":"2024-09-13T08:44:20.904136+0000","last_became_active":"2024-09-13T08:44:20.896473+0000","last_became_peered":"2024-09-13T08:44:20.896473+0000","last_unstale":"2024-09-13T08:44:20.904136+0000","last_undegraded":"2024-09-13T08:44:20.904136+0000","last_fullsized":"2024-09-13T08:44:20.904136+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:27.945015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888239+0000","last_change":"2024-09-13T08:44:20.911434+0000","last_active":"2024-09-13T08:44:21.888239+0000","last_peered":"2024-09-13T08:44:21.888239+0000","last_clean":"2024-09-13T08:44:21.888239+0000","last_became_active":"2024-09-13T08:44:20.911149+0000","last_became_peered":"2024-09-13T08:44:20.911149+0000","last_unstale":"2024-09-13T08:44:21.888239+0000","last_undegraded":"2024-09-13T08:44:21.888239+0000","last_fullsized":"2024-09-13T08:44:21.888239+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:44.154166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889097+0000","last_change":"2024-09-13T08:44:20.888958+0000","last_active":"2024-09-13T08:44:21.889097+0000","last_peered":"2024-09-13T08:44:21.889097+0000","last_clean":"2024-09-13T08:44:21.889097+0000","last_became_active":"2024-09-13T08:44:20.888575+0000","last_became_peered":"2024-09-13T08:44:20.888575+0000","last_unstale":"2024-09-13T08:44:21.889097+0000","last_undegraded":"2024-09-13T08:44:21.889097+0000","last_fullsized":"2024-09-13T08:44:21.889097+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:59.324890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887973+0000","last_change":"2024-09-13T08:44:20.881894+0000","last_active":"2024-09-13T08:44:21.887973+0000","last_peered":"2024-09-13T08:44:21.887973+0000","last_clean":"2024-09-13T08:44:21.887973+0000","last_became_active":"2024-09-13T08:44:20.881644+0000","last_became_peered":"2024-09-13T08:44:20.881644+0000","last_unstale":"2024-09-13T08:44:21.887973+0000","last_undegraded":"2024-09-13T08:44:21.887973+0000","last_fullsized":"2024-09-13T08:44:21.887973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:13.214418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885968+0000","last_change":"2024-09-13T08:44:20.888360+0000","last_active":"2024-09-13T08:44:21.885968+0000","last_peered":"2024-09-13T08:44:21.885968+0000","last_clean":"2024-09-13T08:44:21.885968+0000","last_became_active":"2024-09-13T08:44:20.887708+0000","last_became_peered":"2024-09-13T08:44:20.887708+0000","last_unstale":"2024-09-13T08:44:21.885968+0000","last_undegraded":"2024-09-13T08:44:21.885968+0000","last_fullsized":"2024-09-13T08:44:21.885968+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:35.020541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889239+0000","last_change":"2024-09-13T08:44:20.894756+0000","last_active":"2024-09-13T08:44:21.889239+0000","last_peered":"2024-09-13T08:44:21.889239+0000","last_clean":"2024-09-13T08:44:21.889239+0000","last_became_active":"2024-09-13T08:44:20.893097+0000","last_became_peered":"2024-09-13T08:44:20.893097+0000","last_unstale":"2024-09-13T08:44:21.889239+0000","last_undegraded":"2024-09-13T08:44:21.889239+0000","last_fullsized":"2024-09-13T08:44:21.889239+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:30:17.355606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900325+0000","last_change":"2024-09-13T08:44:20.896015+0000","last_active":"2024-09-13T08:44:20.900325+0000","last_peered":"2024-09-13T08:44:20.900325+0000","last_clean":"2024-09-13T08:44:20.900325+0000","last_became_active":"2024-09-13T08:44:20.895283+0000","last_became_peered":"2024-09-13T08:44:20.895283+0000","last_unstale":"2024-09-13T08:44:20.900325+0000","last_undegraded":"2024-09-13T08:44:20.900325+0000","last_fullsized":"2024-09-13T08:44:20.900325+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:22:11.724821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887020+0000","last_change":"2024-09-13T08:44:20.878267+0000","last_active":"2024-09-13T08:44:21.887020+0000","last_peered":"2024-09-13T08:44:21.887020+0000","last_clean":"2024-09-13T08:44:21.887020+0000","last_became_active":"2024-09-13T08:44:20.877981+0000","last_became_peered":"2024-09-13T08:44:20.877981+0000","last_unstale":"2024-09-13T08:44:21.887020+0000","last_undegraded":"2024-09-13T08:44:21.887020+0000","last_fullsized":"2024-09-13T08:44:21.887020+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:04.578179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.901272+0000","last_change":"2024-09-13T08:44:20.899645+0000","last_active":"2024-09-13T08:44:20.901272+0000","last_peered":"2024-09-13T08:44:20.901272+0000","last_clean":"2024-09-13T08:44:20.901272+0000","last_became_active":"2024-09-13T08:44:20.899314+0000","last_became_peered":"2024-09-13T08:44:20.899314+0000","last_unstale":"2024-09-13T08:44:20.901272+0000","last_undegraded":"2024-09-13T08:44:20.901272+0000","last_fullsized":"2024-09-13T08:44:20.901272+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:41:46.604033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888774+0000","last_change":"2024-09-13T08:44:20.892572+0000","last_active":"2024-09-13T08:44:21.888774+0000","last_peered":"2024-09-13T08:44:21.888774+0000","last_clean":"2024-09-13T08:44:21.888774+0000","last_became_active":"2024-09-13T08:44:20.892205+0000","last_became_peered":"2024-09-13T08:44:20.892205+0000","last_unstale":"2024-09-13T08:44:21.888774+0000","last_undegraded":"2024-09-13T08:44:21.888774+0000","last_fullsized":"2024-09-13T08:44:21.888774+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:40:36.918718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.885561+0000","last_change":"2024-09-13T08:44:20.879961+0000","last_active":"2024-09-13T08:44:20.885561+0000","last_peered":"2024-09-13T08:44:20.885561+0000","last_clean":"2024-09-13T08:44:20.885561+0000","last_became_active":"2024-09-13T08:44:20.879014+0000","last_became_peered":"2024-09-13T08:44:20.879014+0000","last_unstale":"2024-09-13T08:44:20.885561+0000","last_undegraded":"2024-09-13T08:44:20.885561+0000","last_fullsized":"2024-09-13T08:44:20.885561+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:34.230969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909309+0000","last_change":"2024-09-13T08:44:20.899245+0000","last_active":"2024-09-13T08:44:20.909309+0000","last_peered":"2024-09-13T08:44:20.909309+0000","last_clean":"2024-09-13T08:44:20.909309+0000","last_became_active":"2024-09-13T08:44:20.898987+0000","last_became_peered":"2024-09-13T08:44:20.898987+0000","last_unstale":"2024-09-13T08:44:20.909309+0000","last_undegraded":"2024-09-13T08:44:20.909309+0000","last_fullsized":"2024-09-13T08:44:20.909309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:21.285653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887541+0000","last_change":"2024-09-13T08:44:20.894293+0000","last_active":"2024-09-13T08:44:21.887541+0000","last_peered":"2024-09-13T08:44:21.887541+0000","last_clean":"2024-09-13T08:44:21.887541+0000","last_became_active":"2024-09-13T08:44:20.894007+0000","last_became_peered":"2024-09-13T08:44:20.894007+0000","last_unstale":"2024-09-13T08:44:21.887541+0000","last_undegraded":"2024-09-13T08:44:21.887541+0000","last_fullsized":"2024-09-13T08:44:21.887541+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:50.547234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889075+0000","last_change":"2024-09-13T08:44:20.897325+0000","last_active":"2024-09-13T08:44:21.889075+0000","last_peered":"2024-09-13T08:44:21.889075+0000","last_clean":"2024-09-13T08:44:21.889075+0000","last_became_active":"2024-09-13T08:44:20.897017+0000","last_became_peered":"2024-09-13T08:44:20.897017+0000","last_unstale":"2024-09-13T08:44:21.889075+0000","last_undegraded":"2024-09-13T08:44:21.889075+0000","last_fullsized":"2024-09-13T08:44:21.889075+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:40.785046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900297+0000","last_change":"2024-09-13T08:44:20.895543+0000","last_active":"2024-09-13T08:44:20.900297+0000","last_peered":"2024-09-13T08:44:20.900297+0000","last_clean":"2024-09-13T08:44:20.900297+0000","last_became_active":"2024-09-13T08:44:20.894846+0000","last_became_peered":"2024-09-13T08:44:20.894846+0000","last_unstale":"2024-09-13T08:44:20.900297+0000","last_undegraded":"2024-09-13T08:44:20.900297+0000","last_fullsized":"2024-09-13T08:44:20.900297+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:24.570405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.900425+0000","last_change":"2024-09-13T08:44:20.891157+0000","last_active":"2024-09-13T08:44:20.900425+0000","last_peered":"2024-09-13T08:44:20.900425+0000","last_clean":"2024-09-13T08:44:20.900425+0000","last_became_active":"2024-09-13T08:44:20.890886+0000","last_became_peered":"2024-09-13T08:44:20.890886+0000","last_unstale":"2024-09-13T08:44:20.900425+0000","last_undegraded":"2024-09-13T08:44:20.900425+0000","last_fullsized":"2024-09-13T08:44:20.900425+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:43:50.804092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903177+0000","last_change":"2024-09-13T08:44:20.901491+0000","last_active":"2024-09-13T08:44:20.903177+0000","last_peered":"2024-09-13T08:44:20.903177+0000","last_clean":"2024-09-13T08:44:20.903177+0000","last_became_active":"2024-09-13T08:44:20.901136+0000","last_became_peered":"2024-09-13T08:44:20.901136+0000","last_unstale":"2024-09-13T08:44:20.903177+0000","last_undegraded":"2024-09-13T08:44:20.903177+0000","last_fullsized":"2024-09-13T08:44:20.903177+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:18.157926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"123'4","reported_seq":17,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909670+0000","last_change":"2024-09-13T08:44:19.885646+0000","last_active":"2024-09-13T08:44:20.909670+0000","last_peered":"2024-09-13T08:44:20.909670+0000","last_clean":"2024-09-13T08:44:20.909670+0000","last_became_active":"2024-09-13T08:44:19.885357+0000","last_became_peered":"2024-09-13T08:44:19.885357+0000","last_unstale":"2024-09-13T08:44:20.909670+0000","last_undegraded":"2024-09-13T08:44:20.909670+0000","last_fullsized":"2024-09-13T08:44:20.909670+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:28.140984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904090+0000","last_change":"2024-09-13T08:44:20.897017+0000","last_active":"2024-09-13T08:44:20.904090+0000","last_peered":"2024-09-13T08:44:20.904090+0000","last_clean":"2024-09-13T08:44:20.904090+0000","last_became_active":"2024-09-13T08:44:20.896739+0000","last_became_peered":"2024-09-13T08:44:20.896739+0000","last_unstale":"2024-09-13T08:44:20.904090+0000","last_undegraded":"2024-09-13T08:44:20.904090+0000","last_fullsized":"2024-09-13T08:44:20.904090+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:07.781047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"123'1","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909165+0000","last_change":"2024-09-13T08:44:19.891217+0000","last_active":"2024-09-13T08:44:20.909165+0000","last_peered":"2024-09-13T08:44:20.909165+0000","last_clean":"2024-09-13T08:44:20.909165+0000","last_became_active":"2024-09-13T08:44:19.890974+0000","last_became_peered":"2024-09-13T08:44:19.890974+0000","last_unstale":"2024-09-13T08:44:20.909165+0000","last_undegraded":"2024-09-13T08:44:20.909165+0000","last_fullsized":"2024-09-13T08:44:20.909165+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:24.155271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910017+0000","last_change":"2024-09-13T08:44:20.903157+0000","last_active":"2024-09-13T08:44:20.910017+0000","last_peered":"2024-09-13T08:44:20.910017+0000","last_clean":"2024-09-13T08:44:20.910017+0000","last_became_active":"2024-09-13T08:44:20.902892+0000","last_became_peered":"2024-09-13T08:44:20.902892+0000","last_unstale":"2024-09-13T08:44:20.910017+0000","last_undegraded":"2024-09-13T08:44:20.910017+0000","last_fullsized":"2024-09-13T08:44:20.910017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:47.460535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.899809+0000","last_change":"2024-09-13T08:44:19.883574+0000","last_active":"2024-09-13T08:44:20.899809+0000","last_peered":"2024-09-13T08:44:20.899809+0000","last_clean":"2024-09-13T08:44:20.899809+0000","last_became_active":"2024-09-13T08:44:19.883258+0000","last_became_peered":"2024-09-13T08:44:19.883258+0000","last_unstale":"2024-09-13T08:44:20.899809+0000","last_undegraded":"2024-09-13T08:44:20.899809+0000","last_fullsized":"2024-09-13T08:44:20.899809+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:30:48.474033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886338+0000","last_change":"2024-09-13T08:44:20.889023+0000","last_active":"2024-09-13T08:44:21.886338+0000","last_peered":"2024-09-13T08:44:21.886338+0000","last_clean":"2024-09-13T08:44:21.886338+0000","last_became_active":"2024-09-13T08:44:20.886126+0000","last_became_peered":"2024-09-13T08:44:20.886126+0000","last_unstale":"2024-09-13T08:44:21.886338+0000","last_undegraded":"2024-09-13T08:44:21.886338+0000","last_fullsized":"2024-09-13T08:44:21.886338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:41:46.604033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888211+0000","last_change":"2024-09-13T08:44:19.884249+0000","last_active":"2024-09-13T08:44:21.888211+0000","last_peered":"2024-09-13T08:44:21.888211+0000","last_clean":"2024-09-13T08:44:21.888211+0000","last_became_active":"2024-09-13T08:44:19.883916+0000","last_became_peered":"2024-09-13T08:44:19.883916+0000","last_unstale":"2024-09-13T08:44:21.888211+0000","last_undegraded":"2024-09-13T08:44:21.888211+0000","last_fullsized":"2024-09-13T08:44:21.888211+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:01.249969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888145+0000","last_change":"2024-09-13T08:44:20.888079+0000","last_active":"2024-09-13T08:44:21.888145+0000","last_peered":"2024-09-13T08:44:21.888145+0000","last_clean":"2024-09-13T08:44:21.888145+0000","last_became_active":"2024-09-13T08:44:20.886327+0000","last_became_peered":"2024-09-13T08:44:20.886327+0000","last_unstale":"2024-09-13T08:44:21.888145+0000","last_undegraded":"2024-09-13T08:44:21.888145+0000","last_fullsized":"2024-09-13T08:44:21.888145+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:35.470332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887693+0000","last_change":"2024-09-13T08:44:19.878895+0000","last_active":"2024-09-13T08:44:21.887693+0000","last_peered":"2024-09-13T08:44:21.887693+0000","last_clean":"2024-09-13T08:44:21.887693+0000","last_became_active":"2024-09-13T08:44:19.876269+0000","last_became_peered":"2024-09-13T08:44:19.876269+0000","last_unstale":"2024-09-13T08:44:21.887693+0000","last_undegraded":"2024-09-13T08:44:21.887693+0000","last_fullsized":"2024-09-13T08:44:21.887693+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:30.961922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886172+0000","last_change":"2024-09-13T08:44:20.880296+0000","last_active":"2024-09-13T08:44:21.886172+0000","last_peered":"2024-09-13T08:44:21.886172+0000","last_clean":"2024-09-13T08:44:21.886172+0000","last_became_active":"2024-09-13T08:44:20.880037+0000","last_became_peered":"2024-09-13T08:44:20.880037+0000","last_unstale":"2024-09-13T08:44:21.886172+0000","last_undegraded":"2024-09-13T08:44:21.886172+0000","last_fullsized":"2024-09-13T08:44:21.886172+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:47.908231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888221+0000","last_change":"2024-09-13T08:44:19.881761+0000","last_active":"2024-09-13T08:44:21.888221+0000","last_peered":"2024-09-13T08:44:21.888221+0000","last_clean":"2024-09-13T08:44:21.888221+0000","last_became_active":"2024-09-13T08:44:19.881425+0000","last_became_peered":"2024-09-13T08:44:19.881425+0000","last_unstale":"2024-09-13T08:44:21.888221+0000","last_undegraded":"2024-09-13T08:44:21.888221+0000","last_fullsized":"2024-09-13T08:44:21.888221+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:40:50.583674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885677+0000","last_change":"2024-09-13T08:44:20.889069+0000","last_active":"2024-09-13T08:44:21.885677+0000","last_peered":"2024-09-13T08:44:21.885677+0000","last_clean":"2024-09-13T08:44:21.885677+0000","last_became_active":"2024-09-13T08:44:20.888806+0000","last_became_peered":"2024-09-13T08:44:20.888806+0000","last_unstale":"2024-09-13T08:44:21.885677+0000","last_undegraded":"2024-09-13T08:44:21.885677+0000","last_fullsized":"2024-09-13T08:44:21.885677+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:01:47.892851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888408+0000","last_change":"2024-09-13T08:44:19.893478+0000","last_active":"2024-09-13T08:44:20.888408+0000","last_peered":"2024-09-13T08:44:20.888408+0000","last_clean":"2024-09-13T08:44:20.888408+0000","last_became_active":"2024-09-13T08:44:19.893152+0000","last_became_peered":"2024-09-13T08:44:19.893152+0000","last_unstale":"2024-09-13T08:44:20.888408+0000","last_undegraded":"2024-09-13T08:44:20.888408+0000","last_fullsized":"2024-09-13T08:44:20.888408+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:34.210418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887137+0000","last_change":"2024-09-13T08:44:20.886922+0000","last_active":"2024-09-13T08:44:21.887137+0000","last_peered":"2024-09-13T08:44:21.887137+0000","last_clean":"2024-09-13T08:44:21.887137+0000","last_became_active":"2024-09-13T08:44:20.885056+0000","last_became_peered":"2024-09-13T08:44:20.885056+0000","last_unstale":"2024-09-13T08:44:21.887137+0000","last_undegraded":"2024-09-13T08:44:21.887137+0000","last_fullsized":"2024-09-13T08:44:21.887137+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:56.485333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887002+0000","last_change":"2024-09-13T08:44:19.886829+0000","last_active":"2024-09-13T08:44:21.887002+0000","last_peered":"2024-09-13T08:44:21.887002+0000","last_clean":"2024-09-13T08:44:21.887002+0000","last_became_active":"2024-09-13T08:44:19.886539+0000","last_became_peered":"2024-09-13T08:44:19.886539+0000","last_unstale":"2024-09-13T08:44:21.887002+0000","last_undegraded":"2024-09-13T08:44:21.887002+0000","last_fullsized":"2024-09-13T08:44:21.887002+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:07.028864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889088+0000","last_change":"2024-09-13T08:44:20.885538+0000","last_active":"2024-09-13T08:44:21.889088+0000","last_peered":"2024-09-13T08:44:21.889088+0000","last_clean":"2024-09-13T08:44:21.889088+0000","last_became_active":"2024-09-13T08:44:20.883566+0000","last_became_peered":"2024-09-13T08:44:20.883566+0000","last_unstale":"2024-09-13T08:44:21.889088+0000","last_undegraded":"2024-09-13T08:44:21.889088+0000","last_fullsized":"2024-09-13T08:44:21.889088+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:55.531704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886093+0000","last_change":"2024-09-13T08:44:19.884118+0000","last_active":"2024-09-13T08:44:21.886093+0000","last_peered":"2024-09-13T08:44:21.886093+0000","last_clean":"2024-09-13T08:44:21.886093+0000","last_became_active":"2024-09-13T08:44:19.883829+0000","last_became_peered":"2024-09-13T08:44:19.883829+0000","last_unstale":"2024-09-13T08:44:21.886093+0000","last_undegraded":"2024-09-13T08:44:21.886093+0000","last_fullsized":"2024-09-13T08:44:21.886093+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:38.305159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.895032+0000","last_change":"2024-09-13T08:44:20.893149+0000","last_active":"2024-09-13T08:44:20.895032+0000","last_peered":"2024-09-13T08:44:20.895032+0000","last_clean":"2024-09-13T08:44:20.895032+0000","last_became_active":"2024-09-13T08:44:20.892542+0000","last_became_peered":"2024-09-13T08:44:20.892542+0000","last_unstale":"2024-09-13T08:44:20.895032+0000","last_undegraded":"2024-09-13T08:44:20.895032+0000","last_fullsized":"2024-09-13T08:44:20.895032+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:59.576955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888366+0000","last_change":"2024-09-13T08:44:19.883397+0000","last_active":"2024-09-13T08:44:21.888366+0000","last_peered":"2024-09-13T08:44:21.888366+0000","last_clean":"2024-09-13T08:44:21.888366+0000","last_became_active":"2024-09-13T08:44:19.883072+0000","last_became_peered":"2024-09-13T08:44:19.883072+0000","last_unstale":"2024-09-13T08:44:21.888366+0000","last_undegraded":"2024-09-13T08:44:21.888366+0000","last_fullsized":"2024-09-13T08:44:21.888366+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:03:42.127573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886313+0000","last_change":"2024-09-13T08:44:20.906998+0000","last_active":"2024-09-13T08:44:21.886313+0000","last_peered":"2024-09-13T08:44:21.886313+0000","last_clean":"2024-09-13T08:44:21.886313+0000","last_became_active":"2024-09-13T08:44:20.906684+0000","last_became_peered":"2024-09-13T08:44:20.906684+0000","last_unstale":"2024-09-13T08:44:21.886313+0000","last_undegraded":"2024-09-13T08:44:21.886313+0000","last_fullsized":"2024-09-13T08:44:21.886313+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:51.903261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"123'3","reported_seq":16,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.905208+0000","last_change":"2024-09-13T08:44:19.885085+0000","last_active":"2024-09-13T08:44:20.905208+0000","last_peered":"2024-09-13T08:44:20.905208+0000","last_clean":"2024-09-13T08:44:20.905208+0000","last_became_active":"2024-09-13T08:44:19.884807+0000","last_became_peered":"2024-09-13T08:44:19.884807+0000","last_unstale":"2024-09-13T08:44:20.905208+0000","last_undegraded":"2024-09-13T08:44:20.905208+0000","last_fullsized":"2024-09-13T08:44:20.905208+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:31.266566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.909358+0000","last_change":"2024-09-13T08:44:20.907482+0000","last_active":"2024-09-13T08:44:20.909358+0000","last_peered":"2024-09-13T08:44:20.909358+0000","last_clean":"2024-09-13T08:44:20.909358+0000","last_became_active":"2024-09-13T08:44:20.907191+0000","last_became_peered":"2024-09-13T08:44:20.907191+0000","last_unstale":"2024-09-13T08:44:20.909358+0000","last_undegraded":"2024-09-13T08:44:20.909358+0000","last_fullsized":"2024-09-13T08:44:20.909358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:50:18.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"123'6","reported_seq":19,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.916378+0000","last_change":"2024-09-13T08:44:19.884211+0000","last_active":"2024-09-13T08:44:20.916378+0000","last_peered":"2024-09-13T08:44:20.916378+0000","last_clean":"2024-09-13T08:44:20.916378+0000","last_became_active":"2024-09-13T08:44:19.883901+0000","last_became_peered":"2024-09-13T08:44:19.883901+0000","last_unstale":"2024-09-13T08:44:20.916378+0000","last_undegraded":"2024-09-13T08:44:20.916378+0000","last_fullsized":"2024-09-13T08:44:20.916378+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:06.442119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.888350+0000","last_change":"2024-09-13T08:44:20.887134+0000","last_active":"2024-09-13T08:44:20.888350+0000","last_peered":"2024-09-13T08:44:20.888350+0000","last_clean":"2024-09-13T08:44:20.888350+0000","last_became_active":"2024-09-13T08:44:20.886817+0000","last_became_peered":"2024-09-13T08:44:20.886817+0000","last_unstale":"2024-09-13T08:44:20.888350+0000","last_undegraded":"2024-09-13T08:44:20.888350+0000","last_fullsized":"2024-09-13T08:44:20.888350+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:17.100946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.871656+0000","last_change":"2024-09-13T08:44:19.890493+0000","last_active":"2024-09-13T08:44:20.871656+0000","last_peered":"2024-09-13T08:44:20.871656+0000","last_clean":"2024-09-13T08:44:20.871656+0000","last_became_active":"2024-09-13T08:44:19.890164+0000","last_became_peered":"2024-09-13T08:44:19.890164+0000","last_unstale":"2024-09-13T08:44:20.871656+0000","last_undegraded":"2024-09-13T08:44:20.871656+0000","last_fullsized":"2024-09-13T08:44:20.871656+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:29.960427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.902385+0000","last_change":"2024-09-13T08:44:20.899989+0000","last_active":"2024-09-13T08:44:20.902385+0000","last_peered":"2024-09-13T08:44:20.902385+0000","last_clean":"2024-09-13T08:44:20.902385+0000","last_became_active":"2024-09-13T08:44:20.899656+0000","last_became_peered":"2024-09-13T08:44:20.899656+0000","last_unstale":"2024-09-13T08:44:20.902385+0000","last_undegraded":"2024-09-13T08:44:20.902385+0000","last_fullsized":"2024-09-13T08:44:20.902385+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:26.915057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887051+0000","last_change":"2024-09-13T08:44:19.891406+0000","last_active":"2024-09-13T08:44:20.887051+0000","last_peered":"2024-09-13T08:44:20.887051+0000","last_clean":"2024-09-13T08:44:20.887051+0000","last_became_active":"2024-09-13T08:44:19.891080+0000","last_became_peered":"2024-09-13T08:44:19.891080+0000","last_unstale":"2024-09-13T08:44:20.887051+0000","last_undegraded":"2024-09-13T08:44:20.887051+0000","last_fullsized":"2024-09-13T08:44:20.887051+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:56.783762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885672+0000","last_change":"2024-09-13T08:44:20.899331+0000","last_active":"2024-09-13T08:44:21.885672+0000","last_peered":"2024-09-13T08:44:21.885672+0000","last_clean":"2024-09-13T08:44:21.885672+0000","last_became_active":"2024-09-13T08:44:20.898575+0000","last_became_peered":"2024-09-13T08:44:20.898575+0000","last_unstale":"2024-09-13T08:44:21.885672+0000","last_undegraded":"2024-09-13T08:44:21.885672+0000","last_fullsized":"2024-09-13T08:44:21.885672+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:33.518058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"123'8","reported_seq":23,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885528+0000","last_change":"2024-09-13T08:44:19.874633+0000","last_active":"2024-09-13T08:44:21.885528+0000","last_peered":"2024-09-13T08:44:21.885528+0000","last_clean":"2024-09-13T08:44:21.885528+0000","last_became_active":"2024-09-13T08:44:19.874280+0000","last_became_peered":"2024-09-13T08:44:19.874280+0000","last_unstale":"2024-09-13T08:44:21.885528+0000","last_undegraded":"2024-09-13T08:44:21.885528+0000","last_fullsized":"2024-09-13T08:44:21.885528+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:52:54.258397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896363+0000","last_change":"2024-09-13T08:44:20.894103+0000","last_active":"2024-09-13T08:44:20.896363+0000","last_peered":"2024-09-13T08:44:20.896363+0000","last_clean":"2024-09-13T08:44:20.896363+0000","last_became_active":"2024-09-13T08:44:20.893474+0000","last_became_peered":"2024-09-13T08:44:20.893474+0000","last_unstale":"2024-09-13T08:44:20.896363+0000","last_undegraded":"2024-09-13T08:44:20.896363+0000","last_fullsized":"2024-09-13T08:44:20.896363+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:47:49.037689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"123'1","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886178+0000","last_change":"2024-09-13T08:44:19.883066+0000","last_active":"2024-09-13T08:44:21.886178+0000","last_peered":"2024-09-13T08:44:21.886178+0000","last_clean":"2024-09-13T08:44:21.886178+0000","last_became_active":"2024-09-13T08:44:19.882775+0000","last_became_peered":"2024-09-13T08:44:19.882775+0000","last_unstale":"2024-09-13T08:44:21.886178+0000","last_undegraded":"2024-09-13T08:44:21.886178+0000","last_fullsized":"2024-09-13T08:44:21.886178+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:45.641338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.896319+0000","last_change":"2024-09-13T08:44:20.893529+0000","last_active":"2024-09-13T08:44:20.896319+0000","last_peered":"2024-09-13T08:44:20.896319+0000","last_clean":"2024-09-13T08:44:20.896319+0000","last_became_active":"2024-09-13T08:44:20.892947+0000","last_became_peered":"2024-09-13T08:44:20.892947+0000","last_unstale":"2024-09-13T08:44:20.896319+0000","last_undegraded":"2024-09-13T08:44:20.896319+0000","last_fullsized":"2024-09-13T08:44:20.896319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:04.307017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.906158+0000","last_change":"2024-09-13T08:44:19.887455+0000","last_active":"2024-09-13T08:44:20.906158+0000","last_peered":"2024-09-13T08:44:20.906158+0000","last_clean":"2024-09-13T08:44:20.906158+0000","last_became_active":"2024-09-13T08:44:19.887033+0000","last_became_peered":"2024-09-13T08:44:19.887033+0000","last_unstale":"2024-09-13T08:44:20.906158+0000","last_undegraded":"2024-09-13T08:44:20.906158+0000","last_fullsized":"2024-09-13T08:44:20.906158+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:46.897173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888211+0000","last_change":"2024-09-13T08:44:20.890908+0000","last_active":"2024-09-13T08:44:21.888211+0000","last_peered":"2024-09-13T08:44:21.888211+0000","last_clean":"2024-09-13T08:44:21.888211+0000","last_became_active":"2024-09-13T08:44:20.890623+0000","last_became_peered":"2024-09-13T08:44:20.890623+0000","last_unstale":"2024-09-13T08:44:21.888211+0000","last_undegraded":"2024-09-13T08:44:21.888211+0000","last_fullsized":"2024-09-13T08:44:21.888211+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:48.645162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"123'1","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910207+0000","last_change":"2024-09-13T08:44:19.887311+0000","last_active":"2024-09-13T08:44:20.910207+0000","last_peered":"2024-09-13T08:44:20.910207+0000","last_clean":"2024-09-13T08:44:20.910207+0000","last_became_active":"2024-09-13T08:44:19.886919+0000","last_became_peered":"2024-09-13T08:44:19.886919+0000","last_unstale":"2024-09-13T08:44:20.910207+0000","last_undegraded":"2024-09-13T08:44:20.910207+0000","last_fullsized":"2024-09-13T08:44:20.910207+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:40.118017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888609+0000","last_change":"2024-09-13T08:44:20.896009+0000","last_active":"2024-09-13T08:44:21.888609+0000","last_peered":"2024-09-13T08:44:21.888609+0000","last_clean":"2024-09-13T08:44:21.888609+0000","last_became_active":"2024-09-13T08:44:20.895657+0000","last_became_peered":"2024-09-13T08:44:20.895657+0000","last_unstale":"2024-09-13T08:44:21.888609+0000","last_undegraded":"2024-09-13T08:44:21.888609+0000","last_fullsized":"2024-09-13T08:44:21.888609+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:34:38.269551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.882094+0000","last_change":"2024-09-13T08:44:19.879449+0000","last_active":"2024-09-13T08:44:20.882094+0000","last_peered":"2024-09-13T08:44:20.882094+0000","last_clean":"2024-09-13T08:44:20.882094+0000","last_became_active":"2024-09-13T08:44:19.879141+0000","last_became_peered":"2024-09-13T08:44:19.879141+0000","last_unstale":"2024-09-13T08:44:20.882094+0000","last_undegraded":"2024-09-13T08:44:20.882094+0000","last_fullsized":"2024-09-13T08:44:20.882094+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:33.347395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.889310+0000","last_change":"2024-09-13T08:44:20.889962+0000","last_active":"2024-09-13T08:44:21.889310+0000","last_peered":"2024-09-13T08:44:21.889310+0000","last_clean":"2024-09-13T08:44:21.889310+0000","last_became_active":"2024-09-13T08:44:20.889062+0000","last_became_peered":"2024-09-13T08:44:20.889062+0000","last_unstale":"2024-09-13T08:44:21.889310+0000","last_undegraded":"2024-09-13T08:44:21.889310+0000","last_fullsized":"2024-09-13T08:44:21.889310+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:15.964693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"123'2","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.910776+0000","last_change":"2024-09-13T08:44:19.886650+0000","last_active":"2024-09-13T08:44:20.910776+0000","last_peered":"2024-09-13T08:44:20.910776+0000","last_clean":"2024-09-13T08:44:20.910776+0000","last_became_active":"2024-09-13T08:44:19.886380+0000","last_became_peered":"2024-09-13T08:44:19.886380+0000","last_unstale":"2024-09-13T08:44:20.910776+0000","last_undegraded":"2024-09-13T08:44:20.910776+0000","last_fullsized":"2024-09-13T08:44:20.910776+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:20.938256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.907218+0000","last_change":"2024-09-13T08:44:20.905415+0000","last_active":"2024-09-13T08:44:20.907218+0000","last_peered":"2024-09-13T08:44:20.907218+0000","last_clean":"2024-09-13T08:44:20.907218+0000","last_became_active":"2024-09-13T08:44:20.903103+0000","last_became_peered":"2024-09-13T08:44:20.903103+0000","last_unstale":"2024-09-13T08:44:20.907218+0000","last_undegraded":"2024-09-13T08:44:20.907218+0000","last_fullsized":"2024-09-13T08:44:20.907218+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:15.991963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888272+0000","last_change":"2024-09-13T08:44:19.888753+0000","last_active":"2024-09-13T08:44:21.888272+0000","last_peered":"2024-09-13T08:44:21.888272+0000","last_clean":"2024-09-13T08:44:21.888272+0000","last_became_active":"2024-09-13T08:44:19.888377+0000","last_became_peered":"2024-09-13T08:44:19.888377+0000","last_unstale":"2024-09-13T08:44:21.888272+0000","last_undegraded":"2024-09-13T08:44:21.888272+0000","last_fullsized":"2024-09-13T08:44:21.888272+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:33.508644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.880848+0000","last_change":"2024-09-13T08:44:20.874498+0000","last_active":"2024-09-13T08:44:20.880848+0000","last_peered":"2024-09-13T08:44:20.880848+0000","last_clean":"2024-09-13T08:44:20.880848+0000","last_became_active":"2024-09-13T08:44:20.874200+0000","last_became_peered":"2024-09-13T08:44:20.874200+0000","last_unstale":"2024-09-13T08:44:20.880848+0000","last_undegraded":"2024-09-13T08:44:20.880848+0000","last_fullsized":"2024-09-13T08:44:20.880848+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:17.188650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.869586+0000","last_change":"2024-09-13T08:44:19.889479+0000","last_active":"2024-09-13T08:44:20.869586+0000","last_peered":"2024-09-13T08:44:20.869586+0000","last_clean":"2024-09-13T08:44:20.869586+0000","last_became_active":"2024-09-13T08:44:19.889113+0000","last_became_peered":"2024-09-13T08:44:19.889113+0000","last_unstale":"2024-09-13T08:44:20.869586+0000","last_undegraded":"2024-09-13T08:44:20.869586+0000","last_fullsized":"2024-09-13T08:44:20.869586+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:21.723397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885675+0000","last_change":"2024-09-13T08:44:20.875542+0000","last_active":"2024-09-13T08:44:21.885675+0000","last_peered":"2024-09-13T08:44:21.885675+0000","last_clean":"2024-09-13T08:44:21.885675+0000","last_became_active":"2024-09-13T08:44:20.875284+0000","last_became_peered":"2024-09-13T08:44:20.875284+0000","last_unstale":"2024-09-13T08:44:21.885675+0000","last_undegraded":"2024-09-13T08:44:21.885675+0000","last_fullsized":"2024-09-13T08:44:21.885675+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:06.833193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.886621+0000","last_change":"2024-09-13T08:44:19.888095+0000","last_active":"2024-09-13T08:44:20.886621+0000","last_peered":"2024-09-13T08:44:20.886621+0000","last_clean":"2024-09-13T08:44:20.886621+0000","last_became_active":"2024-09-13T08:44:19.887798+0000","last_became_peered":"2024-09-13T08:44:19.887798+0000","last_unstale":"2024-09-13T08:44:20.886621+0000","last_undegraded":"2024-09-13T08:44:20.886621+0000","last_fullsized":"2024-09-13T08:44:20.886621+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:41.762806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.908065+0000","last_change":"2024-09-13T08:44:20.906487+0000","last_active":"2024-09-13T08:44:20.908065+0000","last_peered":"2024-09-13T08:44:20.908065+0000","last_clean":"2024-09-13T08:44:20.908065+0000","last_became_active":"2024-09-13T08:44:20.905996+0000","last_became_peered":"2024-09-13T08:44:20.905996+0000","last_unstale":"2024-09-13T08:44:20.908065+0000","last_undegraded":"2024-09-13T08:44:20.908065+0000","last_fullsized":"2024-09-13T08:44:20.908065+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:44:05.542999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888057+0000","last_change":"2024-09-13T08:44:19.889047+0000","last_active":"2024-09-13T08:44:21.888057+0000","last_peered":"2024-09-13T08:44:21.888057+0000","last_clean":"2024-09-13T08:44:21.888057+0000","last_became_active":"2024-09-13T08:44:19.888682+0000","last_became_peered":"2024-09-13T08:44:19.888682+0000","last_unstale":"2024-09-13T08:44:21.888057+0000","last_undegraded":"2024-09-13T08:44:21.888057+0000","last_fullsized":"2024-09-13T08:44:21.888057+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:59.958964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.888848+0000","last_change":"2024-09-13T08:44:20.890851+0000","last_active":"2024-09-13T08:44:21.888848+0000","last_peered":"2024-09-13T08:44:21.888848+0000","last_clean":"2024-09-13T08:44:21.888848+0000","last_became_active":"2024-09-13T08:44:20.890468+0000","last_became_peered":"2024-09-13T08:44:20.890468+0000","last_unstale":"2024-09-13T08:44:21.888848+0000","last_undegraded":"2024-09-13T08:44:21.888848+0000","last_fullsized":"2024-09-13T08:44:21.888848+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:33:35.035688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.882957+0000","last_change":"2024-09-13T08:44:19.891057+0000","last_active":"2024-09-13T08:44:20.882957+0000","last_peered":"2024-09-13T08:44:20.882957+0000","last_clean":"2024-09-13T08:44:20.882957+0000","last_became_active":"2024-09-13T08:44:19.890715+0000","last_became_peered":"2024-09-13T08:44:19.890715+0000","last_unstale":"2024-09-13T08:44:20.882957+0000","last_undegraded":"2024-09-13T08:44:20.882957+0000","last_fullsized":"2024-09-13T08:44:20.882957+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:42.556834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887713+0000","last_change":"2024-09-13T08:44:20.885289+0000","last_active":"2024-09-13T08:44:20.887713+0000","last_peered":"2024-09-13T08:44:20.887713+0000","last_clean":"2024-09-13T08:44:20.887713+0000","last_became_active":"2024-09-13T08:44:20.884955+0000","last_became_peered":"2024-09-13T08:44:20.884955+0000","last_unstale":"2024-09-13T08:44:20.887713+0000","last_undegraded":"2024-09-13T08:44:20.887713+0000","last_fullsized":"2024-09-13T08:44:20.887713+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:34:55.796847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887748+0000","last_change":"2024-09-13T08:44:19.885326+0000","last_active":"2024-09-13T08:44:21.887748+0000","last_peered":"2024-09-13T08:44:21.887748+0000","last_clean":"2024-09-13T08:44:21.887748+0000","last_became_active":"2024-09-13T08:44:19.885013+0000","last_became_peered":"2024-09-13T08:44:19.885013+0000","last_unstale":"2024-09-13T08:44:21.887748+0000","last_undegraded":"2024-09-13T08:44:21.887748+0000","last_fullsized":"2024-09-13T08:44:21.887748+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:38.273975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.904170+0000","last_change":"2024-09-13T08:44:20.895971+0000","last_active":"2024-09-13T08:44:20.904170+0000","last_peered":"2024-09-13T08:44:20.904170+0000","last_clean":"2024-09-13T08:44:20.904170+0000","last_became_active":"2024-09-13T08:44:20.895684+0000","last_became_peered":"2024-09-13T08:44:20.895684+0000","last_unstale":"2024-09-13T08:44:20.904170+0000","last_undegraded":"2024-09-13T08:44:20.904170+0000","last_fullsized":"2024-09-13T08:44:20.904170+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:59:27.008517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.869523+0000","last_change":"2024-09-13T08:44:19.891325+0000","last_active":"2024-09-13T08:44:20.869523+0000","last_peered":"2024-09-13T08:44:20.869523+0000","last_clean":"2024-09-13T08:44:20.869523+0000","last_became_active":"2024-09-13T08:44:19.890989+0000","last_became_peered":"2024-09-13T08:44:19.890989+0000","last_unstale":"2024-09-13T08:44:20.869523+0000","last_undegraded":"2024-09-13T08:44:20.869523+0000","last_fullsized":"2024-09-13T08:44:20.869523+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:33.713455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885901+0000","last_change":"2024-09-13T08:44:20.890009+0000","last_active":"2024-09-13T08:44:21.885901+0000","last_peered":"2024-09-13T08:44:21.885901+0000","last_clean":"2024-09-13T08:44:21.885901+0000","last_became_active":"2024-09-13T08:44:20.889349+0000","last_became_peered":"2024-09-13T08:44:20.889349+0000","last_unstale":"2024-09-13T08:44:21.885901+0000","last_undegraded":"2024-09-13T08:44:21.885901+0000","last_fullsized":"2024-09-13T08:44:21.885901+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:57.794819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887804+0000","last_change":"2024-09-13T08:44:19.881019+0000","last_active":"2024-09-13T08:44:21.887804+0000","last_peered":"2024-09-13T08:44:21.887804+0000","last_clean":"2024-09-13T08:44:21.887804+0000","last_became_active":"2024-09-13T08:44:19.880556+0000","last_became_peered":"2024-09-13T08:44:19.880556+0000","last_unstale":"2024-09-13T08:44:21.887804+0000","last_undegraded":"2024-09-13T08:44:21.887804+0000","last_fullsized":"2024-09-13T08:44:21.887804+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:20.654141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886376+0000","last_change":"2024-09-13T08:44:20.902297+0000","last_active":"2024-09-13T08:44:21.886376+0000","last_peered":"2024-09-13T08:44:21.886376+0000","last_clean":"2024-09-13T08:44:21.886376+0000","last_became_active":"2024-09-13T08:44:20.902005+0000","last_became_peered":"2024-09-13T08:44:20.902005+0000","last_unstale":"2024-09-13T08:44:21.886376+0000","last_undegraded":"2024-09-13T08:44:21.886376+0000","last_fullsized":"2024-09-13T08:44:21.886376+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:13:33.493726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.885830+0000","last_change":"2024-09-13T08:44:19.878615+0000","last_active":"2024-09-13T08:44:21.885830+0000","last_peered":"2024-09-13T08:44:21.885830+0000","last_clean":"2024-09-13T08:44:21.885830+0000","last_became_active":"2024-09-13T08:44:19.878324+0000","last_became_peered":"2024-09-13T08:44:19.878324+0000","last_unstale":"2024-09-13T08:44:21.885830+0000","last_undegraded":"2024-09-13T08:44:21.885830+0000","last_fullsized":"2024-09-13T08:44:21.885830+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:36.450192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.887405+0000","last_change":"2024-09-13T08:44:20.884536+0000","last_active":"2024-09-13T08:44:21.887405+0000","last_peered":"2024-09-13T08:44:21.887405+0000","last_clean":"2024-09-13T08:44:21.887405+0000","last_became_active":"2024-09-13T08:44:20.880278+0000","last_became_peered":"2024-09-13T08:44:20.880278+0000","last_unstale":"2024-09-13T08:44:21.887405+0000","last_undegraded":"2024-09-13T08:44:21.887405+0000","last_fullsized":"2024-09-13T08:44:21.887405+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:23:39.869307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.887305+0000","last_change":"2024-09-13T08:44:19.891311+0000","last_active":"2024-09-13T08:44:20.887305+0000","last_peered":"2024-09-13T08:44:20.887305+0000","last_clean":"2024-09-13T08:44:20.887305+0000","last_became_active":"2024-09-13T08:44:19.891028+0000","last_became_peered":"2024-09-13T08:44:19.891028+0000","last_unstale":"2024-09-13T08:44:20.887305+0000","last_undegraded":"2024-09-13T08:44:20.887305+0000","last_fullsized":"2024-09-13T08:44:20.887305+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.009484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.902381+0000","last_change":"2024-09-13T08:44:20.900334+0000","last_active":"2024-09-13T08:44:20.902381+0000","last_peered":"2024-09-13T08:44:20.902381+0000","last_clean":"2024-09-13T08:44:20.902381+0000","last_became_active":"2024-09-13T08:44:20.899057+0000","last_became_peered":"2024-09-13T08:44:20.899057+0000","last_unstale":"2024-09-13T08:44:20.902381+0000","last_undegraded":"2024-09-13T08:44:20.902381+0000","last_fullsized":"2024-09-13T08:44:20.902381+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:13:40.187667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.870823+0000","last_change":"2024-09-13T08:44:19.891328+0000","last_active":"2024-09-13T08:44:20.870823+0000","last_peered":"2024-09-13T08:44:20.870823+0000","last_clean":"2024-09-13T08:44:20.870823+0000","last_became_active":"2024-09-13T08:44:19.891013+0000","last_became_peered":"2024-09-13T08:44:19.891013+0000","last_unstale":"2024-09-13T08:44:20.870823+0000","last_undegraded":"2024-09-13T08:44:20.870823+0000","last_fullsized":"2024-09-13T08:44:20.870823+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:20.836057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.903476+0000","last_change":"2024-09-13T08:44:20.896143+0000","last_active":"2024-09-13T08:44:20.903476+0000","last_peered":"2024-09-13T08:44:20.903476+0000","last_clean":"2024-09-13T08:44:20.903476+0000","last_became_active":"2024-09-13T08:44:20.895880+0000","last_became_peered":"2024-09-13T08:44:20.895880+0000","last_unstale":"2024-09-13T08:44:20.903476+0000","last_undegraded":"2024-09-13T08:44:20.903476+0000","last_fullsized":"2024-09-13T08:44:20.903476+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:19.856176+0000","last_clean_scrub_stamp":"2024-09-13T08:44:19.856176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:36.845832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"123'2","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T08:44:21.886550+0000","last_change":"2024-09-13T08:44:19.879464+0000","last_active":"2024-09-13T08:44:21.886550+0000","last_peered":"2024-09-13T08:44:21.886550+0000","last_clean":"2024-09-13T08:44:21.886550+0000","last_became_active":"2024-09-13T08:44:19.878980+0000","last_became_peered":"2024-09-13T08:44:19.878980+0000","last_unstale":"2024-09-13T08:44:21.886550+0000","last_undegraded":"2024-09-13T08:44:21.886550+0000","last_fullsized":"2024-09-13T08:44:21.886550+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:18.845119+0000","last_clean_scrub_stamp":"2024-09-13T08:44:18.845119+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:20:25.767263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":322,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-13T08:44:20.879903+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:44:20.879903+0000","last_peered":"2024-09-13T08:44:20.879903+0000","last_clean":"2024-09-13T08:44:20.879903+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:44:20.879903+0000","last_undegraded":"2024-09-13T08:44:20.879903+0000","last_fullsized":"2024-09-13T08:44:20.879903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509532,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36852,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334988,"statfs":{"total":96636764160,"available":96599027712,"internally_reserved":0,"allocated":1531904,"data_stored":673854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1527808,"data_stored":673820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37436,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334404,"statfs":{"total":96636764160,"available":96598429696,"internally_reserved":0,"allocated":2134016,"data_stored":1264930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542235,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1527808,"data_stored":673820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509532,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1531904,"data_stored":673890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509532,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1536000,"data_stored":674492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509533,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2138112,"data_stored":1265822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1548288,"data_stored":675488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:44:25.561 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:44:25.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.763+0000 7f8a92a00640 1 -- 172.21.15.31:0/2490583048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cb380 msgr2=0x7f8a8c0cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.763+0000 7f8a92a00640 1 --2- 172.21.15.31:0/2490583048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c0cdcc0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005fd0 tx=0x7f8a8402aa00 comp rx=0 tx=0).stop 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 -- 172.21.15.31:0/2490583048 shutdown_connections 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 --2- 172.21.15.31:0/2490583048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c0cdcc0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 --2- 172.21.15.31:0/2490583048 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 0x7f8a8c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 -- 172.21.15.31:0/2490583048 >> 172.21.15.31:0/2490583048 conn(0x7f8a8c0cfbb0 msgr2=0x7f8a8c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 -- 172.21.15.31:0/2490583048 shutdown_connections 2024-09-13T08:44:25.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.764+0000 7f8a92a00640 1 -- 172.21.15.31:0/2490583048 wait complete. 2024-09-13T08:44:25.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.765+0000 7f8a92a00640 1 Processor -- start 2024-09-13T08:44:25.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.765+0000 7f8a92a00640 1 -- start start 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 0x7f8a8c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 0x7f8a8c1f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c0cbe00 con 0x7f8a8c0cc190 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c0cbc80 con 0x7f8a8c0cb380 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.766+0000 7f8a92a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8a8c0cbf80 con 0x7f8a8c161480 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 0x7f8a8c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:59734/0 (socket says 172.21.15.31:59734) 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 -- 172.21.15.31:0/4040436117 learned_addr learned my addr 172.21.15.31:0/4040436117 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a90a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 0x7f8a8c1f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 -- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 msgr2=0x7f8a8c1f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 0x7f8a8c1f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 -- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 msgr2=0x7f8a8c0c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:25.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 0x7f8a8c0c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:25.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 -- 172.21.15.31:0/4040436117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8c14ced0 con 0x7f8a8c0cb380 2024-09-13T08:44:25.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8b400640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 0x7f8a8c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.767+0000 7f8a8be00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c970 tx=0x7f8a7c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.768+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c007bb0 con 0x7f8a8c0cb380 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.768+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a7c007e40 con 0x7f8a8c0cb380 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.768+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c00f690 con 0x7f8a8c0cb380 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.768+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8c14d1b0 con 0x7f8a8c0cb380 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.768+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8c14d770 con 0x7f8a8c0cb380 2024-09-13T08:44:25.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.769+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a8c14dab0 con 0x7f8a8c0cb380 2024-09-13T08:44:25.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.770+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f8a7c00f830 con 0x7f8a8c0cb380 2024-09-13T08:44:25.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.771+0000 7f8a83e00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 0x7f8a6c07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:25.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.771+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f8a7c09bd90 con 0x7f8a8c0cb380 2024-09-13T08:44:25.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.772+0000 7f8a8b400640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 0x7f8a6c07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:25.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.772+0000 7f8a8b400640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 0x7f8a6c07b190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0cd830 tx=0x7f8a78009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:25.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:25.775+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8a8c14dab0 con 0x7f8a8c0cb380 2024-09-13T08:44:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.043+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8a8c06a180 con 0x7f8a8c0cb380 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.049+0000 7f8a83e00640 1 -- 172.21.15.31:0/4040436117 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f8a8c06a180 con 0x7f8a8c0cb380 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:44:26.050 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:44:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 msgr2=0x7f8a6c07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 0x7f8a6c07b190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0cd830 tx=0x7f8a78009040 comp rx=0 tx=0).stop 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 msgr2=0x7f8a8c1f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c970 tx=0x7f8a7c00ce30 comp rx=0 tx=0).stop 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 shutdown_connections 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a6c078cd0 0x7f8a6c07b190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a8c161480 0x7f8a8c1f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a8c0cc190 0x7f8a8c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 --2- 172.21.15.31:0/4040436117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a8c0cb380 0x7f8a8c1f52d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 >> 172.21.15.31:0/4040436117 conn(0x7f8a8c0cfbb0 msgr2=0x7f8a8c15d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:26.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 shutdown_connections 2024-09-13T08:44:26.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.052+0000 7f8a92a00640 1 -- 172.21.15.31:0/4040436117 wait complete. 2024-09-13T08:44:26.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.228+0000 7f364be00640 1 -- 172.21.15.31:0/576790519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0b52d0 msgr2=0x7f364c06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.228+0000 7f364be00640 1 --2- 172.21.15.31:0/576790519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0b52d0 0x7f364c06f9b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f364c0ce2e0 tx=0x7f363402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 -- 172.21.15.31:0/576790519 shutdown_connections 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 --2- 172.21.15.31:0/576790519 >> v2:172.21.15.45:3300/0 conn(0x7f364c0ccb00 0x7f364c152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 --2- 172.21.15.31:0/576790519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0b52d0 0x7f364c06f9b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 -- 172.21.15.31:0/576790519 >> 172.21.15.31:0/576790519 conn(0x7f364c06cb00 msgr2=0x7f364c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 -- 172.21.15.31:0/576790519 shutdown_connections 2024-09-13T08:44:26.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.229+0000 7f364be00640 1 -- 172.21.15.31:0/576790519 wait complete. 2024-09-13T08:44:26.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.230+0000 7f364be00640 1 Processor -- start 2024-09-13T08:44:26.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.230+0000 7f364be00640 1 -- start start 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f364c0b52d0 0x7f364c151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f364c0cc720 0x7f364c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f364c152bc0 con 0x7f364c0ccb00 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f364c0703b0 con 0x7f364c0cc720 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.231+0000 7f364be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f364c152d40 con 0x7f364c0b52d0 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46802/0 (socket says 172.21.15.31:46802) 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 -- 172.21.15.31:0/2989259045 learned_addr learned my addr 172.21.15.31:0/2989259045 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364aa00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f364c0b52d0 0x7f364c151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f3643e00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f364c0cc720 0x7f364c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 -- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f364c0b52d0 msgr2=0x7f364c151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f364c0b52d0 0x7f364c151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 -- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f364c0cc720 msgr2=0x7f364c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f364c0cc720 0x7f364c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.232+0000 7f364b400640 1 -- 172.21.15.31:0/2989259045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c0cbd30 con 0x7f364c0ccb00 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f364b400640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f364400e990 tx=0x7f364400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644004d20 con 0x7f364c0ccb00 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f364400caa0 con 0x7f364c0ccb00 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364c0cc010 con 0x7f364c0ccb00 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644005020 con 0x7f364c0ccb00 2024-09-13T08:44:26.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.233+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c15c2c0 con 0x7f364c0ccb00 2024-09-13T08:44:26.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.234+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364c06f9b0 con 0x7f364c0ccb00 2024-09-13T08:44:26.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.235+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f36440051c0 con 0x7f364c0ccb00 2024-09-13T08:44:26.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.236+0000 7f3642a00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 0x7f362807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:26.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.236+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f364409ba70 con 0x7f364c0ccb00 2024-09-13T08:44:26.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.236+0000 7f364aa00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 0x7f362807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:26.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.237+0000 7f364aa00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 0x7f362807b190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f363c004f60 tx=0x7f363c007e30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:26.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.241+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3644016030 con 0x7f364c0ccb00 2024-09-13T08:44:26.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.475+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f364c155e90 con 0x7f364c0ccb00 2024-09-13T08:44:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.475+0000 7f3642a00640 1 -- 172.21.15.31:0/2989259045 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5225 (secure 0 0 0) 0x7f3644063310 con 0x7f364c0ccb00 2024-09-13T08:44:26.476 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:26.476 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"btime":"2024-09-13T08:44:23:892227+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.477+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 msgr2=0x7f362807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.477+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 0x7f362807b190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f363c004f60 tx=0x7f363c007e30 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.477+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 msgr2=0x7f364c14acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.477+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f364400e990 tx=0x7f364400ee50 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 shutdown_connections 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3628078cd0 0x7f362807b190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c0ccb00 0x7f364c14acc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f364c0cc720 0x7f364c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 --2- 172.21.15.31:0/2989259045 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f364c0b52d0 0x7f364c151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 >> 172.21.15.31:0/2989259045 conn(0x7f364c06cb00 msgr2=0x7f364c153a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 shutdown_connections 2024-09-13T08:44:26.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:26.478+0000 7f364be00640 1 -- 172.21.15.31:0/2989259045 wait complete. 2024-09-13T08:44:26.837 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:44:26.836+0000 7fab67800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:44:26.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:44:20.310429+0000', 'modified': '2024-09-13T08:44:23.328440+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7614': {'gid': 7614, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6835/1889278552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6834', 'nonce': 1889278552}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 1889278552}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7614, 'qdb_cluster': [7614]} 2024-09-13T08:44:26.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:44:26.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 -- 172.21.15.31:0/2239017046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 msgr2=0x7f0dcc073d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 --2- 172.21.15.31:0/2239017046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc073d10 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4005fd0 tx=0x7f0dc402aaa0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 -- 172.21.15.31:0/2239017046 shutdown_connections 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 --2- 172.21.15.31:0/2239017046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc073d10 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 --2- 172.21.15.31:0/2239017046 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc15be30 0x7f0dcc0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 -- 172.21.15.31:0/2239017046 >> 172.21.15.31:0/2239017046 conn(0x7f0dcc0b18f0 msgr2=0x7f0dcc0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 -- 172.21.15.31:0/2239017046 shutdown_connections 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.062+0000 7f0dd1400640 1 -- 172.21.15.31:0/2239017046 wait complete. 2024-09-13T08:44:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.063+0000 7f0dd1400640 1 Processor -- start 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.064+0000 7f0dd1400640 1 -- start start 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.064+0000 7f0dd1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.064+0000 7f0dd1400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc0ced20 0x7f0dcc0adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dd1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dcc15be30 0x7f0dcc0ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dd1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0733e0 con 0x7f0dcc073860 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dd1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0ce9e0 con 0x7f0dcc15be30 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dd1400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc074250 con 0x7f0dcc0ced20 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46818/0 (socket says 172.21.15.31:46818) 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 -- 172.21.15.31:0/4100339993 learned_addr learned my addr 172.21.15.31:0/4100339993 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc0ced20 0x7f0dcc0adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dcb200640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dcc15be30 0x7f0dcc0ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 -- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc0ced20 msgr2=0x7f0dcc0adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc0ced20 0x7f0dcc0adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 -- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dcc15be30 msgr2=0x7f0dcc0ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dcc15be30 0x7f0dcc0ae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.065+0000 7f0dca800640 1 -- 172.21.15.31:0/4100339993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc1f7f10 con 0x7f0dcc073860 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dca800640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00d770 tx=0x7f0dbc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dc2a00640 1 -- 172.21.15.31:0/4100339993 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc017070 con 0x7f0dcc073860 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dc2a00640 1 -- 172.21.15.31:0/4100339993 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dbc009d20 con 0x7f0dcc073860 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dc2a00640 1 -- 172.21.15.31:0/4100339993 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0106c0 con 0x7f0dcc073860 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc1f81b0 con 0x7f0dcc073860 2024-09-13T08:44:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.066+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc1f8770 con 0x7f0dcc073860 2024-09-13T08:44:27.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.068+0000 7f0dc2a00640 1 -- 172.21.15.31:0/4100339993 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f0dbc01a070 con 0x7f0dcc073860 2024-09-13T08:44:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.069+0000 7f0dc2a00640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 0x7f0da807b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.069+0000 7f0dc2a00640 1 -- 172.21.15.31:0/4100339993 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f0dbc09b360 con 0x7f0dcc073860 2024-09-13T08:44:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.070+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 0x7f0da807b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.070+0000 7f0dd4480740 1 Processor -- start 2024-09-13T08:44:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.070+0000 7f0dd4480740 1 -- start start 2024-09-13T08:44:27.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.070+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 0x7f0da807b190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0ce340 tx=0x7f0db4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad4838b0 0x560bad483c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad43efa0 0x560bad4828d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560bad3c1430 con 0x560bad43ea20 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560bad3b15d0 con 0x560bad4838b0 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x560bad3a0a00 con 0x560bad43efa0 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46834/0 (socket says 172.21.15.31:46834) 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 -- 172.21.15.31:0/3565577388 learned_addr learned my addr 172.21.15.31:0/3565577388 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dca800640 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad4838b0 0x560bad483c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dcb200640 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad43efa0 0x560bad4828d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 -- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad43efa0 msgr2=0x560bad4828d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad43efa0 0x560bad4828d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 -- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad4838b0 msgr2=0x560bad483c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad4838b0 0x560bad483c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.071+0000 7f0dc3e00640 1 -- 172.21.15.31:0/3565577388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bad369520 con 0x560bad43ea20 2024-09-13T08:44:27.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.072+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0db400f8d0 tx=0x7f0db4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d062b3b6be9474e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.072+0000 7f0dbb000640 1 -- 172.21.15.31:0/3565577388 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4025070 con 0x560bad43ea20 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.072+0000 7f0dbb000640 1 -- 172.21.15.31:0/3565577388 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0db401b050 con 0x560bad43ea20 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.072+0000 7f0dbb000640 1 -- 172.21.15.31:0/3565577388 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4014de0 con 0x560bad43ea20 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 -- 172.21.15.31:0/3565577388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 msgr2=0x560bad482390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0db400f8d0 tx=0x7f0db4012000 comp rx=0 tx=0).stop 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 -- 172.21.15.31:0/3565577388 shutdown_connections 2024-09-13T08:44:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad43efa0 0x560bad4828d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad43ea20 0x560bad482390 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3565577388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad4838b0 0x560bad483c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 -- 172.21.15.31:0/3565577388 >> 172.21.15.31:0/3565577388 conn(0x560bad439ff0 msgr2=0x560bad43da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 -- 172.21.15.31:0/3565577388 shutdown_connections 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.073+0000 7f0dd4480740 1 -- 172.21.15.31:0/3565577388 wait complete. 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.074+0000 7f0dd4480740 1 -- start start 2024-09-13T08:44:27.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.074+0000 7f0dd4480740 1 Processor -- start 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad4560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad492ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560bad347330 con 0x560bad4673f0 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560bad3a4810 con 0x560bad43efa0 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x560bad111620 con 0x560bad4838b0 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dd4480740 10 client.0 _ll_register_callbacks cb 0x560bad34c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46846/0 (socket says 172.21.15.31:46846) 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 -- 172.21.15.31:0/3778296608 learned_addr learned my addr 172.21.15.31:0/3778296608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dca800640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad4560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad492ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 -- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 msgr2=0x560bad492ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad492ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 -- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 msgr2=0x560bad4560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad4560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.075+0000 7f0dcb200640 1 -- 172.21.15.31:0/3778296608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bad102850 con 0x560bad4673f0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0dcb200640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0dc402b090 tx=0x7f0dc4003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb7c2a515babc84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0daf400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc402ebb0 con 0x560bad4673f0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f0dc402ed50 con 0x560bad4673f0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db401b050 con 0x560bad4673f0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0dc404e050 con 0x560bad4673f0 2024-09-13T08:44:27.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0dd4480740 10 client.7731 fetch_fsmap learned FSMap version 81 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x560bad1796b0 con 0x560bad4673f0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.076+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0db401b050 con 0x560bad4673f0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7f0dc4051020 con 0x560bad4673f0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 10 client.7731 fetch_fsmap finished waiting for FSMap version 81 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 10 client.7731 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 20 client.7731 populate_metadata read hostname 'smithi031' 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad4529a0 0x560bad452d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 4 client.7731 mds_command: new command op to 7614 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 --> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bad3b3520 con 0x560bad4529a0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dca800640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad4529a0 0x560bad452d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0daf400640 10 client.7731 ms_handle_connect on v2:172.21.15.31:6834/1889278552 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.077+0000 7f0dca800640 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad4529a0 0x560bad452d80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10c17624576126e4 server_cookie=893bae81b58f0e3e in_seq=0 out_seq=0 2024-09-13T08:44:27.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.078+0000 7f0daf400640 1 -- 172.21.15.31:0/3778296608 <== mds.0 v2:172.21.15.31:6834/1889278552 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560bad3b3520 con 0x560bad4529a0 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.078+0000 7f0daf400640 10 client.7731 handle_command_reply: tid=0 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.078+0000 7f0dd4480740 1 client.7731 shutdown 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 msgr2=0x560bad480bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0dc402b090 tx=0x7f0dc4003ed0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 shutdown_connections 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad4529a0 0x560bad452d80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480bf0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad492ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 --2- 172.21.15.31:0/3778296608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad4560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0daf400640 0 client.7731 ms_handle_reset on v2:172.21.15.31:6834/1889278552 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 >> 172.21.15.31:0/3778296608 conn(0x560bad43ea20 msgr2=0x560bad438bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 shutdown_connections 2024-09-13T08:44:27.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 1 -- 172.21.15.31:0/3778296608 wait complete. 2024-09-13T08:44:27.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.079+0000 7f0dd4480740 20 client.7731 trim_cache size 0 max 16384 2024-09-13T08:44:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.098+0000 7f0dd4480740 1 Processor -- start 2024-09-13T08:44:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.098+0000 7f0dd4480740 1 -- start start 2024-09-13T08:44:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad47fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad4667c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc200340 con 0x560bad4673f0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0b4b00 con 0x560bad43efa0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc14a100 con 0x560bad4838b0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dc3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46852/0 (socket says 172.21.15.31:46852) 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dc3e00640 1 -- 172.21.15.31:0/553549658 learned_addr learned my addr 172.21.15.31:0/553549658 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dcb200640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad4667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.099+0000 7f0dca800640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad47fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 -- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 msgr2=0x560bad4667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad4667c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 -- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 msgr2=0x560bad47fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad47fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 -- 172.21.15.31:0/553549658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bad454d90 con 0x560bad4673f0 2024-09-13T08:44:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dcb200640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad4667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dca800640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad47fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dc3e00640 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008fd0 tx=0x7f0db4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf2a070fc3cf474 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dba600640 1 -- 172.21.15.31:0/553549658 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4025050 con 0x560bad4673f0 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dba600640 1 -- 172.21.15.31:0/553549658 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0db4031070 con 0x560bad4673f0 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.100+0000 7f0dba600640 1 -- 172.21.15.31:0/553549658 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4020ac0 con 0x560bad4673f0 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 -- 172.21.15.31:0/553549658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 msgr2=0x560bad480210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008fd0 tx=0x7f0db4000ed0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 -- 172.21.15.31:0/553549658 shutdown_connections 2024-09-13T08:44:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4838b0 0x560bad4667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4673f0 0x560bad480210 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 --2- 172.21.15.31:0/553549658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad47fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 -- 172.21.15.31:0/553549658 >> 172.21.15.31:0/553549658 conn(0x560bad43ea20 msgr2=0x560bad478750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 -- 172.21.15.31:0/553549658 shutdown_connections 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.101+0000 7f0dd4480740 1 -- 172.21.15.31:0/553549658 wait complete. 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.102+0000 7f0dd4480740 1 -- start start 2024-09-13T08:44:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.102+0000 7f0dd4480740 1 Processor -- start 2024-09-13T08:44:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad503bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4673f0 0x560bad4a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560bad4815d0 con 0x560bad4838b0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560bad481130 con 0x560bad43efa0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x560bad47cef0 con 0x560bad4673f0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dd4480740 10 client.0 _ll_register_callbacks cb 0x7f0dc4003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46866/0 (socket says 172.21.15.31:46866) 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 -- 172.21.15.31:0/4141282913 learned_addr learned my addr 172.21.15.31:0/4141282913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 -- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4673f0 msgr2=0x560bad4a3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4673f0 0x560bad4a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 -- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 msgr2=0x560bad503bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dca800640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad503bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad503bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dcb200640 1 -- 172.21.15.31:0/4141282913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bad48fbd0 con 0x560bad4838b0 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.103+0000 7f0dca800640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad503bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0dcb200640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009600 tx=0x7f0dc4003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7692039a7e0edc07 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0daf400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40045b0 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f0dc4004750 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db4031070 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0dc4043450 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0dd4480740 10 client.7737 fetch_fsmap learned FSMap version 81 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x560bad5040f0 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.104+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0db4031070 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7f0dc4063080 con 0x560bad4838b0 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 10 client.7737 fetch_fsmap finished waiting for FSMap version 81 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 10 client.7737 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 20 client.7737 populate_metadata read hostname 'smithi031' 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad507860 0x560bad507c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 4 client.7737 mds_command: new command op to 7614 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 --> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] -- command(tid 0: {"prefix": "status"}) -- 0x560bad3aefd0 con 0x560bad507860 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dca800640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad507860 0x560bad507c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0dca800640 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad507860 0x560bad507c40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d07a2529445660d5 server_cookie=5ea1b4d59a492f87 in_seq=0 out_seq=0 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.105+0000 7f0daf400640 10 client.7737 ms_handle_connect on v2:172.21.15.31:6834/1889278552 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0daf400640 1 -- 172.21.15.31:0/4141282913 <== mds.0 v2:172.21.15.31:6834/1889278552 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560bad3aefd0 con 0x560bad507860 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0daf400640 10 client.7737 handle_command_reply: tid=0 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 client.7737 shutdown 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 msgr2=0x560bad4a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009600 tx=0x7f0dc4003f40 comp rx=0 tx=0).stop 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 shutdown_connections 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:6834/1889278552,v1:172.21.15.31:6835/1889278552] conn(0x560bad507860 0x560bad507c40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560bad4838b0 0x560bad4a4070 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x560bad4673f0 0x560bad4a3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 --2- 172.21.15.31:0/4141282913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bad43efa0 0x560bad503bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0daf400640 0 client.7737 ms_handle_reset on v2:172.21.15.31:6834/1889278552 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 >> 172.21.15.31:0/4141282913 conn(0x560bad43ea20 msgr2=0x560bad47c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 shutdown_connections 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 1 -- 172.21.15.31:0/4141282913 wait complete. 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.106+0000 7f0dd4480740 20 client.7737 trim_cache size 0 max 16384 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:44:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 7614, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2405944629999999, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 81, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 124, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 124, 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.518643637 2024-09-13T08:44:27.112 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 msgr2=0x7f0da807b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 0x7f0da807b190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0ce340 tx=0x7f0db4009040 comp rx=0 tx=0).stop 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 msgr2=0x7f0dcc1f4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00d770 tx=0x7f0dbc00dc30 comp rx=0 tx=0).stop 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 shutdown_connections 2024-09-13T08:44:27.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0da8078cd0 0x7f0da807b190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0dcc15be30 0x7f0dcc0ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0dcc0ced20 0x7f0dcc0adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 --2- 172.21.15.31:0/4100339993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dcc073860 0x7f0dcc1f4920 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 >> 172.21.15.31:0/4100339993 conn(0x7f0dcc0b18f0 msgr2=0x7f0dcc0af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 shutdown_connections 2024-09-13T08:44:27.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.107+0000 7f0dd1400640 1 -- 172.21.15.31:0/4100339993 wait complete. 2024-09-13T08:44:27.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:44:27.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 -- 172.21.15.31:0/2290823600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 msgr2=0x7f6c840cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 --2- 172.21.15.31:0/2290823600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c840cf850 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f6c840cc8c0 tx=0x7f6c7402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:27.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 -- 172.21.15.31:0/2290823600 shutdown_connections 2024-09-13T08:44:27.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 --2- 172.21.15.31:0/2290823600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c8415f9e0 0x7f6c8415fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 --2- 172.21.15.31:0/2290823600 >> v2:172.21.15.45:3300/0 conn(0x7f6c840ce3c0 0x7f6c840ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 --2- 172.21.15.31:0/2290823600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c840cf850 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 -- 172.21.15.31:0/2290823600 >> 172.21.15.31:0/2290823600 conn(0x7f6c840cc910 msgr2=0x7f6c840cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 -- 172.21.15.31:0/2290823600 shutdown_connections 2024-09-13T08:44:27.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.284+0000 7f6c89c00640 1 -- 172.21.15.31:0/2290823600 wait complete. 2024-09-13T08:44:27.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.286+0000 7f6c89c00640 1 Processor -- start 2024-09-13T08:44:27.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.286+0000 7f6c89c00640 1 -- start start 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 0x7f6c841f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 0x7f6c8409cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c840cb9d0 con 0x7f6c8415bd70 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6c840cb850 con 0x7f6c8415f9e0 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c89c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6c840cbc50 con 0x7f6c840ce3c0 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c82000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c82000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46888/0 (socket says 172.21.15.31:46888) 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c82000640 1 -- 172.21.15.31:0/1844205867 learned_addr learned my addr 172.21.15.31:0/1844205867 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c82a00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 0x7f6c841f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.287+0000 7f6c83400640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 0x7f6c8409cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 -- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 msgr2=0x7f6c841f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 0x7f6c841f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 -- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 msgr2=0x7f6c8409cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 0x7f6c8409cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 -- 172.21.15.31:0/1844205867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8415d450 con 0x7f6c8415bd70 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82a00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 0x7f6c841f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c83400640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 0x7f6c8409cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c82000640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e740 tx=0x7f6c7000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000ccc0 con 0x7f6c8415bd70 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c7000ee50 con 0x7f6c8415bd70 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70004290 con 0x7f6c8415bd70 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c841f4ce0 con 0x7f6c8415bd70 2024-09-13T08:44:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.288+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c841f52a0 con 0x7f6c8415bd70 2024-09-13T08:44:27.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.289+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8406f5a0 con 0x7f6c8415bd70 2024-09-13T08:44:27.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.290+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f6c700035f0 con 0x7f6c8415bd70 2024-09-13T08:44:27.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.292+0000 7f6c7aa00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 0x7f6c6407b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:27.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.292+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f6c7009bb00 con 0x7f6c8415bd70 2024-09-13T08:44:27.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.292+0000 7f6c82a00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 0x7f6c6407b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:27.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.293+0000 7f6c82a00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 0x7f6c6407b190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c74033000 tx=0x7f6c74004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:27.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.296+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6c7009de20 con 0x7f6c8415bd70 2024-09-13T08:44:27.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.529+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6c840cd170 con 0x7f6c8415bd70 2024-09-13T08:44:27.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.529+0000 7f6c7aa00640 1 -- 172.21.15.31:0/1844205867 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) ==== 53+0+73 (secure 0 0 0) 0x7f6c700639f0 con 0x7f6c8415bd70 2024-09-13T08:44:27.531 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 msgr2=0x7f6c6407b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 0x7f6c6407b190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c74033000 tx=0x7f6c74004730 comp rx=0 tx=0).stop 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 msgr2=0x7f6c8415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e740 tx=0x7f6c7000ec00 comp rx=0 tx=0).stop 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 shutdown_connections 2024-09-13T08:44:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6c64078cd0 0x7f6c6407b190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8415f9e0 0x7f6c8409cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8415bd70 0x7f6c8415b430 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 --2- 172.21.15.31:0/1844205867 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6c840ce3c0 0x7f6c841f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 >> 172.21.15.31:0/1844205867 conn(0x7f6c840cc910 msgr2=0x7f6c840cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 shutdown_connections 2024-09-13T08:44:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27.532+0000 7f6c89c00640 1 -- 172.21.15.31:0/1844205867 wait complete. 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:44:27.549 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:44:27.550 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:44:27.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:27.550 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42264sec preferred_lft 42264sec 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.587 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.588 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:44:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:44:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:44:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:27.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:27.589 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:44:27.681 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:44:27.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:27.685 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:44:27.721 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:44:27.721 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:44:27.721 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:44:27.721 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:44:27.721 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:44:27.722 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:44:27.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:44:27.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:27.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:27.922 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:44:27.958 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:44:27.958 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:44:27.959 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:44:27.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:27.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:44:28.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:44:28.054 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:28.090 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:44:28.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:44:28.091 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:28.166 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:44:28.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:28.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:28.294 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:28.295 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:44:28.321 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:28.321 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:44:28.322 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:44:28.322 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:44:28.322 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:44:28.323 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:44:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:44:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:44:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:44:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:44:28.326 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:44:28.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:28.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:28.384 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:44:28.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:44:28.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:28.483 INFO:teuthology.orchestra.run.smithi031.stdout:client7258 2024-09-13T08:44:28.483 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:28.483 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:28.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:44:28.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 -- 172.21.15.31:0/2270967520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7504004800 msgr2=0x7f7504004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:28.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 --2- 172.21.15.31:0/2270967520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7504004800 0x7f7504004c80 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7508009b50 tx=0x7f7504005830 comp rx=0 tx=0).stop 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 -- 172.21.15.31:0/2270967520 shutdown_connections 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 --2- 172.21.15.31:0/2270967520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7504004800 0x7f7504004c80 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 -- 172.21.15.31:0/2270967520 >> 172.21.15.31:0/2270967520 conn(0x7f751c0cc910 msgr2=0x7f751c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 -- 172.21.15.31:0/2270967520 shutdown_connections 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.722+0000 7f7524000640 1 -- 172.21.15.31:0/2270967520 wait complete. 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.724+0000 7f7524000640 1 Processor -- start 2024-09-13T08:44:28.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.724+0000 7f7524000640 1 -- start start 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7504004800 0x7f751c158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f751c1f8d20 0x7f751c1f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f751c0cb850 con 0x7f751c1f5c30 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f751c0b1a20 con 0x7f7504004800 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7524000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f751c0cb9d0 con 0x7f751c1f8d20 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:28.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46906/0 (socket says 172.21.15.31:46906) 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 -- 172.21.15.31:0/2834408089 learned_addr learned my addr 172.21.15.31:0/2834408089 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7521600640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7504004800 0x7f751c158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7520c00640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f751c1f8d20 0x7f751c1f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 -- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f751c1f8d20 msgr2=0x7f751c1f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f751c1f8d20 0x7f751c1f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 -- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7504004800 msgr2=0x7f751c158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7504004800 0x7f751c158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.725+0000 7f7522000640 1 -- 172.21.15.31:0/2834408089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751c06e590 con 0x7f751c1f5c30 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7522000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cf330 tx=0x7f75140019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514011870 con 0x7f751c1f5c30 2024-09-13T08:44:28.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7514017070 con 0x7f751c1f5c30 2024-09-13T08:44:28.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f751c159490 con 0x7f751c1f5c30 2024-09-13T08:44:28.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514011d80 con 0x7f751c1f5c30 2024-09-13T08:44:28.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.726+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751c159710 con 0x7f751c1f5c30 2024-09-13T08:44:28.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.727+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f751c0ce640 con 0x7f751c1f5c30 2024-09-13T08:44:28.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.728+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f751401c460 con 0x7f751c1f5c30 2024-09-13T08:44:28.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.729+0000 7f7519600640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 0x7f74fc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:28.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.729+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f751409b9f0 con 0x7f751c1f5c30 2024-09-13T08:44:28.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.730+0000 7f7521600640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 0x7f74fc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:28.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.730+0000 7f7521600640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 0x7f74fc07b190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7510001860 tx=0x7f7510001c00 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:28.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.734+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f751c159490 con 0x7f751c1f5c30 2024-09-13T08:44:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.970+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f751c0b0f60 con 0x7f751c1f5c30 2024-09-13T08:44:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.970+0000 7f7519600640 1 -- 172.21.15.31:0/2834408089 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) ==== 53+0+73 (secure 0 0 0) 0x7f7514063980 con 0x7f751c1f5c30 2024-09-13T08:44:28.971 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:44:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 msgr2=0x7f74fc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 0x7f74fc07b190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7510001860 tx=0x7f7510001c00 comp rx=0 tx=0).stop 2024-09-13T08:44:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 msgr2=0x7f751c1f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cf330 tx=0x7f75140019b0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 shutdown_connections 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f74fc078cd0 0x7f74fc07b190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f751c1f5c30 0x7f751c1f7450 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f751c1f8d20 0x7f751c1f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 --2- 172.21.15.31:0/2834408089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7504004800 0x7f751c158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.972+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 >> 172.21.15.31:0/2834408089 conn(0x7f751c0cc910 msgr2=0x7f751c0ad440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.973+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 shutdown_connections 2024-09-13T08:44:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:28.973+0000 7f7524000640 1 -- 172.21.15.31:0/2834408089 wait complete. 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:44:28.989 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:44:28.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:28.990 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:44:29.030 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:44:29.031 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42267sec preferred_lft 42267sec 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:44:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:44:29.033 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:44:29.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.034 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:44:29.129 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:44:29.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.132 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:44:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:44:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:44:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:44:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:44:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:44:29.170 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:44:29.171 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:44:29.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:44:29.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:44:29.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:44:29.171 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:44:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:44:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:44:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:44:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:29.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.369 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:44:29.404 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:44:29.404 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:44:29.405 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:44:29.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:44:29.501 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:44:29.501 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:29.536 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:44:29.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:44:29.537 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:29.611 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:44:29.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:29.741 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:44:29.769 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:44:29.769 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:44:29.769 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:44:29.769 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:44:29.769 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:44:29.770 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:44:29.771 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:44:29.771 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:44:29.771 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:44:29.771 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:44:29.773 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:44:29.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:29.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:29.830 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:44:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:44:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:44:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:44:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:29.922 INFO:teuthology.orchestra.run.smithi045.stdout:client6617 2024-09-13T08:44:29.922 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:29.922 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:29.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:44:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 -- 172.21.15.31:0/381398183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6280b4ca0 msgr2=0x7fc6280d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 --2- 172.21.15.31:0/381398183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6280b4ca0 0x7fc6280d0d40 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc614005fd0 tx=0x7fc61402a4b0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 -- 172.21.15.31:0/381398183 shutdown_connections 2024-09-13T08:44:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 --2- 172.21.15.31:0/381398183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6280b4ca0 0x7fc6280d0d40 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 --2- 172.21.15.31:0/381398183 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6281616c0 0x7fc6280d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 -- 172.21.15.31:0/381398183 >> 172.21.15.31:0/381398183 conn(0x7fc6280d3f30 msgr2=0x7fc6280d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 -- 172.21.15.31:0/381398183 shutdown_connections 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.133+0000 7fc62ce00640 1 -- 172.21.15.31:0/381398183 wait complete. 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.135+0000 7fc62ce00640 1 Processor -- start 2024-09-13T08:44:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.135+0000 7fc62ce00640 1 -- start start 2024-09-13T08:44:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6280b4ca0 0x7fc62815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6280b6d60 0x7fc6280b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280cfee0 con 0x7fc6281616c0 2024-09-13T08:44:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280cfd60 con 0x7fc6280b6d60 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc62ce00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc6280b7920 con 0x7fc6280b4ca0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46918/0 (socket says 172.21.15.31:46918) 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 -- 172.21.15.31:0/3090453603 learned_addr learned my addr 172.21.15.31:0/3090453603 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc625400640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6280b6d60 0x7fc6280b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc625e00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6280b4ca0 0x7fc62815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 -- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6280b4ca0 msgr2=0x7fc62815d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6280b4ca0 0x7fc62815d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.136+0000 7fc626800640 1 -- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6280b6d60 msgr2=0x7fc6280b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc626800640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6280b6d60 0x7fc6280b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc626800640 1 -- 172.21.15.31:0/3090453603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6281f57d0 con 0x7fc6281616c0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc626800640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc62000c7a0 tx=0x7fc62000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620003b70 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc620003d30 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62000aab0 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62815dd30 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.137+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6280a0bf0 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.138+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6280b7db0 con 0x7fc6281616c0 2024-09-13T08:44:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.140+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7fc62000ac50 con 0x7fc6281616c0 2024-09-13T08:44:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.140+0000 7fc61e000640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 0x7fc60807b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.141+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7fc62009b030 con 0x7fc6281616c0 2024-09-13T08:44:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.141+0000 7fc625e00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 0x7fc60807b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.141+0000 7fc625e00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 0x7fc60807b1c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc618005fd0 tx=0x7fc618005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.145+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc620062fc0 con 0x7fc6281616c0 2024-09-13T08:44:30.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.379+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fc6280d22f0 con 0x7fc6281616c0 2024-09-13T08:44:30.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.917+0000 7fc61e000640 1 -- 172.21.15.31:0/3090453603 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v82) ==== 136+0+0 (secure 0 0 0) 0x7fc6200628c0 con 0x7fc6281616c0 2024-09-13T08:44:30.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 msgr2=0x7fc60807b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:30.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 0x7fc60807b1c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc618005fd0 tx=0x7fc618005850 comp rx=0 tx=0).stop 2024-09-13T08:44:30.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 msgr2=0x7fc6280b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:30.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc62000c7a0 tx=0x7fc62000cc60 comp rx=0 tx=0).stop 2024-09-13T08:44:30.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 shutdown_connections 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc608078d00 0x7fc60807b1c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6281616c0 0x7fc6280b1a00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6280b6d60 0x7fc6280b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 --2- 172.21.15.31:0/3090453603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc6280b4ca0 0x7fc62815d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.920+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 >> 172.21.15.31:0/3090453603 conn(0x7fc6280d3f30 msgr2=0x7fc6280b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.921+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 shutdown_connections 2024-09-13T08:44:30.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:30.921+0000 7fc62ce00640 1 -- 172.21.15.31:0/3090453603 wait complete. 2024-09-13T08:44:30.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 -- 172.21.15.31:0/1864937546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b6015be30 msgr2=0x7f2b600cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 --2- 172.21.15.31:0/1864937546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b6015be30 0x7f2b600cdb80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2b54005fd0 tx=0x7f2b5402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 -- 172.21.15.31:0/1864937546 shutdown_connections 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 --2- 172.21.15.31:0/1864937546 >> v2:172.21.15.45:3300/0 conn(0x7f2b60073860 0x7f2b60073d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 --2- 172.21.15.31:0/1864937546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b6015be30 0x7f2b600cdb80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 -- 172.21.15.31:0/1864937546 >> 172.21.15.31:0/1864937546 conn(0x7f2b600b18f0 msgr2=0x7f2b600b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 -- 172.21.15.31:0/1864937546 shutdown_connections 2024-09-13T08:44:31.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.107+0000 7f2b68200640 1 -- 172.21.15.31:0/1864937546 wait complete. 2024-09-13T08:44:31.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.109+0000 7f2b68200640 1 Processor -- start 2024-09-13T08:44:31.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.109+0000 7f2b68200640 1 -- start start 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b600ced20 0x7f2b60158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 0x7f2b601f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60074250 con 0x7f2b6015be30 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b600ce9e0 con 0x7f2b60073860 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b68200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2b600743d0 con 0x7f2b600ced20 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b65800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b66200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 0x7f2b601f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b65800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:59896/0 (socket says 172.21.15.31:59896) 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b65800640 1 -- 172.21.15.31:0/1734595467 learned_addr learned my addr 172.21.15.31:0/1734595467 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.110+0000 7f2b64e00640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b600ced20 0x7f2b60158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 -- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b600ced20 msgr2=0x7f2b60158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b600ced20 0x7f2b60158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 -- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 msgr2=0x7f2b601f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 0x7f2b601f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 -- 172.21.15.31:0/1734595467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6015a1a0 con 0x7f2b60073860 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b66200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 0x7f2b601f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b65800640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2b54034040 tx=0x7f2b54003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5403e030 con 0x7f2b60073860 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b54003c20 con 0x7f2b60073860 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54035040 con 0x7f2b60073860 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.111+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6015a420 con 0x7f2b60073860 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.112+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b6015a9c0 con 0x7f2b60073860 2024-09-13T08:44:31.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.113+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b600748c0 con 0x7f2b60073860 2024-09-13T08:44:31.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.114+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f2b54041030 con 0x7f2b60073860 2024-09-13T08:44:31.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.115+0000 7f2b5d600640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 0x7f2b4007b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.115+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f2b540bb5f0 con 0x7f2b60073860 2024-09-13T08:44:31.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.115+0000 7f2b64e00640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 0x7f2b4007b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.116+0000 7f2b64e00640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 0x7f2b4007b1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b60149440 tx=0x7f2b4c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:31.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.120+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2b6015a1a0 con 0x7f2b60073860 2024-09-13T08:44:31.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.337+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f2b60146040 con 0x7f2b40078d00 2024-09-13T08:44:31.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.339+0000 7f2b5d600640 1 -- 172.21.15.31:0/1734595467 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2b60146040 con 0x7f2b40078d00 2024-09-13T08:44:31.339 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:44:31.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:31.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:44:31.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:31.340 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:44:31.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 msgr2=0x7f2b4007b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 0x7f2b4007b1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b60149440 tx=0x7f2b4c009040 comp rx=0 tx=0).stop 2024-09-13T08:44:31.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 msgr2=0x7f2b6015b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2b54034040 tx=0x7f2b54003820 comp rx=0 tx=0).stop 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 shutdown_connections 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b40078d00 0x7f2b4007b1c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b6015be30 0x7f2b601f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2b600ced20 0x7f2b60158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 --2- 172.21.15.31:0/1734595467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b60073860 0x7f2b6015b1c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 >> 172.21.15.31:0/1734595467 conn(0x7f2b600b18f0 msgr2=0x7f2b600aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 shutdown_connections 2024-09-13T08:44:31.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.341+0000 7f2b68200640 1 -- 172.21.15.31:0/1734595467 wait complete. 2024-09-13T08:44:31.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:44:31.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 -- 172.21.15.31:0/602050280 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f74200cc3e0 msgr2=0x7f74200cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 --2- 172.21.15.31:0/602050280 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f74200cc3e0 0x7f74200cc830 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7418005fd0 tx=0x7f741802a980 comp rx=0 tx=0).stop 2024-09-13T08:44:31.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 -- 172.21.15.31:0/602050280 shutdown_connections 2024-09-13T08:44:31.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 --2- 172.21.15.31:0/602050280 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f74200cc3e0 0x7f74200cc830 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 --2- 172.21.15.31:0/602050280 >> v2:172.21.15.45:3300/0 conn(0x7f7420073ac0 0x7f74200b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 -- 172.21.15.31:0/602050280 >> 172.21.15.31:0/602050280 conn(0x7f74200b1520 msgr2=0x7f74200b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 -- 172.21.15.31:0/602050280 shutdown_connections 2024-09-13T08:44:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.527+0000 7f741fe00640 1 -- 172.21.15.31:0/602050280 wait complete. 2024-09-13T08:44:31.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.529+0000 7f741fe00640 1 Processor -- start 2024-09-13T08:44:31.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.529+0000 7f741fe00640 1 -- start start 2024-09-13T08:44:31.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 0x7f74201e8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f742015bc00 0x7f74201e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74200cace0 con 0x7f7420073ac0 2024-09-13T08:44:31.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f74200cab60 con 0x7f74200cc3e0 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f74200cae60 con 0x7f742015bc00 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46926/0 (socket says 172.21.15.31:46926) 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 -- 172.21.15.31:0/2091951406 learned_addr learned my addr 172.21.15.31:0/2091951406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f7416000640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 0x7f74201e8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741f400640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f742015bc00 0x7f74201e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 -- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f742015bc00 msgr2=0x7f74201e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f742015bc00 0x7f74201e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 -- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 msgr2=0x7f74201e8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 0x7f74201e8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f741ea00640 1 -- 172.21.15.31:0/2091951406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74200cf590 con 0x7f7420073ac0 2024-09-13T08:44:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.530+0000 7f7416000640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 0x7f74201e8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:31.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f741ea00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f741000c3a0 tx=0x7f741000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:31.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410017070 con 0x7f7420073ac0 2024-09-13T08:44:31.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74100053c0 con 0x7f7420073ac0 2024-09-13T08:44:31.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410003850 con 0x7f7420073ac0 2024-09-13T08:44:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7420159990 con 0x7f7420073ac0 2024-09-13T08:44:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.531+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742015e450 con 0x7f7420073ac0 2024-09-13T08:44:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.532+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74200cb380 con 0x7f7420073ac0 2024-09-13T08:44:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.533+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f741001a070 con 0x7f7420073ac0 2024-09-13T08:44:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.535+0000 7f7417400640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 0x7f73fc07b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.535+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f741009a620 con 0x7f7420073ac0 2024-09-13T08:44:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.535+0000 7f7416000640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 0x7f73fc07b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:31.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.536+0000 7f7416000640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 0x7f73fc07b190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74200cc930 tx=0x7f7408005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:31.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.539+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f742015bfe0 con 0x7f7420073ac0 2024-09-13T08:44:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.734+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f74200cb6c0 con 0x7f7420073ac0 2024-09-13T08:44:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.734+0000 7f7417400640 1 -- 172.21.15.31:0/2091951406 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f7410062510 con 0x7f7420073ac0 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 msgr2=0x7f73fc07b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 0x7f73fc07b190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74200cc930 tx=0x7f7408005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 msgr2=0x7f7420159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f741000c3a0 tx=0x7f741000c860 comp rx=0 tx=0).stop 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 shutdown_connections 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc078cd0 0x7f73fc07b190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f742015bc00 0x7f74201e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74200cc3e0 0x7f74201e8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 --2- 172.21.15.31:0/2091951406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7420073ac0 0x7f7420159450 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 >> 172.21.15.31:0/2091951406 conn(0x7f74200b1520 msgr2=0x7f74200b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 shutdown_connections 2024-09-13T08:44:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:31.737+0000 7f741fe00640 1 -- 172.21.15.31:0/2091951406 wait complete. 2024-09-13T08:44:31.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:31.754 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol91320/b6e24bde-0172-4ef1-9bf7-b056d8ab5136) 2024-09-13T08:44:31.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:31.818 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol91320/b6e24bde-0172-4ef1-9bf7-b056d8ab5136/.snap/fake) 2024-09-13T08:44:31.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:31.902 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:44:31.939 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:44:31.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:31.940 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol91320/b6e24bde-0172-4ef1-9bf7-b056d8ab5136) 2024-09-13T08:44:32.041 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:32.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol91320/.meta 2024-09-13T08:44:32.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.102 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp2160/subvol11341/f59e2548-2b3e-42a1-aec9-c9b4fee6170e) 2024-09-13T08:44:32.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.202 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp2160/subvol11341/f59e2548-2b3e-42a1-aec9-c9b4fee6170e/.snap/fake) 2024-09-13T08:44:32.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.287 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:44:32.323 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:44:32.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.323 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp2160/subvol11341/f59e2548-2b3e-42a1-aec9-c9b4fee6170e) 2024-09-13T08:44:32.423 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:32.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp2160/subvol11341/.meta 2024-09-13T08:44:32.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.481 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol42032/62e641f9-94bf-4ada-81f2-46ea1c11402d) 2024-09-13T08:44:32.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.580 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:44:32.619 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:44:32.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:32.620 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol42032/62e641f9-94bf-4ada-81f2-46ea1c11402d) 2024-09-13T08:44:32.721 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:32.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol42032/.meta 2024-09-13T08:44:32.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol91320 2024-09-13T08:44:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 -- 172.21.15.31:0/3819000217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22900cc3e0 msgr2=0x7f22900cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 --2- 172.21.15.31:0/3819000217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22900cc3e0 0x7f22900cc830 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2288005fd0 tx=0x7f228802a980 comp rx=0 tx=0).stop 2024-09-13T08:44:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 -- 172.21.15.31:0/3819000217 shutdown_connections 2024-09-13T08:44:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 --2- 172.21.15.31:0/3819000217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22900cc3e0 0x7f22900cc830 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 --2- 172.21.15.31:0/3819000217 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22900b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 -- 172.21.15.31:0/3819000217 >> 172.21.15.31:0/3819000217 conn(0x7f22900b1520 msgr2=0x7f22900b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 -- 172.21.15.31:0/3819000217 shutdown_connections 2024-09-13T08:44:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.983+0000 7f2296200640 1 -- 172.21.15.31:0/3819000217 wait complete. 2024-09-13T08:44:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.985+0000 7f2296200640 1 Processor -- start 2024-09-13T08:44:32.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.985+0000 7f2296200640 1 -- start start 2024-09-13T08:44:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22901581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22900cc3e0 0x7f2290158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22900cace0 con 0x7f229015bc00 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f22900cab60 con 0x7f22900cc3e0 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2296200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f22900cae60 con 0x7f2290073ac0 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46952/0 (socket says 172.21.15.31:46952) 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 -- 172.21.15.31:0/1971338821 learned_addr learned my addr 172.21.15.31:0/1971338821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f228fe00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22900cc3e0 0x7f2290158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2294e00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22901581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 -- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 msgr2=0x7f22901581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22901581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 -- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22900cc3e0 msgr2=0x7f2290158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22900cc3e0 0x7f2290158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2295800640 1 -- 172.21.15.31:0/1971338821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22900b04b0 con 0x7f229015bc00 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.986+0000 7f2294e00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22901581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f2295800640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2288006010 tx=0x7f2288003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288047070 con 0x7f229015bc00 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2288004540 con 0x7f229015bc00 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288034070 con 0x7f229015bc00 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229015ee40 con 0x7f229015bc00 2024-09-13T08:44:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.987+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22900b0790 con 0x7f229015bc00 2024-09-13T08:44:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.988+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22900cb330 con 0x7f229015bc00 2024-09-13T08:44:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.989+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f228804a070 con 0x7f229015bc00 2024-09-13T08:44:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.990+0000 7f228cc00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 0x7f227007b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.990+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f22880bc260 con 0x7f229015bc00 2024-09-13T08:44:32.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.991+0000 7f2294e00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 0x7f227007b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.991+0000 7f2294e00640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 0x7f227007b1c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2284007fd0 tx=0x7f2284009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:32.995+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f228803f070 con 0x7f229015bc00 2024-09-13T08:44:33.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.218+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol91320", "target": ["mon-mgr", ""]}) -- 0x7f2290070170 con 0x7f2270078d00 2024-09-13T08:44:33.227 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:33.226+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:33.227 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:33.226+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:33.227 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:33.226+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:33.227 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:33.226+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:33.227 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:33.226+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:33.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.269+0000 7f228cc00640 1 -- 172.21.15.31:0/1971338821 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f2290070170 con 0x7f2270078d00 2024-09-13T08:44:33.270 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol91320/b6e24bde-0172-4ef1-9bf7-b056d8ab5136 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 msgr2=0x7f227007b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 0x7f227007b1c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2284007fd0 tx=0x7f2284009040 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 msgr2=0x7f229015ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2288006010 tx=0x7f2288003f10 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 shutdown_connections 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2270078d00 0x7f227007b1c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f229015bc00 0x7f229015ca40 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22900cc3e0 0x7f2290158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 --2- 172.21.15.31:0/1971338821 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2290073ac0 0x7f22901581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.271+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 >> 172.21.15.31:0/1971338821 conn(0x7f22900b1520 msgr2=0x7f22900af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:33.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.272+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 shutdown_connections 2024-09-13T08:44:33.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.272+0000 7f2296200640 1 -- 172.21.15.31:0/1971338821 wait complete. 2024-09-13T08:44:33.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol11341 subvol_grp2160 2024-09-13T08:44:33.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 -- 172.21.15.31:0/1786014978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac0cdf90 msgr2=0x7f19ac0ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 --2- 172.21.15.31:0/1786014978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac0cdf90 0x7f19ac0ad890 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f19a0005fd0 tx=0x7f19a002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 -- 172.21.15.31:0/1786014978 shutdown_connections 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 --2- 172.21.15.31:0/1786014978 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac06e420 0x7f19ac06e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 --2- 172.21.15.31:0/1786014978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac0cdf90 0x7f19ac0ad890 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 -- 172.21.15.31:0/1786014978 >> 172.21.15.31:0/1786014978 conn(0x7f19ac0b1520 msgr2=0x7f19ac0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 -- 172.21.15.31:0/1786014978 shutdown_connections 2024-09-13T08:44:33.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.451+0000 7f19b3200640 1 -- 172.21.15.31:0/1786014978 wait complete. 2024-09-13T08:44:33.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.453+0000 7f19b3200640 1 Processor -- start 2024-09-13T08:44:33.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.453+0000 7f19b3200640 1 -- start start 2024-09-13T08:44:33.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19ac06e420 0x7f19ac158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac0cdf90 0x7f19ac0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0cc560 con 0x7f19ac146f50 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0cc3e0 con 0x7f19ac06e420 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b3200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f19ac0cc6e0 con 0x7f19ac0cdf90 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46980/0 (socket says 172.21.15.31:46980) 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b2800640 1 -- 172.21.15.31:0/4177664388 learned_addr learned my addr 172.21.15.31:0/4177664388 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b1400640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac0cdf90 0x7f19ac0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.454+0000 7f19b1e00640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19ac06e420 0x7f19ac158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 -- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac0cdf90 msgr2=0x7f19ac0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac0cdf90 0x7f19ac0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 -- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19ac06e420 msgr2=0x7f19ac158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19ac06e420 0x7f19ac158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 -- 172.21.15.31:0/4177664388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac0b4530 con 0x7f19ac146f50 2024-09-13T08:44:33.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b2800640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f19a800eaa0 tx=0x7f19a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a800cdb0 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19a8004c10 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac0b4810 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8005020 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.455+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac145670 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.456+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac0ccc70 con 0x7f19ac146f50 2024-09-13T08:44:33.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.458+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f19a80051c0 con 0x7f19ac146f50 2024-09-13T08:44:33.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.459+0000 7f19a6000640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 0x7f199007b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.459+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f19a809bb80 con 0x7f19ac146f50 2024-09-13T08:44:33.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.459+0000 7f19b1e00640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 0x7f199007b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.460+0000 7f19b1e00640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 0x7f199007b190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f19a0033000 tx=0x7f19a0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:33.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.463+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f19a8016030 con 0x7f19ac146f50 2024-09-13T08:44:33.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.685+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol11341", "group_name": "subvol_grp2160", "target": ["mon-mgr", ""]}) -- 0x7f19ac069700 con 0x7f1990078cd0 2024-09-13T08:44:33.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.721+0000 7f19a6000640 1 -- 172.21.15.31:0/4177664388 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+73 (secure 0 0 0) 0x7f19ac069700 con 0x7f1990078cd0 2024-09-13T08:44:33.723 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp2160/subvol11341/f59e2548-2b3e-42a1-aec9-c9b4fee6170e 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 msgr2=0x7f199007b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 0x7f199007b190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f19a0033000 tx=0x7f19a0004730 comp rx=0 tx=0).stop 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 msgr2=0x7f19ac1f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f19a800eaa0 tx=0x7f19a800ef60 comp rx=0 tx=0).stop 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 shutdown_connections 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1990078cd0 0x7f199007b190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19ac146f50 0x7f19ac1f8210 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f19ac0cdf90 0x7f19ac0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 --2- 172.21.15.31:0/4177664388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19ac06e420 0x7f19ac158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 >> 172.21.15.31:0/4177664388 conn(0x7f19ac0b1520 msgr2=0x7f19ac0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:33.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 shutdown_connections 2024-09-13T08:44:33.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.724+0000 7f19b3200640 1 -- 172.21.15.31:0/4177664388 wait complete. 2024-09-13T08:44:33.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol42032 2024-09-13T08:44:33.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 -- 172.21.15.31:0/3400492459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 msgr2=0x7f7ccc0ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 --2- 172.21.15.31:0/3400492459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc0ca8b0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0cfb50 tx=0x7f7cc402aa30 comp rx=0 tx=0).stop 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 -- 172.21.15.31:0/3400492459 shutdown_connections 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 --2- 172.21.15.31:0/3400492459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc0ca8b0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 --2- 172.21.15.31:0/3400492459 >> v2:172.21.15.45:3300/0 conn(0x7f7ccc0cde90 0x7f7ccc0ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.903+0000 7f7cd3800640 1 -- 172.21.15.31:0/3400492459 >> 172.21.15.31:0/3400492459 conn(0x7f7ccc0cfbb0 msgr2=0x7f7ccc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.904+0000 7f7cd3800640 1 -- 172.21.15.31:0/3400492459 shutdown_connections 2024-09-13T08:44:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.904+0000 7f7cd3800640 1 -- 172.21.15.31:0/3400492459 wait complete. 2024-09-13T08:44:33.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.905+0000 7f7cd3800640 1 Processor -- start 2024-09-13T08:44:33.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.905+0000 7f7cd3800640 1 -- start start 2024-09-13T08:44:33.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ccc0cde90 0x7f7ccc1f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 0x7f7ccc157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0cbfd0 con 0x7f7ccc0cc190 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0cbe50 con 0x7f7ccc0cde90 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd3800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc0cadf0 con 0x7f7ccc161480 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47006/0 (socket says 172.21.15.31:47006) 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.906+0000 7f7cd0e00640 1 -- 172.21.15.31:0/2924667653 learned_addr learned my addr 172.21.15.31:0/2924667653 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cc3e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ccc0cde90 0x7f7ccc1f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd1800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 0x7f7ccc157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 -- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 msgr2=0x7f7ccc157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 0x7f7ccc157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 -- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ccc0cde90 msgr2=0x7f7ccc1f57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ccc0cde90 0x7f7ccc1f57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 -- 172.21.15.31:0/2924667653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc0b0240 con 0x7f7ccc0cc190 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd1800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 0x7f7ccc157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd0e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c3a0 tx=0x7f7cbc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc017070 con 0x7f7ccc0cc190 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cbc0053c0 con 0x7f7ccc0cc190 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc003850 con 0x7f7ccc0cc190 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.907+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc15a300 con 0x7f7ccc0cc190 2024-09-13T08:44:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.908+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc15ab40 con 0x7f7ccc0cc190 2024-09-13T08:44:33.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.909+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc0cb2a0 con 0x7f7ccc0cc190 2024-09-13T08:44:33.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.910+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 101695+0+0 (secure 0 0 0) 0x7f7cbc01a070 con 0x7f7ccc0cc190 2024-09-13T08:44:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.911+0000 7f7cc9600640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 0x7f7ca807b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.911+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f7cbc09a5c0 con 0x7f7ccc0cc190 2024-09-13T08:44:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.911+0000 7f7cc3e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 0x7f7ca807b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:33.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.912+0000 7f7cc3e00640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 0x7f7ca807b1c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0aff90 tx=0x7f7cb400b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:33.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.916+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7ccc15a300 con 0x7f7ccc0cc190 2024-09-13T08:44:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:33.942+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f7cbc05e040 con 0x7f7ccc0cc190 2024-09-13T08:44:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.153+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol42032", "target": ["mon-mgr", ""]}) -- 0x7f7ccc0cb5b0 con 0x7f7ca8078d00 2024-09-13T08:44:34.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.184+0000 7f7cc9600640 1 -- 172.21.15.31:0/2924667653 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+144 (secure 0 0 0) 0x7f7ccc0cb5b0 con 0x7f7ca8078d00 2024-09-13T08:44:34.185 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:44:34.185 INFO:teuthology.orchestra.run.smithi031.stdout: "status": { 2024-09-13T08:44:34.185 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "pending", 2024-09-13T08:44:34.185 INFO:teuthology.orchestra.run.smithi031.stdout: "source": { 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout: "volume": "fake", 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout: "subvolume": "fake", 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot": "fake" 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:34.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 msgr2=0x7f7ca807b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 0x7f7ca807b1c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0aff90 tx=0x7f7cb400b390 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 msgr2=0x7f7ccc159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c3a0 tx=0x7f7cbc00c860 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 shutdown_connections 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7ca8078d00 0x7f7ca807b1c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ccc161480 0x7f7ccc157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ccc0cde90 0x7f7ccc1f57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 --2- 172.21.15.31:0/2924667653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ccc0cc190 0x7f7ccc159c70 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.186+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 >> 172.21.15.31:0/2924667653 conn(0x7f7ccc0cfbb0 msgr2=0x7f7ccc15ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.187+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 shutdown_connections 2024-09-13T08:44:34.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.187+0000 7f7cd3800640 1 -- 172.21.15.31:0/2924667653 wait complete. 2024-09-13T08:44:34.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol91320 fake 2024-09-13T08:44:34.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.369+0000 7f1f46e00640 1 -- 172.21.15.31:0/13901081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400745f0 msgr2=0x7f1f400cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.369+0000 7f1f46e00640 1 --2- 172.21.15.31:0/13901081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400745f0 0x7f1f400cdcc0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f1f400d0a70 tx=0x7f1f3802aa10 comp rx=0 tx=0).stop 2024-09-13T08:44:34.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 -- 172.21.15.31:0/13901081 shutdown_connections 2024-09-13T08:44:34.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 --2- 172.21.15.31:0/13901081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400745f0 0x7f1f400cdcc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 --2- 172.21.15.31:0/13901081 >> v2:172.21.15.45:3300/0 conn(0x7f1f400741b0 0x7f1f400cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 --2- 172.21.15.31:0/13901081 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f400cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 -- 172.21.15.31:0/13901081 >> 172.21.15.31:0/13901081 conn(0x7f1f400cfbb0 msgr2=0x7f1f400cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:34.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 -- 172.21.15.31:0/13901081 shutdown_connections 2024-09-13T08:44:34.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.370+0000 7f1f46e00640 1 -- 172.21.15.31:0/13901081 wait complete. 2024-09-13T08:44:34.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.371+0000 7f1f46e00640 1 Processor -- start 2024-09-13T08:44:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.372+0000 7f1f46e00640 1 -- start start 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.372+0000 7f1f46e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.372+0000 7f1f46e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 0x7f1f400aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f46e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f401f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f46e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f400cad70 con 0x7f1f400741b0 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f46e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1f400cabf0 con 0x7f1f400745f0 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f46e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1f400caef0 con 0x7f1f4015acb0 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47020/0 (socket says 172.21.15.31:47020) 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 -- 172.21.15.31:0/2693443406 learned_addr learned my addr 172.21.15.31:0/2693443406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f37400640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 0x7f1f400aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f44e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f401f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 -- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 msgr2=0x7f1f401f8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f401f8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 -- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 msgr2=0x7f1f400aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 0x7f1f400aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 -- 172.21.15.31:0/2693443406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f401f8ff0 con 0x7f1f400741b0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f37400640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 0x7f1f400aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f44e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f401f8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.373+0000 7f1f3fe00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000d750 tx=0x7f1f3000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.374+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30017070 con 0x7f1f400741b0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.374+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f30004310 con 0x7f1f400741b0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.374+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000de60 con 0x7f1f400741b0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.374+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f400adc30 con 0x7f1f400741b0 2024-09-13T08:44:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.374+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f400ae1f0 con 0x7f1f400741b0 2024-09-13T08:44:34.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.375+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f400ae910 con 0x7f1f400741b0 2024-09-13T08:44:34.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.376+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f1f30009c30 con 0x7f1f400741b0 2024-09-13T08:44:34.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.377+0000 7f1f3cc00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 0x7f1f1c07b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.377+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f1f3009b610 con 0x7f1f400741b0 2024-09-13T08:44:34.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.377+0000 7f1f37400640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 0x7f1f1c07b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.378+0000 7f1f37400640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 0x7f1f1c07b220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1f4015f800 tx=0x7f1f28005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:34.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.381+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1f30020020 con 0x7f1f400741b0 2024-09-13T08:44:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.612+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol91320", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7f1f400af210 con 0x7f1f1c078d60 2024-09-13T08:44:34.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.658+0000 7f1f3cc00640 1 -- 172.21.15.31:0/2693443406 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1f28010a60 con 0x7f1f1c078d60 2024-09-13T08:44:34.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 msgr2=0x7f1f1c07b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 0x7f1f1c07b220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1f4015f800 tx=0x7f1f28005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:34.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 msgr2=0x7f1f40167370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000d750 tx=0x7f1f3000dc10 comp rx=0 tx=0).stop 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 shutdown_connections 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1f1c078d60 0x7f1f1c07b220 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1f4015acb0 0x7f1f401f8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f400745f0 0x7f1f400aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 --2- 172.21.15.31:0/2693443406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f400741b0 0x7f1f40167370 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 >> 172.21.15.31:0/2693443406 conn(0x7f1f400cfbb0 msgr2=0x7f1f400cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 shutdown_connections 2024-09-13T08:44:34.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.660+0000 7f1f46e00640 1 -- 172.21.15.31:0/2693443406 wait complete. 2024-09-13T08:44:34.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol11341 fake subvol_grp2160 2024-09-13T08:44:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 -- 172.21.15.31:0/1367084740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800af450 msgr2=0x7fa4800b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 --2- 172.21.15.31:0/1367084740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800af450 0x7fa4800b4940 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa46c005fd0 tx=0x7fa46c02a4e0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 -- 172.21.15.31:0/1367084740 shutdown_connections 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 --2- 172.21.15.31:0/1367084740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800b4e80 0x7fa4800ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 --2- 172.21.15.31:0/1367084740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800af450 0x7fa4800b4940 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 -- 172.21.15.31:0/1367084740 >> 172.21.15.31:0/1367084740 conn(0x7fa4800ccd00 msgr2=0x7fa4800cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 -- 172.21.15.31:0/1367084740 shutdown_connections 2024-09-13T08:44:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.844+0000 7fa486800640 1 -- 172.21.15.31:0/1367084740 wait complete. 2024-09-13T08:44:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.846+0000 7fa486800640 1 Processor -- start 2024-09-13T08:44:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.846+0000 7fa486800640 1 -- start start 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800af450 0x7fa4801f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ce180 0x7fa48015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa480070780 con 0x7fa4800b4e80 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa480070600 con 0x7fa4800af450 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa486800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa4800ceaa0 con 0x7fa4800ce180 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa484a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa484a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46650/0 (socket says 172.21.15.31:46650) 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa484a00640 1 -- 172.21.15.31:0/3988175030 learned_addr learned my addr 172.21.15.31:0/3988175030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa485400640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800af450 0x7fa4801f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.847+0000 7fa485e00640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ce180 0x7fa48015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 -- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ce180 msgr2=0x7fa48015d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ce180 0x7fa48015d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 -- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800af450 msgr2=0x7fa4801f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800af450 0x7fa4801f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 -- 172.21.15.31:0/3988175030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48009c3a0 con 0x7fa4800b4e80 2024-09-13T08:44:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa484a00640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fa46c034040 tx=0x7fa46c003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c040070 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa46c032730 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c032b60 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa480157210 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.848+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48009c940 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.850+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4800b4940 con 0x7fa4800b4e80 2024-09-13T08:44:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.851+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7fa46c043070 con 0x7fa4800b4e80 2024-09-13T08:44:34.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.851+0000 7fa47d600640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 0x7fa46007b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:34.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.852+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7fa46c0bc350 con 0x7fa4800b4e80 2024-09-13T08:44:34.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.852+0000 7fa485400640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 0x7fa46007b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:34.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.852+0000 7fa485400640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 0x7fa46007b220 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa474006fd0 tx=0x7fa474009060 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:34.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:34.856+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa46c083a90 con 0x7fa4800b4e80 2024-09-13T08:44:35.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.086+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol11341", "snap_name": "fake", "group_name": "subvol_grp2160", "target": ["mon-mgr", ""]}) -- 0x7fa4800af9c0 con 0x7fa460078d60 2024-09-13T08:44:35.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.126+0000 7fa47d600640 1 -- 172.21.15.31:0/3988175030 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa4800af9c0 con 0x7fa460078d60 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.128+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 msgr2=0x7fa46007b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.128+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 0x7fa46007b220 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa474006fd0 tx=0x7fa474009060 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 msgr2=0x7fa48015d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fa46c034040 tx=0x7fa46c003bf0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 shutdown_connections 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa460078d60 0x7fa46007b220 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa4800ce180 0x7fa48015d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4800b4e80 0x7fa48015d430 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 --2- 172.21.15.31:0/3988175030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4800af450 0x7fa4801f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 >> 172.21.15.31:0/3988175030 conn(0x7fa4800ccd00 msgr2=0x7fa4800aea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 shutdown_connections 2024-09-13T08:44:35.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.129+0000 7fa486800640 1 -- 172.21.15.31:0/3988175030 wait complete. 2024-09-13T08:44:35.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:35.145 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol91320/.meta) 2024-09-13T08:44:35.201 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:44:35.202 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:44:35.202 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:44:35.202 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol91320/b6e24bde-0172-4ef1-9bf7-b056d8ab5136 2024-09-13T08:44:35.202 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:44:35.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:35.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:35.205 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp2160/subvol11341/.meta) 2024-09-13T08:44:35.263 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:44:35.263 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:44:35.263 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:44:35.263 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/subvol_grp2160/subvol11341/f59e2548-2b3e-42a1-aec9-c9b4fee6170e 2024-09-13T08:44:35.263 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:44:35.264 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:35.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol91320 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.429+0000 7f7e2fc00640 1 -- 172.21.15.31:0/629524545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cc190 msgr2=0x7f7e280ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.429+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/629524545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cc190 0x7f7e280ca8b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7e20005fd0 tx=0x7f7e2002a820 comp rx=0 tx=0).stop 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 -- 172.21.15.31:0/629524545 shutdown_connections 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/629524545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cc190 0x7f7e280ca8b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/629524545 >> v2:172.21.15.45:3300/0 conn(0x7f7e280cde90 0x7f7e280ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 -- 172.21.15.31:0/629524545 >> 172.21.15.31:0/629524545 conn(0x7f7e280cfbb0 msgr2=0x7f7e280cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 -- 172.21.15.31:0/629524545 shutdown_connections 2024-09-13T08:44:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.430+0000 7f7e2fc00640 1 -- 172.21.15.31:0/629524545 wait complete. 2024-09-13T08:44:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.431+0000 7f7e2fc00640 1 Processor -- start 2024-09-13T08:44:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.431+0000 7f7e2fc00640 1 -- start start 2024-09-13T08:44:35.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 0x7f7e280b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cde90 0x7f7e280b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280cbfd0 con 0x7f7e28161480 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280cbe50 con 0x7f7e280cc190 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.432+0000 7f7e2fc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7e280cadf0 con 0x7f7e280cde90 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 0x7f7e280b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cde90 0x7f7e280b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46652/0 (socket says 172.21.15.31:46652) 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 0x7f7e280b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:54892/0 (socket says 172.21.15.31:54892) 2024-09-13T08:44:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 -- 172.21.15.31:0/4029495559 learned_addr learned my addr 172.21.15.31:0/4029495559 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 -- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cde90 msgr2=0x7f7e280b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cde90 0x7f7e280b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 -- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 msgr2=0x7f7e280b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 0x7f7e280b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 -- 172.21.15.31:0/4029495559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e280b54c0 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.433+0000 7f7e2dc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009cd0 tx=0x7f7e20032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.434+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20037040 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.434+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e280b2ba0 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.434+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e20042070 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.434+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20003e60 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.434+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e280b3160 con 0x7f7e28161480 2024-09-13T08:44:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.435+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e280cb2a0 con 0x7f7e28161480 2024-09-13T08:44:35.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.436+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f7e20004280 con 0x7f7e28161480 2024-09-13T08:44:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.437+0000 7f7e25600640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 0x7f7e0407b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.437+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f7e200bbeb0 con 0x7f7e28161480 2024-09-13T08:44:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.437+0000 7f7e2d200640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 0x7f7e0407b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.438+0000 7f7e2d200640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 0x7f7e0407b220 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007fd0 tx=0x7f7e18009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:35.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.441+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7e20083570 con 0x7f7e28161480 2024-09-13T08:44:35.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.663+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol91320", "target": ["mon-mgr", ""]}) -- 0x7f7e280b0e20 con 0x7f7e04078d60 2024-09-13T08:44:35.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.700+0000 7f7e25600640 1 -- 172.21.15.31:0/4029495559 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7e18001410 con 0x7f7e04078d60 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 msgr2=0x7f7e0407b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 0x7f7e0407b220 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007fd0 tx=0x7f7e18009040 comp rx=0 tx=0).stop 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 msgr2=0x7f7e280b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009cd0 tx=0x7f7e20032c60 comp rx=0 tx=0).stop 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 shutdown_connections 2024-09-13T08:44:35.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7e04078d60 0x7f7e0407b220 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e28161480 0x7f7e280b3680 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7e280cde90 0x7f7e280b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 --2- 172.21.15.31:0/4029495559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280cc190 0x7f7e280b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 >> 172.21.15.31:0/4029495559 conn(0x7f7e280cfbb0 msgr2=0x7f7e280d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 shutdown_connections 2024-09-13T08:44:35.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.703+0000 7f7e2fc00640 1 -- 172.21.15.31:0/4029495559 wait complete. 2024-09-13T08:44:35.709 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.708+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.709 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.708+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.709 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.708+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.709 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.708+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.709 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.708+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol11341 subvol_grp2160 2024-09-13T08:44:35.730 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.725+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.730 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.725+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.730 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.725+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.730 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.726+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.731 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:35.726+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:44:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 -- 172.21.15.31:0/2657476817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3460072560 msgr2=0x7f34600cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 --2- 172.21.15.31:0/2657476817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3460072560 0x7f34600cb740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3450009830 tx=0x7f345002a4a0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 -- 172.21.15.31:0/2657476817 shutdown_connections 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 --2- 172.21.15.31:0/2657476817 >> v2:172.21.15.45:3300/0 conn(0x7f34600cbc80 0x7f34600b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 --2- 172.21.15.31:0/2657476817 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3460072560 0x7f34600cb740 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 -- 172.21.15.31:0/2657476817 >> 172.21.15.31:0/2657476817 conn(0x7f34600cfbb0 msgr2=0x7f34600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 -- 172.21.15.31:0/2657476817 shutdown_connections 2024-09-13T08:44:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.894+0000 7f3468800640 1 -- 172.21.15.31:0/2657476817 wait complete. 2024-09-13T08:44:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.896+0000 7f3468800640 1 Processor -- start 2024-09-13T08:44:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.896+0000 7f3468800640 1 -- start start 2024-09-13T08:44:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34600cbc80 0x7f346015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346015bfc0 0x7f346015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f346006e740 con 0x7f3460072560 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f346006e5c0 con 0x7f346015bfc0 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3468800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f34600b2f20 con 0x7f34600cbc80 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3465e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3465e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46680/0 (socket says 172.21.15.31:46680) 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3465e00640 1 -- 172.21.15.31:0/4201856373 learned_addr learned my addr 172.21.15.31:0/4201856373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3466800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346015bfc0 0x7f346015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.897+0000 7f3465400640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34600cbc80 0x7f346015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 -- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34600cbc80 msgr2=0x7f346015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34600cbc80 0x7f346015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 -- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346015bfc0 msgr2=0x7f346015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346015bfc0 0x7f346015e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 -- 172.21.15.31:0/4201856373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34600c87f0 con 0x7f3460072560 2024-09-13T08:44:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3465e00640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f345400c500 tx=0x7f345400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3454016020 con 0x7f3460072560 2024-09-13T08:44:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3454012920 con 0x7f3460072560 2024-09-13T08:44:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3454012d70 con 0x7f3460072560 2024-09-13T08:44:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34600c8ad0 con 0x7f3460072560 2024-09-13T08:44:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.898+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34600c9090 con 0x7f3460072560 2024-09-13T08:44:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.899+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34600b33b0 con 0x7f3460072560 2024-09-13T08:44:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.901+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f3454019020 con 0x7f3460072560 2024-09-13T08:44:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.902+0000 7f345a000640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 0x7f344407b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.902+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7f345409b440 con 0x7f3460072560 2024-09-13T08:44:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.902+0000 7f3465400640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 0x7f344407b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.903+0000 7f3465400640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 0x7f344407b220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3450034000 tx=0x7f3450004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:35.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:35.906+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3454020020 con 0x7f3460072560 2024-09-13T08:44:36.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.139+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol11341", "group_name": "subvol_grp2160", "target": ["mon-mgr", ""]}) -- 0x7f34600cb310 con 0x7f3444078d60 2024-09-13T08:44:36.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.175+0000 7f345a000640 1 -- 172.21.15.31:0/4201856373 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f34600cb310 con 0x7f3444078d60 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 msgr2=0x7f344407b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 0x7f344407b220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3450034000 tx=0x7f3450004730 comp rx=0 tx=0).stop 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 msgr2=0x7f3460153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f345400c500 tx=0x7f345400c9c0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 shutdown_connections 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3444078d60 0x7f344407b220 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346015bfc0 0x7f346015e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f34600cbc80 0x7f346015ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 --2- 172.21.15.31:0/4201856373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3460072560 0x7f3460153180 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 >> 172.21.15.31:0/4201856373 conn(0x7f34600cfbb0 msgr2=0x7f34600b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:36.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 shutdown_connections 2024-09-13T08:44:36.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.178+0000 7f3468800640 1 -- 172.21.15.31:0/4201856373 wait complete. 2024-09-13T08:44:36.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol42032 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 -- 172.21.15.31:0/692271149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 msgr2=0x7f09f40b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 --2- 172.21.15.31:0/692271149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40b29e0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f09ec005fd0 tx=0x7f09ec02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 -- 172.21.15.31:0/692271149 shutdown_connections 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 --2- 172.21.15.31:0/692271149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40b29e0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 --2- 172.21.15.31:0/692271149 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09f4072560 0x7f09f40cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.361+0000 7f09fb800640 1 -- 172.21.15.31:0/692271149 >> 172.21.15.31:0/692271149 conn(0x7f09f40cfbb0 msgr2=0x7f09f40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.362+0000 7f09fb800640 1 -- 172.21.15.31:0/692271149 shutdown_connections 2024-09-13T08:44:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.362+0000 7f09fb800640 1 -- 172.21.15.31:0/692271149 wait complete. 2024-09-13T08:44:36.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.363+0000 7f09fb800640 1 Processor -- start 2024-09-13T08:44:36.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.363+0000 7f09fb800640 1 -- start start 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09f4072560 0x7f09f40ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f415bfc0 0x7f09f415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09f406e740 con 0x7f09f40cbc80 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f09f406e5c0 con 0x7f09f415bfc0 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09fb800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f09f40b2f20 con 0x7f09f4072560 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46698/0 (socket says 172.21.15.31:46698) 2024-09-13T08:44:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.364+0000 7f09ebe00640 1 -- 172.21.15.31:0/2260954368 learned_addr learned my addr 172.21.15.31:0/2260954368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09f8e00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09f4072560 0x7f09f40ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09f9800640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f415bfc0 0x7f09f415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 -- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09f4072560 msgr2=0x7f09f40ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09f4072560 0x7f09f40ceb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 -- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f415bfc0 msgr2=0x7f09f415c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f415bfc0 0x7f09f415c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 -- 172.21.15.31:0/2260954368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f41f8620 con 0x7f09f40cbc80 2024-09-13T08:44:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09ebe00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f40cbc80 0x7f09f40cf060 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00d950 tx=0x7f09dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.365+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc017070 con 0x7f09f40cbc80 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.366+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09dc009d20 con 0x7f09f40cbc80 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.366+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc0106c0 con 0x7f09f40cbc80 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.366+0000 7f09fb800640 1 -- 172.21.15.31:0/2260954368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f41f8900 con 0x7f09f40cbc80 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.366+0000 7f09fb800640 1 -- 172.21.15.31:0/2260954368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f41f8ec0 con 0x7f09f40cbc80 2024-09-13T08:44:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.367+0000 7f09fb800640 1 -- 172.21.15.31:0/2260954368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f40c34d0 con 0x7f09f40cbc80 2024-09-13T08:44:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.368+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f09dc010860 con 0x7f09f40cbc80 2024-09-13T08:44:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.369+0000 7f09f1600640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f09d0078d60 0x7f09d007b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.369+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5797+0+0 (secure 0 0 0) 0x7f09dc09b5b0 con 0x7f09f40cbc80 2024-09-13T08:44:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.369+0000 7f09f8e00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f09d0078d60 0x7f09d007b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.370+0000 7f09f8e00640 1 --2- 172.21.15.31:0/2260954368 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f09d0078d60 0x7f09d007b220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09e4001720 tx=0x7f09e4007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.373+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f09f40c34d0 con 0x7f09f40cbc80 2024-09-13T08:44:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.595+0000 7f09fb800640 1 -- 172.21.15.31:0/2260954368 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol42032", "target": ["mon-mgr", ""]}) -- 0x7f09f41456a0 con 0x7f09d0078d60 2024-09-13T08:44:36.615 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:44:36.614+0000 7fe16b200640 -1 mgr.server reply reply (11) Resource temporarily unavailable subvolume 'subvol42032' is not ready for operation rm (use --force to override) 2024-09-13T08:44:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.614+0000 7f09f1600640 1 -- 172.21.15.31:0/2260954368 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -11 subvolume 'subvol42032' is not ready for operation rm (use --force to override)) ==== 87+0+0 (secure 0 0 0) 0x7f09f41456a0 con 0x7f09d0078d60 2024-09-13T08:44:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:Error EAGAIN: subvolume 'subvol42032' is not ready for operation rm (use --force to override) 2024-09-13T08:44:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-09-13T08:44:36.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:36.625 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol42032/.meta) 2024-09-13T08:44:36.682 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:44:36.682 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:44:36.682 INFO:teuthology.orchestra.run.smithi031.stdout:type = clone 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol42032/62e641f9-94bf-4ada-81f2-46ea1c11402d 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:state = pending 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:[source] 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:volume = fake 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:subvolume = fake 2024-09-13T08:44:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:snapshot = fake 2024-09-13T08:44:36.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol42032 --force 2024-09-13T08:44:36.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 -- 172.21.15.31:0/2499566077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af8004d40 msgr2=0x7f5af80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 --2- 172.21.15.31:0/2499566077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af8004d40 0x7f5af80051c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c560 tx=0x7f5af8005830 comp rx=0 tx=0).stop 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b16000640 1 -- 172.21.15.31:0/2499566077 reap_dead start 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 -- 172.21.15.31:0/2499566077 shutdown_connections 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 -- 172.21.15.31:0/2499566077 >> 172.21.15.31:0/2499566077 conn(0x7f5b100cfbb0 msgr2=0x7f5b100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 -- 172.21.15.31:0/2499566077 shutdown_connections 2024-09-13T08:44:36.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.854+0000 7f5b18000640 1 -- 172.21.15.31:0/2499566077 wait complete. 2024-09-13T08:44:36.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.856+0000 7f5b18000640 1 Processor -- start 2024-09-13T08:44:36.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.856+0000 7f5b18000640 1 -- start start 2024-09-13T08:44:36.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5af8004d40 0x7f5b100b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b10157450 0x7f5b1015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b100cbfd0 con 0x7f5b1014a060 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b100cbe50 con 0x7f5b10157450 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b18000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5b10072560 con 0x7f5af8004d40 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b14c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b14c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46704/0 (socket says 172.21.15.31:46704) 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.857+0000 7f5b14c00640 1 -- 172.21.15.31:0/1342779443 learned_addr learned my addr 172.21.15.31:0/1342779443 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b15600640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5af8004d40 0x7f5b100b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b16000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b10157450 0x7f5b1015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 -- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5af8004d40 msgr2=0x7f5b100b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5af8004d40 0x7f5b100b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 -- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b10157450 msgr2=0x7f5b1015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b10157450 0x7f5b1015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 -- 172.21.15.31:0/1342779443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b101f4780 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b14c00640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5afc004e80 tx=0x7f5afc001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5afc010890 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5afc016070 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b101f49f0 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5afc010da0 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.858+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1014a4e0 con 0x7f5b1014a060 2024-09-13T08:44:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.859+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b100cb740 con 0x7f5b1014a060 2024-09-13T08:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.861+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f5afc01b460 con 0x7f5b1014a060 2024-09-13T08:44:36.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.862+0000 7f5b0d600640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 0x7f5af007b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:36.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.862+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5797+0+0 (secure 0 0 0) 0x7f5afc09bcc0 con 0x7f5b1014a060 2024-09-13T08:44:36.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.862+0000 7f5b15600640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 0x7f5af007b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:36.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.863+0000 7f5b15600640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 0x7f5af007b250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b100c7df0 tx=0x7f5b04005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:36.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:36.866+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5afc063a80 con 0x7f5b1014a060 2024-09-13T08:44:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.099+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol42032", "force": true, "target": ["mon-mgr", ""]}) -- 0x7f5b100cb310 con 0x7f5af0078d90 2024-09-13T08:44:37.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:44:37.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:44:37.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.135+0000 7f5b0d600640 1 -- 172.21.15.31:0/1342779443 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5b100cb310 con 0x7f5af0078d90 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 msgr2=0x7f5af007b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 0x7f5af007b250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b100c7df0 tx=0x7f5b04005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 msgr2=0x7f5b10156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5afc004e80 tx=0x7f5afc001710 comp rx=0 tx=0).stop 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 shutdown_connections 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5af0078d90 0x7f5af007b250 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b10157450 0x7f5b1015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b1014a060 0x7f5b10156f10 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 --2- 172.21.15.31:0/1342779443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5af8004d40 0x7f5b100b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 >> 172.21.15.31:0/1342779443 conn(0x7f5b100cfbb0 msgr2=0x7f5b1015d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:37.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 shutdown_connections 2024-09-13T08:44:37.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.138+0000 7f5b18000640 1 -- 172.21.15.31:0/1342779443 wait complete. 2024-09-13T08:44:37.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-13T08:44:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 -- 172.21.15.31:0/1227572549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c380d0560 msgr2=0x7f0c380d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/1227572549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c380d0560 0x7f0c380d09a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0c380d2960 tx=0x7f0c3002aa00 comp rx=0 tx=0).stop 2024-09-13T08:44:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 -- 172.21.15.31:0/1227572549 shutdown_connections 2024-09-13T08:44:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/1227572549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c380d0560 0x7f0c380d09a0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/1227572549 >> v2:172.21.15.45:3300/0 conn(0x7f0c380cee70 0x7f0c380cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/1227572549 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c3815df10 0x7f0c380ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 -- 172.21.15.31:0/1227572549 >> 172.21.15.31:0/1227572549 conn(0x7f0c380d3e20 msgr2=0x7f0c380d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 -- 172.21.15.31:0/1227572549 shutdown_connections 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.370+0000 7f0c3dc00640 1 -- 172.21.15.31:0/1227572549 wait complete. 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.372+0000 7f0c3dc00640 1 Processor -- start 2024-09-13T08:44:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.372+0000 7f0c3dc00640 1 -- start start 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c380cee70 0x7f0c3815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c380d0560 0x7f0c3814f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0c380d1060 con 0x7f0c3815df10 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c380d0ee0 con 0x7f0c380cee70 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3dc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0c380d11e0 con 0x7f0c380d0560 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46734/0 (socket says 172.21.15.31:46734) 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 -- 172.21.15.31:0/2347768011 learned_addr learned my addr 172.21.15.31:0/2347768011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c37e00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c380cee70 0x7f0c3815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c37400640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c380d0560 0x7f0c3814f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 -- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c380d0560 msgr2=0x7f0c3814f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c380d0560 0x7f0c3814f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 -- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c380cee70 msgr2=0x7f0c3815b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c380cee70 0x7f0c3815b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.373+0000 7f0c3d200640 1 -- 172.21.15.31:0/2347768011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3814f9d0 con 0x7f0c3815df10 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c3d200640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f0c3002aff0 tx=0x7f0c30003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30046070 con 0x7f0c3815df10 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c3002f660 con 0x7f0c3815df10 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3815af00 con 0x7f0c3815df10 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30033050 con 0x7f0c3815df10 2024-09-13T08:44:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.374+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c380c89e0 con 0x7f0c3815df10 2024-09-13T08:44:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.375+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c380698c0 con 0x7f0c3815df10 2024-09-13T08:44:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.376+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 101746+0+0 (secure 0 0 0) 0x7f0c30049070 con 0x7f0c3815df10 2024-09-13T08:44:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.377+0000 7f0c2fe00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 0x7f0c1807b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.377+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5781+0+0 (secure 0 0 0) 0x7f0c300bbf40 con 0x7f0c3815df10 2024-09-13T08:44:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.377+0000 7f0c37e00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 0x7f0c1807b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.378+0000 7f0c37e00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 0x7f0c1807b250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0c380b9e70 tx=0x7f0c28005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.381+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0c380698c0 con 0x7f0c3815df10 2024-09-13T08:44:37.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.601+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f0c380c8d30 con 0x7f0c18078d90 2024-09-13T08:44:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.605+0000 7f0c2fe00640 1 -- 172.21.15.31:0/2347768011 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f0c380c8d30 con 0x7f0c18078d90 2024-09-13T08:44:37.606 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-09-13T08:44:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 msgr2=0x7f0c1807b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 0x7f0c1807b250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0c380b9e70 tx=0x7f0c28005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 msgr2=0x7f0c380cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f0c3002aff0 tx=0x7f0c30003ea0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 shutdown_connections 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0c18078d90 0x7f0c1807b250 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c3815df10 0x7f0c380cd0c0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0c380d0560 0x7f0c3814f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 --2- 172.21.15.31:0/2347768011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c380cee70 0x7f0c3815b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 >> 172.21.15.31:0/2347768011 conn(0x7f0c380d3e20 msgr2=0x7f0c380b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 shutdown_connections 2024-09-13T08:44:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37.607+0000 7f0c3dc00640 1 -- 172.21.15.31:0/2347768011 wait complete. 2024-09-13T08:44:37.624 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:44:37.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:44:37.711 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:44:37.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:44:37.720 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:44:37.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:37.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:37.901 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:44:37.902 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:37.902 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:37.902 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:37.938 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:37.938 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:38.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:44:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 -- 172.21.15.31:0/1799765108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 msgr2=0x7fcb900ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 --2- 172.21.15.31:0/1799765108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900ce290 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcb900b5020 tx=0x7fcb7c02a540 comp rx=0 tx=0).stop 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 -- 172.21.15.31:0/1799765108 shutdown_connections 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 --2- 172.21.15.31:0/1799765108 >> v2:172.21.15.45:3300/0 conn(0x7fcb90073860 0x7fcb90073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 --2- 172.21.15.31:0/1799765108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900ce290 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 -- 172.21.15.31:0/1799765108 >> 172.21.15.31:0/1799765108 conn(0x7fcb900b18f0 msgr2=0x7fcb900b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 -- 172.21.15.31:0/1799765108 shutdown_connections 2024-09-13T08:44:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.214+0000 7fcb97800640 1 -- 172.21.15.31:0/1799765108 wait complete. 2024-09-13T08:44:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.216+0000 7fcb97800640 1 Processor -- start 2024-09-13T08:44:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.216+0000 7fcb97800640 1 -- start start 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 0x7fcb9015f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9015be30 0x7fcb901f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900733e0 con 0x7fcb900ced20 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900ce9e0 con 0x7fcb9015be30 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.217+0000 7fcb97800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcb900741f0 con 0x7fcb90073860 2024-09-13T08:44:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb94e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 0x7fcb9015f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb95800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9015be30 0x7fcb901f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb94e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 0x7fcb9015f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:53046/0 (socket says 172.21.15.31:53046) 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46756/0 (socket says 172.21.15.31:46756) 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb94e00640 1 -- 172.21.15.31:0/4282211740 learned_addr learned my addr 172.21.15.31:0/4282211740 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 -- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 msgr2=0x7fcb9015f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 0x7fcb9015f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 -- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9015be30 msgr2=0x7fcb901f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9015be30 0x7fcb901f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 -- 172.21.15.31:0/4282211740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb900acd40 con 0x7fcb900ced20 2024-09-13T08:44:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.218+0000 7fcb8fe00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c031000 tx=0x7fcb7c032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.219+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c040070 con 0x7fcb900ced20 2024-09-13T08:44:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.219+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb7c005330 con 0x7fcb900ced20 2024-09-13T08:44:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.219+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb901f5480 con 0x7fcb900ced20 2024-09-13T08:44:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.219+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c003800 con 0x7fcb900ced20 2024-09-13T08:44:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.219+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb901f5a40 con 0x7fcb900ced20 2024-09-13T08:44:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.220+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb90073cb0 con 0x7fcb900ced20 2024-09-13T08:44:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.221+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fcb7c039040 con 0x7fcb900ced20 2024-09-13T08:44:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.222+0000 7fcb8cc00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 0x7fcb7007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.222+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5781+0+0 (secure 0 0 0) 0x7fcb7c0bc580 con 0x7fcb900ced20 2024-09-13T08:44:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.223+0000 7fcb94e00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 0x7fcb7007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.223+0000 7fcb94e00640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 0x7fcb7007b340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb900c95b0 tx=0x7fcb80007f50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:38.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.227+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcb7c083c40 con 0x7fcb900ced20 2024-09-13T08:44:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.435+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcb9009de70 con 0x7fcb900ced20 2024-09-13T08:44:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.436+0000 7fcb8cc00640 1 -- 172.21.15.31:0/4282211740 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v126) ==== 80+0+0 (secure 0 0 0) 0x7fcb7c088670 con 0x7fcb900ced20 2024-09-13T08:44:38.437 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:44:38.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 msgr2=0x7fcb7007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 0x7fcb7007b340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb900c95b0 tx=0x7fcb80007f50 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 msgr2=0x7fcb900cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c031000 tx=0x7fcb7c032d50 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 shutdown_connections 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb70078e80 0x7fcb7007b340 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9015be30 0x7fcb901f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb900ced20 0x7fcb900cadd0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 --2- 172.21.15.31:0/4282211740 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb90073860 0x7fcb9015f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 >> 172.21.15.31:0/4282211740 conn(0x7fcb900b18f0 msgr2=0x7fcb900cbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 shutdown_connections 2024-09-13T08:44:38.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.438+0000 7fcb97800640 1 -- 172.21.15.31:0/4282211740 wait complete. 2024-09-13T08:44:38.455 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:44:38.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:38.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:44:38.554 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:44:38.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:38.555 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:44:38.597 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:38.598 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:38.598 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:38.636 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:38.636 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:38.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:44:38.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 -- 172.21.15.31:0/625045859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71415b540 msgr2=0x7ff71406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 --2- 172.21.15.31:0/625045859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71415b540 0x7ff71406fd60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff70c005fd0 tx=0x7ff70c02a840 comp rx=0 tx=0).stop 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 -- 172.21.15.31:0/625045859 shutdown_connections 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 --2- 172.21.15.31:0/625045859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71415b540 0x7ff71406fd60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 --2- 172.21.15.31:0/625045859 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b0f0 0x7ff71406f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 -- 172.21.15.31:0/625045859 >> 172.21.15.31:0/625045859 conn(0x7ff7140cc7a0 msgr2=0x7ff7140ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 -- 172.21.15.31:0/625045859 shutdown_connections 2024-09-13T08:44:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.879+0000 7ff719600640 1 -- 172.21.15.31:0/625045859 wait complete. 2024-09-13T08:44:38.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.881+0000 7ff719600640 1 Processor -- start 2024-09-13T08:44:38.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.881+0000 7ff719600640 1 -- start start 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7140b17a0 0x7ff7141f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b540 0x7ff71414ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff71415efc0 con 0x7ff71415b0f0 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff71415ee40 con 0x7ff7140b17a0 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff719600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff71415f140 con 0x7ff71415b540 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff713400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff713400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46760/0 (socket says 172.21.15.31:46760) 2024-09-13T08:44:38.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.882+0000 7ff713400640 1 -- 172.21.15.31:0/3980354764 learned_addr learned my addr 172.21.15.31:0/3980354764 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713e00640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7140b17a0 0x7ff7141f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff718c00640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b540 0x7ff71414ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 -- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b540 msgr2=0x7ff71414ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b540 0x7ff71414ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 -- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7140b17a0 msgr2=0x7ff7141f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7140b17a0 0x7ff7141f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 -- 172.21.15.31:0/3980354764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7141f34f0 con 0x7ff71415b0f0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff713400640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff70000ea60 tx=0x7ff70000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70000cde0 con 0x7ff71415b0f0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff700004020 con 0x7ff71415b0f0 2024-09-13T08:44:38.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7140c4f80 con 0x7ff71415b0f0 2024-09-13T08:44:38.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.883+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700004450 con 0x7ff71415b0f0 2024-09-13T08:44:38.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.884+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7141503c0 con 0x7ff71415b0f0 2024-09-13T08:44:38.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.885+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7140b1b80 con 0x7ff71415b0f0 2024-09-13T08:44:38.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.886+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff7000035f0 con 0x7ff71415b0f0 2024-09-13T08:44:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.887+0000 7ff70be00640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 0x7ff6f407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.887+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7ff70009c090 con 0x7ff71415b0f0 2024-09-13T08:44:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.887+0000 7ff713e00640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 0x7ff6f407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.888+0000 7ff713e00640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 0x7ff6f407b340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff704000fb0 tx=0x7ff704009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:38.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:38.892+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff7141f34f0 con 0x7ff71415b0f0 2024-09-13T08:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.100+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff7140b5ca0 con 0x7ff71415b0f0 2024-09-13T08:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.100+0000 7ff70be00640 1 -- 172.21.15.31:0/3980354764 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v127) ==== 80+0+0 (secure 0 0 0) 0x7ff700063c10 con 0x7ff71415b0f0 2024-09-13T08:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 msgr2=0x7ff6f407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 0x7ff6f407b340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff704000fb0 tx=0x7ff704009040 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 msgr2=0x7ff7141543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff70000ea60 tx=0x7ff70000ef20 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 shutdown_connections 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6f4078e80 0x7ff6f407b340 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff71415b540 0x7ff71414ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff71415b0f0 0x7ff7141543f0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 --2- 172.21.15.31:0/3980354764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7140b17a0 0x7ff7141f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 >> 172.21.15.31:0/3980354764 conn(0x7ff7140cc7a0 msgr2=0x7ff7140cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:39.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 shutdown_connections 2024-09-13T08:44:39.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.103+0000 7ff719600640 1 -- 172.21.15.31:0/3980354764 wait complete. 2024-09-13T08:44:39.119 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:44:39.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:39.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:44:39.185 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:44:39.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:44:39.186 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:44:39.223 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:39.223 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:39.223 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:39.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:39.259 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:44:39.259 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:39.259 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:39.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:39.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 -- 172.21.15.31:0/3676701806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1400160620 msgr2=0x7f14000c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3676701806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1400160620 0x7f14000c7a80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f13f8005fd0 tx=0x7f13f802a820 comp rx=0 tx=0).stop 2024-09-13T08:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 -- 172.21.15.31:0/3676701806 shutdown_connections 2024-09-13T08:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3676701806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1400160620 0x7f14000c7a80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3676701806 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140015bdf0 0x7f140015f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 -- 172.21.15.31:0/3676701806 >> 172.21.15.31:0/3676701806 conn(0x7f14000d0360 msgr2=0x7f14000d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 -- 172.21.15.31:0/3676701806 shutdown_connections 2024-09-13T08:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.461+0000 7f13ffe00640 1 -- 172.21.15.31:0/3676701806 wait complete. 2024-09-13T08:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.463+0000 7f13ffe00640 1 Processor -- start 2024-09-13T08:44:39.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.463+0000 7f13ffe00640 1 -- start start 2024-09-13T08:44:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14000cd6f0 0x7f140015f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140009a290 0x7f140009a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14000c8350 con 0x7f140015bdf0 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14000c81d0 con 0x7f14000cd6f0 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ffe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f14000c84d0 con 0x7f140009a290 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46762/0 (socket says 172.21.15.31:46762) 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 -- 172.21.15.31:0/3388184795 learned_addr learned my addr 172.21.15.31:0/3388184795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13fea00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14000cd6f0 0x7f140015f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13ff400640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140009a290 0x7f140009a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 -- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140009a290 msgr2=0x7f140009a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140009a290 0x7f140009a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 -- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14000cd6f0 msgr2=0x7f140015f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.464+0000 7f13f7e00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14000cd6f0 0x7f140015f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13f7e00640 1 -- 172.21.15.31:0/3388184795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140015d9d0 con 0x7f140015bdf0 2024-09-13T08:44:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13f7e00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c490 tx=0x7f13e800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8016020 con 0x7f140015bdf0 2024-09-13T08:44:39.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13e8012920 con 0x7f140015bdf0 2024-09-13T08:44:39.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8012d50 con 0x7f140015bdf0 2024-09-13T08:44:39.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140015d410 con 0x7f140015bdf0 2024-09-13T08:44:39.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.465+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14001f7a90 con 0x7f140015bdf0 2024-09-13T08:44:39.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.466+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14000c7a80 con 0x7f140015bdf0 2024-09-13T08:44:39.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.468+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f13e8019020 con 0x7f140015bdf0 2024-09-13T08:44:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.468+0000 7f13f6a00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 0x7f13dc07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.469+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7f13e809b500 con 0x7f140015bdf0 2024-09-13T08:44:39.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.469+0000 7f13fea00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 0x7f13dc07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.469+0000 7f13fea00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 0x7f13dc07b340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14000998b0 tx=0x7f13f0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:39.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.473+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f13e8020020 con 0x7f140015bdf0 2024-09-13T08:44:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.708+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14000c97d0 con 0x7f140015bdf0 2024-09-13T08:44:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.708+0000 7f13f6a00640 1 -- 172.21.15.31:0/3388184795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5225 (secure 0 0 0) 0x7f140015d9d0 con 0x7f140015bdf0 2024-09-13T08:44:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:39.710 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-13T08:44:30:915768+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:39.710 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-13T08:44:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 msgr2=0x7f13dc07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 0x7f13dc07b340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14000998b0 tx=0x7f13f0005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 msgr2=0x7f14001f8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c490 tx=0x7f13e800c950 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 shutdown_connections 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f13dc078e80 0x7f13dc07b340 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f140009a290 0x7f140009a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140015bdf0 0x7f14001f8290 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 --2- 172.21.15.31:0/3388184795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14000cd6f0 0x7f140015f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 >> 172.21.15.31:0/3388184795 conn(0x7f14000d0360 msgr2=0x7f14000c90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 shutdown_connections 2024-09-13T08:44:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.711+0000 7f13ffe00640 1 -- 172.21.15.31:0/3388184795 wait complete. 2024-09-13T08:44:39.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.890+0000 7f490aa00640 1 -- 172.21.15.31:0/3817061055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 msgr2=0x7f48e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 --2- 172.21.15.31:0/3817061055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f48e800b9c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00e3d0 tx=0x7f48e8005830 comp rx=0 tx=0).stop 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 -- 172.21.15.31:0/3817061055 shutdown_connections 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 --2- 172.21.15.31:0/3817061055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f48e800b9c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 -- 172.21.15.31:0/3817061055 >> 172.21.15.31:0/3817061055 conn(0x7f49040cc910 msgr2=0x7f49040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 -- 172.21.15.31:0/3817061055 shutdown_connections 2024-09-13T08:44:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.891+0000 7f490aa00640 1 -- 172.21.15.31:0/3817061055 wait complete. 2024-09-13T08:44:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.892+0000 7f490aa00640 1 Processor -- start 2024-09-13T08:44:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.893+0000 7f490aa00640 1 -- start start 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49040ae620 0x7f49040aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 0x7f49041f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4904074140 con 0x7f48e80051c0 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4904073fc0 con 0x7f49040ae620 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f490aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f49040742c0 con 0x7f49041f4ba0 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46784/0 (socket says 172.21.15.31:46784) 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903e00640 1 -- 172.21.15.31:0/599077422 learned_addr learned my addr 172.21.15.31:0/599077422 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903400640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49040ae620 0x7f49040aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4908a00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 0x7f49041f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903e00640 1 -- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 msgr2=0x7f49041f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.894+0000 7f4903e00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 0x7f49041f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f4903e00640 1 -- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49040ae620 msgr2=0x7f49040aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f4903e00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49040ae620 0x7f49040aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f4903e00640 1 -- 172.21.15.31:0/599077422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49040acf50 con 0x7f48e80051c0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f4908a00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 0x7f49041f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f4903e00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006680 tx=0x7f48f40066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4015070 con 0x7f48e80051c0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48f4011920 con 0x7f48e80051c0 2024-09-13T08:44:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4011d70 con 0x7f48e80051c0 2024-09-13T08:44:39.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49040ad230 con 0x7f48e80051c0 2024-09-13T08:44:39.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.895+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49040ad7f0 con 0x7f48e80051c0 2024-09-13T08:44:39.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.896+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49040cc0d0 con 0x7f48e80051c0 2024-09-13T08:44:39.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.897+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f48f4018070 con 0x7f48e80051c0 2024-09-13T08:44:39.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.898+0000 7f48fbe00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 0x7f48e407b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:39.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.899+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7f48f409c2f0 con 0x7f48e80051c0 2024-09-13T08:44:39.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.899+0000 7f4903400640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 0x7f48e407b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:39.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.899+0000 7f4903400640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 0x7f48e407b370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4904156710 tx=0x7f48f0006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:39.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:39.903+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f48f4063820 con 0x7f48e80051c0 2024-09-13T08:44:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.136+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49040b2090 con 0x7f48e80051c0 2024-09-13T08:44:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.137+0000 7f48fbe00640 1 -- 172.21.15.31:0/599077422 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5225 (secure 0 0 0) 0x7f48f4068250 con 0x7f48e80051c0 2024-09-13T08:44:40.138 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:40.138 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-13T08:44:30:915768+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:40.138 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-13T08:44:40.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 msgr2=0x7f48e407b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 0x7f48e407b370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4904156710 tx=0x7f48f0006b10 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 msgr2=0x7f490415ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006680 tx=0x7f48f40066b0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 shutdown_connections 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e4078eb0 0x7f48e407b370 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f49041f4ba0 0x7f49041f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49040ae620 0x7f49040aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 --2- 172.21.15.31:0/599077422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48e80051c0 0x7f490415ece0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.139+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 >> 172.21.15.31:0/599077422 conn(0x7f49040cc910 msgr2=0x7f490406e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.140+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 shutdown_connections 2024-09-13T08:44:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.140+0000 7f490aa00640 1 -- 172.21.15.31:0/599077422 wait complete. 2024-09-13T08:44:40.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 -- 172.21.15.31:0/1881488537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040cc720 msgr2=0x7fd10406f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 --2- 172.21.15.31:0/1881488537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040cc720 0x7fd10406f9b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fd1040cec50 tx=0x7fd0f402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 -- 172.21.15.31:0/1881488537 shutdown_connections 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 --2- 172.21.15.31:0/1881488537 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040701a0 0x7fd1040b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 --2- 172.21.15.31:0/1881488537 >> v2:172.21.15.45:3300/0 conn(0x7fd1040ccb00 0x7fd1040b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 --2- 172.21.15.31:0/1881488537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040cc720 0x7fd10406f9b0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.315+0000 7fd108a00640 1 -- 172.21.15.31:0/1881488537 >> 172.21.15.31:0/1881488537 conn(0x7fd10406cb00 msgr2=0x7fd10406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.316+0000 7fd108a00640 1 -- 172.21.15.31:0/1881488537 shutdown_connections 2024-09-13T08:44:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.316+0000 7fd108a00640 1 -- 172.21.15.31:0/1881488537 wait complete. 2024-09-13T08:44:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.317+0000 7fd108a00640 1 Processor -- start 2024-09-13T08:44:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.317+0000 7fd108a00640 1 -- start start 2024-09-13T08:44:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040cc720 0x7fd104155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1040ccb00 0x7fd1040cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd104152800 con 0x7fd1040701a0 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd104152680 con 0x7fd1040ccb00 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd108a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd104152980 con 0x7fd1040cc720 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd103400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd103400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46800/0 (socket says 172.21.15.31:46800) 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.318+0000 7fd103400640 1 -- 172.21.15.31:0/1034769766 learned_addr learned my addr 172.21.15.31:0/1034769766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd0faa00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040cc720 0x7fd104155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103e00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1040ccb00 0x7fd1040cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 -- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040cc720 msgr2=0x7fd104155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040cc720 0x7fd104155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 -- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1040ccb00 msgr2=0x7fd1040cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1040ccb00 0x7fd1040cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 -- 172.21.15.31:0/1034769766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1041f5560 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.319+0000 7fd103400640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.320+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f403f070 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.320+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f40047a0 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.320+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4035030 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.320+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1040ca630 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.320+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1041f4f10 con 0x7fd1040701a0 2024-09-13T08:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.321+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10406f9b0 con 0x7fd1040701a0 2024-09-13T08:44:40.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.322+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fd0f4042070 con 0x7fd1040701a0 2024-09-13T08:44:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.323+0000 7fd0fbe00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 0x7fd0e007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.323+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7fd0f40bc2e0 con 0x7fd1040701a0 2024-09-13T08:44:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.323+0000 7fd0faa00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 0x7fd0e007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.324+0000 7fd0faa00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 0x7fd0e007b340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd104151fc0 tx=0x7fd0ec00b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:40.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.327+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd0f4083980 con 0x7fd1040701a0 2024-09-13T08:44:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.523+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd10415f670 con 0x7fd1040701a0 2024-09-13T08:44:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.524+0000 7fd0fbe00640 1 -- 172.21.15.31:0/1034769766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16796 (secure 0 0 0) 0x7fd0f40883b0 con 0x7fd1040701a0 2024-09-13T08:44:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:40.525 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:38.789026+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:18.106733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:19.279918+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 msgr2=0x7fd0e007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 0x7fd0e007b340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd104151fc0 tx=0x7fd0ec00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 msgr2=0x7fd1040ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f4003910 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 shutdown_connections 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd0e0078e80 0x7fd0e007b340 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1040ccb00 0x7fd1040cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1040cc720 0x7fd104155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 --2- 172.21.15.31:0/1034769766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1040701a0 0x7fd1040ca0f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.526+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 >> 172.21.15.31:0/1034769766 conn(0x7fd10406cb00 msgr2=0x7fd10406e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.527+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 shutdown_connections 2024-09-13T08:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.527+0000 7fd108a00640 1 -- 172.21.15.31:0/1034769766 wait complete. 2024-09-13T08:44:40.543 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:44:40.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1297137694 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 msgr2=0x7fb1f415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1297137694 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415b1d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009860 tx=0x7fb1e402a530 comp rx=0 tx=0).stop 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1297137694 shutdown_connections 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1297137694 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415b1d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1297137694 >> 172.21.15.31:0/1297137694 conn(0x7fb1f40b1c00 msgr2=0x7fb1f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1297137694 shutdown_connections 2024-09-13T08:44:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.707+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1297137694 wait complete. 2024-09-13T08:44:40.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.709+0000 7fb1f8a00640 1 Processor -- start 2024-09-13T08:44:40.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.709+0000 7fb1f8a00640 1 -- start start 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f40c3780 0x7fb1f4151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f40732c0 con 0x7fb1f40ca740 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f4152140 con 0x7fb1f40c3780 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1f8a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f40cb300 con 0x7fb1f40ccb00 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46814/0 (socket says 172.21.15.31:46814) 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.710+0000 7fb1ebe00640 1 -- 172.21.15.31:0/1243324932 learned_addr learned my addr 172.21.15.31:0/1243324932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1f2800640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1f1e00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f40c3780 0x7fb1f4151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 -- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 msgr2=0x7fb1f415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 -- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f40c3780 msgr2=0x7fb1f4151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f40c3780 0x7fb1f4151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 -- 172.21.15.31:0/1243324932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f40b2fc0 con 0x7fb1f40ca740 2024-09-13T08:44:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ebe00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c480 tx=0x7fb1dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0052d0 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f40b32a0 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1dc005490 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.711+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc012890 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.712+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f40b3860 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.713+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f40cd9b0 con 0x7fb1f40ca740 2024-09-13T08:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.714+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fb1dc012a30 con 0x7fb1f40ca740 2024-09-13T08:44:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.715+0000 7fb1ea000640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 0x7fb1d007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.715+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7fb1dc09b110 con 0x7fb1f40ca740 2024-09-13T08:44:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.715+0000 7fb1f1e00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 0x7fb1d007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.716+0000 7fb1f1e00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 0x7fb1d007b340 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4034000 tx=0x7fb1e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.719+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb1dc062650 con 0x7fb1f40ca740 2024-09-13T08:44:40.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.953+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1f40ce790 con 0x7fb1f40ca740 2024-09-13T08:44:40.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.953+0000 7fb1ea000640 1 -- 172.21.15.31:0/1243324932 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5225 (secure 0 0 0) 0x7fb1dc067080 con 0x7fb1f40ca740 2024-09-13T08:44:40.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:40.954 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-13T08:44:30:915768+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 msgr2=0x7fb1d007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 0x7fb1d007b340 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4034000 tx=0x7fb1e4004730 comp rx=0 tx=0).stop 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 msgr2=0x7fb1f415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c480 tx=0x7fb1dc00c940 comp rx=0 tx=0).stop 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 shutdown_connections 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1d0078e80 0x7fb1d007b340 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb1f40ccb00 0x7fb1f415a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1f40ca740 0x7fb1f415a310 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 --2- 172.21.15.31:0/1243324932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f40c3780 0x7fb1f4151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 >> 172.21.15.31:0/1243324932 conn(0x7fb1f40b1c00 msgr2=0x7fb1f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 shutdown_connections 2024-09-13T08:44:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:40.956+0000 7fb1f8a00640 1 -- 172.21.15.31:0/1243324932 wait complete. 2024-09-13T08:44:40.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 -- 172.21.15.31:0/2646347236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe820005230 msgr2=0x7fe82000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 --2- 172.21.15.31:0/2646347236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe820005230 0x7fe82000b9c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fe83400c6e0 tx=0x7fe820005830 comp rx=0 tx=0).stop 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 -- 172.21.15.31:0/2646347236 shutdown_connections 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 --2- 172.21.15.31:0/2646347236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe820005230 0x7fe82000b9c0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 -- 172.21.15.31:0/2646347236 >> 172.21.15.31:0/2646347236 conn(0x7fe83c0cc910 msgr2=0x7fe83c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 -- 172.21.15.31:0/2646347236 shutdown_connections 2024-09-13T08:44:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.132+0000 7fe841200640 1 -- 172.21.15.31:0/2646347236 wait complete. 2024-09-13T08:44:41.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.134+0000 7fe841200640 1 Processor -- start 2024-09-13T08:44:41.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.134+0000 7fe841200640 1 -- start start 2024-09-13T08:44:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe820005230 0x7fe83c09c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 0x7fe83c1f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0b1d60 con 0x7fe83c14b140 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0cc620 con 0x7fe83c1f86b0 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe841200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c0b1ee0 con 0x7fe820005230 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe831400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.135+0000 7fe831400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46830/0 (socket says 172.21.15.31:46830) 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 -- 172.21.15.31:0/1258548019 learned_addr learned my addr 172.21.15.31:0/1258548019 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe839e00640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe820005230 0x7fe83c09c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe83a800640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 0x7fe83c1f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 -- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe820005230 msgr2=0x7fe83c09c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe820005230 0x7fe83c09c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 -- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 msgr2=0x7fe83c1f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 0x7fe83c1f8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 -- 172.21.15.31:0/1258548019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c0b2e60 con 0x7fe83c14b140 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe83a800640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 0x7fe83c1f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe831400640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fe824011850 tx=0x7fe824011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82400fd50 con 0x7fe83c14b140 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe824016070 con 0x7fe83c14b140 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.136+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c0b3140 con 0x7fe83c14b140 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.137+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8240126b0 con 0x7fe83c14b140 2024-09-13T08:44:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.137+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c14b5a0 con 0x7fe83c14b140 2024-09-13T08:44:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.138+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c0ce600 con 0x7fe83c14b140 2024-09-13T08:44:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.139+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fe824018030 con 0x7fe83c14b140 2024-09-13T08:44:41.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.140+0000 7fe832a00640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 0x7fe81807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.140+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7fe82409bcd0 con 0x7fe83c14b140 2024-09-13T08:44:41.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.140+0000 7fe839e00640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 0x7fe81807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.141+0000 7fe839e00640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 0x7fe81807b370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe82c006650 tx=0x7fe82c006250 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.144+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe824063240 con 0x7fe83c14b140 2024-09-13T08:44:41.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.376+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe83c0af5d0 con 0x7fe83c14b140 2024-09-13T08:44:41.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.377+0000 7fe832a00640 1 -- 172.21.15.31:0/1258548019 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5225 (secure 0 0 0) 0x7fe824067c70 con 0x7fe83c14b140 2024-09-13T08:44:41.378 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:41.378 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-13T08:44:30:915768+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:20.310429+0000","modified":"2024-09-13T08:44:23.328440+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7614":{"gid":7614,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/1889278552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1889278552},{"type":"v1","addr":"172.21.15.31:6835","nonce":1889278552}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7614,"qdb_cluster":[7614]},"id":9}]} 2024-09-13T08:44:41.378 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-13T08:44:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 msgr2=0x7fe81807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 0x7fe81807b370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe82c006650 tx=0x7fe82c006250 comp rx=0 tx=0).stop 2024-09-13T08:44:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 msgr2=0x7fe83c1f8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fe824011850 tx=0x7fe824011d10 comp rx=0 tx=0).stop 2024-09-13T08:44:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 shutdown_connections 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe818078eb0 0x7fe81807b370 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe83c1f86b0 0x7fe83c1f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c14b140 0x7fe83c1f8170 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 --2- 172.21.15.31:0/1258548019 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe820005230 0x7fe83c09c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 >> 172.21.15.31:0/1258548019 conn(0x7fe83c0cc910 msgr2=0x7fe83c15b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 shutdown_connections 2024-09-13T08:44:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.379+0000 7fe841200640 1 -- 172.21.15.31:0/1258548019 wait complete. 2024-09-13T08:44:41.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.554+0000 7f9a17200640 1 -- 172.21.15.31:0/3666707251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180cf2f0 msgr2=0x7f9a1815fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.554+0000 7f9a17200640 1 --2- 172.21.15.31:0/3666707251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180cf2f0 0x7f9a1815fa90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9a180b2640 tx=0x7f9a0402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 -- 172.21.15.31:0/3666707251 shutdown_connections 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 --2- 172.21.15.31:0/3666707251 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 --2- 172.21.15.31:0/3666707251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180cf2f0 0x7f9a1815fa90 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 --2- 172.21.15.31:0/3666707251 >> v2:172.21.15.45:3300/0 conn(0x7f9a180caa00 0x7f9a18161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 -- 172.21.15.31:0/3666707251 >> 172.21.15.31:0/3666707251 conn(0x7f9a1806cb00 msgr2=0x7f9a1806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 -- 172.21.15.31:0/3666707251 shutdown_connections 2024-09-13T08:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.555+0000 7f9a17200640 1 -- 172.21.15.31:0/3666707251 wait complete. 2024-09-13T08:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.556+0000 7f9a17200640 1 Processor -- start 2024-09-13T08:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.556+0000 7f9a17200640 1 -- start start 2024-09-13T08:44:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a180cf2f0 0x7f9a180c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a180ce3c0 con 0x7f9a180caa00 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a180ce240 con 0x7f9a180cf2f0 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.557+0000 7f9a17200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9a180ce540 con 0x7f9a180cc3e0 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46858/0 (socket says 172.21.15.31:46858) 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 -- 172.21.15.31:0/119610625 learned_addr learned my addr 172.21.15.31:0/119610625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a16800640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a180cf2f0 0x7f9a180c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15400640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 -- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 msgr2=0x7f9a180b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 -- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a180cf2f0 msgr2=0x7f9a180c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a180cf2f0 0x7f9a180c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 -- 172.21.15.31:0/119610625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a181f5430 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15400640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a15e00640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800d7a0 tx=0x7f9a0800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08017070 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a08009cc0 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.558+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080106c0 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.559+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a181f5710 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.559+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a180c5850 con 0x7f9a180caa00 2024-09-13T08:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.560+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1815f0d0 con 0x7f9a180caa00 2024-09-13T08:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.561+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9a0801a070 con 0x7f9a180caa00 2024-09-13T08:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.562+0000 7f9a0e000640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 0x7f99f807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.562+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7f9a0809b6d0 con 0x7f9a180caa00 2024-09-13T08:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.562+0000 7f9a15400640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 0x7f99f807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.563+0000 7f9a15400640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 0x7f99f807b340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a04033000 tx=0x7f9a04004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.566+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9a1815f0d0 con 0x7f9a180caa00 2024-09-13T08:44:41.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.762+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a180ce9f0 con 0x7f9a180caa00 2024-09-13T08:44:41.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.763+0000 7f9a0e000640 1 -- 172.21.15.31:0/119610625 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16796 (secure 0 0 0) 0x7f9a080632e0 con 0x7f9a180caa00 2024-09-13T08:44:41.764 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:41.765 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:38.789026+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:18.106733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:19.279918+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 msgr2=0x7f99f807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 0x7f99f807b340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a04033000 tx=0x7f9a04004730 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 msgr2=0x7f9a181479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800d7a0 tx=0x7f9a0800dc60 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 shutdown_connections 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f99f8078e80 0x7f99f807b340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a180cf2f0 0x7f9a180c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a180cc3e0 0x7f9a180b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 --2- 172.21.15.31:0/119610625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a180caa00 0x7f9a181479e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 >> 172.21.15.31:0/119610625 conn(0x7f9a1806cb00 msgr2=0x7f9a180adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 shutdown_connections 2024-09-13T08:44:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.766+0000 7f9a17200640 1 -- 172.21.15.31:0/119610625 wait complete. 2024-09-13T08:44:41.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:44:41.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 -- 172.21.15.31:0/768003424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780c9350 msgr2=0x7f86780d0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 --2- 172.21.15.31:0/768003424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780c9350 0x7f86780d0ea0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f8660005fd0 tx=0x7f866002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 -- 172.21.15.31:0/768003424 shutdown_connections 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 --2- 172.21.15.31:0/768003424 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780cea60 0x7f86780ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 --2- 172.21.15.31:0/768003424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780c9350 0x7f86780d0ea0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 -- 172.21.15.31:0/768003424 >> 172.21.15.31:0/768003424 conn(0x7f86780d3f50 msgr2=0x7f86780d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 -- 172.21.15.31:0/768003424 shutdown_connections 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.944+0000 7f8677200640 1 -- 172.21.15.31:0/768003424 wait complete. 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.946+0000 7f8677200640 1 Processor -- start 2024-09-13T08:44:41.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.946+0000 7f8677200640 1 -- start start 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780c9350 0x7f8678157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 0x7f86780cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86780cf430 con 0x7f86780cea60 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f86780d15f0 con 0x7f867815e160 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8677200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f86780cf5b0 con 0x7f86780c9350 2024-09-13T08:44:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46878/0 (socket says 172.21.15.31:46878) 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 -- 172.21.15.31:0/2156457880 learned_addr learned my addr 172.21.15.31:0/2156457880 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8675e00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780c9350 0x7f8678157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f8676800640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 0x7f86780cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 -- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780c9350 msgr2=0x7f8678157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780c9350 0x7f8678157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 -- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 msgr2=0x7f86780cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.947+0000 7f866fe00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 0x7f86780cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f866fe00640 1 -- 172.21.15.31:0/2156457880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86780b7f50 con 0x7f86780cea60 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f8676800640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 0x7f86780cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f866fe00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f86600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866003f070 con 0x7f86780cea60 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8660005600 con 0x7f86780cea60 2024-09-13T08:44:41.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660035030 con 0x7f86780cea60 2024-09-13T08:44:41.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86780b8230 con 0x7f86780cea60 2024-09-13T08:44:41.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.948+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86780b87f0 con 0x7f86780cea60 2024-09-13T08:44:41.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.949+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86780ceef0 con 0x7f86780cea60 2024-09-13T08:44:41.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.950+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f8660042070 con 0x7f86780cea60 2024-09-13T08:44:41.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.951+0000 7f866e000640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 0x7f865407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:41.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.952+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5797+0+0 (secure 0 0 0) 0x7f86600bbfe0 con 0x7f86780cea60 2024-09-13T08:44:41.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.952+0000 7f8675e00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 0x7f865407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:41.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.952+0000 7f8675e00640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 0x7f865407b340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8668001720 tx=0x7f8668009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:41.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:41.956+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8660083710 con 0x7f86780cea60 2024-09-13T08:44:42.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.188+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f86781ea850 con 0x7f86780cea60 2024-09-13T08:44:42.189 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:44:42.189+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:44:42.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.191+0000 7f866e000640 1 -- 172.21.15.31:0/2156457880 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) ==== 161+0+0 (secure 0 0 0) 0x7f8660088140 con 0x7f86780cea60 2024-09-13T08:44:42.192 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:44:42.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 msgr2=0x7f865407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:42.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 0x7f865407b340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8668001720 tx=0x7f8668009040 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 msgr2=0x7f86780b8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f86600038a0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 shutdown_connections 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8654078e80 0x7f865407b340 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f867815e160 0x7f86780cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86780cea60 0x7f86780b8ff0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 --2- 172.21.15.31:0/2156457880 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f86780c9350 0x7f8678157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.194+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 >> 172.21.15.31:0/2156457880 conn(0x7f86780d3f50 msgr2=0x7f86780c8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.195+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 shutdown_connections 2024-09-13T08:44:42.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.195+0000 7f8677200640 1 -- 172.21.15.31:0/2156457880 wait complete. 2024-09-13T08:44:42.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:44:42.262 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:44:42.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 -- 172.21.15.31:0/4258543425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5040b40c0 msgr2=0x7ff504167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:42.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 --2- 172.21.15.31:0/4258543425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5040b40c0 0x7ff504167470 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc005fd0 tx=0x7ff4fc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 -- 172.21.15.31:0/4258543425 shutdown_connections 2024-09-13T08:44:42.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 --2- 172.21.15.31:0/4258543425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5040b40c0 0x7ff504167470 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 -- 172.21.15.31:0/4258543425 >> 172.21.15.31:0/4258543425 conn(0x7ff5040ccd00 msgr2=0x7ff5040cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:42.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 -- 172.21.15.31:0/4258543425 shutdown_connections 2024-09-13T08:44:42.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.377+0000 7ff503200640 1 -- 172.21.15.31:0/4258543425 wait complete. 2024-09-13T08:44:42.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.379+0000 7ff503200640 1 Processor -- start 2024-09-13T08:44:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.379+0000 7ff503200640 1 -- start start 2024-09-13T08:44:42.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:42.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 0x7ff5040c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:42.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 0x7ff5041d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:42.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5040ce320 con 0x7ff504070140 2024-09-13T08:44:42.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff50406ff20 con 0x7ff5040b40c0 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff503200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff5040b2350 con 0x7ff5040b1f70 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff501e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff501e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46924/0 (socket says 172.21.15.31:46924) 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff501e00640 1 -- 172.21.15.31:0/991813933 learned_addr learned my addr 172.21.15.31:0/991813933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff501400640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 0x7ff5040c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.380+0000 7ff502800640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 0x7ff5041d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 -- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 msgr2=0x7ff5040c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 0x7ff5040c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 -- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 msgr2=0x7ff5041d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 0x7ff5041d38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 -- 172.21.15.31:0/991813933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50409d770 con 0x7ff504070140 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501400640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 0x7ff5040c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff502800640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 0x7ff5041d38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff501e00640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c450 tx=0x7ff4f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40052d0 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4f4005470 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4012870 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50409da50 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.381+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5040c66c0 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.383+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5040af370 con 0x7ff504070140 2024-09-13T08:44:42.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.384+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff4f40035f0 con 0x7ff504070140 2024-09-13T08:44:42.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.385+0000 7ff4fa000640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 0x7ff4e407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:42.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.385+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5839+0+0 (secure 0 0 0) 0x7ff4f409b0e0 con 0x7ff504070140 2024-09-13T08:44:42.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.385+0000 7ff501400640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 0x7ff4e407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:42.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.386+0000 7ff501400640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 0x7ff4e407b340 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0007a50 tx=0x7ff4f0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:42.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.389+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff5040af370 con 0x7ff504070140 2024-09-13T08:44:42.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:42.619+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff5040688b0 con 0x7ff504070140 2024-09-13T08:44:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.193+0000 7ff4fa000640 1 -- 172.21.15.31:0/991813933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) ==== 104+0+0 (secure 0 0 0) 0x7ff4f4062d10 con 0x7ff504070140 2024-09-13T08:44:43.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 msgr2=0x7ff4e407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:43.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 0x7ff4e407b340 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0007a50 tx=0x7ff4f0009040 comp rx=0 tx=0).stop 2024-09-13T08:44:43.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 msgr2=0x7ff5040c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:43.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c450 tx=0x7ff4f400c910 comp rx=0 tx=0).stop 2024-09-13T08:44:43.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 shutdown_connections 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4e4078e80 0x7ff4e407b340 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5040b40c0 0x7ff5041d38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff5040b1f70 0x7ff5040c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 --2- 172.21.15.31:0/991813933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff504070140 0x7ff5040c6be0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 >> 172.21.15.31:0/991813933 conn(0x7ff5040ccd00 msgr2=0x7ff5040b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 shutdown_connections 2024-09-13T08:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.196+0000 7ff503200640 1 -- 172.21.15.31:0/991813933 wait complete. 2024-09-13T08:44:43.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:44:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 -- 172.21.15.31:0/3693468928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f233c003a50 msgr2=0x7f233c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:43.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 --2- 172.21.15.31:0/3693468928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f233c003a50 0x7f233c003e70 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f2348009b50 tx=0x7f233c005830 comp rx=0 tx=0).stop 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f2357e00640 1 -- 172.21.15.31:0/3693468928 reap_dead start 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 -- 172.21.15.31:0/3693468928 shutdown_connections 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 -- 172.21.15.31:0/3693468928 >> 172.21.15.31:0/3693468928 conn(0x7f23580d1b90 msgr2=0x7f23580d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 -- 172.21.15.31:0/3693468928 shutdown_connections 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.387+0000 7f235d200640 1 -- 172.21.15.31:0/3693468928 wait complete. 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.389+0000 7f235d200640 1 Processor -- start 2024-09-13T08:44:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.389+0000 7f235d200640 1 -- start start 2024-09-13T08:44:43.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f233c003a50 0x7f23581f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:43.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:43.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 0x7f23580c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:43.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f23580b55a0 con 0x7f23581f5100 2024-09-13T08:44:43.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f23580cc9f0 con 0x7f23580c8c60 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f235d200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f23580b5720 con 0x7f233c003a50 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46944/0 (socket says 172.21.15.31:46944) 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 -- 172.21.15.31:0/674184035 learned_addr learned my addr 172.21.15.31:0/674184035 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f2357400640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f233c003a50 0x7f23581f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f2357e00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 0x7f23580c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 -- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f233c003a50 msgr2=0x7f23581f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f233c003a50 0x7f23581f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 -- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 msgr2=0x7f23580c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 0x7f23580c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:43.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.390+0000 7f234ea00640 1 -- 172.21.15.31:0/674184035 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235815d3e0 con 0x7f23581f5100 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f2357e00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 0x7f23580c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f234ea00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2340006680 tx=0x7f23400066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340015070 con 0x7f23581f5100 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23400119c0 con 0x7f23581f5100 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340011e10 con 0x7f23581f5100 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235815d6c0 con 0x7f23581f5100 2024-09-13T08:44:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.391+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235815dc80 con 0x7f23581f5100 2024-09-13T08:44:43.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.392+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23580b5370 con 0x7f23581f5100 2024-09-13T08:44:43.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.393+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f2340018070 con 0x7f23581f5100 2024-09-13T08:44:43.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.394+0000 7f234fe00640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 0x7f233407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:43.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.395+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5839+0+0 (secure 0 0 0) 0x7f234009c280 con 0x7f23581f5100 2024-09-13T08:44:43.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.395+0000 7f2357400640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 0x7f233407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:43.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.395+0000 7f2357400640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 0x7f233407b340 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2348035000 tx=0x7f234802d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:43.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.398+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f23580b5370 con 0x7f23581f5100 2024-09-13T08:44:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:43.592+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f23580b5060 con 0x7f23581f5100 2024-09-13T08:44:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.195+0000 7f234fe00640 1 -- 172.21.15.31:0/674184035 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v129) ==== 181+0+0 (secure 0 0 0) 0x7f2340063790 con 0x7f23581f5100 2024-09-13T08:44:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:44:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 msgr2=0x7f233407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 0x7f233407b340 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2348035000 tx=0x7f234802d040 comp rx=0 tx=0).stop 2024-09-13T08:44:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 msgr2=0x7f23581f5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2340006680 tx=0x7f23400066b0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 shutdown_connections 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2334078e80 0x7f233407b340 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23580c8c60 0x7f23580c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23581f5100 0x7f23581f5580 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 --2- 172.21.15.31:0/674184035 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f233c003a50 0x7f23581f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.198+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 >> 172.21.15.31:0/674184035 conn(0x7f23580d1b90 msgr2=0x7f23580d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.199+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 shutdown_connections 2024-09-13T08:44:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.199+0000 7f235d200640 1 -- 172.21.15.31:0/674184035 wait complete. 2024-09-13T08:44:44.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 -- 172.21.15.31:0/33984965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93940cbe50 msgr2=0x7f93940b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 --2- 172.21.15.31:0/33984965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93940cbe50 0x7f93940b29e0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f938c005fd0 tx=0x7f938c02a940 comp rx=0 tx=0).stop 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 -- 172.21.15.31:0/33984965 shutdown_connections 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 --2- 172.21.15.31:0/33984965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93940cbe50 0x7f93940b29e0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 -- 172.21.15.31:0/33984965 >> 172.21.15.31:0/33984965 conn(0x7f93940cfbb0 msgr2=0x7f93940cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:44.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 -- 172.21.15.31:0/33984965 shutdown_connections 2024-09-13T08:44:44.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.387+0000 7f939bc00640 1 -- 172.21.15.31:0/33984965 wait complete. 2024-09-13T08:44:44.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.388+0000 7f939bc00640 1 Processor -- start 2024-09-13T08:44:44.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 -- start start 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9394072560 0x7f939415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940cbe50 0x7f93941f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f93940b2f20 con 0x7f939415bfc0 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93940d0a00 con 0x7f93940cbe50 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.389+0000 7f939bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f93940b30a0 con 0x7f9394072560 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46964/0 (socket says 172.21.15.31:46964) 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 -- 172.21.15.31:0/3975131371 learned_addr learned my addr 172.21.15.31:0/3975131371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399200640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9394072560 0x7f939415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9393e00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940cbe50 0x7f93941f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 -- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9394072560 msgr2=0x7f939415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9394072560 0x7f939415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 -- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940cbe50 msgr2=0x7f93941f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940cbe50 0x7f93941f8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.390+0000 7f9399c00640 1 -- 172.21.15.31:0/3975131371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939415ef60 con 0x7f939415bfc0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f9399c00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f938c009cd0 tx=0x7f938c031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c030430 con 0x7f939415bfc0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f938c004240 con 0x7f939415bfc0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939415f240 con 0x7f939415bfc0 2024-09-13T08:44:44.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c004670 con 0x7f939415bfc0 2024-09-13T08:44:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.391+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93941f8db0 con 0x7f939415bfc0 2024-09-13T08:44:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.392+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93940b29e0 con 0x7f939415bfc0 2024-09-13T08:44:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.393+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f938c039440 con 0x7f939415bfc0 2024-09-13T08:44:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.394+0000 7f9390c00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 0x7f937407b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.394+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5401+0+0 (secure 0 0 0) 0x7f938c0bca60 con 0x7f939415bfc0 2024-09-13T08:44:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.395+0000 7f9399200640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 0x7f937407b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:44.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.395+0000 7f9399200640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 0x7f937407b370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9388000f60 tx=0x7f9388009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:44.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.399+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f938c084320 con 0x7f939415bfc0 2024-09-13T08:44:44.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:44.593+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f93940cb310 con 0x7f939415bfc0 2024-09-13T08:44:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.197+0000 7f9390c00640 1 -- 172.21.15.31:0/3975131371 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v130) ==== 169+0+0 (secure 0 0 0) 0x7f938c088d50 con 0x7f939415bfc0 2024-09-13T08:44:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:44:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 msgr2=0x7f937407b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 0x7f937407b370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9388000f60 tx=0x7f9388009040 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 msgr2=0x7f939415e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f938c009cd0 tx=0x7f938c031ea0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 shutdown_connections 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9374078eb0 0x7f937407b370 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f939415bfc0 0x7f939415e4a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940cbe50 0x7f93941f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 --2- 172.21.15.31:0/3975131371 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9394072560 0x7f939415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 >> 172.21.15.31:0/3975131371 conn(0x7f93940cfbb0 msgr2=0x7f93940ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.201+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 shutdown_connections 2024-09-13T08:44:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.202+0000 7f939bc00640 1 -- 172.21.15.31:0/3975131371 wait complete. 2024-09-13T08:44:45.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:44:45.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 -- 172.21.15.31:0/121235032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 msgr2=0x7fe98c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 --2- 172.21.15.31:0/121235032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe98c003e70 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fe99800c310 tx=0x7fe98c005830 comp rx=0 tx=0).stop 2024-09-13T08:44:45.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 -- 172.21.15.31:0/121235032 shutdown_connections 2024-09-13T08:44:45.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 --2- 172.21.15.31:0/121235032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe98c003e70 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 -- 172.21.15.31:0/121235032 >> 172.21.15.31:0/121235032 conn(0x7fe9a80b18f0 msgr2=0x7fe9a80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.397+0000 7fe9aca00640 1 -- 172.21.15.31:0/121235032 shutdown_connections 2024-09-13T08:44:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.398+0000 7fe9aca00640 1 -- 172.21.15.31:0/121235032 wait complete. 2024-09-13T08:44:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.399+0000 7fe9aca00640 1 Processor -- start 2024-09-13T08:44:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.399+0000 7fe9aca00640 1 -- start start 2024-09-13T08:44:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9a80c9bd0 0x7fe9a81f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a80cb2d0 0x7fe9a80cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a806f710 con 0x7fe98c003a50 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a806f590 con 0x7fe9a80cb2d0 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9aca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a806e660 con 0x7fe9a80c9bd0 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9a5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9a5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55266/0 (socket says 172.21.15.31:55266) 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9a5e00640 1 -- 172.21.15.31:0/1326016639 learned_addr learned my addr 172.21.15.31:0/1326016639 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe99d400640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9a80c9bd0 0x7fe9a81f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.400+0000 7fe9a6800640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a80cb2d0 0x7fe9a80cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 -- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9a80c9bd0 msgr2=0x7fe9a81f9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9a80c9bd0 0x7fe9a81f9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 -- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a80cb2d0 msgr2=0x7fe9a80cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a80cb2d0 0x7fe9a80cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 -- 172.21.15.31:0/1326016639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a8153420 con 0x7fe98c003a50 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe9a5e00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe99800c400 tx=0x7fe998004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.401+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe998031040 con 0x7fe98c003a50 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.402+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe998006420 con 0x7fe98c003a50 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.402+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe998006c70 con 0x7fe98c003a50 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.402+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a81f9c60 con 0x7fe98c003a50 2024-09-13T08:44:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.402+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a80ca330 con 0x7fe98c003a50 2024-09-13T08:44:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.403+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a80b2d10 con 0x7fe98c003a50 2024-09-13T08:44:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.404+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fe998034070 con 0x7fe98c003a50 2024-09-13T08:44:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.405+0000 7fe99ea00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 0x7fe98407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.405+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7fe9980bb3d0 con 0x7fe98c003a50 2024-09-13T08:44:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.405+0000 7fe99d400640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 0x7fe98407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.406+0000 7fe99d400640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 0x7fe98407b340 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9a80cc070 tx=0x7fe990005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.409+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe998082d50 con 0x7fe98c003a50 2024-09-13T08:44:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.606+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fe9a8154320 con 0x7fe98c003a50 2024-09-13T08:44:45.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.610+0000 7fe99ea00640 1 -- 172.21.15.31:0/1326016639 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v41) ==== 71+0+0 (secure 0 0 0) 0x7fe998087780 con 0x7fe98c003a50 2024-09-13T08:44:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 msgr2=0x7fe98407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 0x7fe98407b340 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9a80cc070 tx=0x7fe990005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 msgr2=0x7fe9a8152ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe99800c400 tx=0x7fe998004bc0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 shutdown_connections 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe984078e80 0x7fe98407b340 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a80cb2d0 0x7fe9a80cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe9a80c9bd0 0x7fe9a81f9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 --2- 172.21.15.31:0/1326016639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe98c003a50 0x7fe9a8152ee0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 >> 172.21.15.31:0/1326016639 conn(0x7fe9a80b18f0 msgr2=0x7fe9a8154bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 shutdown_connections 2024-09-13T08:44:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.613+0000 7fe9aca00640 1 -- 172.21.15.31:0/1326016639 wait complete. 2024-09-13T08:44:45.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 -- 172.21.15.31:0/1298509033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56980051c0 msgr2=0x7f569800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 --2- 172.21.15.31:0/1298509033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56980051c0 0x7f569800b9c0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c610 tx=0x7f5698005830 comp rx=0 tx=0).stop 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56b9c00640 1 -- 172.21.15.31:0/1298509033 reap_dead start 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 -- 172.21.15.31:0/1298509033 shutdown_connections 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 -- 172.21.15.31:0/1298509033 >> 172.21.15.31:0/1298509033 conn(0x7f56b406cb00 msgr2=0x7f56b406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 -- 172.21.15.31:0/1298509033 shutdown_connections 2024-09-13T08:44:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.791+0000 7f56ba600640 1 -- 172.21.15.31:0/1298509033 wait complete. 2024-09-13T08:44:45.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.793+0000 7f56ba600640 1 Processor -- start 2024-09-13T08:44:45.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.793+0000 7f56ba600640 1 -- start start 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56980051c0 0x7f56b414aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56b41475d0 0x7f56b41f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56b406f9b0 con 0x7f56b41f4b50 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56b40703b0 con 0x7f56980051c0 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56ba600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f56b406fb30 con 0x7f56b41475d0 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56b9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56b9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55274/0 (socket says 172.21.15.31:55274) 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56b9c00640 1 -- 172.21.15.31:0/4217419051 learned_addr learned my addr 172.21.15.31:0/4217419051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.794+0000 7f56abe00640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56b41475d0 0x7f56b41f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9200640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56980051c0 0x7f56b414aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 -- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56b41475d0 msgr2=0x7f56b41f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56b41475d0 0x7f56b41f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 -- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56980051c0 msgr2=0x7f56b414aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56980051c0 0x7f56b414aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 -- 172.21.15.31:0/4217419051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b4149f90 con 0x7f56b41f4b50 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b9c00640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f56ac02b3b0 tx=0x7f56ac004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac02bb10 con 0x7f56b41f4b50 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56ac02bdc0 con 0x7f56b41f4b50 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0062d0 con 0x7f56b41f4b50 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.795+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b41f5140 con 0x7f56b41f4b50 2024-09-13T08:44:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.796+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b414a250 con 0x7f56b41f4b50 2024-09-13T08:44:45.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.797+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b40cee50 con 0x7f56b41f4b50 2024-09-13T08:44:45.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.798+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f56ac006470 con 0x7f56b41f4b50 2024-09-13T08:44:45.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.799+0000 7f56b1600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 0x7f569007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:45.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.799+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f56ac0bb570 con 0x7f56b41f4b50 2024-09-13T08:44:45.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.799+0000 7f56b9200640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 0x7f569007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:45.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.800+0000 7f56b9200640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 0x7f569007b340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f56b40b26c0 tx=0x7f56a4005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:45.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:45.803+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f56b40cee50 con 0x7f56b41f4b50 2024-09-13T08:44:46.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.073+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7f56b40b4bc0 con 0x7f56b41f4b50 2024-09-13T08:44:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.199+0000 7f56b1600640 1 -- 172.21.15.31:0/4217419051 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v415) ==== 148+0+0 (secure 0 0 0) 0x7f56b40b4bc0 con 0x7f56b41f4b50 2024-09-13T08:44:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 msgr2=0x7f569007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 0x7f569007b340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f56b40b26c0 tx=0x7f56a4005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 msgr2=0x7f56b415e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f56ac02b3b0 tx=0x7f56ac004bb0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 shutdown_connections 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5690078e80 0x7f569007b340 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b41f4b50 0x7f56b415e0f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.201+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56b41475d0 0x7f56b41f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.202+0000 7f56ba600640 1 --2- 172.21.15.31:0/4217419051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56980051c0 0x7f56b414aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.202+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 >> 172.21.15.31:0/4217419051 conn(0x7f56b406cb00 msgr2=0x7f56b40b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.202+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 shutdown_connections 2024-09-13T08:44:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.202+0000 7f56ba600640 1 -- 172.21.15.31:0/4217419051 wait complete. 2024-09-13T08:44:46.218 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-13T08:44:46.218 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:44:46.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:46.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 -- 172.21.15.31:0/2714011931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815bd70 msgr2=0x7f2be80cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2714011931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815bd70 0x7f2be80cf850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009830 tx=0x7f2bdc02a510 comp rx=0 tx=0).stop 2024-09-13T08:44:46.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 -- 172.21.15.31:0/2714011931 shutdown_connections 2024-09-13T08:44:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2714011931 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815fe20 0x7f2be80cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2714011931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815bd70 0x7f2be80cf850 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 -- 172.21.15.31:0/2714011931 >> 172.21.15.31:0/2714011931 conn(0x7f2be80cc910 msgr2=0x7f2be80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 -- 172.21.15.31:0/2714011931 shutdown_connections 2024-09-13T08:44:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.378+0000 7f2bf0400640 1 -- 172.21.15.31:0/2714011931 wait complete. 2024-09-13T08:44:46.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.380+0000 7f2bf0400640 1 Processor -- start 2024-09-13T08:44:46.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.380+0000 7f2bf0400640 1 -- start start 2024-09-13T08:44:46.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815bd70 0x7f2be8159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815fe20 0x7f2be806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80cec50 con 0x7f2be815f9e0 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80cead0 con 0x7f2be815fe20 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bf0400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80cb850 con 0x7f2be815bd70 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bed000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bed000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55282/0 (socket says 172.21.15.31:55282) 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bed000640 1 -- 172.21.15.31:0/2983563417 learned_addr learned my addr 172.21.15.31:0/2983563417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2beda00640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815bd70 0x7f2be8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.381+0000 7f2bee400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815fe20 0x7f2be806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 -- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815bd70 msgr2=0x7f2be8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815bd70 0x7f2be8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 -- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815fe20 msgr2=0x7f2be806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815fe20 0x7f2be806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 -- 172.21.15.31:0/2983563417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80c4690 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bed000640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c490 tx=0x7f2bd400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011790 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bd4017070 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be81f8720 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011ce0 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.382+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be8159540 con 0x7f2be815f9e0 2024-09-13T08:44:46.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.384+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be80cf850 con 0x7f2be815f9e0 2024-09-13T08:44:46.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.385+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f2bd401c460 con 0x7f2be815f9e0 2024-09-13T08:44:46.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.386+0000 7f2be5600640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 0x7f2bc807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.386+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f2bd409af60 con 0x7f2be815f9e0 2024-09-13T08:44:46.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.386+0000 7f2beda00640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 0x7f2bc807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.387+0000 7f2beda00640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 0x7f2bc807b370 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc034000 tx=0x7f2bdc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.390+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2bd4062810 con 0x7f2be815f9e0 2024-09-13T08:44:46.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.587+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2be8072920 con 0x7f2be815f9e0 2024-09-13T08:44:46.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.588+0000 7f2be5600640 1 -- 172.21.15.31:0/2983563417 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+13061 (secure 0 0 0) 0x7f2bd4067240 con 0x7f2be815f9e0 2024-09-13T08:44:46.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:46.589 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":130,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:45.195527+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6835/1889278552":"2024-09-14T08:44:42.189358+0000","172.21.15.31:6834/1889278552":"2024-09-14T08:44:42.189358+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:46.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 msgr2=0x7f2bc807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 0x7f2bc807b370 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc034000 tx=0x7f2bdc004730 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 msgr2=0x7f2be809d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c490 tx=0x7f2bd400c950 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 shutdown_connections 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2bc8078eb0 0x7f2bc807b370 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be815fe20 0x7f2be806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2be815f9e0 0x7f2be809d4a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 --2- 172.21.15.31:0/2983563417 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2be815bd70 0x7f2be8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 >> 172.21.15.31:0/2983563417 conn(0x7f2be80cc910 msgr2=0x7f2be815b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 shutdown_connections 2024-09-13T08:44:46.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.590+0000 7f2bf0400640 1 -- 172.21.15.31:0/2983563417 wait complete. 2024-09-13T08:44:46.607 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:44:46.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.770+0000 7f7b17a00640 1 -- 172.21.15.31:0/1545506161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7af4004800 msgr2=0x7f7af4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.770+0000 7f7b17a00640 1 --2- 172.21.15.31:0/1545506161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7af4004800 0x7f7af4004c80 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7b1006db40 tx=0x7f7af4005830 comp rx=0 tx=0).stop 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.771+0000 7f7b15a00640 1 -- 172.21.15.31:0/1545506161 reap_dead start 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.771+0000 7f7b17a00640 1 -- 172.21.15.31:0/1545506161 shutdown_connections 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.771+0000 7f7b17a00640 1 -- 172.21.15.31:0/1545506161 >> 172.21.15.31:0/1545506161 conn(0x7f7b100cfbb0 msgr2=0x7f7b100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.771+0000 7f7b17a00640 1 -- 172.21.15.31:0/1545506161 shutdown_connections 2024-09-13T08:44:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.771+0000 7f7b17a00640 1 -- 172.21.15.31:0/1545506161 wait complete. 2024-09-13T08:44:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.772+0000 7f7b17a00640 1 Processor -- start 2024-09-13T08:44:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.772+0000 7f7b17a00640 1 -- start start 2024-09-13T08:44:46.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 0x7f7b100b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b100c5080 0x7f7b100c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1006e740 con 0x7f7b100c4c20 2024-09-13T08:44:46.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1006e5c0 con 0x7f7b100c5080 2024-09-13T08:44:46.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.773+0000 7f7b17a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7b100cb740 con 0x7f7af4004800 2024-09-13T08:44:46.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55290/0 (socket says 172.21.15.31:55290) 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 -- 172.21.15.31:0/340031255 learned_addr learned my addr 172.21.15.31:0/340031255 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b15000640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 0x7f7b100b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b15a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b100c5080 0x7f7b100c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 -- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 msgr2=0x7f7b100b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 0x7f7b100b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 -- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b100c5080 msgr2=0x7f7b100c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b100c5080 0x7f7b100c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 -- 172.21.15.31:0/340031255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b100c9c10 con 0x7f7b100c4c20 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b15000640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 0x7f7b100b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:46.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b07e00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7af800bac0 tx=0x7f7af8034c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:46.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8039040 con 0x7f7b100c4c20 2024-09-13T08:44:46.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7af8005110 con 0x7f7b100c4c20 2024-09-13T08:44:46.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.774+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b100c8a90 con 0x7f7b100c4c20 2024-09-13T08:44:46.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.775+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af800b620 con 0x7f7b100c4c20 2024-09-13T08:44:46.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.775+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b100c9050 con 0x7f7b100c4c20 2024-09-13T08:44:46.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.776+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1006fa60 con 0x7f7b100c4c20 2024-09-13T08:44:46.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.777+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f7af8044020 con 0x7f7b100c4c20 2024-09-13T08:44:46.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.778+0000 7f7b0d600640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 0x7f7aec07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.778+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f7af80baca0 con 0x7f7b100c4c20 2024-09-13T08:44:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.778+0000 7f7b15000640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 0x7f7aec07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.779+0000 7f7b15000640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 0x7f7aec07b340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b00000fb0 tx=0x7f7b00006720 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.782+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7af804b020 con 0x7f7b100c4c20 2024-09-13T08:44:46.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.977+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7b10159db0 con 0x7f7b100c4c20 2024-09-13T08:44:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.978+0000 7f7b0d600640 1 -- 172.21.15.31:0/340031255 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7f7af8082b60 con 0x7f7b100c4c20 2024-09-13T08:44:46.979 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:44:46.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 msgr2=0x7f7aec07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 0x7f7aec07b340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b00000fb0 tx=0x7f7b00006720 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 msgr2=0x7f7b100c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7af800bac0 tx=0x7f7af8034c90 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 shutdown_connections 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7aec078e80 0x7f7aec07b340 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b100c5080 0x7f7b100c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b100c4c20 0x7f7b100c55c0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 --2- 172.21.15.31:0/340031255 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7af4004800 0x7f7b100b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 >> 172.21.15.31:0/340031255 conn(0x7f7b100cfbb0 msgr2=0x7f7b1015aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.980+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 shutdown_connections 2024-09-13T08:44:46.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:46.981+0000 7f7b17a00640 1 -- 172.21.15.31:0/340031255 wait complete. 2024-09-13T08:44:46.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-13T08:44:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 -- 172.21.15.31:0/288222418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4100cd6f0 msgr2=0x7fb4101600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 --2- 172.21.15.31:0/288222418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4100cd6f0 0x7fb4101600e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005fd0 tx=0x7fb3fc02a540 comp rx=0 tx=0).stop 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 -- 172.21.15.31:0/288222418 shutdown_connections 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 --2- 172.21.15.31:0/288222418 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb410160620 0x7fb4100c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 --2- 172.21.15.31:0/288222418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4100cd6f0 0x7fb4101600e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 -- 172.21.15.31:0/288222418 >> 172.21.15.31:0/288222418 conn(0x7fb4100d0360 msgr2=0x7fb4100d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 -- 172.21.15.31:0/288222418 shutdown_connections 2024-09-13T08:44:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.158+0000 7fb416c00640 1 -- 172.21.15.31:0/288222418 wait complete. 2024-09-13T08:44:47.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.160+0000 7fb416c00640 1 Processor -- start 2024-09-13T08:44:47.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.160+0000 7fb416c00640 1 -- start start 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4100cd6f0 0x7fb4101575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 0x7fb4101f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4100c8140 con 0x7fb41015f190 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4100c7fc0 con 0x7fb41015bdf0 2024-09-13T08:44:47.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb4100c82c0 con 0x7fb4100cd6f0 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.161+0000 7fb416200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55296/0 (socket says 172.21.15.31:55296) 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 -- 172.21.15.31:0/3302133775 learned_addr learned my addr 172.21.15.31:0/3302133775 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb415800640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4100cd6f0 0x7fb4101575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb414e00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 0x7fb4101f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 -- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4100cd6f0 msgr2=0x7fb4101575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4100cd6f0 0x7fb4101575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 -- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 msgr2=0x7fb4101f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 0x7fb4101f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 -- 172.21.15.31:0/3302133775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb410157ba0 con 0x7fb41015f190 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb414e00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 0x7fb4101f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.162+0000 7fb416200640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb40800ea40 tx=0x7fb40800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.163+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40800cde0 con 0x7fb41015f190 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.163+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb408004c10 con 0x7fb41015f190 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.163+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408005020 con 0x7fb41015f190 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.163+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb410157e80 con 0x7fb41015f190 2024-09-13T08:44:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.163+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb410158440 con 0x7fb41015f190 2024-09-13T08:44:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.164+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4100c7a80 con 0x7fb41015f190 2024-09-13T08:44:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.165+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fb4080051c0 con 0x7fb41015f190 2024-09-13T08:44:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.166+0000 7fb40d600640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 0x7fb3f007b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.167+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7fb40809bf10 con 0x7fb41015f190 2024-09-13T08:44:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.167+0000 7fb415800640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 0x7fb3f007b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.167+0000 7fb415800640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 0x7fb3f007b370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb4040017c0 tx=0x7fb404009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:47.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.171+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb408063770 con 0x7fb41015f190 2024-09-13T08:44:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:47.441+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7fb410072860 con 0x7fb41015f190 2024-09-13T08:44:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.206+0000 7fb40d600640 1 -- 172.21.15.31:0/3302133775 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v417) ==== 154+0+0 (secure 0 0 0) 0x7fb4080681a0 con 0x7fb41015f190 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 msgr2=0x7fb3f007b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 0x7fb3f007b370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb4040017c0 tx=0x7fb404009040 comp rx=0 tx=0).stop 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 msgr2=0x7fb410158ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb40800ea40 tx=0x7fb40800ef00 comp rx=0 tx=0).stop 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 shutdown_connections 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb3f0078eb0 0x7fb3f007b370 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41015f190 0x7fb410158ab0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb41015bdf0 0x7fb4101f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 --2- 172.21.15.31:0/3302133775 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4100cd6f0 0x7fb4101575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 >> 172.21.15.31:0/3302133775 conn(0x7fb4100d0360 msgr2=0x7fb4100b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:48.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 shutdown_connections 2024-09-13T08:44:48.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.210+0000 7fb416c00640 1 -- 172.21.15.31:0/3302133775 wait complete. 2024-09-13T08:44:48.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:44:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 -- 172.21.15.31:0/2158955339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0071720 msgr2=0x7f12e0073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 --2- 172.21.15.31:0/2158955339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0071720 0x7f12e0073590 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009830 tx=0x7f12c802a530 comp rx=0 tx=0).stop 2024-09-13T08:44:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 -- 172.21.15.31:0/2158955339 shutdown_connections 2024-09-13T08:44:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 --2- 172.21.15.31:0/2158955339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0071720 0x7f12e0073590 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 -- 172.21.15.31:0/2158955339 >> 172.21.15.31:0/2158955339 conn(0x7f12e00d0bb0 msgr2=0x7f12e00d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 -- 172.21.15.31:0/2158955339 shutdown_connections 2024-09-13T08:44:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.396+0000 7f12e5400640 1 -- 172.21.15.31:0/2158955339 wait complete. 2024-09-13T08:44:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.397+0000 7f12e5400640 1 Processor -- start 2024-09-13T08:44:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.397+0000 7f12e5400640 1 -- start start 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e0071720 0x7f12e0151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f12e015e100 0x7f12e0154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0166050 con 0x7f12e0073ad0 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0165ed0 con 0x7f12e0071720 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.398+0000 7f12e5400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f12e01661d0 con 0x7f12e015e100 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e0071720 0x7f12e0151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55304/0 (socket says 172.21.15.31:55304) 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 -- 172.21.15.31:0/2042917681 learned_addr learned my addr 172.21.15.31:0/2042917681 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12e4a00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f12e015e100 0x7f12e0154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 -- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f12e015e100 msgr2=0x7f12e0154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f12e015e100 0x7f12e0154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 -- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e0071720 msgr2=0x7f12e0151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e0071720 0x7f12e0151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.399+0000 7f12d7e00640 1 -- 172.21.15.31:0/2042917681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e0165720 con 0x7f12e0073ad0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12d7e00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f12c802ab20 tx=0x7f12c8032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8040070 con 0x7f12e0073ad0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12c8005310 con 0x7f12e0073ad0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8003800 con 0x7f12e0073ad0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e0165a00 con 0x7f12e0073ad0 2024-09-13T08:44:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.400+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e014f120 con 0x7f12e0073ad0 2024-09-13T08:44:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.401+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e0165c40 con 0x7f12e0073ad0 2024-09-13T08:44:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.402+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f12c8039040 con 0x7f12e0073ad0 2024-09-13T08:44:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.403+0000 7f12dcc00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 0x7f12bc07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.403+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f12c80bc220 con 0x7f12e0073ad0 2024-09-13T08:44:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.403+0000 7f12dfe00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 0x7f12bc07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.404+0000 7f12dfe00640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 0x7f12bc07b340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f12d00046b0 tx=0x7f12d0007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.407+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f12e0165c40 con 0x7f12e0073ad0 2024-09-13T08:44:48.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.603+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f12e0073590 con 0x7f12e0073ad0 2024-09-13T08:44:48.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.603+0000 7f12dcc00640 1 -- 172.21.15.31:0/2042917681 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v41) ==== 59+0+1998 (secure 0 0 0) 0x7f12c8083bf0 con 0x7f12e0073ad0 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:44:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:44:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:44:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:44:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:44:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:44:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 msgr2=0x7f12bc07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 0x7f12bc07b340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f12d00046b0 tx=0x7f12d0007e90 comp rx=0 tx=0).stop 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 msgr2=0x7f12e014ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f12c802ab20 tx=0x7f12c8032cd0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 shutdown_connections 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f12bc078e80 0x7f12bc07b340 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f12e015e100 0x7f12e0154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e0073ad0 0x7f12e014ebe0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 --2- 172.21.15.31:0/2042917681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e0071720 0x7f12e0151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 >> 172.21.15.31:0/2042917681 conn(0x7f12e00d0bb0 msgr2=0x7f12e00c72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 shutdown_connections 2024-09-13T08:44:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.606+0000 7f12e5400640 1 -- 172.21.15.31:0/2042917681 wait complete. 2024-09-13T08:44:48.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:44:48.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 -- 172.21.15.31:0/2723960976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 msgr2=0x7f41e80d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 --2- 172.21.15.31:0/2723960976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e80d28f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009820 tx=0x7f41d802a510 comp rx=0 tx=0).stop 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 -- 172.21.15.31:0/2723960976 shutdown_connections 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 --2- 172.21.15.31:0/2723960976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e80d28f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 -- 172.21.15.31:0/2723960976 >> 172.21.15.31:0/2723960976 conn(0x7f41e80d1dd0 msgr2=0x7f41e80d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 -- 172.21.15.31:0/2723960976 shutdown_connections 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.785+0000 7f41ee800640 1 -- 172.21.15.31:0/2723960976 wait complete. 2024-09-13T08:44:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.786+0000 7f41ee800640 1 Processor -- start 2024-09-13T08:44:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 -- start start 2024-09-13T08:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41e80ceb50 0x7f41e81f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 0x7f41e81f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41e80b6070 con 0x7f41e815bec0 2024-09-13T08:44:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41e80b5ef0 con 0x7f41e80cf050 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.787+0000 7f41ee800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f41e80b61f0 con 0x7f41e80ceb50 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55314/0 (socket says 172.21.15.31:55314) 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 -- 172.21.15.31:0/2922991439 learned_addr learned my addr 172.21.15.31:0/2922991439 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 -- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41e80ceb50 msgr2=0x7f41e81f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41e80ceb50 0x7f41e81f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 -- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 msgr2=0x7f41e81f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41dea00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 0x7f41e81f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 0x7f41e81f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41e7e00640 1 -- 172.21.15.31:0/2922991439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e81f7910 con 0x7f41e815bec0 2024-09-13T08:44:48.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.788+0000 7f41dea00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 0x7f41e81f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:48.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41e7e00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c7d0 tx=0x7f41e000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:48.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0003b70 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41e0003d10 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e000aab0 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e80c59a0 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.789+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e80c5f60 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.790+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e80b6710 con 0x7f41e815bec0 2024-09-13T08:44:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.791+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f41e000ac50 con 0x7f41e815bec0 2024-09-13T08:44:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.792+0000 7f41dfe00640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 0x7f41c407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.792+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f41e009b160 con 0x7f41e815bec0 2024-09-13T08:44:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.792+0000 7f41e7400640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 0x7f41c407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:48.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.793+0000 7f41e7400640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 0x7f41c407b340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41d8034000 tx=0x7f41d80046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:48.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.797+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f41e81f7910 con 0x7f41e815bec0 2024-09-13T08:44:48.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.992+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f41e80761d0 con 0x7f41e815bec0 2024-09-13T08:44:48.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:48.997+0000 7f41dfe00640 1 -- 172.21.15.31:0/2922991439 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f41e0063010 con 0x7f41e815bec0 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 msgr2=0x7f41c407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 0x7f41c407b340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41d8034000 tx=0x7f41d80046c0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 msgr2=0x7f41e815cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c7d0 tx=0x7f41e000cc90 comp rx=0 tx=0).stop 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 shutdown_connections 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f41c4078e80 0x7f41c407b340 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41e815bec0 0x7f41e815cdd0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80cf050 0x7f41e81f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 --2- 172.21.15.31:0/2922991439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f41e80ceb50 0x7f41e81f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 >> 172.21.15.31:0/2922991439 conn(0x7f41e80d1dd0 msgr2=0x7f41e8070860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:49.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 shutdown_connections 2024-09-13T08:44:49.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.000+0000 7f41ee800640 1 -- 172.21.15.31:0/2922991439 wait complete. 2024-09-13T08:44:49.018 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:44:49.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:49.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:44:49.179 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:44:49.179 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:49.180 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:49.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:49.216 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:49.216 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:49.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:49.253 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:44:49.253 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:49.292 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:44:49.292 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:44:49.292 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:44:49.330 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:44:49.330 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:44:49.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:49.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 -- 172.21.15.31:0/850531320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d600cf850 msgr2=0x7f2d600ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 --2- 172.21.15.31:0/850531320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d600cf850 0x7f2d600ce3c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d600cfe60 tx=0x7f2d4c02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 -- 172.21.15.31:0/850531320 shutdown_connections 2024-09-13T08:44:49.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 --2- 172.21.15.31:0/850531320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d600cf850 0x7f2d600ce3c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 -- 172.21.15.31:0/850531320 >> 172.21.15.31:0/850531320 conn(0x7f2d600cc910 msgr2=0x7f2d600cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:49.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 -- 172.21.15.31:0/850531320 shutdown_connections 2024-09-13T08:44:49.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.492+0000 7f2d67600640 1 -- 172.21.15.31:0/850531320 wait complete. 2024-09-13T08:44:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.494+0000 7f2d67600640 1 Processor -- start 2024-09-13T08:44:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.494+0000 7f2d67600640 1 -- start start 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d600ce900 0x7f2d600c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 0x7f2d600c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601600f0 con 0x7f2d600cf850 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d600cb850 con 0x7f2d6015bd70 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d67600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2d600c74a0 con 0x7f2d600ce900 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55326/0 (socket says 172.21.15.31:55326) 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d5fe00640 1 -- 172.21.15.31:0/3761289858 learned_addr learned my addr 172.21.15.31:0/3761289858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d65600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 0x7f2d600c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.495+0000 7f2d64c00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d600ce900 0x7f2d600c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 -- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d600ce900 msgr2=0x7f2d600c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d600ce900 0x7f2d600c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 -- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 msgr2=0x7f2d600c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 0x7f2d600c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 -- 172.21.15.31:0/3761289858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d600c76d0 con 0x7f2d600cf850 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d65600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 0x7f2d600c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5fe00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c02ab90 tx=0x7f2d4c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.496+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c040070 con 0x7f2d600cf850 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.497+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d4c005310 con 0x7f2d600cf850 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.497+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0047a0 con 0x7f2d600cf850 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.497+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d600c79b0 con 0x7f2d600cf850 2024-09-13T08:44:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.497+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d600c7f70 con 0x7f2d600cf850 2024-09-13T08:44:49.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.498+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d600cc3b0 con 0x7f2d600cf850 2024-09-13T08:44:49.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.499+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f2d4c039040 con 0x7f2d600cf850 2024-09-13T08:44:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.500+0000 7f2d5cc00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 0x7f2d4007b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.500+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7f2d4c0bc290 con 0x7f2d600cf850 2024-09-13T08:44:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.500+0000 7f2d64c00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 0x7f2d4007b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.501+0000 7f2d64c00640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 0x7f2d4007b370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000f60 tx=0x7f2d54009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.504+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2d4c083c60 con 0x7f2d600cf850 2024-09-13T08:44:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.739+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d600aeb60 con 0x7f2d600cf850 2024-09-13T08:44:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.739+0000 7f2d5cc00640 1 -- 172.21.15.31:0/3761289858 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+3911 (secure 0 0 0) 0x7f2d4c088690 con 0x7f2d600cf850 2024-09-13T08:44:49.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:49.741 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"btime":"2024-09-13T08:44:43:191535+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7153,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1055764836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1055764836},{"type":"v1","addr":"172.21.15.45:6801","nonce":1055764836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3639477503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3639477503},{"type":"v1","addr":"172.21.15.45:6835","nonce":3639477503}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2247490578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2247490578},{"type":"v1","addr":"172.21.15.31:6833","nonce":2247490578}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":7899,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3578927764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3578927764},{"type":"v1","addr":"172.21.15.31:6835","nonce":3578927764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[]} 2024-09-13T08:44:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2024-09-13T08:44:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 msgr2=0x7f2d4007b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 0x7f2d4007b370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000f60 tx=0x7f2d54009040 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 msgr2=0x7f2d601533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c02ab90 tx=0x7f2d4c032c10 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 shutdown_connections 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d40078eb0 0x7f2d4007b370 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6015bd70 0x7f2d600c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d600cf850 0x7f2d601533d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 --2- 172.21.15.31:0/3761289858 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d600ce900 0x7f2d600c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 >> 172.21.15.31:0/3761289858 conn(0x7f2d600cc910 msgr2=0x7f2d600acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.742+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 shutdown_connections 2024-09-13T08:44:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.743+0000 7f2d67600640 1 -- 172.21.15.31:0/3761289858 wait complete. 2024-09-13T08:44:49.759 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:44:49.760 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:44:49.760 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:44:49.760 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:49.760 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:44:49.760 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:44:49.761 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:44:49.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:49.761 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:44:49.761 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:44:49.761 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:44:49.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:49.761 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:44:49.761 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:44:49.762 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:44:49.762 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:44:49.775 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:44:49.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:44:49.776 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:44:49.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:44:49.778 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:44:49.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:44:49.779 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:44:49.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:44:49.821 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:44:49.821 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:49.823 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:44:49.823 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:49.824 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:44:49.824 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:49.826 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:44:49.826 DEBUG:teuthology.parallel:result is None 2024-09-13T08:44:49.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:44:49.953 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:44:49.953 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:44:49.955 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-13T08:44:49.954+0000 7f6fcdc00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-13T08:44:49.963 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:44:49.963 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:44:49.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 -- 172.21.15.31:0/4241568469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 msgr2=0x7fed94073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 --2- 172.21.15.31:0/4241568469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed94073cf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fed8c005fd0 tx=0x7fed8c02aaa0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 -- 172.21.15.31:0/4241568469 shutdown_connections 2024-09-13T08:44:49.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 --2- 172.21.15.31:0/4241568469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed94073cf0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 --2- 172.21.15.31:0/4241568469 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed940ced20 0x7fed940ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 -- 172.21.15.31:0/4241568469 >> 172.21.15.31:0/4241568469 conn(0x7fed940b18f0 msgr2=0x7fed940b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 -- 172.21.15.31:0/4241568469 shutdown_connections 2024-09-13T08:44:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.992+0000 7fed99400640 1 -- 172.21.15.31:0/4241568469 wait complete. 2024-09-13T08:44:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.994+0000 7fed99400640 1 Processor -- start 2024-09-13T08:44:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.994+0000 7fed99400640 1 -- start start 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed940ced20 0x7fed9414d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed9415be30 0x7fed94150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed94074230 con 0x7fed94073860 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed940ce9e0 con 0x7fed940ced20 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed99400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fed940743b0 con 0x7fed9415be30 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55362/0 (socket says 172.21.15.31:55362) 2024-09-13T08:44:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 -- 172.21.15.31:0/1210697470 learned_addr learned my addr 172.21.15.31:0/1210697470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed89e00640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed940ced20 0x7fed9414d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed93200640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed9415be30 0x7fed94150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 -- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed9415be30 msgr2=0x7fed94150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed9415be30 0x7fed94150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 -- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed940ced20 msgr2=0x7fed9414d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed940ced20 0x7fed9414d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.995+0000 7fed92800640 1 -- 172.21.15.31:0/1210697470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed94150ee0 con 0x7fed94073860 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed92800640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fed8400c3a0 tx=0x7fed8400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84017070 con 0x7fed94073860 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed840053c0 con 0x7fed94073860 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84003850 con 0x7fed94073860 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9414d780 con 0x7fed94073860 2024-09-13T08:44:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.996+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9409aab0 con 0x7fed94073860 2024-09-13T08:44:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.998+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed940748a0 con 0x7fed94073860 2024-09-13T08:44:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:49.999+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fed8401a070 con 0x7fed94073860 2024-09-13T08:44:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.000+0000 7fed8b400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 0x7fed7007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.000+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4987+0+0 (secure 0 0 0) 0x7fed8409a5d0 con 0x7fed94073860 2024-09-13T08:44:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.000+0000 7fed89e00640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 0x7fed7007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.001+0000 7fed89e00640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 0x7fed7007b340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fed941f9840 tx=0x7fed7c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:50.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.006+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fed94150ee0 con 0x7fed94073860 2024-09-13T08:44:50.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:50.217+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fed940ada70 con 0x7fed94073860 2024-09-13T08:44:51.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.221+0000 7fed8b400640 1 -- 172.21.15.31:0/1210697470 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) ==== 97+0+0 (secure 0 0 0) 0x7fed8409a030 con 0x7fed94073860 2024-09-13T08:44:51.222 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 msgr2=0x7fed7007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 0x7fed7007b340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fed941f9840 tx=0x7fed7c005f90 comp rx=0 tx=0).stop 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 msgr2=0x7fed9414cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fed8400c3a0 tx=0x7fed8400c860 comp rx=0 tx=0).stop 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 shutdown_connections 2024-09-13T08:44:51.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fed70078e80 0x7fed7007b340 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fed9415be30 0x7fed94150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed940ced20 0x7fed9414d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 --2- 172.21.15.31:0/1210697470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed94073860 0x7fed9414cba0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 >> 172.21.15.31:0/1210697470 conn(0x7fed940b18f0 msgr2=0x7fed9406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 shutdown_connections 2024-09-13T08:44:51.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.225+0000 7fed99400640 1 -- 172.21.15.31:0/1210697470 wait complete. 2024-09-13T08:44:51.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:44:51.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 -- 172.21.15.31:0/630572772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc152cb0 msgr2=0x7f55bc0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 --2- 172.21.15.31:0/630572772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc152cb0 0x7f55bc0cc4b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009830 tx=0x7f55ac02a4a0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 -- 172.21.15.31:0/630572772 shutdown_connections 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 --2- 172.21.15.31:0/630572772 >> v2:172.21.15.45:3300/0 conn(0x7f55bc15fed0 0x7f55bc06f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 --2- 172.21.15.31:0/630572772 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc152cb0 0x7f55bc0cc4b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 -- 172.21.15.31:0/630572772 >> 172.21.15.31:0/630572772 conn(0x7f55bc06cb00 msgr2=0x7f55bc06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 -- 172.21.15.31:0/630572772 shutdown_connections 2024-09-13T08:44:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.406+0000 7f55c0a00640 1 -- 172.21.15.31:0/630572772 wait complete. 2024-09-13T08:44:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.408+0000 7f55c0a00640 1 Processor -- start 2024-09-13T08:44:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.408+0000 7f55c0a00640 1 -- start start 2024-09-13T08:44:51.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cc9f0 0x7f55bc151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc15fed0 0x7f55bc15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc070060 con 0x7f55bc152cb0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc06fee0 con 0x7f55bc15fed0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55c0a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc0701e0 con 0x7f55bc0cc9f0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55b1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55b1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55374/0 (socket says 172.21.15.31:55374) 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55b1e00640 1 -- 172.21.15.31:0/3863043710 learned_addr learned my addr 172.21.15.31:0/3863043710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55ba800640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cc9f0 0x7f55bc151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.409+0000 7f55bb200640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc15fed0 0x7f55bc15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 -- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cc9f0 msgr2=0x7f55bc151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cc9f0 0x7f55bc151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 -- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc15fed0 msgr2=0x7f55bc15dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc15fed0 0x7f55bc15dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 -- 172.21.15.31:0/3863043710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1f49e0 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b1e00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f55a4001220 tx=0x7f55a400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4016020 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55a400a930 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a400ad80 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.410+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc1f4c80 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.411+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc1f5240 con 0x7f55bc152cb0 2024-09-13T08:44:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.412+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc0cc4b0 con 0x7f55bc152cb0 2024-09-13T08:44:51.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.413+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f55a4019020 con 0x7f55bc152cb0 2024-09-13T08:44:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.414+0000 7f55b3400640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 0x7f559807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.414+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4901+0+0 (secure 0 0 0) 0x7f55a409b1c0 con 0x7f55bc152cb0 2024-09-13T08:44:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.414+0000 7f55ba800640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 0x7f559807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.415+0000 7f55ba800640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 0x7f559807b340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55ac034000 tx=0x7f55ac004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:51.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.418+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f55a4020020 con 0x7f55bc152cb0 2024-09-13T08:44:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.683+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f55bc0ce190 con 0x7f55bc152cb0 2024-09-13T08:44:51.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.683+0000 7f55b3400640 1 -- 172.21.15.31:0/3863043710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7f55a4062ab0 con 0x7f55bc152cb0 2024-09-13T08:44:51.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:51.684 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:44:51.684 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:44:51.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.685 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:44:51.685 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:44:51.685 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.685 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.686 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:51.687 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:44:51.689 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:44:51.701 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 msgr2=0x7f559807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 0x7f559807b340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55ac034000 tx=0x7f55ac004730 comp rx=0 tx=0).stop 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 msgr2=0x7f55bc0c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f55a4001220 tx=0x7f55a400cb70 comp rx=0 tx=0).stop 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 shutdown_connections 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5598078e80 0x7f559807b340 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc15fed0 0x7f55bc15dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc152cb0 0x7f55bc0c84c0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 --2- 172.21.15.31:0/3863043710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cc9f0 0x7f55bc151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.685+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 >> 172.21.15.31:0/3863043710 conn(0x7f55bc06cb00 msgr2=0x7f55bc06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:51.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.686+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 shutdown_connections 2024-09-13T08:44:51.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.686+0000 7f55c0a00640 1 -- 172.21.15.31:0/3863043710 wait complete. 2024-09-13T08:44:51.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:44:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.864+0000 7f76d7800640 1 -- 172.21.15.31:0/4292480599 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 msgr2=0x7f76d00cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.864+0000 7f76d7800640 1 --2- 172.21.15.31:0/4292480599 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d00cc300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009860 tx=0x7f76b802a5a0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.864+0000 7f76d7800640 1 -- 172.21.15.31:0/4292480599 shutdown_connections 2024-09-13T08:44:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.864+0000 7f76d7800640 1 --2- 172.21.15.31:0/4292480599 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d00cc300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.864+0000 7f76d7800640 1 -- 172.21.15.31:0/4292480599 >> 172.21.15.31:0/4292480599 conn(0x7f76d00cc910 msgr2=0x7f76d00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.865+0000 7f76d7800640 1 -- 172.21.15.31:0/4292480599 shutdown_connections 2024-09-13T08:44:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.865+0000 7f76d7800640 1 -- 172.21.15.31:0/4292480599 wait complete. 2024-09-13T08:44:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.866+0000 7f76d7800640 1 Processor -- start 2024-09-13T08:44:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.866+0000 7f76d7800640 1 -- start start 2024-09-13T08:44:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d01f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76d015dfc0 0x7f76d01f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76d00b1da0 con 0x7f76d00b1910 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f76d00740a0 con 0x7f76d015dfc0 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d7800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f76d00b1f20 con 0x7f76d00ca5c0 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55382/0 (socket says 172.21.15.31:55382) 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.867+0000 7f76d4e00640 1 -- 172.21.15.31:0/1480141060 learned_addr learned my addr 172.21.15.31:0/1480141060 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76c7e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d01f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d5800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76d015dfc0 0x7f76d01f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 -- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 msgr2=0x7f76d01f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d01f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 -- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76d015dfc0 msgr2=0x7f76d01f84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76d015dfc0 0x7f76d01f84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 -- 172.21.15.31:0/1480141060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d009b670 con 0x7f76d00b1910 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76c7e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d01f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d4e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c430 tx=0x7f76c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c0017070 con 0x7f76d00b1910 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c00053e0 con 0x7f76d00b1910 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c0003850 con 0x7f76d00b1910 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.868+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d009b950 con 0x7f76d00b1910 2024-09-13T08:44:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.869+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d009bf10 con 0x7f76d00b1910 2024-09-13T08:44:51.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.870+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d00cc300 con 0x7f76d00b1910 2024-09-13T08:44:51.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.871+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f76c001a070 con 0x7f76d00b1910 2024-09-13T08:44:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.872+0000 7f76cd600640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 0x7f76ac07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.872+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4901+0+0 (secure 0 0 0) 0x7f76c009a680 con 0x7f76d00b1910 2024-09-13T08:44:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.872+0000 7f76c7e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 0x7f76ac07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.873+0000 7f76c7e00640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 0x7f76ac07b340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76b8034000 tx=0x7f76b8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:51.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:51.876+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f76c0062580 con 0x7f76d00b1910 2024-09-13T08:44:52.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.074+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f76d00cd070 con 0x7f76d00b1910 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.075+0000 7f76cd600640 1 -- 172.21.15.31:0/1480141060 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) ==== 84+0+65 (secure 0 0 0) 0x7f76c0061e80 con 0x7f76d00b1910 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:44:52.076 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:44:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 msgr2=0x7f76ac07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 0x7f76ac07b340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76b8034000 tx=0x7f76b8004730 comp rx=0 tx=0).stop 2024-09-13T08:44:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 msgr2=0x7f76d009d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c430 tx=0x7f76c000c8f0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 shutdown_connections 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f76ac078e80 0x7f76ac07b340 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76d015dfc0 0x7f76d01f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f76d00ca5c0 0x7f76d01f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 --2- 172.21.15.31:0/1480141060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76d00b1910 0x7f76d009d040 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 >> 172.21.15.31:0/1480141060 conn(0x7f76d00cc910 msgr2=0x7f76d00b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 shutdown_connections 2024-09-13T08:44:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.077+0000 7f76d7800640 1 -- 172.21.15.31:0/1480141060 wait complete. 2024-09-13T08:44:52.094 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:44:52.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.259+0000 7f03ae400640 1 -- 172.21.15.31:0/3867073363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a815bfc0 msgr2=0x7f03a8072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.259+0000 7f03ae400640 1 --2- 172.21.15.31:0/3867073363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a815bfc0 0x7f03a8072560 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0398009830 tx=0x7f039802a540 comp rx=0 tx=0).stop 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.260+0000 7f03ae400640 1 -- 172.21.15.31:0/3867073363 shutdown_connections 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.260+0000 7f03ae400640 1 --2- 172.21.15.31:0/3867073363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a815bfc0 0x7f03a8072560 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.260+0000 7f03ae400640 1 -- 172.21.15.31:0/3867073363 >> 172.21.15.31:0/3867073363 conn(0x7f03a80cfbb0 msgr2=0x7f03a80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.260+0000 7f03ae400640 1 -- 172.21.15.31:0/3867073363 shutdown_connections 2024-09-13T08:44:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.260+0000 7f03ae400640 1 -- 172.21.15.31:0/3867073363 wait complete. 2024-09-13T08:44:52.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.261+0000 7f03ae400640 1 Processor -- start 2024-09-13T08:44:52.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.261+0000 7f03ae400640 1 -- start start 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 0x7f03a8152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 0x7f03a8157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03a80cbfd0 con 0x7f03a806fa60 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03a80cbe50 con 0x7f03a80ceab0 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.262+0000 7f03ae400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03a80b30f0 con 0x7f03a815bfc0 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55396/0 (socket says 172.21.15.31:55396) 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 -- 172.21.15.31:0/240965778 learned_addr learned my addr 172.21.15.31:0/240965778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:52.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7e00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 0x7f03a8157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f039ea00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 0x7f03a8152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 -- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 msgr2=0x7f03a8152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 0x7f03a8152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 -- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 msgr2=0x7f03a8157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 0x7f03a8157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7400640 1 -- 172.21.15.31:0/240965778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a80c36e0 con 0x7f03a806fa60 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f03a7e00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 0x7f03a8157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.263+0000 7f039ea00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 0x7f03a8152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f03a7400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f039802ab30 tx=0x7f0398003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398005c90 con 0x7f03a806fa60 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0398005020 con 0x7f03a806fa60 2024-09-13T08:44:52.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398005450 con 0x7f03a806fa60 2024-09-13T08:44:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a81f4e30 con 0x7f03a806fa60 2024-09-13T08:44:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.264+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a81f53f0 con 0x7f03a806fa60 2024-09-13T08:44:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.265+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a8072560 con 0x7f03a806fa60 2024-09-13T08:44:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.266+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f03980317c0 con 0x7f03a806fa60 2024-09-13T08:44:52.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.267+0000 7f039fe00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 0x7f038407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:52.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.267+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4901+0+0 (secure 0 0 0) 0x7f03980bbfc0 con 0x7f03a806fa60 2024-09-13T08:44:52.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.267+0000 7f039ea00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 0x7f038407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:52.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.268+0000 7f039ea00640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 0x7f038407b340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0390001710 tx=0x7f0390009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:52.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.271+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f039803f030 con 0x7f03a806fa60 2024-09-13T08:44:52.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:52.468+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f03a80cb310 con 0x7f03a806fa60 2024-09-13T08:44:53.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.233+0000 7f039fe00640 1 -- 172.21.15.31:0/240965778 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) ==== 138+0+0 (secure 0 0 0) 0x7f0398084030 con 0x7f03a806fa60 2024-09-13T08:44:53.234 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:44:53.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.236+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 msgr2=0x7f038407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:53.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.236+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 0x7f038407b340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0390001710 tx=0x7f0390009040 comp rx=0 tx=0).stop 2024-09-13T08:44:53.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.236+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 msgr2=0x7f03a814f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:53.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.236+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f039802ab30 tx=0x7f0398003c10 comp rx=0 tx=0).stop 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 shutdown_connections 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0384078e80 0x7f038407b340 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03a80ceab0 0x7f03a8157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03a815bfc0 0x7f03a8152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 --2- 172.21.15.31:0/240965778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03a806fa60 0x7f03a814f590 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.237+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 >> 172.21.15.31:0/240965778 conn(0x7f03a80cfbb0 msgr2=0x7f03a80b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:53.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.238+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 shutdown_connections 2024-09-13T08:44:53.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.238+0000 7f03ae400640 1 -- 172.21.15.31:0/240965778 wait complete. 2024-09-13T08:44:53.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:44:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 -- 172.21.15.31:0/673636692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b300cbe50 msgr2=0x7f0b300b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 --2- 172.21.15.31:0/673636692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b300cbe50 0x7f0b300b29e0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005fd0 tx=0x7f0b2802ab40 comp rx=0 tx=0).stop 2024-09-13T08:44:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 -- 172.21.15.31:0/673636692 shutdown_connections 2024-09-13T08:44:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 --2- 172.21.15.31:0/673636692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b300cbe50 0x7f0b300b29e0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 -- 172.21.15.31:0/673636692 >> 172.21.15.31:0/673636692 conn(0x7f0b300cfbb0 msgr2=0x7f0b300cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:53.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 -- 172.21.15.31:0/673636692 shutdown_connections 2024-09-13T08:44:53.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.434+0000 7f0b35e00640 1 -- 172.21.15.31:0/673636692 wait complete. 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.435+0000 7f0b35e00640 1 Processor -- start 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 -- start start 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 0x7f0b30146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3015bfc0 0x7f0b300c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b300b30a0 con 0x7f0b30072560 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b300b2f20 con 0x7f0b3015bfc0 2024-09-13T08:44:53.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.436+0000 7f0b35e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0b300b3220 con 0x7f0b300cbe50 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55408/0 (socket says 172.21.15.31:55408) 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 -- 172.21.15.31:0/263242526 learned_addr learned my addr 172.21.15.31:0/263242526 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2f400640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3015bfc0 0x7f0b300c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b26000640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 0x7f0b30146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 -- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 msgr2=0x7f0b30146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 0x7f0b30146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 -- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3015bfc0 msgr2=0x7f0b300c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3015bfc0 0x7f0b300c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b2ea00640 1 -- 172.21.15.31:0/263242526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b300c6540 con 0x7f0b30072560 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.437+0000 7f0b26000640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 0x7f0b30146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:53.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b2ea00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c400 tx=0x7f0b2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20017070 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b200053c0 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b300c8130 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20003850 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.438+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3014f430 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.439+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30074610 con 0x7f0b30072560 2024-09-13T08:44:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.441+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f0b2001a070 con 0x7f0b30072560 2024-09-13T08:44:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.441+0000 7f0b27e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 0x7f0b0c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.442+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5271+0+0 (secure 0 0 0) 0x7f0b2009aa30 con 0x7f0b30072560 2024-09-13T08:44:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.442+0000 7f0b26000640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 0x7f0b0c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.442+0000 7f0b26000640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 0x7f0b0c07b340 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b300cb630 tx=0x7f0b18009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.445+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0b30074610 con 0x7f0b30072560 2024-09-13T08:44:53.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:53.643+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0b300b29e0 con 0x7f0b30072560 2024-09-13T08:44:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.236+0000 7f0b27e00640 1 -- 172.21.15.31:0/263242526 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) ==== 170+0+0 (secure 0 0 0) 0x7f0b20021020 con 0x7f0b30072560 2024-09-13T08:44:54.315 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:44:54.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 msgr2=0x7f0b0c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 0x7f0b0c07b340 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b300cb630 tx=0x7f0b18009040 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 msgr2=0x7f0b3015c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c400 tx=0x7f0b2000c8c0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 shutdown_connections 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0b0c078e80 0x7f0b0c07b340 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3015bfc0 0x7f0b300c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0b300cbe50 0x7f0b30146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 --2- 172.21.15.31:0/263242526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b30072560 0x7f0b3015c4f0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 >> 172.21.15.31:0/263242526 conn(0x7f0b300cfbb0 msgr2=0x7f0b30158540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 shutdown_connections 2024-09-13T08:44:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.239+0000 7f0b35e00640 1 -- 172.21.15.31:0/263242526 wait complete. 2024-09-13T08:44:54.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 -- 172.21.15.31:0/13490509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c06fa60 msgr2=0x7f574c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 --2- 172.21.15.31:0/13490509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c06fa60 0x7f574c0cb740 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f574c069dd0 tx=0x7f573402a560 comp rx=0 tx=0).stop 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 -- 172.21.15.31:0/13490509 shutdown_connections 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 --2- 172.21.15.31:0/13490509 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fec0 0x7f574c0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 --2- 172.21.15.31:0/13490509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c06fa60 0x7f574c0cb740 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 --2- 172.21.15.31:0/13490509 >> v2:172.21.15.45:3300/0 conn(0x7f574c15bfc0 0x7f574c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 -- 172.21.15.31:0/13490509 >> 172.21.15.31:0/13490509 conn(0x7f574c0cfbb0 msgr2=0x7f574c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.482+0000 7f5753800640 1 -- 172.21.15.31:0/13490509 shutdown_connections 2024-09-13T08:44:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.483+0000 7f5753800640 1 -- 172.21.15.31:0/13490509 wait complete. 2024-09-13T08:44:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.484+0000 7f5753800640 1 Processor -- start 2024-09-13T08:44:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.484+0000 7f5753800640 1 -- start start 2024-09-13T08:44:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fa60 0x7f574c1f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574c15bfc0 0x7f574c0cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f574c0b2b60 con 0x7f574c1562e0 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f574c0b29e0 con 0x7f574c15bfc0 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5753800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f574c0b2ce0 con 0x7f574c06fa60 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.485+0000 7f5751800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55422/0 (socket says 172.21.15.31:55422) 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 -- 172.21.15.31:0/1907629820 learned_addr learned my addr 172.21.15.31:0/1907629820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5750e00640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fa60 0x7f574c1f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5743e00640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574c15bfc0 0x7f574c0cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 -- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fa60 msgr2=0x7f574c1f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fa60 0x7f574c1f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 -- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574c15bfc0 msgr2=0x7f574c0cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574c15bfc0 0x7f574c0cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 -- 172.21.15.31:0/1907629820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c1f4da0 con 0x7f574c1562e0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.486+0000 7f5751800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f574400e980 tx=0x7f574400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.487+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744004d20 con 0x7f574c1562e0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.487+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f574400cb20 con 0x7f574c1562e0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.487+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744005020 con 0x7f574c1562e0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.487+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c1f5080 con 0x7f574c1562e0 2024-09-13T08:44:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.487+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c1f5920 con 0x7f574c1562e0 2024-09-13T08:44:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.488+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574c0b3200 con 0x7f574c1562e0 2024-09-13T08:44:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.489+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f57440051c0 con 0x7f574c1562e0 2024-09-13T08:44:54.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.490+0000 7f5749600640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 0x7f572807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.490+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5653+0+0 (secure 0 0 0) 0x7f574409be30 con 0x7f574c1562e0 2024-09-13T08:44:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.491+0000 7f5750e00640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 0x7f572807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.491+0000 7f5750e00640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 0x7f572807b340 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f573c00bb50 tx=0x7f573c00b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:54.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.495+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f57440a1020 con 0x7f574c1562e0 2024-09-13T08:44:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:54.745+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f574c0cb310 con 0x7f574c1562e0 2024-09-13T08:44:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.259+0000 7f5749600640 1 -- 172.21.15.31:0/1907629820 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v86) ==== 174+0+0 (secure 0 0 0) 0x7f5744016030 con 0x7f574c1562e0 2024-09-13T08:44:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 20 and data pool 21 2024-09-13T08:44:55.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.268+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 msgr2=0x7f572807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.268+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 0x7f572807b340 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f573c00bb50 tx=0x7f573c00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.268+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 msgr2=0x7f574c0c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.268+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f574400e980 tx=0x7f574400ee40 comp rx=0 tx=0).stop 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 shutdown_connections 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5728078e80 0x7f572807b340 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574c1562e0 0x7f574c0c7620 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f574c15bfc0 0x7f574c0cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 --2- 172.21.15.31:0/1907629820 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f574c06fa60 0x7f574c1f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.269+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 >> 172.21.15.31:0/1907629820 conn(0x7f574c0cfbb0 msgr2=0x7f574c0cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:55.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.271+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 shutdown_connections 2024-09-13T08:44:55.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.271+0000 7f5753800640 1 -- 172.21.15.31:0/1907629820 wait complete. 2024-09-13T08:44:55.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 -- 172.21.15.31:0/139320456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3950004d40 msgr2=0x7f39500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 --2- 172.21.15.31:0/139320456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3950004d40 0x7f39500051c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f3964006f90 tx=0x7f3950005830 comp rx=0 tx=0).stop 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f396be00640 1 -- 172.21.15.31:0/139320456 reap_dead start 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 -- 172.21.15.31:0/139320456 shutdown_connections 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 -- 172.21.15.31:0/139320456 >> 172.21.15.31:0/139320456 conn(0x7f396c0cc910 msgr2=0x7f396c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 -- 172.21.15.31:0/139320456 shutdown_connections 2024-09-13T08:44:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.464+0000 7f3972200640 1 -- 172.21.15.31:0/139320456 wait complete. 2024-09-13T08:44:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.466+0000 7f3972200640 1 Processor -- start 2024-09-13T08:44:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.466+0000 7f3972200640 1 -- start start 2024-09-13T08:44:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3950004d40 0x7f396c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f396c1f49e0 0x7f396c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0cb9d0 con 0x7f396c0b0e80 2024-09-13T08:44:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0cb850 con 0x7f3950004d40 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3972200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f396c0cc360 con 0x7f396c1f49e0 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f396be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f396be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45050/0 (socket says 172.21.15.31:45050) 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f396be00640 1 -- 172.21.15.31:0/3244003878 learned_addr learned my addr 172.21.15.31:0/3244003878 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f3962a00640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f396c1f49e0 0x7f396c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.467+0000 7f396b400640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3950004d40 0x7f396c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 -- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f396c1f49e0 msgr2=0x7f396c0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f396c1f49e0 0x7f396c0b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 -- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3950004d40 msgr2=0x7f396c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3950004d40 0x7f396c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 -- 172.21.15.31:0/3244003878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c0b12a0 con 0x7f396c0b0e80 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f396be00640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f3964031040 tx=0x7f3964003bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964001c30 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3964003c20 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39640016f0 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1f5070 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.468+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1f5630 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.470+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c15fe20 con 0x7f396c0b0e80 2024-09-13T08:44:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.471+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f3964035020 con 0x7f396c0b0e80 2024-09-13T08:44:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.472+0000 7f3963e00640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 0x7f394807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.472+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5753+0+0 (secure 0 0 0) 0x7f39640bb2b0 con 0x7f396c0b0e80 2024-09-13T08:44:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.472+0000 7f396b400640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 0x7f394807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.472+0000 7f396b400640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 0x7f394807b370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c009500 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.476+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f396403d070 con 0x7f396c0b0e80 2024-09-13T08:44:55.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.674+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f396c1f4dc0 con 0x7f396c0b0e80 2024-09-13T08:44:55.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.675+0000 7f3963e00640 1 -- 172.21.15.31:0/3244003878 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16771 (secure 0 0 0) 0x7f3964082810 con 0x7f396c0b0e80 2024-09-13T08:44:55.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:55.679 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":138,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:54.747687+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:52.478415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:53.653411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:55.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 msgr2=0x7f394807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 0x7f394807b370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c009500 comp rx=0 tx=0).stop 2024-09-13T08:44:55.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 msgr2=0x7f396c1f8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f3964031040 tx=0x7f3964003bb0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 shutdown_connections 2024-09-13T08:44:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3948078eb0 0x7f394807b370 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c0b0e80 0x7f396c1f8400 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f396c1f49e0 0x7f396c0b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 --2- 172.21.15.31:0/3244003878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3950004d40 0x7f396c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 >> 172.21.15.31:0/3244003878 conn(0x7f396c0cc910 msgr2=0x7f396c15b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:55.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 shutdown_connections 2024-09-13T08:44:55.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.678+0000 7f3972200640 1 -- 172.21.15.31:0/3244003878 wait complete. 2024-09-13T08:44:55.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.856+0000 7ff811600640 1 -- 172.21.15.31:0/3662925164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c0cbc50 msgr2=0x7ff80c0cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.856+0000 7ff811600640 1 --2- 172.21.15.31:0/3662925164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c0cbc50 0x7ff80c0cc0a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff804009be0 tx=0x7ff80402a7e0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.857+0000 7ff811600640 1 -- 172.21.15.31:0/3662925164 shutdown_connections 2024-09-13T08:44:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.857+0000 7ff811600640 1 --2- 172.21.15.31:0/3662925164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c0cbc50 0x7ff80c0cc0a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.857+0000 7ff811600640 1 -- 172.21.15.31:0/3662925164 >> 172.21.15.31:0/3662925164 conn(0x7ff80c0cc910 msgr2=0x7ff80c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.857+0000 7ff811600640 1 -- 172.21.15.31:0/3662925164 shutdown_connections 2024-09-13T08:44:55.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.857+0000 7ff811600640 1 -- 172.21.15.31:0/3662925164 wait complete. 2024-09-13T08:44:55.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.858+0000 7ff811600640 1 Processor -- start 2024-09-13T08:44:55.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.858+0000 7ff811600640 1 -- start start 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff80c0cf850 0x7ff80c1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c15bd70 0x7ff80c0afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1600f0 con 0x7ff80c0cbc50 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0cead0 con 0x7ff80c0cf850 2024-09-13T08:44:55.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff811600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c0b3d20 con 0x7ff80c15bd70 2024-09-13T08:44:55.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.859+0000 7ff80aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45062/0 (socket says 172.21.15.31:45062) 2024-09-13T08:44:55.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 -- 172.21.15.31:0/926161749 learned_addr learned my addr 172.21.15.31:0/926161749 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:55.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80b400640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c15bd70 0x7ff80c0afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff803e00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff80c0cf850 0x7ff80c1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 -- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c15bd70 msgr2=0x7ff80c0afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c15bd70 0x7ff80c0afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 -- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff80c0cf850 msgr2=0x7ff80c1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff80c0cf850 0x7ff80c1523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 -- 172.21.15.31:0/926161749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c0b31a0 con 0x7ff80c0cbc50 2024-09-13T08:44:55.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.860+0000 7ff80aa00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c480 tx=0x7ff7fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.861+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc017070 con 0x7ff80c0cbc50 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.861+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7fc0035f0 con 0x7ff80c0cbc50 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.861+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c0b3480 con 0x7ff80c0cbc50 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.861+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc003a40 con 0x7ff80c0cbc50 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.861+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c0b3a40 con 0x7ff80c0cbc50 2024-09-13T08:44:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.862+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c0cb950 con 0x7ff80c0cbc50 2024-09-13T08:44:55.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.863+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff7fc01a070 con 0x7ff80c0cbc50 2024-09-13T08:44:55.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.864+0000 7ff802a00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 0x7ff7e807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:55.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.864+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5753+0+0 (secure 0 0 0) 0x7ff7fc09a950 con 0x7ff80c0cbc50 2024-09-13T08:44:55.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.864+0000 7ff803e00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 0x7ff7e807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:55.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.865+0000 7ff803e00640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 0x7ff7e807b340 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0ae120 tx=0x7ff7f4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:55.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:55.868+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff7fc061f80 con 0x7ff80c0cbc50 2024-09-13T08:44:56.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.069+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff80c0af120 con 0x7ff80c0cbc50 2024-09-13T08:44:56.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.070+0000 7ff802a00640 1 -- 172.21.15.31:0/926161749 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16771 (secure 0 0 0) 0x7ff7fc0669b0 con 0x7ff80c0cbc50 2024-09-13T08:44:56.072 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":138,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:54.747687+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:52.478415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:53.653411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.072+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 msgr2=0x7ff7e807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.072+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 0x7ff7e807b340 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0ae120 tx=0x7ff7f4009040 comp rx=0 tx=0).stop 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.072+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 msgr2=0x7ff80c15d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.072+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c480 tx=0x7ff7fc00c940 comp rx=0 tx=0).stop 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 shutdown_connections 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff7e8078e80 0x7ff7e807b340 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff80c15bd70 0x7ff80c0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff80c0cf850 0x7ff80c1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 --2- 172.21.15.31:0/926161749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0cbc50 0x7ff80c15d480 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 >> 172.21.15.31:0/926161749 conn(0x7ff80c0cc910 msgr2=0x7ff80c074160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 shutdown_connections 2024-09-13T08:44:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.073+0000 7ff811600640 1 -- 172.21.15.31:0/926161749 wait complete. 2024-09-13T08:44:56.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:44:56.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.254+0000 7ff6f7e00640 1 -- 172.21.15.31:0/4064013143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 msgr2=0x7ff6f80ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:56.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.254+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/4064013143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f80ca020 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80cdd00 tx=0x7ff6e802a5f0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 -- 172.21.15.31:0/4064013143 shutdown_connections 2024-09-13T08:44:56.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/4064013143 >> v2:172.21.15.45:3300/0 conn(0x7ff6f806f970 0x7ff6f80ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/4064013143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f80ca020 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 -- 172.21.15.31:0/4064013143 >> 172.21.15.31:0/4064013143 conn(0x7ff6f80b1520 msgr2=0x7ff6f80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:56.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 -- 172.21.15.31:0/4064013143 shutdown_connections 2024-09-13T08:44:56.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.255+0000 7ff6f7e00640 1 -- 172.21.15.31:0/4064013143 wait complete. 2024-09-13T08:44:56.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.256+0000 7ff6f7e00640 1 Processor -- start 2024-09-13T08:44:56.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.256+0000 7ff6f7e00640 1 -- start start 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6f806f530 0x7ff6f81f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6f806f970 0x7ff6f81ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f815e560 con 0x7ff6f815abf0 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f80ca5e0 con 0x7ff6f806f530 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.257+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f815e6e0 con 0x7ff6f806f970 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45074/0 (socket says 172.21.15.31:45074) 2024-09-13T08:44:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 -- 172.21.15.31:0/2120177994 learned_addr learned my addr 172.21.15.31:0/2120177994 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f6a00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6f806f530 0x7ff6f81f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6efe00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6f806f970 0x7ff6f81ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 -- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6f806f970 msgr2=0x7ff6f81ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6f806f970 0x7ff6f81ee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 -- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6f806f530 msgr2=0x7ff6f81f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6f806f530 0x7ff6f81f5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:56.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 -- 172.21.15.31:0/2120177994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f80c3500 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.258+0000 7ff6f7400640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000eaa0 tx=0x7ff6f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.259+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000cdb0 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.259+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6f0004c10 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.259+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f80c37e0 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.259+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0005020 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.259+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f80c3da0 con 0x7ff6f815abf0 2024-09-13T08:44:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.260+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f8151ad0 con 0x7ff6f815abf0 2024-09-13T08:44:56.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.262+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff6f00051c0 con 0x7ff6f815abf0 2024-09-13T08:44:56.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.263+0000 7ff6eea00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 0x7ff6d407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:56.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.263+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5753+0+0 (secure 0 0 0) 0x7ff6f009bf20 con 0x7ff6f815abf0 2024-09-13T08:44:56.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.263+0000 7ff6f6a00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 0x7ff6d407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:56.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.264+0000 7ff6f6a00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 0x7ff6d407b340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8033000 tx=0x7ff6e8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:56.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.269+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff6f0016030 con 0x7ff6f815abf0 2024-09-13T08:44:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:56.507+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff6f8073490 con 0x7ff6f815abf0 2024-09-13T08:44:57.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.278+0000 7ff6eea00640 1 -- 172.21.15.31:0/2120177994 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) ==== 118+0+0 (secure 0 0 0) 0x7ff6f009d0b0 con 0x7ff6f815abf0 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.281+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 msgr2=0x7ff6d407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.281+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 0x7ff6d407b340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8033000 tx=0x7ff6e8004730 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.281+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 msgr2=0x7ff6f815d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.281+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000eaa0 tx=0x7ff6f000ef60 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 shutdown_connections 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff6d4078e80 0x7ff6d407b340 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6f815abf0 0x7ff6f815d030 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6f806f970 0x7ff6f81ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 --2- 172.21.15.31:0/2120177994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6f806f530 0x7ff6f81f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 >> 172.21.15.31:0/2120177994 conn(0x7ff6f80b1520 msgr2=0x7ff6f80cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 shutdown_connections 2024-09-13T08:44:57.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.282+0000 7ff6f7e00640 1 -- 172.21.15.31:0/2120177994 wait complete. 2024-09-13T08:44:57.303 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:44:57.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 -- 172.21.15.31:0/2575820939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 msgr2=0x7f9d9c0ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 --2- 172.21.15.31:0/2575820939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c0ca8b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f9d94005fd0 tx=0x7f9d9402aa10 comp rx=0 tx=0).stop 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 -- 172.21.15.31:0/2575820939 shutdown_connections 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 --2- 172.21.15.31:0/2575820939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c0ca8b0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 --2- 172.21.15.31:0/2575820939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d9c0cde90 0x7f9d9c0ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 -- 172.21.15.31:0/2575820939 >> 172.21.15.31:0/2575820939 conn(0x7f9d9c0cfbb0 msgr2=0x7f9d9c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 -- 172.21.15.31:0/2575820939 shutdown_connections 2024-09-13T08:44:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.471+0000 7f9da1800640 1 -- 172.21.15.31:0/2575820939 wait complete. 2024-09-13T08:44:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.472+0000 7f9da1800640 1 Processor -- start 2024-09-13T08:44:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.472+0000 7f9da1800640 1 -- start start 2024-09-13T08:44:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 0x7f9d9c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d9c161480 0x7f9d9c0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c0cbfd0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c0cbe50 con 0x7f9d9c0cde90 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.473+0000 7f9da1800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9c0cadf0 con 0x7f9d9c161480 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45078/0 (socket says 172.21.15.31:45078) 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 -- 172.21.15.31:0/602816545 learned_addr learned my addr 172.21.15.31:0/602816545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 -- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d9c161480 msgr2=0x7f9d9c0738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d9c161480 0x7f9d9c0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d92000640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 0x7f9d9c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 -- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 msgr2=0x7f9d9c14f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 0x7f9d9c14f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 -- 172.21.15.31:0/602816545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9c1f6a00 con 0x7f9d9c0cc190 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d92000640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 0x7f9d9c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d9aa00640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c3a0 tx=0x7f9d8c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.474+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c017070 con 0x7f9d9c0cc190 2024-09-13T08:44:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.475+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d8c0053c0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.475+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c003850 con 0x7f9d9c0cc190 2024-09-13T08:44:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.475+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9c1f6ce0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.475+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d9c1f72a0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.476+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d9c0d06a0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.477+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9d8c01a070 con 0x7f9d9c0cc190 2024-09-13T08:44:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.478+0000 7f9d93400640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 0x7f9d7807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.478+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f9d8c09a8d0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.478+0000 7f9d92000640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 0x7f9d7807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.479+0000 7f9d92000640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 0x7f9d7807b340 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c074140 tx=0x7f9d84005e50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:57.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.482+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9d9c0d06a0 con 0x7f9d9c0cc190 2024-09-13T08:44:57.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:57.715+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9d9c0cb2a0 con 0x7f9d9c0cc190 2024-09-13T08:44:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.284+0000 7f9d93400640 1 -- 172.21.15.31:0/602816545 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) ==== 122+0+0 (secure 0 0 0) 0x7f9d8c09a0c0 con 0x7f9d9c0cc190 2024-09-13T08:44:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 msgr2=0x7f9d7807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 0x7f9d7807b340 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c074140 tx=0x7f9d84005e50 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 msgr2=0x7f9d9c1f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c3a0 tx=0x7f9d8c00c860 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 shutdown_connections 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d78078e80 0x7f9d7807b340 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d9c161480 0x7f9d9c0738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d9c0cde90 0x7f9d9c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 --2- 172.21.15.31:0/602816545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d9c0cc190 0x7f9d9c1f7aa0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.287+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 >> 172.21.15.31:0/602816545 conn(0x7f9d9c0cfbb0 msgr2=0x7f9d9c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.288+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 shutdown_connections 2024-09-13T08:44:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.288+0000 7f9da1800640 1 -- 172.21.15.31:0/602816545 wait complete. 2024-09-13T08:44:58.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:44:58.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:44:58.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 -- 172.21.15.31:0/3016143273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 msgr2=0x7ffbd0005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/3016143273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbd0005d90 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400bb20 tx=0x7ffbd0006940 comp rx=0 tx=0).stop 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbeb400640 1 -- 172.21.15.31:0/3016143273 reap_dead start 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 -- 172.21.15.31:0/3016143273 shutdown_connections 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 -- 172.21.15.31:0/3016143273 >> 172.21.15.31:0/3016143273 conn(0x7ffbec0cc910 msgr2=0x7ffbec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 -- 172.21.15.31:0/3016143273 shutdown_connections 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.470+0000 7ffbf1e00640 1 -- 172.21.15.31:0/3016143273 wait complete. 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.472+0000 7ffbf1e00640 1 Processor -- start 2024-09-13T08:44:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.472+0000 7ffbf1e00640 1 -- start start 2024-09-13T08:44:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffbec1f5d30 0x7ffbec1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbec0c4ba0 0x7ffbec15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec0cf760 con 0x7ffbd0005910 2024-09-13T08:44:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec0cdfa0 con 0x7ffbec0c4ba0 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbf1e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ffbec0ca4e0 con 0x7ffbec1f5d30 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45092/0 (socket says 172.21.15.31:45092) 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 -- 172.21.15.31:0/2298529960 learned_addr learned my addr 172.21.15.31:0/2298529960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 -- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffbec1f5d30 msgr2=0x7ffbec1f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffbec1f5d30 0x7ffbec1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 -- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbec0c4ba0 msgr2=0x7ffbec15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbec0c4ba0 0x7ffbec15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.473+0000 7ffbeaa00640 1 -- 172.21.15.31:0/2298529960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbec0c37f0 con 0x7ffbd0005910 2024-09-13T08:44:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbeaa00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00e5c0 tx=0x7ffbdc00ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc015070 con 0x7ffbd0005910 2024-09-13T08:44:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffbdc00c9a0 con 0x7ffbd0005910 2024-09-13T08:44:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbec1f62a0 con 0x7ffbd0005910 2024-09-13T08:44:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00cdf0 con 0x7ffbd0005910 2024-09-13T08:44:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.474+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbec1f6860 con 0x7ffbd0005910 2024-09-13T08:44:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.475+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbec050f70 con 0x7ffbd0005910 2024-09-13T08:44:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.476+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ffbdc018070 con 0x7ffbd0005910 2024-09-13T08:44:58.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.477+0000 7ffbe3e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 0x7ffbc807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.477+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7ffbdc09b8f0 con 0x7ffbd0005910 2024-09-13T08:44:58.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.477+0000 7ffbe2000640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 0x7ffbc807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.478+0000 7ffbe2000640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 0x7ffbc807b370 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4036000 tx=0x7ffbd4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.481+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ffbec050f70 con 0x7ffbd0005910 2024-09-13T08:44:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.714+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbec0c39c0 con 0x7ffbd0005910 2024-09-13T08:44:58.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.715+0000 7ffbe3e00640 1 -- 172.21.15.31:0/2298529960 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5221 (secure 0 0 0) 0x7ffbdc063530 con 0x7ffbd0005910 2024-09-13T08:44:58.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:58.717 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"btime":"2024-09-13T08:44:58:284276+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:44:58.717 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2024-09-13T08:44:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 msgr2=0x7ffbc807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 0x7ffbc807b370 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4036000 tx=0x7ffbd4004730 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 msgr2=0x7ffbec0c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00e5c0 tx=0x7ffbdc00ea80 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 shutdown_connections 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ffbc8078eb0 0x7ffbc807b370 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbec0c4ba0 0x7ffbec15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ffbec1f5d30 0x7ffbec1f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 --2- 172.21.15.31:0/2298529960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbd0005910 0x7ffbec0c5eb0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 >> 172.21.15.31:0/2298529960 conn(0x7ffbec0cc910 msgr2=0x7ffbec0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 shutdown_connections 2024-09-13T08:44:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.717+0000 7ffbf1e00640 1 -- 172.21.15.31:0/2298529960 wait complete. 2024-09-13T08:44:58.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.894+0000 7f5334a00640 1 -- 172.21.15.31:0/2774435182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5314004800 msgr2=0x7f5314004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.894+0000 7f5334a00640 1 --2- 172.21.15.31:0/2774435182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5314004800 0x7f5314004c80 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5318009b80 tx=0x7f5314005830 comp rx=0 tx=0).stop 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.894+0000 7f532f200640 1 -- 172.21.15.31:0/2774435182 reap_dead start 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.894+0000 7f5334a00640 1 -- 172.21.15.31:0/2774435182 shutdown_connections 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.895+0000 7f5334a00640 1 -- 172.21.15.31:0/2774435182 >> 172.21.15.31:0/2774435182 conn(0x7f533006cb00 msgr2=0x7f533006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.895+0000 7f5334a00640 1 -- 172.21.15.31:0/2774435182 shutdown_connections 2024-09-13T08:44:58.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.895+0000 7f5334a00640 1 -- 172.21.15.31:0/2774435182 wait complete. 2024-09-13T08:44:58.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.896+0000 7f5334a00640 1 Processor -- start 2024-09-13T08:44:58.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.896+0000 7f5334a00640 1 -- start start 2024-09-13T08:44:58.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5314004800 0x7f53300b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5330151270 0x7f53300b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5330152680 con 0x7f53300cbb40 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53300703b0 con 0x7f5314004800 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f5334a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5330152800 con 0x7f5330151270 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f532f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f532f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45104/0 (socket says 172.21.15.31:45104) 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.897+0000 7f532f200640 1 -- 172.21.15.31:0/2203041508 learned_addr learned my addr 172.21.15.31:0/2203041508 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532e800640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5314004800 0x7f53300b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f5325e00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5330151270 0x7f53300b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 -- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5330151270 msgr2=0x7f53300b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5330151270 0x7f53300b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 -- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5314004800 msgr2=0x7f53300b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5314004800 0x7f53300b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:58.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 -- 172.21.15.31:0/2203041508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53301f7d90 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f532f200640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f5328005fd0 tx=0x7f5328010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328003970 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5328015070 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328004030 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.898+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53301f8070 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.899+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53300b13f0 con 0x7f53300cbb40 2024-09-13T08:44:58.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.900+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f533006f9b0 con 0x7f53300cbb40 2024-09-13T08:44:58.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.901+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f53280041d0 con 0x7f53300cbb40 2024-09-13T08:44:58.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.902+0000 7f5327400640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 0x7f530c07b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:58.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.902+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f532809bca0 con 0x7f53300cbb40 2024-09-13T08:44:58.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.902+0000 7f532e800640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 0x7f530c07b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:58.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.903+0000 7f532e800640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 0x7f530c07b370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5320001eb0 tx=0x7f5320006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:58.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:58.906+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5328063200 con 0x7f53300cbb40 2024-09-13T08:44:59.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.103+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f533006f190 con 0x7f53300cbb40 2024-09-13T08:44:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.104+0000 7f5327400640 1 -- 172.21.15.31:0/2203041508 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16758 (secure 0 0 0) 0x7f5328067c30 con 0x7f53300cbb40 2024-09-13T08:44:59.106 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:59.106 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":139,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:56.260431+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:52.478415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:53.653411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:44:59.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 msgr2=0x7f530c07b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 0x7f530c07b370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5320001eb0 tx=0x7f5320006040 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 msgr2=0x7f53300ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f5328005fd0 tx=0x7f5328010f40 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 shutdown_connections 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f530c078eb0 0x7f530c07b370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53300cbb40 0x7f53300ca0f0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5330151270 0x7f53300b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 --2- 172.21.15.31:0/2203041508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5314004800 0x7f53300b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 >> 172.21.15.31:0/2203041508 conn(0x7f533006cb00 msgr2=0x7f53300c7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 shutdown_connections 2024-09-13T08:44:59.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.106+0000 7f5334a00640 1 -- 172.21.15.31:0/2203041508 wait complete. 2024-09-13T08:44:59.124 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:44:59.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.284+0000 7ff63e000640 1 -- 172.21.15.31:0/860931918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c003a50 msgr2=0x7ff61c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.284+0000 7ff63e000640 1 --2- 172.21.15.31:0/860931918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c003a50 0x7ff61c003e70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff628006f80 tx=0x7ff61c005830 comp rx=0 tx=0).stop 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.284+0000 7ff637e00640 1 -- 172.21.15.31:0/860931918 reap_dead start 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.284+0000 7ff63e000640 1 -- 172.21.15.31:0/860931918 shutdown_connections 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.284+0000 7ff63e000640 1 -- 172.21.15.31:0/860931918 >> 172.21.15.31:0/860931918 conn(0x7ff6380cc910 msgr2=0x7ff6380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.285+0000 7ff63e000640 1 -- 172.21.15.31:0/860931918 shutdown_connections 2024-09-13T08:44:59.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.285+0000 7ff63e000640 1 -- 172.21.15.31:0/860931918 wait complete. 2024-09-13T08:44:59.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.286+0000 7ff63e000640 1 Processor -- start 2024-09-13T08:44:59.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.286+0000 7ff63e000640 1 -- start start 2024-09-13T08:44:59.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.287+0000 7ff63e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff61c003a50 0x7ff6380acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.287+0000 7ff63e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff63e000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6380b2160 0x7ff63814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff63e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff63815bcb0 con 0x7ff63815da70 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff63e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff63815bb30 con 0x7ff61c003a50 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff63e000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff63815be30 con 0x7ff6380b2160 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45114/0 (socket says 172.21.15.31:45114) 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 -- 172.21.15.31:0/3193921795 learned_addr learned my addr 172.21.15.31:0/3193921795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff637400640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff61c003a50 0x7ff6380acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff637e00640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6380b2160 0x7ff63814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 -- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6380b2160 msgr2=0x7ff63814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6380b2160 0x7ff63814e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 -- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff61c003a50 msgr2=0x7ff6380acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff61c003a50 0x7ff6380acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.288+0000 7ff62ea00640 1 -- 172.21.15.31:0/3193921795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63809dce0 con 0x7ff63815da70 2024-09-13T08:44:59.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff62ea00640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff620011ab0 tx=0x7ff620011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:59.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62001e070 con 0x7ff63815da70 2024-09-13T08:44:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff62000fe70 con 0x7ff63815da70 2024-09-13T08:44:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63815cf90 con 0x7ff63815da70 2024-09-13T08:44:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6200042b0 con 0x7ff63815da70 2024-09-13T08:44:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.289+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63815d550 con 0x7ff63815da70 2024-09-13T08:44:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.290+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63806fa60 con 0x7ff63815da70 2024-09-13T08:44:59.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.293+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff620021070 con 0x7ff63815da70 2024-09-13T08:44:59.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.295+0000 7ff62fe00640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 0x7ff61407b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.295+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7ff62009c690 con 0x7ff63815da70 2024-09-13T08:44:59.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.295+0000 7ff637400640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 0x7ff61407b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.296+0000 7ff637400640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 0x7ff61407b370 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff628006f80 tx=0x7ff628001930 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:59.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.298+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff620063dc0 con 0x7ff63815da70 2024-09-13T08:44:59.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.492+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff638070370 con 0x7ff614078eb0 2024-09-13T08:44:59.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.496+0000 7ff62fe00640 1 -- 172.21.15.31:0/3193921795 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252470 (secure 0 0 0) 0x7ff638070370 con 0x7ff614078eb0 2024-09-13T08:44:59.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:59.502 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:44:59.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 msgr2=0x7ff61407b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 0x7ff61407b370 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff628006f80 tx=0x7ff628001930 comp rx=0 tx=0).stop 2024-09-13T08:44:59.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 msgr2=0x7ff6380b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff620011ab0 tx=0x7ff620011f70 comp rx=0 tx=0).stop 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 shutdown_connections 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff614078eb0 0x7ff61407b370 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff6380b2160 0x7ff63814e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63815da70 0x7ff6380b1c20 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 --2- 172.21.15.31:0/3193921795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff61c003a50 0x7ff6380acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 >> 172.21.15.31:0/3193921795 conn(0x7ff6380cc910 msgr2=0x7ff638073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 shutdown_connections 2024-09-13T08:44:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.500+0000 7ff63e000640 1 -- 172.21.15.31:0/3193921795 wait complete. 2024-09-13T08:44:59.518 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-13T08:44:58.998048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280868,"kb_used_data":14712,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693852,"statfs":{"total":773094113280,"available":772806504448,"internally_reserved":0,"allocated":15065088,"data_stored":7304596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.761713"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275654+0000","last_change":"2024-09-13T08:44:55.266523+0000","last_active":"2024-09-13T08:44:56.275654+0000","last_peered":"2024-09-13T08:44:56.275654+0000","last_clean":"2024-09-13T08:44:56.275654+0000","last_became_active":"2024-09-13T08:44:55.266261+0000","last_became_peered":"2024-09-13T08:44:55.266261+0000","last_unstale":"2024-09-13T08:44:56.275654+0000","last_undegraded":"2024-09-13T08:44:56.275654+0000","last_fullsized":"2024-09-13T08:44:56.275654+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:52.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275619+0000","last_change":"2024-09-13T08:44:55.271585+0000","last_active":"2024-09-13T08:44:56.275619+0000","last_peered":"2024-09-13T08:44:56.275619+0000","last_clean":"2024-09-13T08:44:56.275619+0000","last_became_active":"2024-09-13T08:44:55.271296+0000","last_became_peered":"2024-09-13T08:44:55.271296+0000","last_unstale":"2024-09-13T08:44:56.275619+0000","last_undegraded":"2024-09-13T08:44:56.275619+0000","last_fullsized":"2024-09-13T08:44:56.275619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:35.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490609+0000","last_change":"2024-09-13T08:44:55.271212+0000","last_active":"2024-09-13T08:44:56.490609+0000","last_peered":"2024-09-13T08:44:56.490609+0000","last_clean":"2024-09-13T08:44:56.490609+0000","last_became_active":"2024-09-13T08:44:55.269819+0000","last_became_peered":"2024-09-13T08:44:55.269819+0000","last_unstale":"2024-09-13T08:44:56.490609+0000","last_undegraded":"2024-09-13T08:44:56.490609+0000","last_fullsized":"2024-09-13T08:44:56.490609+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:31:43.108217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291262+0000","last_change":"2024-09-13T08:44:55.276181+0000","last_active":"2024-09-13T08:44:56.291262+0000","last_peered":"2024-09-13T08:44:56.291262+0000","last_clean":"2024-09-13T08:44:56.291262+0000","last_became_active":"2024-09-13T08:44:55.275890+0000","last_became_peered":"2024-09-13T08:44:55.275890+0000","last_unstale":"2024-09-13T08:44:56.291262+0000","last_undegraded":"2024-09-13T08:44:56.291262+0000","last_fullsized":"2024-09-13T08:44:56.291262+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:54.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274336+0000","last_change":"2024-09-13T08:44:55.281453+0000","last_active":"2024-09-13T08:44:56.274336+0000","last_peered":"2024-09-13T08:44:56.274336+0000","last_clean":"2024-09-13T08:44:56.274336+0000","last_became_active":"2024-09-13T08:44:55.281180+0000","last_became_peered":"2024-09-13T08:44:55.281180+0000","last_unstale":"2024-09-13T08:44:56.274336+0000","last_undegraded":"2024-09-13T08:44:56.274336+0000","last_fullsized":"2024-09-13T08:44:56.274336+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:20.981313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490942+0000","last_change":"2024-09-13T08:44:55.272511+0000","last_active":"2024-09-13T08:44:56.490942+0000","last_peered":"2024-09-13T08:44:56.490942+0000","last_clean":"2024-09-13T08:44:56.490942+0000","last_became_active":"2024-09-13T08:44:55.272193+0000","last_became_peered":"2024-09-13T08:44:55.272193+0000","last_unstale":"2024-09-13T08:44:56.490942+0000","last_undegraded":"2024-09-13T08:44:56.490942+0000","last_fullsized":"2024-09-13T08:44:56.490942+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:50.855940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491207+0000","last_change":"2024-09-13T08:44:55.275891+0000","last_active":"2024-09-13T08:44:56.491207+0000","last_peered":"2024-09-13T08:44:56.491207+0000","last_clean":"2024-09-13T08:44:56.491207+0000","last_became_active":"2024-09-13T08:44:55.275561+0000","last_became_peered":"2024-09-13T08:44:55.275561+0000","last_unstale":"2024-09-13T08:44:56.491207+0000","last_undegraded":"2024-09-13T08:44:56.491207+0000","last_fullsized":"2024-09-13T08:44:56.491207+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.773464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274552+0000","last_change":"2024-09-13T08:44:55.279852+0000","last_active":"2024-09-13T08:44:56.274552+0000","last_peered":"2024-09-13T08:44:56.274552+0000","last_clean":"2024-09-13T08:44:56.274552+0000","last_became_active":"2024-09-13T08:44:55.279589+0000","last_became_peered":"2024-09-13T08:44:55.279589+0000","last_unstale":"2024-09-13T08:44:56.274552+0000","last_undegraded":"2024-09-13T08:44:56.274552+0000","last_fullsized":"2024-09-13T08:44:56.274552+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:08.147436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491619+0000","last_change":"2024-09-13T08:44:55.269462+0000","last_active":"2024-09-13T08:44:56.491619+0000","last_peered":"2024-09-13T08:44:56.491619+0000","last_clean":"2024-09-13T08:44:56.491619+0000","last_became_active":"2024-09-13T08:44:55.268887+0000","last_became_peered":"2024-09-13T08:44:55.268887+0000","last_unstale":"2024-09-13T08:44:56.491619+0000","last_undegraded":"2024-09-13T08:44:56.491619+0000","last_fullsized":"2024-09-13T08:44:56.491619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:03.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490630+0000","last_change":"2024-09-13T08:44:55.275559+0000","last_active":"2024-09-13T08:44:56.490630+0000","last_peered":"2024-09-13T08:44:56.490630+0000","last_clean":"2024-09-13T08:44:56.490630+0000","last_became_active":"2024-09-13T08:44:55.275309+0000","last_became_peered":"2024-09-13T08:44:55.275309+0000","last_unstale":"2024-09-13T08:44:56.490630+0000","last_undegraded":"2024-09-13T08:44:56.490630+0000","last_fullsized":"2024-09-13T08:44:56.490630+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:16.375360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491656+0000","last_change":"2024-09-13T08:44:55.277621+0000","last_active":"2024-09-13T08:44:56.491656+0000","last_peered":"2024-09-13T08:44:56.491656+0000","last_clean":"2024-09-13T08:44:56.491656+0000","last_became_active":"2024-09-13T08:44:55.277320+0000","last_became_peered":"2024-09-13T08:44:55.277320+0000","last_unstale":"2024-09-13T08:44:56.491656+0000","last_undegraded":"2024-09-13T08:44:56.491656+0000","last_fullsized":"2024-09-13T08:44:56.491656+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:05.337043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491423+0000","last_change":"2024-09-13T08:44:55.283842+0000","last_active":"2024-09-13T08:44:56.491423+0000","last_peered":"2024-09-13T08:44:56.491423+0000","last_clean":"2024-09-13T08:44:56.491423+0000","last_became_active":"2024-09-13T08:44:55.283530+0000","last_became_peered":"2024-09-13T08:44:55.283530+0000","last_unstale":"2024-09-13T08:44:56.491423+0000","last_undegraded":"2024-09-13T08:44:56.491423+0000","last_fullsized":"2024-09-13T08:44:56.491423+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:48.243969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274280+0000","last_change":"2024-09-13T08:44:55.281165+0000","last_active":"2024-09-13T08:44:56.274280+0000","last_peered":"2024-09-13T08:44:56.274280+0000","last_clean":"2024-09-13T08:44:56.274280+0000","last_became_active":"2024-09-13T08:44:55.280881+0000","last_became_peered":"2024-09-13T08:44:55.280881+0000","last_unstale":"2024-09-13T08:44:56.274280+0000","last_undegraded":"2024-09-13T08:44:56.274280+0000","last_fullsized":"2024-09-13T08:44:56.274280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:26.280541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490939+0000","last_change":"2024-09-13T08:44:55.278452+0000","last_active":"2024-09-13T08:44:56.490939+0000","last_peered":"2024-09-13T08:44:56.490939+0000","last_clean":"2024-09-13T08:44:56.490939+0000","last_became_active":"2024-09-13T08:44:55.277773+0000","last_became_peered":"2024-09-13T08:44:55.277773+0000","last_unstale":"2024-09-13T08:44:56.490939+0000","last_undegraded":"2024-09-13T08:44:56.490939+0000","last_fullsized":"2024-09-13T08:44:56.490939+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:00.979830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273962+0000","last_change":"2024-09-13T08:44:55.276406+0000","last_active":"2024-09-13T08:44:56.273962+0000","last_peered":"2024-09-13T08:44:56.273962+0000","last_clean":"2024-09-13T08:44:56.273962+0000","last_became_active":"2024-09-13T08:44:55.275942+0000","last_became_peered":"2024-09-13T08:44:55.275942+0000","last_unstale":"2024-09-13T08:44:56.273962+0000","last_undegraded":"2024-09-13T08:44:56.273962+0000","last_fullsized":"2024-09-13T08:44:56.273962+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:34.927349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491121+0000","last_change":"2024-09-13T08:44:55.269060+0000","last_active":"2024-09-13T08:44:56.491121+0000","last_peered":"2024-09-13T08:44:56.491121+0000","last_clean":"2024-09-13T08:44:56.491121+0000","last_became_active":"2024-09-13T08:44:55.268768+0000","last_became_peered":"2024-09-13T08:44:55.268768+0000","last_unstale":"2024-09-13T08:44:56.491121+0000","last_undegraded":"2024-09-13T08:44:56.491121+0000","last_fullsized":"2024-09-13T08:44:56.491121+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:06.221239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490791+0000","last_change":"2024-09-13T08:44:55.265806+0000","last_active":"2024-09-13T08:44:56.490791+0000","last_peered":"2024-09-13T08:44:56.490791+0000","last_clean":"2024-09-13T08:44:56.490791+0000","last_became_active":"2024-09-13T08:44:55.265539+0000","last_became_peered":"2024-09-13T08:44:55.265539+0000","last_unstale":"2024-09-13T08:44:56.490791+0000","last_undegraded":"2024-09-13T08:44:56.490791+0000","last_fullsized":"2024-09-13T08:44:56.490791+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:58.558715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274078+0000","last_change":"2024-09-13T08:44:55.276617+0000","last_active":"2024-09-13T08:44:56.274078+0000","last_peered":"2024-09-13T08:44:56.274078+0000","last_clean":"2024-09-13T08:44:56.274078+0000","last_became_active":"2024-09-13T08:44:55.276113+0000","last_became_peered":"2024-09-13T08:44:55.276113+0000","last_unstale":"2024-09-13T08:44:56.274078+0000","last_undegraded":"2024-09-13T08:44:56.274078+0000","last_fullsized":"2024-09-13T08:44:56.274078+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:31:23.862370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275616+0000","last_change":"2024-09-13T08:44:55.271898+0000","last_active":"2024-09-13T08:44:56.275616+0000","last_peered":"2024-09-13T08:44:56.275616+0000","last_clean":"2024-09-13T08:44:56.275616+0000","last_became_active":"2024-09-13T08:44:55.270304+0000","last_became_peered":"2024-09-13T08:44:55.270304+0000","last_unstale":"2024-09-13T08:44:56.275616+0000","last_undegraded":"2024-09-13T08:44:56.275616+0000","last_fullsized":"2024-09-13T08:44:56.275616+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:23.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275343+0000","last_change":"2024-09-13T08:44:55.266185+0000","last_active":"2024-09-13T08:44:56.275343+0000","last_peered":"2024-09-13T08:44:56.275343+0000","last_clean":"2024-09-13T08:44:56.275343+0000","last_became_active":"2024-09-13T08:44:55.265585+0000","last_became_peered":"2024-09-13T08:44:55.265585+0000","last_unstale":"2024-09-13T08:44:56.275343+0000","last_undegraded":"2024-09-13T08:44:56.275343+0000","last_fullsized":"2024-09-13T08:44:56.275343+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:12.967100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490206+0000","last_change":"2024-09-13T08:44:55.265991+0000","last_active":"2024-09-13T08:44:56.490206+0000","last_peered":"2024-09-13T08:44:56.490206+0000","last_clean":"2024-09-13T08:44:56.490206+0000","last_became_active":"2024-09-13T08:44:55.265718+0000","last_became_peered":"2024-09-13T08:44:55.265718+0000","last_unstale":"2024-09-13T08:44:56.490206+0000","last_undegraded":"2024-09-13T08:44:56.490206+0000","last_fullsized":"2024-09-13T08:44:56.490206+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:15.970484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491321+0000","last_change":"2024-09-13T08:44:55.278743+0000","last_active":"2024-09-13T08:44:56.491321+0000","last_peered":"2024-09-13T08:44:56.491321+0000","last_clean":"2024-09-13T08:44:56.491321+0000","last_became_active":"2024-09-13T08:44:55.278150+0000","last_became_peered":"2024-09-13T08:44:55.278150+0000","last_unstale":"2024-09-13T08:44:56.491321+0000","last_undegraded":"2024-09-13T08:44:56.491321+0000","last_fullsized":"2024-09-13T08:44:56.491321+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:09:39.633701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290845+0000","last_change":"2024-09-13T08:44:55.277149+0000","last_active":"2024-09-13T08:44:56.290845+0000","last_peered":"2024-09-13T08:44:56.290845+0000","last_clean":"2024-09-13T08:44:56.290845+0000","last_became_active":"2024-09-13T08:44:55.272353+0000","last_became_peered":"2024-09-13T08:44:55.272353+0000","last_unstale":"2024-09-13T08:44:56.290845+0000","last_undegraded":"2024-09-13T08:44:56.290845+0000","last_fullsized":"2024-09-13T08:44:56.290845+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:15.506354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273975+0000","last_change":"2024-09-13T08:44:55.287273+0000","last_active":"2024-09-13T08:44:56.273975+0000","last_peered":"2024-09-13T08:44:56.273975+0000","last_clean":"2024-09-13T08:44:56.273975+0000","last_became_active":"2024-09-13T08:44:55.287015+0000","last_became_peered":"2024-09-13T08:44:55.287015+0000","last_unstale":"2024-09-13T08:44:56.273975+0000","last_undegraded":"2024-09-13T08:44:56.273975+0000","last_fullsized":"2024-09-13T08:44:56.273975+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:02:22.270131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276133+0000","last_change":"2024-09-13T08:44:55.279126+0000","last_active":"2024-09-13T08:44:56.276133+0000","last_peered":"2024-09-13T08:44:56.276133+0000","last_clean":"2024-09-13T08:44:56.276133+0000","last_became_active":"2024-09-13T08:44:55.277830+0000","last_became_peered":"2024-09-13T08:44:55.277830+0000","last_unstale":"2024-09-13T08:44:56.276133+0000","last_undegraded":"2024-09-13T08:44:56.276133+0000","last_fullsized":"2024-09-13T08:44:56.276133+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:19:05.504717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275710+0000","last_change":"2024-09-13T08:44:55.267775+0000","last_active":"2024-09-13T08:44:56.275710+0000","last_peered":"2024-09-13T08:44:56.275710+0000","last_clean":"2024-09-13T08:44:56.275710+0000","last_became_active":"2024-09-13T08:44:55.267475+0000","last_became_peered":"2024-09-13T08:44:55.267475+0000","last_unstale":"2024-09-13T08:44:56.275710+0000","last_undegraded":"2024-09-13T08:44:56.275710+0000","last_fullsized":"2024-09-13T08:44:56.275710+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:32.535999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276292+0000","last_change":"2024-09-13T08:44:55.277675+0000","last_active":"2024-09-13T08:44:56.276292+0000","last_peered":"2024-09-13T08:44:56.276292+0000","last_clean":"2024-09-13T08:44:56.276292+0000","last_became_active":"2024-09-13T08:44:55.276706+0000","last_became_peered":"2024-09-13T08:44:55.276706+0000","last_unstale":"2024-09-13T08:44:56.276292+0000","last_undegraded":"2024-09-13T08:44:56.276292+0000","last_fullsized":"2024-09-13T08:44:56.276292+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:34:25.213862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276412+0000","last_change":"2024-09-13T08:44:55.265599+0000","last_active":"2024-09-13T08:44:56.276412+0000","last_peered":"2024-09-13T08:44:56.276412+0000","last_clean":"2024-09-13T08:44:56.276412+0000","last_became_active":"2024-09-13T08:44:55.265308+0000","last_became_peered":"2024-09-13T08:44:55.265308+0000","last_unstale":"2024-09-13T08:44:56.276412+0000","last_undegraded":"2024-09-13T08:44:56.276412+0000","last_fullsized":"2024-09-13T08:44:56.276412+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:47.203294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275417+0000","last_change":"2024-09-13T08:44:55.279330+0000","last_active":"2024-09-13T08:44:56.275417+0000","last_peered":"2024-09-13T08:44:56.275417+0000","last_clean":"2024-09-13T08:44:56.275417+0000","last_became_active":"2024-09-13T08:44:55.279035+0000","last_became_peered":"2024-09-13T08:44:55.279035+0000","last_unstale":"2024-09-13T08:44:56.275417+0000","last_undegraded":"2024-09-13T08:44:56.275417+0000","last_fullsized":"2024-09-13T08:44:56.275417+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:35:58.221542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491202+0000","last_change":"2024-09-13T08:44:55.282381+0000","last_active":"2024-09-13T08:44:56.491202+0000","last_peered":"2024-09-13T08:44:56.491202+0000","last_clean":"2024-09-13T08:44:56.491202+0000","last_became_active":"2024-09-13T08:44:55.278456+0000","last_became_peered":"2024-09-13T08:44:55.278456+0000","last_unstale":"2024-09-13T08:44:56.491202+0000","last_undegraded":"2024-09-13T08:44:56.491202+0000","last_fullsized":"2024-09-13T08:44:56.491202+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:37.964503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274746+0000","last_change":"2024-09-13T08:44:55.278069+0000","last_active":"2024-09-13T08:44:56.274746+0000","last_peered":"2024-09-13T08:44:56.274746+0000","last_clean":"2024-09-13T08:44:56.274746+0000","last_became_active":"2024-09-13T08:44:55.277331+0000","last_became_peered":"2024-09-13T08:44:55.277331+0000","last_unstale":"2024-09-13T08:44:56.274746+0000","last_undegraded":"2024-09-13T08:44:56.274746+0000","last_fullsized":"2024-09-13T08:44:56.274746+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:05:30.487512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274690+0000","last_change":"2024-09-13T08:44:55.273556+0000","last_active":"2024-09-13T08:44:56.274690+0000","last_peered":"2024-09-13T08:44:56.274690+0000","last_clean":"2024-09-13T08:44:56.274690+0000","last_became_active":"2024-09-13T08:44:55.273032+0000","last_became_peered":"2024-09-13T08:44:55.273032+0000","last_unstale":"2024-09-13T08:44:56.274690+0000","last_undegraded":"2024-09-13T08:44:56.274690+0000","last_fullsized":"2024-09-13T08:44:56.274690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:25:30.196431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274035+0000","last_change":"2024-09-13T08:44:55.275193+0000","last_active":"2024-09-13T08:44:56.274035+0000","last_peered":"2024-09-13T08:44:56.274035+0000","last_clean":"2024-09-13T08:44:56.274035+0000","last_became_active":"2024-09-13T08:44:55.274867+0000","last_became_peered":"2024-09-13T08:44:55.274867+0000","last_unstale":"2024-09-13T08:44:56.274035+0000","last_undegraded":"2024-09-13T08:44:56.274035+0000","last_fullsized":"2024-09-13T08:44:56.274035+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:32.172098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":19,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490756+0000","last_change":"2024-09-13T08:44:54.266444+0000","last_active":"2024-09-13T08:44:56.490756+0000","last_peered":"2024-09-13T08:44:56.490756+0000","last_clean":"2024-09-13T08:44:56.490756+0000","last_became_active":"2024-09-13T08:44:54.265401+0000","last_became_peered":"2024-09-13T08:44:54.265401+0000","last_unstale":"2024-09-13T08:44:56.490756+0000","last_undegraded":"2024-09-13T08:44:56.490756+0000","last_fullsized":"2024-09-13T08:44:56.490756+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:58.947701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492317+0000","last_change":"2024-09-13T08:44:55.273520+0000","last_active":"2024-09-13T08:44:56.492317+0000","last_peered":"2024-09-13T08:44:56.492317+0000","last_clean":"2024-09-13T08:44:56.492317+0000","last_became_active":"2024-09-13T08:44:55.272959+0000","last_became_peered":"2024-09-13T08:44:55.272959+0000","last_unstale":"2024-09-13T08:44:56.492317+0000","last_undegraded":"2024-09-13T08:44:56.492317+0000","last_fullsized":"2024-09-13T08:44:56.492317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:56.726509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491787+0000","last_change":"2024-09-13T08:44:54.260286+0000","last_active":"2024-09-13T08:44:56.491787+0000","last_peered":"2024-09-13T08:44:56.491787+0000","last_clean":"2024-09-13T08:44:56.491787+0000","last_became_active":"2024-09-13T08:44:54.258226+0000","last_became_peered":"2024-09-13T08:44:54.258226+0000","last_unstale":"2024-09-13T08:44:56.491787+0000","last_undegraded":"2024-09-13T08:44:56.491787+0000","last_fullsized":"2024-09-13T08:44:56.491787+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:18.867194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491324+0000","last_change":"2024-09-13T08:44:55.284369+0000","last_active":"2024-09-13T08:44:56.491324+0000","last_peered":"2024-09-13T08:44:56.491324+0000","last_clean":"2024-09-13T08:44:56.491324+0000","last_became_active":"2024-09-13T08:44:55.284041+0000","last_became_peered":"2024-09-13T08:44:55.284041+0000","last_unstale":"2024-09-13T08:44:56.491324+0000","last_undegraded":"2024-09-13T08:44:56.491324+0000","last_fullsized":"2024-09-13T08:44:56.491324+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:36.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490728+0000","last_change":"2024-09-13T08:44:54.265229+0000","last_active":"2024-09-13T08:44:56.490728+0000","last_peered":"2024-09-13T08:44:56.490728+0000","last_clean":"2024-09-13T08:44:56.490728+0000","last_became_active":"2024-09-13T08:44:54.264945+0000","last_became_peered":"2024-09-13T08:44:54.264945+0000","last_unstale":"2024-09-13T08:44:56.490728+0000","last_undegraded":"2024-09-13T08:44:56.490728+0000","last_fullsized":"2024-09-13T08:44:56.490728+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:29.389478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291266+0000","last_change":"2024-09-13T08:44:55.267465+0000","last_active":"2024-09-13T08:44:56.291266+0000","last_peered":"2024-09-13T08:44:56.291266+0000","last_clean":"2024-09-13T08:44:56.291266+0000","last_became_active":"2024-09-13T08:44:55.267183+0000","last_became_peered":"2024-09-13T08:44:55.267183+0000","last_unstale":"2024-09-13T08:44:56.291266+0000","last_undegraded":"2024-09-13T08:44:56.291266+0000","last_fullsized":"2024-09-13T08:44:56.291266+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:42.198294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490291+0000","last_change":"2024-09-13T08:44:54.263949+0000","last_active":"2024-09-13T08:44:56.490291+0000","last_peered":"2024-09-13T08:44:56.490291+0000","last_clean":"2024-09-13T08:44:56.490291+0000","last_became_active":"2024-09-13T08:44:54.263668+0000","last_became_peered":"2024-09-13T08:44:54.263668+0000","last_unstale":"2024-09-13T08:44:56.490291+0000","last_undegraded":"2024-09-13T08:44:56.490291+0000","last_fullsized":"2024-09-13T08:44:56.490291+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:16.769289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274797+0000","last_change":"2024-09-13T08:44:55.262769+0000","last_active":"2024-09-13T08:44:56.274797+0000","last_peered":"2024-09-13T08:44:56.274797+0000","last_clean":"2024-09-13T08:44:56.274797+0000","last_became_active":"2024-09-13T08:44:55.262451+0000","last_became_peered":"2024-09-13T08:44:55.262451+0000","last_unstale":"2024-09-13T08:44:56.274797+0000","last_undegraded":"2024-09-13T08:44:56.274797+0000","last_fullsized":"2024-09-13T08:44:56.274797+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:05.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275544+0000","last_change":"2024-09-13T08:44:54.253005+0000","last_active":"2024-09-13T08:44:56.275544+0000","last_peered":"2024-09-13T08:44:56.275544+0000","last_clean":"2024-09-13T08:44:56.275544+0000","last_became_active":"2024-09-13T08:44:54.252685+0000","last_became_peered":"2024-09-13T08:44:54.252685+0000","last_unstale":"2024-09-13T08:44:56.275544+0000","last_undegraded":"2024-09-13T08:44:56.275544+0000","last_fullsized":"2024-09-13T08:44:56.275544+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:36:21.690288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291461+0000","last_change":"2024-09-13T08:44:55.272203+0000","last_active":"2024-09-13T08:44:56.291461+0000","last_peered":"2024-09-13T08:44:56.291461+0000","last_clean":"2024-09-13T08:44:56.291461+0000","last_became_active":"2024-09-13T08:44:55.271918+0000","last_became_peered":"2024-09-13T08:44:55.271918+0000","last_unstale":"2024-09-13T08:44:56.291461+0000","last_undegraded":"2024-09-13T08:44:56.291461+0000","last_fullsized":"2024-09-13T08:44:56.291461+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:30.597352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273576+0000","last_change":"2024-09-13T08:44:54.258728+0000","last_active":"2024-09-13T08:44:56.273576+0000","last_peered":"2024-09-13T08:44:56.273576+0000","last_clean":"2024-09-13T08:44:56.273576+0000","last_became_active":"2024-09-13T08:44:54.258408+0000","last_became_peered":"2024-09-13T08:44:54.258408+0000","last_unstale":"2024-09-13T08:44:56.273576+0000","last_undegraded":"2024-09-13T08:44:56.273576+0000","last_fullsized":"2024-09-13T08:44:56.273576+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:38:59.558456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274872+0000","last_change":"2024-09-13T08:44:55.281750+0000","last_active":"2024-09-13T08:44:56.274872+0000","last_peered":"2024-09-13T08:44:56.274872+0000","last_clean":"2024-09-13T08:44:56.274872+0000","last_became_active":"2024-09-13T08:44:55.281369+0000","last_became_peered":"2024-09-13T08:44:55.281369+0000","last_unstale":"2024-09-13T08:44:56.274872+0000","last_undegraded":"2024-09-13T08:44:56.274872+0000","last_fullsized":"2024-09-13T08:44:56.274872+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:52.352802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490749+0000","last_change":"2024-09-13T08:44:54.256616+0000","last_active":"2024-09-13T08:44:56.490749+0000","last_peered":"2024-09-13T08:44:56.490749+0000","last_clean":"2024-09-13T08:44:56.490749+0000","last_became_active":"2024-09-13T08:44:54.256352+0000","last_became_peered":"2024-09-13T08:44:54.256352+0000","last_unstale":"2024-09-13T08:44:56.490749+0000","last_undegraded":"2024-09-13T08:44:56.490749+0000","last_fullsized":"2024-09-13T08:44:56.490749+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:29.514973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290838+0000","last_change":"2024-09-13T08:44:55.266843+0000","last_active":"2024-09-13T08:44:56.290838+0000","last_peered":"2024-09-13T08:44:56.290838+0000","last_clean":"2024-09-13T08:44:56.290838+0000","last_became_active":"2024-09-13T08:44:55.266583+0000","last_became_peered":"2024-09-13T08:44:55.266583+0000","last_unstale":"2024-09-13T08:44:56.290838+0000","last_undegraded":"2024-09-13T08:44:56.290838+0000","last_fullsized":"2024-09-13T08:44:56.290838+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:49.797055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.489903+0000","last_change":"2024-09-13T08:44:54.269348+0000","last_active":"2024-09-13T08:44:56.489903+0000","last_peered":"2024-09-13T08:44:56.489903+0000","last_clean":"2024-09-13T08:44:56.489903+0000","last_became_active":"2024-09-13T08:44:54.268761+0000","last_became_peered":"2024-09-13T08:44:54.268761+0000","last_unstale":"2024-09-13T08:44:56.489903+0000","last_undegraded":"2024-09-13T08:44:56.489903+0000","last_fullsized":"2024-09-13T08:44:56.489903+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:14.558510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291364+0000","last_change":"2024-09-13T08:44:55.271361+0000","last_active":"2024-09-13T08:44:56.291364+0000","last_peered":"2024-09-13T08:44:56.291364+0000","last_clean":"2024-09-13T08:44:56.291364+0000","last_became_active":"2024-09-13T08:44:55.271070+0000","last_became_peered":"2024-09-13T08:44:55.271070+0000","last_unstale":"2024-09-13T08:44:56.291364+0000","last_undegraded":"2024-09-13T08:44:56.291364+0000","last_fullsized":"2024-09-13T08:44:56.291364+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:31.811236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491500+0000","last_change":"2024-09-13T08:44:54.262267+0000","last_active":"2024-09-13T08:44:56.491500+0000","last_peered":"2024-09-13T08:44:56.491500+0000","last_clean":"2024-09-13T08:44:56.491500+0000","last_became_active":"2024-09-13T08:44:54.261926+0000","last_became_peered":"2024-09-13T08:44:54.261926+0000","last_unstale":"2024-09-13T08:44:56.491500+0000","last_undegraded":"2024-09-13T08:44:56.491500+0000","last_fullsized":"2024-09-13T08:44:56.491500+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:24.478636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491400+0000","last_change":"2024-09-13T08:44:55.279445+0000","last_active":"2024-09-13T08:44:56.491400+0000","last_peered":"2024-09-13T08:44:56.491400+0000","last_clean":"2024-09-13T08:44:56.491400+0000","last_became_active":"2024-09-13T08:44:55.271480+0000","last_became_peered":"2024-09-13T08:44:55.271480+0000","last_unstale":"2024-09-13T08:44:56.491400+0000","last_undegraded":"2024-09-13T08:44:56.491400+0000","last_fullsized":"2024-09-13T08:44:56.491400+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:04.111848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274178+0000","last_change":"2024-09-13T08:44:54.259956+0000","last_active":"2024-09-13T08:44:56.274178+0000","last_peered":"2024-09-13T08:44:56.274178+0000","last_clean":"2024-09-13T08:44:56.274178+0000","last_became_active":"2024-09-13T08:44:54.259304+0000","last_became_peered":"2024-09-13T08:44:54.259304+0000","last_unstale":"2024-09-13T08:44:56.274178+0000","last_undegraded":"2024-09-13T08:44:56.274178+0000","last_fullsized":"2024-09-13T08:44:56.274178+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:01.426416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492145+0000","last_change":"2024-09-13T08:44:55.278564+0000","last_active":"2024-09-13T08:44:56.492145+0000","last_peered":"2024-09-13T08:44:56.492145+0000","last_clean":"2024-09-13T08:44:56.492145+0000","last_became_active":"2024-09-13T08:44:55.276693+0000","last_became_peered":"2024-09-13T08:44:55.276693+0000","last_unstale":"2024-09-13T08:44:56.492145+0000","last_undegraded":"2024-09-13T08:44:56.492145+0000","last_fullsized":"2024-09-13T08:44:56.492145+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:10.647195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":21,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492022+0000","last_change":"2024-09-13T08:44:54.262744+0000","last_active":"2024-09-13T08:44:56.492022+0000","last_peered":"2024-09-13T08:44:56.492022+0000","last_clean":"2024-09-13T08:44:56.492022+0000","last_became_active":"2024-09-13T08:44:54.262327+0000","last_became_peered":"2024-09-13T08:44:54.262327+0000","last_unstale":"2024-09-13T08:44:56.492022+0000","last_undegraded":"2024-09-13T08:44:56.492022+0000","last_fullsized":"2024-09-13T08:44:56.492022+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:53.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290851+0000","last_change":"2024-09-13T08:44:55.284935+0000","last_active":"2024-09-13T08:44:56.290851+0000","last_peered":"2024-09-13T08:44:56.290851+0000","last_clean":"2024-09-13T08:44:56.290851+0000","last_became_active":"2024-09-13T08:44:55.284660+0000","last_became_peered":"2024-09-13T08:44:55.284660+0000","last_unstale":"2024-09-13T08:44:56.290851+0000","last_undegraded":"2024-09-13T08:44:56.290851+0000","last_fullsized":"2024-09-13T08:44:56.290851+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:59.543608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490153+0000","last_change":"2024-09-13T08:44:54.264285+0000","last_active":"2024-09-13T08:44:56.490153+0000","last_peered":"2024-09-13T08:44:56.490153+0000","last_clean":"2024-09-13T08:44:56.490153+0000","last_became_active":"2024-09-13T08:44:54.264028+0000","last_became_peered":"2024-09-13T08:44:54.264028+0000","last_unstale":"2024-09-13T08:44:56.490153+0000","last_undegraded":"2024-09-13T08:44:56.490153+0000","last_fullsized":"2024-09-13T08:44:56.490153+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:29.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275724+0000","last_change":"2024-09-13T08:44:55.280824+0000","last_active":"2024-09-13T08:44:56.275724+0000","last_peered":"2024-09-13T08:44:56.275724+0000","last_clean":"2024-09-13T08:44:56.275724+0000","last_became_active":"2024-09-13T08:44:55.280527+0000","last_became_peered":"2024-09-13T08:44:55.280527+0000","last_unstale":"2024-09-13T08:44:56.275724+0000","last_undegraded":"2024-09-13T08:44:56.275724+0000","last_fullsized":"2024-09-13T08:44:56.275724+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:48.020108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491455+0000","last_change":"2024-09-13T08:44:54.260694+0000","last_active":"2024-09-13T08:44:56.491455+0000","last_peered":"2024-09-13T08:44:56.491455+0000","last_clean":"2024-09-13T08:44:56.491455+0000","last_became_active":"2024-09-13T08:44:54.260412+0000","last_became_peered":"2024-09-13T08:44:54.260412+0000","last_unstale":"2024-09-13T08:44:56.491455+0000","last_undegraded":"2024-09-13T08:44:56.491455+0000","last_fullsized":"2024-09-13T08:44:56.491455+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:02.170087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290706+0000","last_change":"2024-09-13T08:44:55.277787+0000","last_active":"2024-09-13T08:44:56.290706+0000","last_peered":"2024-09-13T08:44:56.290706+0000","last_clean":"2024-09-13T08:44:56.290706+0000","last_became_active":"2024-09-13T08:44:55.277490+0000","last_became_peered":"2024-09-13T08:44:55.277490+0000","last_unstale":"2024-09-13T08:44:56.290706+0000","last_undegraded":"2024-09-13T08:44:56.290706+0000","last_fullsized":"2024-09-13T08:44:56.290706+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:52.158716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275671+0000","last_change":"2024-09-13T08:44:54.256931+0000","last_active":"2024-09-13T08:44:56.275671+0000","last_peered":"2024-09-13T08:44:56.275671+0000","last_clean":"2024-09-13T08:44:56.275671+0000","last_became_active":"2024-09-13T08:44:54.256635+0000","last_became_peered":"2024-09-13T08:44:54.256635+0000","last_unstale":"2024-09-13T08:44:56.275671+0000","last_undegraded":"2024-09-13T08:44:56.275671+0000","last_fullsized":"2024-09-13T08:44:56.275671+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:51.526164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291079+0000","last_change":"2024-09-13T08:44:55.273932+0000","last_active":"2024-09-13T08:44:56.291079+0000","last_peered":"2024-09-13T08:44:56.291079+0000","last_clean":"2024-09-13T08:44:56.291079+0000","last_became_active":"2024-09-13T08:44:55.273133+0000","last_became_peered":"2024-09-13T08:44:55.273133+0000","last_unstale":"2024-09-13T08:44:56.291079+0000","last_undegraded":"2024-09-13T08:44:56.291079+0000","last_fullsized":"2024-09-13T08:44:56.291079+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:30.524014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":23,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274659+0000","last_change":"2024-09-13T08:44:54.256980+0000","last_active":"2024-09-13T08:44:56.274659+0000","last_peered":"2024-09-13T08:44:56.274659+0000","last_clean":"2024-09-13T08:44:56.274659+0000","last_became_active":"2024-09-13T08:44:54.256693+0000","last_became_peered":"2024-09-13T08:44:54.256693+0000","last_unstale":"2024-09-13T08:44:56.274659+0000","last_undegraded":"2024-09-13T08:44:56.274659+0000","last_fullsized":"2024-09-13T08:44:56.274659+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:28.513665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274234+0000","last_change":"2024-09-13T08:44:55.285849+0000","last_active":"2024-09-13T08:44:56.274234+0000","last_peered":"2024-09-13T08:44:56.274234+0000","last_clean":"2024-09-13T08:44:56.274234+0000","last_became_active":"2024-09-13T08:44:55.285566+0000","last_became_peered":"2024-09-13T08:44:55.285566+0000","last_unstale":"2024-09-13T08:44:56.274234+0000","last_undegraded":"2024-09-13T08:44:56.274234+0000","last_fullsized":"2024-09-13T08:44:56.274234+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:22.285510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291496+0000","last_change":"2024-09-13T08:44:54.259191+0000","last_active":"2024-09-13T08:44:56.291496+0000","last_peered":"2024-09-13T08:44:56.291496+0000","last_clean":"2024-09-13T08:44:56.291496+0000","last_became_active":"2024-09-13T08:44:54.258798+0000","last_became_peered":"2024-09-13T08:44:54.258798+0000","last_unstale":"2024-09-13T08:44:56.291496+0000","last_undegraded":"2024-09-13T08:44:56.291496+0000","last_fullsized":"2024-09-13T08:44:56.291496+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:48.023861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276214+0000","last_change":"2024-09-13T08:44:55.278575+0000","last_active":"2024-09-13T08:44:56.276214+0000","last_peered":"2024-09-13T08:44:56.276214+0000","last_clean":"2024-09-13T08:44:56.276214+0000","last_became_active":"2024-09-13T08:44:55.276763+0000","last_became_peered":"2024-09-13T08:44:55.276763+0000","last_unstale":"2024-09-13T08:44:56.276214+0000","last_undegraded":"2024-09-13T08:44:56.276214+0000","last_fullsized":"2024-09-13T08:44:56.276214+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:56:58.603337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275776+0000","last_change":"2024-09-13T08:44:54.259077+0000","last_active":"2024-09-13T08:44:56.275776+0000","last_peered":"2024-09-13T08:44:56.275776+0000","last_clean":"2024-09-13T08:44:56.275776+0000","last_became_active":"2024-09-13T08:44:54.258694+0000","last_became_peered":"2024-09-13T08:44:54.258694+0000","last_unstale":"2024-09-13T08:44:56.275776+0000","last_undegraded":"2024-09-13T08:44:56.275776+0000","last_fullsized":"2024-09-13T08:44:56.275776+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:25.036242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274042+0000","last_change":"2024-09-13T08:44:55.286464+0000","last_active":"2024-09-13T08:44:56.274042+0000","last_peered":"2024-09-13T08:44:56.274042+0000","last_clean":"2024-09-13T08:44:56.274042+0000","last_became_active":"2024-09-13T08:44:55.286137+0000","last_became_peered":"2024-09-13T08:44:55.286137+0000","last_unstale":"2024-09-13T08:44:56.274042+0000","last_undegraded":"2024-09-13T08:44:56.274042+0000","last_fullsized":"2024-09-13T08:44:56.274042+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:16.803489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492273+0000","last_change":"2024-09-13T08:44:54.263421+0000","last_active":"2024-09-13T08:44:56.492273+0000","last_peered":"2024-09-13T08:44:56.492273+0000","last_clean":"2024-09-13T08:44:56.492273+0000","last_became_active":"2024-09-13T08:44:54.262563+0000","last_became_peered":"2024-09-13T08:44:54.262563+0000","last_unstale":"2024-09-13T08:44:56.492273+0000","last_undegraded":"2024-09-13T08:44:56.492273+0000","last_fullsized":"2024-09-13T08:44:56.492273+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:44.116362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491142+0000","last_change":"2024-09-13T08:44:55.276553+0000","last_active":"2024-09-13T08:44:56.491142+0000","last_peered":"2024-09-13T08:44:56.491142+0000","last_clean":"2024-09-13T08:44:56.491142+0000","last_became_active":"2024-09-13T08:44:55.276152+0000","last_became_peered":"2024-09-13T08:44:55.276152+0000","last_unstale":"2024-09-13T08:44:56.491142+0000","last_undegraded":"2024-09-13T08:44:56.491142+0000","last_fullsized":"2024-09-13T08:44:56.491142+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:07.680327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276293+0000","last_change":"2024-09-13T08:44:54.260419+0000","last_active":"2024-09-13T08:44:56.276293+0000","last_peered":"2024-09-13T08:44:56.276293+0000","last_clean":"2024-09-13T08:44:56.276293+0000","last_became_active":"2024-09-13T08:44:54.260132+0000","last_became_peered":"2024-09-13T08:44:54.260132+0000","last_unstale":"2024-09-13T08:44:56.276293+0000","last_undegraded":"2024-09-13T08:44:56.276293+0000","last_fullsized":"2024-09-13T08:44:56.276293+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:30.607909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491419+0000","last_change":"2024-09-13T08:44:55.273138+0000","last_active":"2024-09-13T08:44:56.491419+0000","last_peered":"2024-09-13T08:44:56.491419+0000","last_clean":"2024-09-13T08:44:56.491419+0000","last_became_active":"2024-09-13T08:44:55.272839+0000","last_became_peered":"2024-09-13T08:44:55.272839+0000","last_unstale":"2024-09-13T08:44:56.491419+0000","last_undegraded":"2024-09-13T08:44:56.491419+0000","last_fullsized":"2024-09-13T08:44:56.491419+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:14.425364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490771+0000","last_change":"2024-09-13T08:44:54.263813+0000","last_active":"2024-09-13T08:44:56.490771+0000","last_peered":"2024-09-13T08:44:56.490771+0000","last_clean":"2024-09-13T08:44:56.490771+0000","last_became_active":"2024-09-13T08:44:54.262871+0000","last_became_peered":"2024-09-13T08:44:54.262871+0000","last_unstale":"2024-09-13T08:44:56.490771+0000","last_undegraded":"2024-09-13T08:44:56.490771+0000","last_fullsized":"2024-09-13T08:44:56.490771+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:53.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275758+0000","last_change":"2024-09-13T08:44:55.279333+0000","last_active":"2024-09-13T08:44:56.275758+0000","last_peered":"2024-09-13T08:44:56.275758+0000","last_clean":"2024-09-13T08:44:56.275758+0000","last_became_active":"2024-09-13T08:44:55.278510+0000","last_became_peered":"2024-09-13T08:44:55.278510+0000","last_unstale":"2024-09-13T08:44:56.275758+0000","last_undegraded":"2024-09-13T08:44:56.275758+0000","last_fullsized":"2024-09-13T08:44:56.275758+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:47.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290900+0000","last_change":"2024-09-13T08:44:54.267401+0000","last_active":"2024-09-13T08:44:56.290900+0000","last_peered":"2024-09-13T08:44:56.290900+0000","last_clean":"2024-09-13T08:44:56.290900+0000","last_became_active":"2024-09-13T08:44:54.267099+0000","last_became_peered":"2024-09-13T08:44:54.267099+0000","last_unstale":"2024-09-13T08:44:56.290900+0000","last_undegraded":"2024-09-13T08:44:56.290900+0000","last_fullsized":"2024-09-13T08:44:56.290900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:19.972272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490446+0000","last_change":"2024-09-13T08:44:55.275417+0000","last_active":"2024-09-13T08:44:56.490446+0000","last_peered":"2024-09-13T08:44:56.490446+0000","last_clean":"2024-09-13T08:44:56.490446+0000","last_became_active":"2024-09-13T08:44:55.274975+0000","last_became_peered":"2024-09-13T08:44:55.274975+0000","last_unstale":"2024-09-13T08:44:56.490446+0000","last_undegraded":"2024-09-13T08:44:56.490446+0000","last_fullsized":"2024-09-13T08:44:56.490446+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:35.665465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290954+0000","last_change":"2024-09-13T08:44:54.262253+0000","last_active":"2024-09-13T08:44:56.290954+0000","last_peered":"2024-09-13T08:44:56.290954+0000","last_clean":"2024-09-13T08:44:56.290954+0000","last_became_active":"2024-09-13T08:44:54.258536+0000","last_became_peered":"2024-09-13T08:44:54.258536+0000","last_unstale":"2024-09-13T08:44:56.290954+0000","last_undegraded":"2024-09-13T08:44:56.290954+0000","last_fullsized":"2024-09-13T08:44:56.290954+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:48.331106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275128+0000","last_change":"2024-09-13T08:44:55.277516+0000","last_active":"2024-09-13T08:44:56.275128+0000","last_peered":"2024-09-13T08:44:56.275128+0000","last_clean":"2024-09-13T08:44:56.275128+0000","last_became_active":"2024-09-13T08:44:55.276968+0000","last_became_peered":"2024-09-13T08:44:55.276968+0000","last_unstale":"2024-09-13T08:44:56.275128+0000","last_undegraded":"2024-09-13T08:44:56.275128+0000","last_fullsized":"2024-09-13T08:44:56.275128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:00.122856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290830+0000","last_change":"2024-09-13T08:44:54.267335+0000","last_active":"2024-09-13T08:44:56.290830+0000","last_peered":"2024-09-13T08:44:56.290830+0000","last_clean":"2024-09-13T08:44:56.290830+0000","last_became_active":"2024-09-13T08:44:54.266819+0000","last_became_peered":"2024-09-13T08:44:54.266819+0000","last_unstale":"2024-09-13T08:44:56.290830+0000","last_undegraded":"2024-09-13T08:44:56.290830+0000","last_fullsized":"2024-09-13T08:44:56.290830+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:06.861965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291360+0000","last_change":"2024-09-13T08:44:55.270799+0000","last_active":"2024-09-13T08:44:56.291360+0000","last_peered":"2024-09-13T08:44:56.291360+0000","last_clean":"2024-09-13T08:44:56.291360+0000","last_became_active":"2024-09-13T08:44:55.270510+0000","last_became_peered":"2024-09-13T08:44:55.270510+0000","last_unstale":"2024-09-13T08:44:56.291360+0000","last_undegraded":"2024-09-13T08:44:56.291360+0000","last_fullsized":"2024-09-13T08:44:56.291360+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:02.322294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291576+0000","last_change":"2024-09-13T08:44:54.260321+0000","last_active":"2024-09-13T08:44:56.291576+0000","last_peered":"2024-09-13T08:44:56.291576+0000","last_clean":"2024-09-13T08:44:56.291576+0000","last_became_active":"2024-09-13T08:44:54.256112+0000","last_became_peered":"2024-09-13T08:44:54.256112+0000","last_unstale":"2024-09-13T08:44:56.291576+0000","last_undegraded":"2024-09-13T08:44:56.291576+0000","last_fullsized":"2024-09-13T08:44:56.291576+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:26.247091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291444+0000","last_change":"2024-09-13T08:44:55.270694+0000","last_active":"2024-09-13T08:44:56.291444+0000","last_peered":"2024-09-13T08:44:56.291444+0000","last_clean":"2024-09-13T08:44:56.291444+0000","last_became_active":"2024-09-13T08:44:55.270395+0000","last_became_peered":"2024-09-13T08:44:55.270395+0000","last_unstale":"2024-09-13T08:44:56.291444+0000","last_undegraded":"2024-09-13T08:44:56.291444+0000","last_fullsized":"2024-09-13T08:44:56.291444+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:08.047668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290003+0000","last_change":"2024-09-13T08:44:54.269125+0000","last_active":"2024-09-13T08:44:56.290003+0000","last_peered":"2024-09-13T08:44:56.290003+0000","last_clean":"2024-09-13T08:44:56.290003+0000","last_became_active":"2024-09-13T08:44:54.268832+0000","last_became_peered":"2024-09-13T08:44:54.268832+0000","last_unstale":"2024-09-13T08:44:56.290003+0000","last_undegraded":"2024-09-13T08:44:56.290003+0000","last_fullsized":"2024-09-13T08:44:56.290003+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:36.832036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275992+0000","last_change":"2024-09-13T08:44:55.265866+0000","last_active":"2024-09-13T08:44:56.275992+0000","last_peered":"2024-09-13T08:44:56.275992+0000","last_clean":"2024-09-13T08:44:56.275992+0000","last_became_active":"2024-09-13T08:44:55.265212+0000","last_became_peered":"2024-09-13T08:44:55.265212+0000","last_unstale":"2024-09-13T08:44:56.275992+0000","last_undegraded":"2024-09-13T08:44:56.275992+0000","last_fullsized":"2024-09-13T08:44:56.275992+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:23.600901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492246+0000","last_change":"2024-09-13T08:44:54.262846+0000","last_active":"2024-09-13T08:44:56.492246+0000","last_peered":"2024-09-13T08:44:56.492246+0000","last_clean":"2024-09-13T08:44:56.492246+0000","last_became_active":"2024-09-13T08:44:54.262376+0000","last_became_peered":"2024-09-13T08:44:54.262376+0000","last_unstale":"2024-09-13T08:44:56.492246+0000","last_undegraded":"2024-09-13T08:44:56.492246+0000","last_fullsized":"2024-09-13T08:44:56.492246+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:30.802195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275248+0000","last_change":"2024-09-13T08:44:55.271553+0000","last_active":"2024-09-13T08:44:56.275248+0000","last_peered":"2024-09-13T08:44:56.275248+0000","last_clean":"2024-09-13T08:44:56.275248+0000","last_became_active":"2024-09-13T08:44:55.271278+0000","last_became_peered":"2024-09-13T08:44:55.271278+0000","last_unstale":"2024-09-13T08:44:56.275248+0000","last_undegraded":"2024-09-13T08:44:56.275248+0000","last_fullsized":"2024-09-13T08:44:56.275248+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:12.548581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491978+0000","last_change":"2024-09-13T08:44:54.255976+0000","last_active":"2024-09-13T08:44:56.491978+0000","last_peered":"2024-09-13T08:44:56.491978+0000","last_clean":"2024-09-13T08:44:56.491978+0000","last_became_active":"2024-09-13T08:44:54.255663+0000","last_became_peered":"2024-09-13T08:44:54.255663+0000","last_unstale":"2024-09-13T08:44:56.491978+0000","last_undegraded":"2024-09-13T08:44:56.491978+0000","last_fullsized":"2024-09-13T08:44:56.491978+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:29.514973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491452+0000","last_change":"2024-09-13T08:44:55.285736+0000","last_active":"2024-09-13T08:44:56.491452+0000","last_peered":"2024-09-13T08:44:56.491452+0000","last_clean":"2024-09-13T08:44:56.491452+0000","last_became_active":"2024-09-13T08:44:55.285497+0000","last_became_peered":"2024-09-13T08:44:55.285497+0000","last_unstale":"2024-09-13T08:44:56.491452+0000","last_undegraded":"2024-09-13T08:44:56.491452+0000","last_fullsized":"2024-09-13T08:44:56.491452+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:01:24.452026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275873+0000","last_change":"2024-09-13T08:44:54.252490+0000","last_active":"2024-09-13T08:44:56.275873+0000","last_peered":"2024-09-13T08:44:56.275873+0000","last_clean":"2024-09-13T08:44:56.275873+0000","last_became_active":"2024-09-13T08:44:54.252216+0000","last_became_peered":"2024-09-13T08:44:54.252216+0000","last_unstale":"2024-09-13T08:44:56.275873+0000","last_undegraded":"2024-09-13T08:44:56.275873+0000","last_fullsized":"2024-09-13T08:44:56.275873+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:01:44.009297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291381+0000","last_change":"2024-09-13T08:44:55.280041+0000","last_active":"2024-09-13T08:44:56.291381+0000","last_peered":"2024-09-13T08:44:56.291381+0000","last_clean":"2024-09-13T08:44:56.291381+0000","last_became_active":"2024-09-13T08:44:55.279759+0000","last_became_peered":"2024-09-13T08:44:55.279759+0000","last_unstale":"2024-09-13T08:44:56.291381+0000","last_undegraded":"2024-09-13T08:44:56.291381+0000","last_fullsized":"2024-09-13T08:44:56.291381+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:54:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490150+0000","last_change":"2024-09-13T08:44:54.264718+0000","last_active":"2024-09-13T08:44:56.490150+0000","last_peered":"2024-09-13T08:44:56.490150+0000","last_clean":"2024-09-13T08:44:56.490150+0000","last_became_active":"2024-09-13T08:44:54.264448+0000","last_became_peered":"2024-09-13T08:44:54.264448+0000","last_unstale":"2024-09-13T08:44:56.490150+0000","last_undegraded":"2024-09-13T08:44:56.490150+0000","last_fullsized":"2024-09-13T08:44:56.490150+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:23.920024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290771+0000","last_change":"2024-09-13T08:44:55.265806+0000","last_active":"2024-09-13T08:44:56.290771+0000","last_peered":"2024-09-13T08:44:56.290771+0000","last_clean":"2024-09-13T08:44:56.290771+0000","last_became_active":"2024-09-13T08:44:55.265520+0000","last_became_peered":"2024-09-13T08:44:55.265520+0000","last_unstale":"2024-09-13T08:44:56.290771+0000","last_undegraded":"2024-09-13T08:44:56.290771+0000","last_fullsized":"2024-09-13T08:44:56.290771+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:43:51.191345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490338+0000","last_change":"2024-09-13T08:44:54.260295+0000","last_active":"2024-09-13T08:44:56.490338+0000","last_peered":"2024-09-13T08:44:56.490338+0000","last_clean":"2024-09-13T08:44:56.490338+0000","last_became_active":"2024-09-13T08:44:54.260033+0000","last_became_peered":"2024-09-13T08:44:54.260033+0000","last_unstale":"2024-09-13T08:44:56.490338+0000","last_undegraded":"2024-09-13T08:44:56.490338+0000","last_fullsized":"2024-09-13T08:44:56.490338+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:35.398014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.489955+0000","last_change":"2024-09-13T08:44:55.266487+0000","last_active":"2024-09-13T08:44:56.489955+0000","last_peered":"2024-09-13T08:44:56.489955+0000","last_clean":"2024-09-13T08:44:56.489955+0000","last_became_active":"2024-09-13T08:44:55.266244+0000","last_became_peered":"2024-09-13T08:44:55.266244+0000","last_unstale":"2024-09-13T08:44:56.489955+0000","last_undegraded":"2024-09-13T08:44:56.489955+0000","last_fullsized":"2024-09-13T08:44:56.489955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:01:33.630012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490614+0000","last_change":"2024-09-13T08:44:54.259464+0000","last_active":"2024-09-13T08:44:56.490614+0000","last_peered":"2024-09-13T08:44:56.490614+0000","last_clean":"2024-09-13T08:44:56.490614+0000","last_became_active":"2024-09-13T08:44:54.259135+0000","last_became_peered":"2024-09-13T08:44:54.259135+0000","last_unstale":"2024-09-13T08:44:56.490614+0000","last_undegraded":"2024-09-13T08:44:56.490614+0000","last_fullsized":"2024-09-13T08:44:56.490614+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:35.443612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491390+0000","last_change":"2024-09-13T08:44:55.276016+0000","last_active":"2024-09-13T08:44:56.491390+0000","last_peered":"2024-09-13T08:44:56.491390+0000","last_clean":"2024-09-13T08:44:56.491390+0000","last_became_active":"2024-09-13T08:44:55.275632+0000","last_became_peered":"2024-09-13T08:44:55.275632+0000","last_unstale":"2024-09-13T08:44:56.491390+0000","last_undegraded":"2024-09-13T08:44:56.491390+0000","last_fullsized":"2024-09-13T08:44:56.491390+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:07.354992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491397+0000","last_change":"2024-09-13T08:44:54.262412+0000","last_active":"2024-09-13T08:44:56.491397+0000","last_peered":"2024-09-13T08:44:56.491397+0000","last_clean":"2024-09-13T08:44:56.491397+0000","last_became_active":"2024-09-13T08:44:54.261172+0000","last_became_peered":"2024-09-13T08:44:54.261172+0000","last_unstale":"2024-09-13T08:44:56.491397+0000","last_undegraded":"2024-09-13T08:44:56.491397+0000","last_fullsized":"2024-09-13T08:44:56.491397+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:06.671286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":351,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273381+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:44:56.273381+0000","last_peered":"2024-09-13T08:44:56.273381+0000","last_clean":"2024-09-13T08:44:56.273381+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:44:56.273381+0000","last_undegraded":"2024-09-13T08:44:56.273381+0000","last_fullsized":"2024-09-13T08:44:56.273381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1732608,"data_stored":768052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509539,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1720320,"data_stored":761708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":2351104,"data_stored":1360226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542242,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1753088,"data_stored":770268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":762324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1724416,"data_stored":761730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509540,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1358490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1724416,"data_stored":761798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:44:59.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:44:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 -- 172.21.15.31:0/478320682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0ce900 msgr2=0x7fd28c0cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 --2- 172.21.15.31:0/478320682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0ce900 0x7fd28c0cbc50 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fd284005fd0 tx=0x7fd28402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 -- 172.21.15.31:0/478320682 shutdown_connections 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 --2- 172.21.15.31:0/478320682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0ce900 0x7fd28c0cbc50 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 --2- 172.21.15.31:0/478320682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0cf850 0x7fd28c0ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 -- 172.21.15.31:0/478320682 >> 172.21.15.31:0/478320682 conn(0x7fd28c0cc910 msgr2=0x7fd28c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 -- 172.21.15.31:0/478320682 shutdown_connections 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.717+0000 7fd291c00640 1 -- 172.21.15.31:0/478320682 wait complete. 2024-09-13T08:44:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.718+0000 7fd291c00640 1 Processor -- start 2024-09-13T08:44:59.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.719+0000 7fd291c00640 1 -- start start 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.719+0000 7fd291c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0ce900 0x7fd28c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd291c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd291c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28c15bd70 0x7fd28c06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd291c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c1600f0 con 0x7fd28c0cf850 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd291c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c0cb850 con 0x7fd28c15bd70 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd291c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c06ea30 con 0x7fd28c0ce900 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45144/0 (socket says 172.21.15.31:45144) 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 -- 172.21.15.31:0/3128204048 learned_addr learned my addr 172.21.15.31:0/3128204048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd28aa00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0ce900 0x7fd28c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd28b400640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28c15bd70 0x7fd28c06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 -- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0ce900 msgr2=0x7fd28c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0ce900 0x7fd28c06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 -- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28c15bd70 msgr2=0x7fd28c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28c15bd70 0x7fd28c06fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.720+0000 7fd282000640 1 -- 172.21.15.31:0/3128204048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c1f5370 con 0x7fd28c0cf850 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd282000640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd27400d770 tx=0x7fd27400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274017070 con 0x7fd28c0cf850 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd274009d20 con 0x7fd28c0cf850 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274004280 con 0x7fd28c0cf850 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c1f4760 con 0x7fd28c0cf850 2024-09-13T08:44:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.721+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c1f4d20 con 0x7fd28c0cf850 2024-09-13T08:44:59.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.722+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c0ce3c0 con 0x7fd28c0cf850 2024-09-13T08:44:59.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.723+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fd2740035f0 con 0x7fd28c0cf850 2024-09-13T08:44:59.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.724+0000 7fd283400640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 0x7fd26807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:44:59.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.724+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fd27409b5e0 con 0x7fd28c0cf850 2024-09-13T08:44:59.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.725+0000 7fd28aa00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 0x7fd26807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:44:59.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.725+0000 7fd28aa00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 0x7fd26807b370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd27c000f60 tx=0x7fd27c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:44:59.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.729+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd274062bd0 con 0x7fd28c0cf850 2024-09-13T08:44:59.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.921+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd28c1f7a40 con 0x7fd268078eb0 2024-09-13T08:44:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.925+0000 7fd283400640 1 -- 172.21.15.31:0/3128204048 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252470 (secure 0 0 0) 0x7fd28c1f7a40 con 0x7fd268078eb0 2024-09-13T08:44:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:44:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 msgr2=0x7fd26807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 0x7fd26807b370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd27c000f60 tx=0x7fd27c009040 comp rx=0 tx=0).stop 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 msgr2=0x7fd28c0b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd27400d770 tx=0x7fd27400dc30 comp rx=0 tx=0).stop 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 shutdown_connections 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd268078eb0 0x7fd26807b370 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd28c15bd70 0x7fd28c06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd28c0cf850 0x7fd28c0b4df0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 --2- 172.21.15.31:0/3128204048 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd28c0ce900 0x7fd28c06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:44:59.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 >> 172.21.15.31:0/3128204048 conn(0x7fd28c0cc910 msgr2=0x7fd28c0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:44:59.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 shutdown_connections 2024-09-13T08:44:59.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:44:59.930+0000 7fd291c00640 1 -- 172.21.15.31:0/3128204048 wait complete. 2024-09-13T08:44:59.948 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-13T08:44:58.998048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280868,"kb_used_data":14712,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693852,"statfs":{"total":773094113280,"available":772806504448,"internally_reserved":0,"allocated":15065088,"data_stored":7304596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.761713"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275654+0000","last_change":"2024-09-13T08:44:55.266523+0000","last_active":"2024-09-13T08:44:56.275654+0000","last_peered":"2024-09-13T08:44:56.275654+0000","last_clean":"2024-09-13T08:44:56.275654+0000","last_became_active":"2024-09-13T08:44:55.266261+0000","last_became_peered":"2024-09-13T08:44:55.266261+0000","last_unstale":"2024-09-13T08:44:56.275654+0000","last_undegraded":"2024-09-13T08:44:56.275654+0000","last_fullsized":"2024-09-13T08:44:56.275654+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:52.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275619+0000","last_change":"2024-09-13T08:44:55.271585+0000","last_active":"2024-09-13T08:44:56.275619+0000","last_peered":"2024-09-13T08:44:56.275619+0000","last_clean":"2024-09-13T08:44:56.275619+0000","last_became_active":"2024-09-13T08:44:55.271296+0000","last_became_peered":"2024-09-13T08:44:55.271296+0000","last_unstale":"2024-09-13T08:44:56.275619+0000","last_undegraded":"2024-09-13T08:44:56.275619+0000","last_fullsized":"2024-09-13T08:44:56.275619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:35.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490609+0000","last_change":"2024-09-13T08:44:55.271212+0000","last_active":"2024-09-13T08:44:56.490609+0000","last_peered":"2024-09-13T08:44:56.490609+0000","last_clean":"2024-09-13T08:44:56.490609+0000","last_became_active":"2024-09-13T08:44:55.269819+0000","last_became_peered":"2024-09-13T08:44:55.269819+0000","last_unstale":"2024-09-13T08:44:56.490609+0000","last_undegraded":"2024-09-13T08:44:56.490609+0000","last_fullsized":"2024-09-13T08:44:56.490609+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:31:43.108217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291262+0000","last_change":"2024-09-13T08:44:55.276181+0000","last_active":"2024-09-13T08:44:56.291262+0000","last_peered":"2024-09-13T08:44:56.291262+0000","last_clean":"2024-09-13T08:44:56.291262+0000","last_became_active":"2024-09-13T08:44:55.275890+0000","last_became_peered":"2024-09-13T08:44:55.275890+0000","last_unstale":"2024-09-13T08:44:56.291262+0000","last_undegraded":"2024-09-13T08:44:56.291262+0000","last_fullsized":"2024-09-13T08:44:56.291262+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:54.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274336+0000","last_change":"2024-09-13T08:44:55.281453+0000","last_active":"2024-09-13T08:44:56.274336+0000","last_peered":"2024-09-13T08:44:56.274336+0000","last_clean":"2024-09-13T08:44:56.274336+0000","last_became_active":"2024-09-13T08:44:55.281180+0000","last_became_peered":"2024-09-13T08:44:55.281180+0000","last_unstale":"2024-09-13T08:44:56.274336+0000","last_undegraded":"2024-09-13T08:44:56.274336+0000","last_fullsized":"2024-09-13T08:44:56.274336+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:20.981313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490942+0000","last_change":"2024-09-13T08:44:55.272511+0000","last_active":"2024-09-13T08:44:56.490942+0000","last_peered":"2024-09-13T08:44:56.490942+0000","last_clean":"2024-09-13T08:44:56.490942+0000","last_became_active":"2024-09-13T08:44:55.272193+0000","last_became_peered":"2024-09-13T08:44:55.272193+0000","last_unstale":"2024-09-13T08:44:56.490942+0000","last_undegraded":"2024-09-13T08:44:56.490942+0000","last_fullsized":"2024-09-13T08:44:56.490942+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:50.855940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491207+0000","last_change":"2024-09-13T08:44:55.275891+0000","last_active":"2024-09-13T08:44:56.491207+0000","last_peered":"2024-09-13T08:44:56.491207+0000","last_clean":"2024-09-13T08:44:56.491207+0000","last_became_active":"2024-09-13T08:44:55.275561+0000","last_became_peered":"2024-09-13T08:44:55.275561+0000","last_unstale":"2024-09-13T08:44:56.491207+0000","last_undegraded":"2024-09-13T08:44:56.491207+0000","last_fullsized":"2024-09-13T08:44:56.491207+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.773464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274552+0000","last_change":"2024-09-13T08:44:55.279852+0000","last_active":"2024-09-13T08:44:56.274552+0000","last_peered":"2024-09-13T08:44:56.274552+0000","last_clean":"2024-09-13T08:44:56.274552+0000","last_became_active":"2024-09-13T08:44:55.279589+0000","last_became_peered":"2024-09-13T08:44:55.279589+0000","last_unstale":"2024-09-13T08:44:56.274552+0000","last_undegraded":"2024-09-13T08:44:56.274552+0000","last_fullsized":"2024-09-13T08:44:56.274552+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:08.147436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491619+0000","last_change":"2024-09-13T08:44:55.269462+0000","last_active":"2024-09-13T08:44:56.491619+0000","last_peered":"2024-09-13T08:44:56.491619+0000","last_clean":"2024-09-13T08:44:56.491619+0000","last_became_active":"2024-09-13T08:44:55.268887+0000","last_became_peered":"2024-09-13T08:44:55.268887+0000","last_unstale":"2024-09-13T08:44:56.491619+0000","last_undegraded":"2024-09-13T08:44:56.491619+0000","last_fullsized":"2024-09-13T08:44:56.491619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:03.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490630+0000","last_change":"2024-09-13T08:44:55.275559+0000","last_active":"2024-09-13T08:44:56.490630+0000","last_peered":"2024-09-13T08:44:56.490630+0000","last_clean":"2024-09-13T08:44:56.490630+0000","last_became_active":"2024-09-13T08:44:55.275309+0000","last_became_peered":"2024-09-13T08:44:55.275309+0000","last_unstale":"2024-09-13T08:44:56.490630+0000","last_undegraded":"2024-09-13T08:44:56.490630+0000","last_fullsized":"2024-09-13T08:44:56.490630+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:16.375360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491656+0000","last_change":"2024-09-13T08:44:55.277621+0000","last_active":"2024-09-13T08:44:56.491656+0000","last_peered":"2024-09-13T08:44:56.491656+0000","last_clean":"2024-09-13T08:44:56.491656+0000","last_became_active":"2024-09-13T08:44:55.277320+0000","last_became_peered":"2024-09-13T08:44:55.277320+0000","last_unstale":"2024-09-13T08:44:56.491656+0000","last_undegraded":"2024-09-13T08:44:56.491656+0000","last_fullsized":"2024-09-13T08:44:56.491656+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:05.337043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491423+0000","last_change":"2024-09-13T08:44:55.283842+0000","last_active":"2024-09-13T08:44:56.491423+0000","last_peered":"2024-09-13T08:44:56.491423+0000","last_clean":"2024-09-13T08:44:56.491423+0000","last_became_active":"2024-09-13T08:44:55.283530+0000","last_became_peered":"2024-09-13T08:44:55.283530+0000","last_unstale":"2024-09-13T08:44:56.491423+0000","last_undegraded":"2024-09-13T08:44:56.491423+0000","last_fullsized":"2024-09-13T08:44:56.491423+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:48.243969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274280+0000","last_change":"2024-09-13T08:44:55.281165+0000","last_active":"2024-09-13T08:44:56.274280+0000","last_peered":"2024-09-13T08:44:56.274280+0000","last_clean":"2024-09-13T08:44:56.274280+0000","last_became_active":"2024-09-13T08:44:55.280881+0000","last_became_peered":"2024-09-13T08:44:55.280881+0000","last_unstale":"2024-09-13T08:44:56.274280+0000","last_undegraded":"2024-09-13T08:44:56.274280+0000","last_fullsized":"2024-09-13T08:44:56.274280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:37:26.280541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490939+0000","last_change":"2024-09-13T08:44:55.278452+0000","last_active":"2024-09-13T08:44:56.490939+0000","last_peered":"2024-09-13T08:44:56.490939+0000","last_clean":"2024-09-13T08:44:56.490939+0000","last_became_active":"2024-09-13T08:44:55.277773+0000","last_became_peered":"2024-09-13T08:44:55.277773+0000","last_unstale":"2024-09-13T08:44:56.490939+0000","last_undegraded":"2024-09-13T08:44:56.490939+0000","last_fullsized":"2024-09-13T08:44:56.490939+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:00.979830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273962+0000","last_change":"2024-09-13T08:44:55.276406+0000","last_active":"2024-09-13T08:44:56.273962+0000","last_peered":"2024-09-13T08:44:56.273962+0000","last_clean":"2024-09-13T08:44:56.273962+0000","last_became_active":"2024-09-13T08:44:55.275942+0000","last_became_peered":"2024-09-13T08:44:55.275942+0000","last_unstale":"2024-09-13T08:44:56.273962+0000","last_undegraded":"2024-09-13T08:44:56.273962+0000","last_fullsized":"2024-09-13T08:44:56.273962+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:34.927349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491121+0000","last_change":"2024-09-13T08:44:55.269060+0000","last_active":"2024-09-13T08:44:56.491121+0000","last_peered":"2024-09-13T08:44:56.491121+0000","last_clean":"2024-09-13T08:44:56.491121+0000","last_became_active":"2024-09-13T08:44:55.268768+0000","last_became_peered":"2024-09-13T08:44:55.268768+0000","last_unstale":"2024-09-13T08:44:56.491121+0000","last_undegraded":"2024-09-13T08:44:56.491121+0000","last_fullsized":"2024-09-13T08:44:56.491121+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:06.221239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490791+0000","last_change":"2024-09-13T08:44:55.265806+0000","last_active":"2024-09-13T08:44:56.490791+0000","last_peered":"2024-09-13T08:44:56.490791+0000","last_clean":"2024-09-13T08:44:56.490791+0000","last_became_active":"2024-09-13T08:44:55.265539+0000","last_became_peered":"2024-09-13T08:44:55.265539+0000","last_unstale":"2024-09-13T08:44:56.490791+0000","last_undegraded":"2024-09-13T08:44:56.490791+0000","last_fullsized":"2024-09-13T08:44:56.490791+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:58.558715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274078+0000","last_change":"2024-09-13T08:44:55.276617+0000","last_active":"2024-09-13T08:44:56.274078+0000","last_peered":"2024-09-13T08:44:56.274078+0000","last_clean":"2024-09-13T08:44:56.274078+0000","last_became_active":"2024-09-13T08:44:55.276113+0000","last_became_peered":"2024-09-13T08:44:55.276113+0000","last_unstale":"2024-09-13T08:44:56.274078+0000","last_undegraded":"2024-09-13T08:44:56.274078+0000","last_fullsized":"2024-09-13T08:44:56.274078+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:31:23.862370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275616+0000","last_change":"2024-09-13T08:44:55.271898+0000","last_active":"2024-09-13T08:44:56.275616+0000","last_peered":"2024-09-13T08:44:56.275616+0000","last_clean":"2024-09-13T08:44:56.275616+0000","last_became_active":"2024-09-13T08:44:55.270304+0000","last_became_peered":"2024-09-13T08:44:55.270304+0000","last_unstale":"2024-09-13T08:44:56.275616+0000","last_undegraded":"2024-09-13T08:44:56.275616+0000","last_fullsized":"2024-09-13T08:44:56.275616+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:23.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275343+0000","last_change":"2024-09-13T08:44:55.266185+0000","last_active":"2024-09-13T08:44:56.275343+0000","last_peered":"2024-09-13T08:44:56.275343+0000","last_clean":"2024-09-13T08:44:56.275343+0000","last_became_active":"2024-09-13T08:44:55.265585+0000","last_became_peered":"2024-09-13T08:44:55.265585+0000","last_unstale":"2024-09-13T08:44:56.275343+0000","last_undegraded":"2024-09-13T08:44:56.275343+0000","last_fullsized":"2024-09-13T08:44:56.275343+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:12.967100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490206+0000","last_change":"2024-09-13T08:44:55.265991+0000","last_active":"2024-09-13T08:44:56.490206+0000","last_peered":"2024-09-13T08:44:56.490206+0000","last_clean":"2024-09-13T08:44:56.490206+0000","last_became_active":"2024-09-13T08:44:55.265718+0000","last_became_peered":"2024-09-13T08:44:55.265718+0000","last_unstale":"2024-09-13T08:44:56.490206+0000","last_undegraded":"2024-09-13T08:44:56.490206+0000","last_fullsized":"2024-09-13T08:44:56.490206+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:15.970484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491321+0000","last_change":"2024-09-13T08:44:55.278743+0000","last_active":"2024-09-13T08:44:56.491321+0000","last_peered":"2024-09-13T08:44:56.491321+0000","last_clean":"2024-09-13T08:44:56.491321+0000","last_became_active":"2024-09-13T08:44:55.278150+0000","last_became_peered":"2024-09-13T08:44:55.278150+0000","last_unstale":"2024-09-13T08:44:56.491321+0000","last_undegraded":"2024-09-13T08:44:56.491321+0000","last_fullsized":"2024-09-13T08:44:56.491321+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:09:39.633701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290845+0000","last_change":"2024-09-13T08:44:55.277149+0000","last_active":"2024-09-13T08:44:56.290845+0000","last_peered":"2024-09-13T08:44:56.290845+0000","last_clean":"2024-09-13T08:44:56.290845+0000","last_became_active":"2024-09-13T08:44:55.272353+0000","last_became_peered":"2024-09-13T08:44:55.272353+0000","last_unstale":"2024-09-13T08:44:56.290845+0000","last_undegraded":"2024-09-13T08:44:56.290845+0000","last_fullsized":"2024-09-13T08:44:56.290845+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:15.506354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273975+0000","last_change":"2024-09-13T08:44:55.287273+0000","last_active":"2024-09-13T08:44:56.273975+0000","last_peered":"2024-09-13T08:44:56.273975+0000","last_clean":"2024-09-13T08:44:56.273975+0000","last_became_active":"2024-09-13T08:44:55.287015+0000","last_became_peered":"2024-09-13T08:44:55.287015+0000","last_unstale":"2024-09-13T08:44:56.273975+0000","last_undegraded":"2024-09-13T08:44:56.273975+0000","last_fullsized":"2024-09-13T08:44:56.273975+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:02:22.270131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276133+0000","last_change":"2024-09-13T08:44:55.279126+0000","last_active":"2024-09-13T08:44:56.276133+0000","last_peered":"2024-09-13T08:44:56.276133+0000","last_clean":"2024-09-13T08:44:56.276133+0000","last_became_active":"2024-09-13T08:44:55.277830+0000","last_became_peered":"2024-09-13T08:44:55.277830+0000","last_unstale":"2024-09-13T08:44:56.276133+0000","last_undegraded":"2024-09-13T08:44:56.276133+0000","last_fullsized":"2024-09-13T08:44:56.276133+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:19:05.504717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275710+0000","last_change":"2024-09-13T08:44:55.267775+0000","last_active":"2024-09-13T08:44:56.275710+0000","last_peered":"2024-09-13T08:44:56.275710+0000","last_clean":"2024-09-13T08:44:56.275710+0000","last_became_active":"2024-09-13T08:44:55.267475+0000","last_became_peered":"2024-09-13T08:44:55.267475+0000","last_unstale":"2024-09-13T08:44:56.275710+0000","last_undegraded":"2024-09-13T08:44:56.275710+0000","last_fullsized":"2024-09-13T08:44:56.275710+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:32.535999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276292+0000","last_change":"2024-09-13T08:44:55.277675+0000","last_active":"2024-09-13T08:44:56.276292+0000","last_peered":"2024-09-13T08:44:56.276292+0000","last_clean":"2024-09-13T08:44:56.276292+0000","last_became_active":"2024-09-13T08:44:55.276706+0000","last_became_peered":"2024-09-13T08:44:55.276706+0000","last_unstale":"2024-09-13T08:44:56.276292+0000","last_undegraded":"2024-09-13T08:44:56.276292+0000","last_fullsized":"2024-09-13T08:44:56.276292+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:34:25.213862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276412+0000","last_change":"2024-09-13T08:44:55.265599+0000","last_active":"2024-09-13T08:44:56.276412+0000","last_peered":"2024-09-13T08:44:56.276412+0000","last_clean":"2024-09-13T08:44:56.276412+0000","last_became_active":"2024-09-13T08:44:55.265308+0000","last_became_peered":"2024-09-13T08:44:55.265308+0000","last_unstale":"2024-09-13T08:44:56.276412+0000","last_undegraded":"2024-09-13T08:44:56.276412+0000","last_fullsized":"2024-09-13T08:44:56.276412+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:47.203294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275417+0000","last_change":"2024-09-13T08:44:55.279330+0000","last_active":"2024-09-13T08:44:56.275417+0000","last_peered":"2024-09-13T08:44:56.275417+0000","last_clean":"2024-09-13T08:44:56.275417+0000","last_became_active":"2024-09-13T08:44:55.279035+0000","last_became_peered":"2024-09-13T08:44:55.279035+0000","last_unstale":"2024-09-13T08:44:56.275417+0000","last_undegraded":"2024-09-13T08:44:56.275417+0000","last_fullsized":"2024-09-13T08:44:56.275417+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:35:58.221542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491202+0000","last_change":"2024-09-13T08:44:55.282381+0000","last_active":"2024-09-13T08:44:56.491202+0000","last_peered":"2024-09-13T08:44:56.491202+0000","last_clean":"2024-09-13T08:44:56.491202+0000","last_became_active":"2024-09-13T08:44:55.278456+0000","last_became_peered":"2024-09-13T08:44:55.278456+0000","last_unstale":"2024-09-13T08:44:56.491202+0000","last_undegraded":"2024-09-13T08:44:56.491202+0000","last_fullsized":"2024-09-13T08:44:56.491202+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:05:37.964503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274746+0000","last_change":"2024-09-13T08:44:55.278069+0000","last_active":"2024-09-13T08:44:56.274746+0000","last_peered":"2024-09-13T08:44:56.274746+0000","last_clean":"2024-09-13T08:44:56.274746+0000","last_became_active":"2024-09-13T08:44:55.277331+0000","last_became_peered":"2024-09-13T08:44:55.277331+0000","last_unstale":"2024-09-13T08:44:56.274746+0000","last_undegraded":"2024-09-13T08:44:56.274746+0000","last_fullsized":"2024-09-13T08:44:56.274746+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:05:30.487512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274690+0000","last_change":"2024-09-13T08:44:55.273556+0000","last_active":"2024-09-13T08:44:56.274690+0000","last_peered":"2024-09-13T08:44:56.274690+0000","last_clean":"2024-09-13T08:44:56.274690+0000","last_became_active":"2024-09-13T08:44:55.273032+0000","last_became_peered":"2024-09-13T08:44:55.273032+0000","last_unstale":"2024-09-13T08:44:56.274690+0000","last_undegraded":"2024-09-13T08:44:56.274690+0000","last_fullsized":"2024-09-13T08:44:56.274690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:25:30.196431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274035+0000","last_change":"2024-09-13T08:44:55.275193+0000","last_active":"2024-09-13T08:44:56.274035+0000","last_peered":"2024-09-13T08:44:56.274035+0000","last_clean":"2024-09-13T08:44:56.274035+0000","last_became_active":"2024-09-13T08:44:55.274867+0000","last_became_peered":"2024-09-13T08:44:55.274867+0000","last_unstale":"2024-09-13T08:44:56.274035+0000","last_undegraded":"2024-09-13T08:44:56.274035+0000","last_fullsized":"2024-09-13T08:44:56.274035+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:32.172098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":19,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490756+0000","last_change":"2024-09-13T08:44:54.266444+0000","last_active":"2024-09-13T08:44:56.490756+0000","last_peered":"2024-09-13T08:44:56.490756+0000","last_clean":"2024-09-13T08:44:56.490756+0000","last_became_active":"2024-09-13T08:44:54.265401+0000","last_became_peered":"2024-09-13T08:44:54.265401+0000","last_unstale":"2024-09-13T08:44:56.490756+0000","last_undegraded":"2024-09-13T08:44:56.490756+0000","last_fullsized":"2024-09-13T08:44:56.490756+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:58.947701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492317+0000","last_change":"2024-09-13T08:44:55.273520+0000","last_active":"2024-09-13T08:44:56.492317+0000","last_peered":"2024-09-13T08:44:56.492317+0000","last_clean":"2024-09-13T08:44:56.492317+0000","last_became_active":"2024-09-13T08:44:55.272959+0000","last_became_peered":"2024-09-13T08:44:55.272959+0000","last_unstale":"2024-09-13T08:44:56.492317+0000","last_undegraded":"2024-09-13T08:44:56.492317+0000","last_fullsized":"2024-09-13T08:44:56.492317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:15:56.726509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491787+0000","last_change":"2024-09-13T08:44:54.260286+0000","last_active":"2024-09-13T08:44:56.491787+0000","last_peered":"2024-09-13T08:44:56.491787+0000","last_clean":"2024-09-13T08:44:56.491787+0000","last_became_active":"2024-09-13T08:44:54.258226+0000","last_became_peered":"2024-09-13T08:44:54.258226+0000","last_unstale":"2024-09-13T08:44:56.491787+0000","last_undegraded":"2024-09-13T08:44:56.491787+0000","last_fullsized":"2024-09-13T08:44:56.491787+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:18.867194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491324+0000","last_change":"2024-09-13T08:44:55.284369+0000","last_active":"2024-09-13T08:44:56.491324+0000","last_peered":"2024-09-13T08:44:56.491324+0000","last_clean":"2024-09-13T08:44:56.491324+0000","last_became_active":"2024-09-13T08:44:55.284041+0000","last_became_peered":"2024-09-13T08:44:55.284041+0000","last_unstale":"2024-09-13T08:44:56.491324+0000","last_undegraded":"2024-09-13T08:44:56.491324+0000","last_fullsized":"2024-09-13T08:44:56.491324+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:36.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490728+0000","last_change":"2024-09-13T08:44:54.265229+0000","last_active":"2024-09-13T08:44:56.490728+0000","last_peered":"2024-09-13T08:44:56.490728+0000","last_clean":"2024-09-13T08:44:56.490728+0000","last_became_active":"2024-09-13T08:44:54.264945+0000","last_became_peered":"2024-09-13T08:44:54.264945+0000","last_unstale":"2024-09-13T08:44:56.490728+0000","last_undegraded":"2024-09-13T08:44:56.490728+0000","last_fullsized":"2024-09-13T08:44:56.490728+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:56:29.389478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291266+0000","last_change":"2024-09-13T08:44:55.267465+0000","last_active":"2024-09-13T08:44:56.291266+0000","last_peered":"2024-09-13T08:44:56.291266+0000","last_clean":"2024-09-13T08:44:56.291266+0000","last_became_active":"2024-09-13T08:44:55.267183+0000","last_became_peered":"2024-09-13T08:44:55.267183+0000","last_unstale":"2024-09-13T08:44:56.291266+0000","last_undegraded":"2024-09-13T08:44:56.291266+0000","last_fullsized":"2024-09-13T08:44:56.291266+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:42.198294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490291+0000","last_change":"2024-09-13T08:44:54.263949+0000","last_active":"2024-09-13T08:44:56.490291+0000","last_peered":"2024-09-13T08:44:56.490291+0000","last_clean":"2024-09-13T08:44:56.490291+0000","last_became_active":"2024-09-13T08:44:54.263668+0000","last_became_peered":"2024-09-13T08:44:54.263668+0000","last_unstale":"2024-09-13T08:44:56.490291+0000","last_undegraded":"2024-09-13T08:44:56.490291+0000","last_fullsized":"2024-09-13T08:44:56.490291+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:16.769289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274797+0000","last_change":"2024-09-13T08:44:55.262769+0000","last_active":"2024-09-13T08:44:56.274797+0000","last_peered":"2024-09-13T08:44:56.274797+0000","last_clean":"2024-09-13T08:44:56.274797+0000","last_became_active":"2024-09-13T08:44:55.262451+0000","last_became_peered":"2024-09-13T08:44:55.262451+0000","last_unstale":"2024-09-13T08:44:56.274797+0000","last_undegraded":"2024-09-13T08:44:56.274797+0000","last_fullsized":"2024-09-13T08:44:56.274797+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:05.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275544+0000","last_change":"2024-09-13T08:44:54.253005+0000","last_active":"2024-09-13T08:44:56.275544+0000","last_peered":"2024-09-13T08:44:56.275544+0000","last_clean":"2024-09-13T08:44:56.275544+0000","last_became_active":"2024-09-13T08:44:54.252685+0000","last_became_peered":"2024-09-13T08:44:54.252685+0000","last_unstale":"2024-09-13T08:44:56.275544+0000","last_undegraded":"2024-09-13T08:44:56.275544+0000","last_fullsized":"2024-09-13T08:44:56.275544+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:36:21.690288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291461+0000","last_change":"2024-09-13T08:44:55.272203+0000","last_active":"2024-09-13T08:44:56.291461+0000","last_peered":"2024-09-13T08:44:56.291461+0000","last_clean":"2024-09-13T08:44:56.291461+0000","last_became_active":"2024-09-13T08:44:55.271918+0000","last_became_peered":"2024-09-13T08:44:55.271918+0000","last_unstale":"2024-09-13T08:44:56.291461+0000","last_undegraded":"2024-09-13T08:44:56.291461+0000","last_fullsized":"2024-09-13T08:44:56.291461+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:30.597352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273576+0000","last_change":"2024-09-13T08:44:54.258728+0000","last_active":"2024-09-13T08:44:56.273576+0000","last_peered":"2024-09-13T08:44:56.273576+0000","last_clean":"2024-09-13T08:44:56.273576+0000","last_became_active":"2024-09-13T08:44:54.258408+0000","last_became_peered":"2024-09-13T08:44:54.258408+0000","last_unstale":"2024-09-13T08:44:56.273576+0000","last_undegraded":"2024-09-13T08:44:56.273576+0000","last_fullsized":"2024-09-13T08:44:56.273576+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:38:59.558456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274872+0000","last_change":"2024-09-13T08:44:55.281750+0000","last_active":"2024-09-13T08:44:56.274872+0000","last_peered":"2024-09-13T08:44:56.274872+0000","last_clean":"2024-09-13T08:44:56.274872+0000","last_became_active":"2024-09-13T08:44:55.281369+0000","last_became_peered":"2024-09-13T08:44:55.281369+0000","last_unstale":"2024-09-13T08:44:56.274872+0000","last_undegraded":"2024-09-13T08:44:56.274872+0000","last_fullsized":"2024-09-13T08:44:56.274872+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:52.352802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490749+0000","last_change":"2024-09-13T08:44:54.256616+0000","last_active":"2024-09-13T08:44:56.490749+0000","last_peered":"2024-09-13T08:44:56.490749+0000","last_clean":"2024-09-13T08:44:56.490749+0000","last_became_active":"2024-09-13T08:44:54.256352+0000","last_became_peered":"2024-09-13T08:44:54.256352+0000","last_unstale":"2024-09-13T08:44:56.490749+0000","last_undegraded":"2024-09-13T08:44:56.490749+0000","last_fullsized":"2024-09-13T08:44:56.490749+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:29.514973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290838+0000","last_change":"2024-09-13T08:44:55.266843+0000","last_active":"2024-09-13T08:44:56.290838+0000","last_peered":"2024-09-13T08:44:56.290838+0000","last_clean":"2024-09-13T08:44:56.290838+0000","last_became_active":"2024-09-13T08:44:55.266583+0000","last_became_peered":"2024-09-13T08:44:55.266583+0000","last_unstale":"2024-09-13T08:44:56.290838+0000","last_undegraded":"2024-09-13T08:44:56.290838+0000","last_fullsized":"2024-09-13T08:44:56.290838+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:49.797055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.489903+0000","last_change":"2024-09-13T08:44:54.269348+0000","last_active":"2024-09-13T08:44:56.489903+0000","last_peered":"2024-09-13T08:44:56.489903+0000","last_clean":"2024-09-13T08:44:56.489903+0000","last_became_active":"2024-09-13T08:44:54.268761+0000","last_became_peered":"2024-09-13T08:44:54.268761+0000","last_unstale":"2024-09-13T08:44:56.489903+0000","last_undegraded":"2024-09-13T08:44:56.489903+0000","last_fullsized":"2024-09-13T08:44:56.489903+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:14.558510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291364+0000","last_change":"2024-09-13T08:44:55.271361+0000","last_active":"2024-09-13T08:44:56.291364+0000","last_peered":"2024-09-13T08:44:56.291364+0000","last_clean":"2024-09-13T08:44:56.291364+0000","last_became_active":"2024-09-13T08:44:55.271070+0000","last_became_peered":"2024-09-13T08:44:55.271070+0000","last_unstale":"2024-09-13T08:44:56.291364+0000","last_undegraded":"2024-09-13T08:44:56.291364+0000","last_fullsized":"2024-09-13T08:44:56.291364+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:31.811236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491500+0000","last_change":"2024-09-13T08:44:54.262267+0000","last_active":"2024-09-13T08:44:56.491500+0000","last_peered":"2024-09-13T08:44:56.491500+0000","last_clean":"2024-09-13T08:44:56.491500+0000","last_became_active":"2024-09-13T08:44:54.261926+0000","last_became_peered":"2024-09-13T08:44:54.261926+0000","last_unstale":"2024-09-13T08:44:56.491500+0000","last_undegraded":"2024-09-13T08:44:56.491500+0000","last_fullsized":"2024-09-13T08:44:56.491500+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:24.478636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491400+0000","last_change":"2024-09-13T08:44:55.279445+0000","last_active":"2024-09-13T08:44:56.491400+0000","last_peered":"2024-09-13T08:44:56.491400+0000","last_clean":"2024-09-13T08:44:56.491400+0000","last_became_active":"2024-09-13T08:44:55.271480+0000","last_became_peered":"2024-09-13T08:44:55.271480+0000","last_unstale":"2024-09-13T08:44:56.491400+0000","last_undegraded":"2024-09-13T08:44:56.491400+0000","last_fullsized":"2024-09-13T08:44:56.491400+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:04.111848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274178+0000","last_change":"2024-09-13T08:44:54.259956+0000","last_active":"2024-09-13T08:44:56.274178+0000","last_peered":"2024-09-13T08:44:56.274178+0000","last_clean":"2024-09-13T08:44:56.274178+0000","last_became_active":"2024-09-13T08:44:54.259304+0000","last_became_peered":"2024-09-13T08:44:54.259304+0000","last_unstale":"2024-09-13T08:44:56.274178+0000","last_undegraded":"2024-09-13T08:44:56.274178+0000","last_fullsized":"2024-09-13T08:44:56.274178+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:01.426416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492145+0000","last_change":"2024-09-13T08:44:55.278564+0000","last_active":"2024-09-13T08:44:56.492145+0000","last_peered":"2024-09-13T08:44:56.492145+0000","last_clean":"2024-09-13T08:44:56.492145+0000","last_became_active":"2024-09-13T08:44:55.276693+0000","last_became_peered":"2024-09-13T08:44:55.276693+0000","last_unstale":"2024-09-13T08:44:56.492145+0000","last_undegraded":"2024-09-13T08:44:56.492145+0000","last_fullsized":"2024-09-13T08:44:56.492145+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:10.647195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":21,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492022+0000","last_change":"2024-09-13T08:44:54.262744+0000","last_active":"2024-09-13T08:44:56.492022+0000","last_peered":"2024-09-13T08:44:56.492022+0000","last_clean":"2024-09-13T08:44:56.492022+0000","last_became_active":"2024-09-13T08:44:54.262327+0000","last_became_peered":"2024-09-13T08:44:54.262327+0000","last_unstale":"2024-09-13T08:44:56.492022+0000","last_undegraded":"2024-09-13T08:44:56.492022+0000","last_fullsized":"2024-09-13T08:44:56.492022+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:53.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290851+0000","last_change":"2024-09-13T08:44:55.284935+0000","last_active":"2024-09-13T08:44:56.290851+0000","last_peered":"2024-09-13T08:44:56.290851+0000","last_clean":"2024-09-13T08:44:56.290851+0000","last_became_active":"2024-09-13T08:44:55.284660+0000","last_became_peered":"2024-09-13T08:44:55.284660+0000","last_unstale":"2024-09-13T08:44:56.290851+0000","last_undegraded":"2024-09-13T08:44:56.290851+0000","last_fullsized":"2024-09-13T08:44:56.290851+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:59.543608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490153+0000","last_change":"2024-09-13T08:44:54.264285+0000","last_active":"2024-09-13T08:44:56.490153+0000","last_peered":"2024-09-13T08:44:56.490153+0000","last_clean":"2024-09-13T08:44:56.490153+0000","last_became_active":"2024-09-13T08:44:54.264028+0000","last_became_peered":"2024-09-13T08:44:54.264028+0000","last_unstale":"2024-09-13T08:44:56.490153+0000","last_undegraded":"2024-09-13T08:44:56.490153+0000","last_fullsized":"2024-09-13T08:44:56.490153+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:20:29.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275724+0000","last_change":"2024-09-13T08:44:55.280824+0000","last_active":"2024-09-13T08:44:56.275724+0000","last_peered":"2024-09-13T08:44:56.275724+0000","last_clean":"2024-09-13T08:44:56.275724+0000","last_became_active":"2024-09-13T08:44:55.280527+0000","last_became_peered":"2024-09-13T08:44:55.280527+0000","last_unstale":"2024-09-13T08:44:56.275724+0000","last_undegraded":"2024-09-13T08:44:56.275724+0000","last_fullsized":"2024-09-13T08:44:56.275724+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:48.020108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491455+0000","last_change":"2024-09-13T08:44:54.260694+0000","last_active":"2024-09-13T08:44:56.491455+0000","last_peered":"2024-09-13T08:44:56.491455+0000","last_clean":"2024-09-13T08:44:56.491455+0000","last_became_active":"2024-09-13T08:44:54.260412+0000","last_became_peered":"2024-09-13T08:44:54.260412+0000","last_unstale":"2024-09-13T08:44:56.491455+0000","last_undegraded":"2024-09-13T08:44:56.491455+0000","last_fullsized":"2024-09-13T08:44:56.491455+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:02.170087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290706+0000","last_change":"2024-09-13T08:44:55.277787+0000","last_active":"2024-09-13T08:44:56.290706+0000","last_peered":"2024-09-13T08:44:56.290706+0000","last_clean":"2024-09-13T08:44:56.290706+0000","last_became_active":"2024-09-13T08:44:55.277490+0000","last_became_peered":"2024-09-13T08:44:55.277490+0000","last_unstale":"2024-09-13T08:44:56.290706+0000","last_undegraded":"2024-09-13T08:44:56.290706+0000","last_fullsized":"2024-09-13T08:44:56.290706+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:52.158716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275671+0000","last_change":"2024-09-13T08:44:54.256931+0000","last_active":"2024-09-13T08:44:56.275671+0000","last_peered":"2024-09-13T08:44:56.275671+0000","last_clean":"2024-09-13T08:44:56.275671+0000","last_became_active":"2024-09-13T08:44:54.256635+0000","last_became_peered":"2024-09-13T08:44:54.256635+0000","last_unstale":"2024-09-13T08:44:56.275671+0000","last_undegraded":"2024-09-13T08:44:56.275671+0000","last_fullsized":"2024-09-13T08:44:56.275671+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:51.526164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291079+0000","last_change":"2024-09-13T08:44:55.273932+0000","last_active":"2024-09-13T08:44:56.291079+0000","last_peered":"2024-09-13T08:44:56.291079+0000","last_clean":"2024-09-13T08:44:56.291079+0000","last_became_active":"2024-09-13T08:44:55.273133+0000","last_became_peered":"2024-09-13T08:44:55.273133+0000","last_unstale":"2024-09-13T08:44:56.291079+0000","last_undegraded":"2024-09-13T08:44:56.291079+0000","last_fullsized":"2024-09-13T08:44:56.291079+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:30.524014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":23,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274659+0000","last_change":"2024-09-13T08:44:54.256980+0000","last_active":"2024-09-13T08:44:56.274659+0000","last_peered":"2024-09-13T08:44:56.274659+0000","last_clean":"2024-09-13T08:44:56.274659+0000","last_became_active":"2024-09-13T08:44:54.256693+0000","last_became_peered":"2024-09-13T08:44:54.256693+0000","last_unstale":"2024-09-13T08:44:56.274659+0000","last_undegraded":"2024-09-13T08:44:56.274659+0000","last_fullsized":"2024-09-13T08:44:56.274659+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:28.513665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274234+0000","last_change":"2024-09-13T08:44:55.285849+0000","last_active":"2024-09-13T08:44:56.274234+0000","last_peered":"2024-09-13T08:44:56.274234+0000","last_clean":"2024-09-13T08:44:56.274234+0000","last_became_active":"2024-09-13T08:44:55.285566+0000","last_became_peered":"2024-09-13T08:44:55.285566+0000","last_unstale":"2024-09-13T08:44:56.274234+0000","last_undegraded":"2024-09-13T08:44:56.274234+0000","last_fullsized":"2024-09-13T08:44:56.274234+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:22.285510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291496+0000","last_change":"2024-09-13T08:44:54.259191+0000","last_active":"2024-09-13T08:44:56.291496+0000","last_peered":"2024-09-13T08:44:56.291496+0000","last_clean":"2024-09-13T08:44:56.291496+0000","last_became_active":"2024-09-13T08:44:54.258798+0000","last_became_peered":"2024-09-13T08:44:54.258798+0000","last_unstale":"2024-09-13T08:44:56.291496+0000","last_undegraded":"2024-09-13T08:44:56.291496+0000","last_fullsized":"2024-09-13T08:44:56.291496+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:48.023861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276214+0000","last_change":"2024-09-13T08:44:55.278575+0000","last_active":"2024-09-13T08:44:56.276214+0000","last_peered":"2024-09-13T08:44:56.276214+0000","last_clean":"2024-09-13T08:44:56.276214+0000","last_became_active":"2024-09-13T08:44:55.276763+0000","last_became_peered":"2024-09-13T08:44:55.276763+0000","last_unstale":"2024-09-13T08:44:56.276214+0000","last_undegraded":"2024-09-13T08:44:56.276214+0000","last_fullsized":"2024-09-13T08:44:56.276214+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:56:58.603337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275776+0000","last_change":"2024-09-13T08:44:54.259077+0000","last_active":"2024-09-13T08:44:56.275776+0000","last_peered":"2024-09-13T08:44:56.275776+0000","last_clean":"2024-09-13T08:44:56.275776+0000","last_became_active":"2024-09-13T08:44:54.258694+0000","last_became_peered":"2024-09-13T08:44:54.258694+0000","last_unstale":"2024-09-13T08:44:56.275776+0000","last_undegraded":"2024-09-13T08:44:56.275776+0000","last_fullsized":"2024-09-13T08:44:56.275776+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:25.036242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.274042+0000","last_change":"2024-09-13T08:44:55.286464+0000","last_active":"2024-09-13T08:44:56.274042+0000","last_peered":"2024-09-13T08:44:56.274042+0000","last_clean":"2024-09-13T08:44:56.274042+0000","last_became_active":"2024-09-13T08:44:55.286137+0000","last_became_peered":"2024-09-13T08:44:55.286137+0000","last_unstale":"2024-09-13T08:44:56.274042+0000","last_undegraded":"2024-09-13T08:44:56.274042+0000","last_fullsized":"2024-09-13T08:44:56.274042+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:02:16.803489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492273+0000","last_change":"2024-09-13T08:44:54.263421+0000","last_active":"2024-09-13T08:44:56.492273+0000","last_peered":"2024-09-13T08:44:56.492273+0000","last_clean":"2024-09-13T08:44:56.492273+0000","last_became_active":"2024-09-13T08:44:54.262563+0000","last_became_peered":"2024-09-13T08:44:54.262563+0000","last_unstale":"2024-09-13T08:44:56.492273+0000","last_undegraded":"2024-09-13T08:44:56.492273+0000","last_fullsized":"2024-09-13T08:44:56.492273+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:44.116362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491142+0000","last_change":"2024-09-13T08:44:55.276553+0000","last_active":"2024-09-13T08:44:56.491142+0000","last_peered":"2024-09-13T08:44:56.491142+0000","last_clean":"2024-09-13T08:44:56.491142+0000","last_became_active":"2024-09-13T08:44:55.276152+0000","last_became_peered":"2024-09-13T08:44:55.276152+0000","last_unstale":"2024-09-13T08:44:56.491142+0000","last_undegraded":"2024-09-13T08:44:56.491142+0000","last_fullsized":"2024-09-13T08:44:56.491142+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:07.680327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.276293+0000","last_change":"2024-09-13T08:44:54.260419+0000","last_active":"2024-09-13T08:44:56.276293+0000","last_peered":"2024-09-13T08:44:56.276293+0000","last_clean":"2024-09-13T08:44:56.276293+0000","last_became_active":"2024-09-13T08:44:54.260132+0000","last_became_peered":"2024-09-13T08:44:54.260132+0000","last_unstale":"2024-09-13T08:44:56.276293+0000","last_undegraded":"2024-09-13T08:44:56.276293+0000","last_fullsized":"2024-09-13T08:44:56.276293+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:53:30.607909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491419+0000","last_change":"2024-09-13T08:44:55.273138+0000","last_active":"2024-09-13T08:44:56.491419+0000","last_peered":"2024-09-13T08:44:56.491419+0000","last_clean":"2024-09-13T08:44:56.491419+0000","last_became_active":"2024-09-13T08:44:55.272839+0000","last_became_peered":"2024-09-13T08:44:55.272839+0000","last_unstale":"2024-09-13T08:44:56.491419+0000","last_undegraded":"2024-09-13T08:44:56.491419+0000","last_fullsized":"2024-09-13T08:44:56.491419+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:14.425364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490771+0000","last_change":"2024-09-13T08:44:54.263813+0000","last_active":"2024-09-13T08:44:56.490771+0000","last_peered":"2024-09-13T08:44:56.490771+0000","last_clean":"2024-09-13T08:44:56.490771+0000","last_became_active":"2024-09-13T08:44:54.262871+0000","last_became_peered":"2024-09-13T08:44:54.262871+0000","last_unstale":"2024-09-13T08:44:56.490771+0000","last_undegraded":"2024-09-13T08:44:56.490771+0000","last_fullsized":"2024-09-13T08:44:56.490771+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:58:53.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275758+0000","last_change":"2024-09-13T08:44:55.279333+0000","last_active":"2024-09-13T08:44:56.275758+0000","last_peered":"2024-09-13T08:44:56.275758+0000","last_clean":"2024-09-13T08:44:56.275758+0000","last_became_active":"2024-09-13T08:44:55.278510+0000","last_became_peered":"2024-09-13T08:44:55.278510+0000","last_unstale":"2024-09-13T08:44:56.275758+0000","last_undegraded":"2024-09-13T08:44:56.275758+0000","last_fullsized":"2024-09-13T08:44:56.275758+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:41:47.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290900+0000","last_change":"2024-09-13T08:44:54.267401+0000","last_active":"2024-09-13T08:44:56.290900+0000","last_peered":"2024-09-13T08:44:56.290900+0000","last_clean":"2024-09-13T08:44:56.290900+0000","last_became_active":"2024-09-13T08:44:54.267099+0000","last_became_peered":"2024-09-13T08:44:54.267099+0000","last_unstale":"2024-09-13T08:44:56.290900+0000","last_undegraded":"2024-09-13T08:44:56.290900+0000","last_fullsized":"2024-09-13T08:44:56.290900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:42:19.972272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490446+0000","last_change":"2024-09-13T08:44:55.275417+0000","last_active":"2024-09-13T08:44:56.490446+0000","last_peered":"2024-09-13T08:44:56.490446+0000","last_clean":"2024-09-13T08:44:56.490446+0000","last_became_active":"2024-09-13T08:44:55.274975+0000","last_became_peered":"2024-09-13T08:44:55.274975+0000","last_unstale":"2024-09-13T08:44:56.490446+0000","last_undegraded":"2024-09-13T08:44:56.490446+0000","last_fullsized":"2024-09-13T08:44:56.490446+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:59:35.665465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290954+0000","last_change":"2024-09-13T08:44:54.262253+0000","last_active":"2024-09-13T08:44:56.290954+0000","last_peered":"2024-09-13T08:44:56.290954+0000","last_clean":"2024-09-13T08:44:56.290954+0000","last_became_active":"2024-09-13T08:44:54.258536+0000","last_became_peered":"2024-09-13T08:44:54.258536+0000","last_unstale":"2024-09-13T08:44:56.290954+0000","last_undegraded":"2024-09-13T08:44:56.290954+0000","last_fullsized":"2024-09-13T08:44:56.290954+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:56:48.331106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275128+0000","last_change":"2024-09-13T08:44:55.277516+0000","last_active":"2024-09-13T08:44:56.275128+0000","last_peered":"2024-09-13T08:44:56.275128+0000","last_clean":"2024-09-13T08:44:56.275128+0000","last_became_active":"2024-09-13T08:44:55.276968+0000","last_became_peered":"2024-09-13T08:44:55.276968+0000","last_unstale":"2024-09-13T08:44:56.275128+0000","last_undegraded":"2024-09-13T08:44:56.275128+0000","last_fullsized":"2024-09-13T08:44:56.275128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:00.122856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290830+0000","last_change":"2024-09-13T08:44:54.267335+0000","last_active":"2024-09-13T08:44:56.290830+0000","last_peered":"2024-09-13T08:44:56.290830+0000","last_clean":"2024-09-13T08:44:56.290830+0000","last_became_active":"2024-09-13T08:44:54.266819+0000","last_became_peered":"2024-09-13T08:44:54.266819+0000","last_unstale":"2024-09-13T08:44:56.290830+0000","last_undegraded":"2024-09-13T08:44:56.290830+0000","last_fullsized":"2024-09-13T08:44:56.290830+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:06.861965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291360+0000","last_change":"2024-09-13T08:44:55.270799+0000","last_active":"2024-09-13T08:44:56.291360+0000","last_peered":"2024-09-13T08:44:56.291360+0000","last_clean":"2024-09-13T08:44:56.291360+0000","last_became_active":"2024-09-13T08:44:55.270510+0000","last_became_peered":"2024-09-13T08:44:55.270510+0000","last_unstale":"2024-09-13T08:44:56.291360+0000","last_undegraded":"2024-09-13T08:44:56.291360+0000","last_fullsized":"2024-09-13T08:44:56.291360+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:02.322294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291576+0000","last_change":"2024-09-13T08:44:54.260321+0000","last_active":"2024-09-13T08:44:56.291576+0000","last_peered":"2024-09-13T08:44:56.291576+0000","last_clean":"2024-09-13T08:44:56.291576+0000","last_became_active":"2024-09-13T08:44:54.256112+0000","last_became_peered":"2024-09-13T08:44:54.256112+0000","last_unstale":"2024-09-13T08:44:56.291576+0000","last_undegraded":"2024-09-13T08:44:56.291576+0000","last_fullsized":"2024-09-13T08:44:56.291576+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:26.247091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291444+0000","last_change":"2024-09-13T08:44:55.270694+0000","last_active":"2024-09-13T08:44:56.291444+0000","last_peered":"2024-09-13T08:44:56.291444+0000","last_clean":"2024-09-13T08:44:56.291444+0000","last_became_active":"2024-09-13T08:44:55.270395+0000","last_became_peered":"2024-09-13T08:44:55.270395+0000","last_unstale":"2024-09-13T08:44:56.291444+0000","last_undegraded":"2024-09-13T08:44:56.291444+0000","last_fullsized":"2024-09-13T08:44:56.291444+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:08.047668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290003+0000","last_change":"2024-09-13T08:44:54.269125+0000","last_active":"2024-09-13T08:44:56.290003+0000","last_peered":"2024-09-13T08:44:56.290003+0000","last_clean":"2024-09-13T08:44:56.290003+0000","last_became_active":"2024-09-13T08:44:54.268832+0000","last_became_peered":"2024-09-13T08:44:54.268832+0000","last_unstale":"2024-09-13T08:44:56.290003+0000","last_undegraded":"2024-09-13T08:44:56.290003+0000","last_fullsized":"2024-09-13T08:44:56.290003+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:01:36.832036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275992+0000","last_change":"2024-09-13T08:44:55.265866+0000","last_active":"2024-09-13T08:44:56.275992+0000","last_peered":"2024-09-13T08:44:56.275992+0000","last_clean":"2024-09-13T08:44:56.275992+0000","last_became_active":"2024-09-13T08:44:55.265212+0000","last_became_peered":"2024-09-13T08:44:55.265212+0000","last_unstale":"2024-09-13T08:44:56.275992+0000","last_undegraded":"2024-09-13T08:44:56.275992+0000","last_fullsized":"2024-09-13T08:44:56.275992+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:00:23.600901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.492246+0000","last_change":"2024-09-13T08:44:54.262846+0000","last_active":"2024-09-13T08:44:56.492246+0000","last_peered":"2024-09-13T08:44:56.492246+0000","last_clean":"2024-09-13T08:44:56.492246+0000","last_became_active":"2024-09-13T08:44:54.262376+0000","last_became_peered":"2024-09-13T08:44:54.262376+0000","last_unstale":"2024-09-13T08:44:56.492246+0000","last_undegraded":"2024-09-13T08:44:56.492246+0000","last_fullsized":"2024-09-13T08:44:56.492246+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:30.802195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275248+0000","last_change":"2024-09-13T08:44:55.271553+0000","last_active":"2024-09-13T08:44:56.275248+0000","last_peered":"2024-09-13T08:44:56.275248+0000","last_clean":"2024-09-13T08:44:56.275248+0000","last_became_active":"2024-09-13T08:44:55.271278+0000","last_became_peered":"2024-09-13T08:44:55.271278+0000","last_unstale":"2024-09-13T08:44:56.275248+0000","last_undegraded":"2024-09-13T08:44:56.275248+0000","last_fullsized":"2024-09-13T08:44:56.275248+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:12.548581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491978+0000","last_change":"2024-09-13T08:44:54.255976+0000","last_active":"2024-09-13T08:44:56.491978+0000","last_peered":"2024-09-13T08:44:56.491978+0000","last_clean":"2024-09-13T08:44:56.491978+0000","last_became_active":"2024-09-13T08:44:54.255663+0000","last_became_peered":"2024-09-13T08:44:54.255663+0000","last_unstale":"2024-09-13T08:44:56.491978+0000","last_undegraded":"2024-09-13T08:44:56.491978+0000","last_fullsized":"2024-09-13T08:44:56.491978+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:29.514973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491452+0000","last_change":"2024-09-13T08:44:55.285736+0000","last_active":"2024-09-13T08:44:56.491452+0000","last_peered":"2024-09-13T08:44:56.491452+0000","last_clean":"2024-09-13T08:44:56.491452+0000","last_became_active":"2024-09-13T08:44:55.285497+0000","last_became_peered":"2024-09-13T08:44:55.285497+0000","last_unstale":"2024-09-13T08:44:56.491452+0000","last_undegraded":"2024-09-13T08:44:56.491452+0000","last_fullsized":"2024-09-13T08:44:56.491452+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:01:24.452026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.275873+0000","last_change":"2024-09-13T08:44:54.252490+0000","last_active":"2024-09-13T08:44:56.275873+0000","last_peered":"2024-09-13T08:44:56.275873+0000","last_clean":"2024-09-13T08:44:56.275873+0000","last_became_active":"2024-09-13T08:44:54.252216+0000","last_became_peered":"2024-09-13T08:44:54.252216+0000","last_unstale":"2024-09-13T08:44:56.275873+0000","last_undegraded":"2024-09-13T08:44:56.275873+0000","last_fullsized":"2024-09-13T08:44:56.275873+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:01:44.009297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.291381+0000","last_change":"2024-09-13T08:44:55.280041+0000","last_active":"2024-09-13T08:44:56.291381+0000","last_peered":"2024-09-13T08:44:56.291381+0000","last_clean":"2024-09-13T08:44:56.291381+0000","last_became_active":"2024-09-13T08:44:55.279759+0000","last_became_peered":"2024-09-13T08:44:55.279759+0000","last_unstale":"2024-09-13T08:44:56.291381+0000","last_undegraded":"2024-09-13T08:44:56.291381+0000","last_fullsized":"2024-09-13T08:44:56.291381+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:54:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490150+0000","last_change":"2024-09-13T08:44:54.264718+0000","last_active":"2024-09-13T08:44:56.490150+0000","last_peered":"2024-09-13T08:44:56.490150+0000","last_clean":"2024-09-13T08:44:56.490150+0000","last_became_active":"2024-09-13T08:44:54.264448+0000","last_became_peered":"2024-09-13T08:44:54.264448+0000","last_unstale":"2024-09-13T08:44:56.490150+0000","last_undegraded":"2024-09-13T08:44:56.490150+0000","last_fullsized":"2024-09-13T08:44:56.490150+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:23.920024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.290771+0000","last_change":"2024-09-13T08:44:55.265806+0000","last_active":"2024-09-13T08:44:56.290771+0000","last_peered":"2024-09-13T08:44:56.290771+0000","last_clean":"2024-09-13T08:44:56.290771+0000","last_became_active":"2024-09-13T08:44:55.265520+0000","last_became_peered":"2024-09-13T08:44:55.265520+0000","last_unstale":"2024-09-13T08:44:56.290771+0000","last_undegraded":"2024-09-13T08:44:56.290771+0000","last_fullsized":"2024-09-13T08:44:56.290771+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:43:51.191345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490338+0000","last_change":"2024-09-13T08:44:54.260295+0000","last_active":"2024-09-13T08:44:56.490338+0000","last_peered":"2024-09-13T08:44:56.490338+0000","last_clean":"2024-09-13T08:44:56.490338+0000","last_became_active":"2024-09-13T08:44:54.260033+0000","last_became_peered":"2024-09-13T08:44:54.260033+0000","last_unstale":"2024-09-13T08:44:56.490338+0000","last_undegraded":"2024-09-13T08:44:56.490338+0000","last_fullsized":"2024-09-13T08:44:56.490338+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:35.398014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.489955+0000","last_change":"2024-09-13T08:44:55.266487+0000","last_active":"2024-09-13T08:44:56.489955+0000","last_peered":"2024-09-13T08:44:56.489955+0000","last_clean":"2024-09-13T08:44:56.489955+0000","last_became_active":"2024-09-13T08:44:55.266244+0000","last_became_peered":"2024-09-13T08:44:55.266244+0000","last_unstale":"2024-09-13T08:44:56.489955+0000","last_undegraded":"2024-09-13T08:44:56.489955+0000","last_fullsized":"2024-09-13T08:44:56.489955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:01:33.630012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.490614+0000","last_change":"2024-09-13T08:44:54.259464+0000","last_active":"2024-09-13T08:44:56.490614+0000","last_peered":"2024-09-13T08:44:56.490614+0000","last_clean":"2024-09-13T08:44:56.490614+0000","last_became_active":"2024-09-13T08:44:54.259135+0000","last_became_peered":"2024-09-13T08:44:54.259135+0000","last_unstale":"2024-09-13T08:44:56.490614+0000","last_undegraded":"2024-09-13T08:44:56.490614+0000","last_fullsized":"2024-09-13T08:44:56.490614+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:35.443612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491390+0000","last_change":"2024-09-13T08:44:55.276016+0000","last_active":"2024-09-13T08:44:56.491390+0000","last_peered":"2024-09-13T08:44:56.491390+0000","last_clean":"2024-09-13T08:44:56.491390+0000","last_became_active":"2024-09-13T08:44:55.275632+0000","last_became_peered":"2024-09-13T08:44:55.275632+0000","last_unstale":"2024-09-13T08:44:56.491390+0000","last_undegraded":"2024-09-13T08:44:56.491390+0000","last_fullsized":"2024-09-13T08:44:56.491390+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:54.233456+0000","last_clean_scrub_stamp":"2024-09-13T08:44:54.233456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:07.354992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.491397+0000","last_change":"2024-09-13T08:44:54.262412+0000","last_active":"2024-09-13T08:44:56.491397+0000","last_peered":"2024-09-13T08:44:56.491397+0000","last_clean":"2024-09-13T08:44:56.491397+0000","last_became_active":"2024-09-13T08:44:54.261172+0000","last_became_peered":"2024-09-13T08:44:54.261172+0000","last_unstale":"2024-09-13T08:44:56.491397+0000","last_undegraded":"2024-09-13T08:44:56.491397+0000","last_fullsized":"2024-09-13T08:44:56.491397+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:44:53.224415+0000","last_clean_scrub_stamp":"2024-09-13T08:44:53.224415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:06.671286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":351,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-13T08:44:56.273381+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:44:56.273381+0000","last_peered":"2024-09-13T08:44:56.273381+0000","last_clean":"2024-09-13T08:44:56.273381+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:44:56.273381+0000","last_undegraded":"2024-09-13T08:44:56.273381+0000","last_fullsized":"2024-09-13T08:44:56.273381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1732608,"data_stored":768052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509539,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1720320,"data_stored":761708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":2351104,"data_stored":1360226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542242,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1753088,"data_stored":770268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":762324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1724416,"data_stored":761730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509540,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1358490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1724416,"data_stored":761798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:44:59.952 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:44:59.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 -- 172.21.15.31:0/152449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a0c004d90 msgr2=0x7f4a0c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/152449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a0c004d90 0x7f4a0c005210 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c6c0 tx=0x7f4a0c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a27e00640 1 -- 172.21.15.31:0/152449 reap_dead start 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 -- 172.21.15.31:0/152449 shutdown_connections 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 -- 172.21.15.31:0/152449 >> 172.21.15.31:0/152449 conn(0x7f4a280b1520 msgr2=0x7f4a280b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 -- 172.21.15.31:0/152449 shutdown_connections 2024-09-13T08:45:00.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.155+0000 7f4a2ce00640 1 -- 172.21.15.31:0/152449 wait complete. 2024-09-13T08:45:00.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.157+0000 7f4a2ce00640 1 Processor -- start 2024-09-13T08:45:00.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.157+0000 7f4a2ce00640 1 -- start start 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a0c004d90 0x7f4a280c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a281f8160 0x7f4a2815e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280cc9d0 con 0x7f4a280c84f0 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280cc850 con 0x7f4a281f8160 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a2ce00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4a280ccb50 con 0x7f4a0c004d90 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a1ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a1ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45166/0 (socket says 172.21.15.31:45166) 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a1ea00640 1 -- 172.21.15.31:0/3644899950 learned_addr learned my addr 172.21.15.31:0/3644899950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a27400640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a0c004d90 0x7f4a280c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.158+0000 7f4a27e00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a281f8160 0x7f4a2815e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 -- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a0c004d90 msgr2=0x7f4a280c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a0c004d90 0x7f4a280c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 -- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a281f8160 msgr2=0x7f4a2815e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a281f8160 0x7f4a2815e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 -- 172.21.15.31:0/3644899950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a280c7450 con 0x7f4a280c84f0 2024-09-13T08:45:00.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1ea00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4a100067f0 tx=0x7f4a10001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10015070 con 0x7f4a280c84f0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a10011880 con 0x7f4a280c84f0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10011cd0 con 0x7f4a280c84f0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a280c7730 con 0x7f4a280c84f0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.159+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a280c7cf0 con 0x7f4a280c84f0 2024-09-13T08:45:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.160+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a280b1310 con 0x7f4a280c84f0 2024-09-13T08:45:00.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.162+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f4a10018070 con 0x7f4a280c84f0 2024-09-13T08:45:00.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.163+0000 7f4a1fe00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 0x7f4a0407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.163+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f4a1009be80 con 0x7f4a280c84f0 2024-09-13T08:45:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.163+0000 7f4a27400640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 0x7f4a0407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.163+0000 7f4a27400640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 0x7f4a0407b340 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009c50 tx=0x7f4a18005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.167+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4a280b1310 con 0x7f4a280c84f0 2024-09-13T08:45:00.221 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:45:00.219+0000 7fd5c1400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:45:00.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.438+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4a28069fa0 con 0x7f4a280c84f0 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.441+0000 7f4a1fe00640 1 -- 172.21.15.31:0/3644899950 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7f4a28069fa0 con 0x7f4a280c84f0 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:45:00.443 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:45:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 msgr2=0x7f4a0407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 0x7f4a0407b340 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009c50 tx=0x7f4a18005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 msgr2=0x7f4a281f8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4a100067f0 tx=0x7f4a10001800 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 shutdown_connections 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4a04078e80 0x7f4a0407b340 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a281f8160 0x7f4a2815e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a280c84f0 0x7f4a281f8ae0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 --2- 172.21.15.31:0/3644899950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4a0c004d90 0x7f4a280c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 >> 172.21.15.31:0/3644899950 conn(0x7f4a280b1520 msgr2=0x7f4a2806eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 shutdown_connections 2024-09-13T08:45:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.444+0000 7f4a2ce00640 1 -- 172.21.15.31:0/3644899950 wait complete. 2024-09-13T08:45:00.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:00.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 -- 172.21.15.31:0/886784634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 msgr2=0x7f0d8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 --2- 172.21.15.31:0/886784634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0d8c004c80 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009b30 tx=0x7f0d8c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0da9e00640 1 -- 172.21.15.31:0/886784634 reap_dead start 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 -- 172.21.15.31:0/886784634 shutdown_connections 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 -- 172.21.15.31:0/886784634 >> 172.21.15.31:0/886784634 conn(0x7f0da40cc910 msgr2=0x7f0da40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 -- 172.21.15.31:0/886784634 shutdown_connections 2024-09-13T08:45:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.622+0000 7f0dabe00640 1 -- 172.21.15.31:0/886784634 wait complete. 2024-09-13T08:45:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.624+0000 7f0dabe00640 1 Processor -- start 2024-09-13T08:45:00.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.624+0000 7f0dabe00640 1 -- start start 2024-09-13T08:45:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 0x7f0da406e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da40c8280 0x7f0da40c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0da40b1da0 con 0x7f0d8c004800 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0da4074150 con 0x7f0da40c8280 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0dabe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0da40b1f20 con 0x7f0da415c260 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0da9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0da9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45196/0 (socket says 172.21.15.31:45196) 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.625+0000 7f0da9400640 1 -- 172.21.15.31:0/1149164447 learned_addr learned my addr 172.21.15.31:0/1149164447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da8a00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 0x7f0da406e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9e00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da40c8280 0x7f0da40c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 -- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 msgr2=0x7f0da406e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 0x7f0da406e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 -- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da40c8280 msgr2=0x7f0da40c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da40c8280 0x7f0da40c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 -- 172.21.15.31:0/1149164447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da415ca90 con 0x7f0d8c004800 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da8a00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 0x7f0da406e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da9400640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f0d98010aa0 tx=0x7f0d98010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9800ee90 con 0x7f0d8c004800 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d98005020 con 0x7f0d8c004800 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da409abb0 con 0x7f0d8c004800 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.626+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98005430 con 0x7f0d8c004800 2024-09-13T08:45:00.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.627+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da40c7000 con 0x7f0d8c004800 2024-09-13T08:45:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.627+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da40cc300 con 0x7f0d8c004800 2024-09-13T08:45:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.629+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f0d980047a0 con 0x7f0d8c004800 2024-09-13T08:45:00.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.630+0000 7f0da1600640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 0x7f0d8407b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:00.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.630+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f0d9809c590 con 0x7f0d8c004800 2024-09-13T08:45:00.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.630+0000 7f0da8a00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 0x7f0d8407b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:00.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.631+0000 7f0da8a00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 0x7f0d8407b510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0d90034000 tx=0x7f0d9002d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.634+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0d98064140 con 0x7f0d8c004800 2024-09-13T08:45:00.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.865+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0da40af5d0 con 0x7f0d8c004800 2024-09-13T08:45:00.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.866+0000 7f0da1600640 1 -- 172.21.15.31:0/1149164447 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5221 (secure 0 0 0) 0x7f0d98063a40 con 0x7f0d8c004800 2024-09-13T08:45:00.868 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:00.868 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"btime":"2024-09-13T08:44:58:284276+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:45:00.868 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2024-09-13T08:45:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 msgr2=0x7f0d8407b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 0x7f0d8407b510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0d90034000 tx=0x7f0d9002d040 comp rx=0 tx=0).stop 2024-09-13T08:45:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 msgr2=0x7f0da409d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f0d98010aa0 tx=0x7f0d98010f60 comp rx=0 tx=0).stop 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 shutdown_connections 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0d84079050 0x7f0d8407b510 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da40c8280 0x7f0da40c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0da415c260 0x7f0da406e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 --2- 172.21.15.31:0/1149164447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d8c004800 0x7f0da409d5a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 >> 172.21.15.31:0/1149164447 conn(0x7f0da40cc910 msgr2=0x7f0da40ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 shutdown_connections 2024-09-13T08:45:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:00.868+0000 7f0dabe00640 1 -- 172.21.15.31:0/1149164447 wait complete. 2024-09-13T08:45:01.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:44:54.747650+0000', 'modified': '2024-09-13T08:44:57.717583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7989}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7989': {'gid': 7989, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6834/1614002559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 1614002559}, {'type': 'v1', 'addr': '172.21.15.31:6834', 'nonce': 1614002559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7989, 'qdb_cluster': [7989]} 2024-09-13T08:45:01.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:45:01.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:45:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 -- 172.21.15.31:0/1406657314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c06e670 msgr2=0x7fdb6c0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 --2- 172.21.15.31:0/1406657314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0cdf70 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fdb54005fd0 tx=0x7fdb5402a540 comp rx=0 tx=0).stop 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 -- 172.21.15.31:0/1406657314 shutdown_connections 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 --2- 172.21.15.31:0/1406657314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0cdf70 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 --2- 172.21.15.31:0/1406657314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 0x7fdb6c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 -- 172.21.15.31:0/1406657314 >> 172.21.15.31:0/1406657314 conn(0x7fdb6c0cc7a0 msgr2=0x7fdb6c0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 -- 172.21.15.31:0/1406657314 shutdown_connections 2024-09-13T08:45:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.451+0000 7fdb6b200640 1 -- 172.21.15.31:0/1406657314 wait complete. 2024-09-13T08:45:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.453+0000 7fdb6b200640 1 Processor -- start 2024-09-13T08:45:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.453+0000 7fdb6b200640 1 -- start start 2024-09-13T08:45:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 0x7fdb6c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c070040 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c06fec0 con 0x7fdb6c06e670 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6c0701c0 con 0x7fdb6c15bb70 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb61400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb61400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45210/0 (socket says 172.21.15.31:45210) 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb61400640 1 -- 172.21.15.31:0/873847179 learned_addr learned my addr 172.21.15.31:0/873847179 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb69e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.454+0000 7fdb6a800640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 0x7fdb6c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 -- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 msgr2=0x7fdb6c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 0x7fdb6c0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 -- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb6c06e670 msgr2=0x7fdb6c0c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 -- 172.21.15.31:0/873847179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c1f7f90 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb61400640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009850 tx=0x7fdb540038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb62a00640 1 -- 172.21.15.31:0/873847179 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5403f070 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb62a00640 1 -- 172.21.15.31:0/873847179 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb540047a0 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6c0b3c40 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.455+0000 7fdb62a00640 1 -- 172.21.15.31:0/873847179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54035030 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.456+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6c0b4200 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.458+0000 7fdb62a00640 1 -- 172.21.15.31:0/873847179 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fdb54042070 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.459+0000 7fdb62a00640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 0x7fdb4807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.459+0000 7fdb62a00640 1 -- 172.21.15.31:0/873847179 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fdb540bc0c0 con 0x7fdb6c0ce4b0 2024-09-13T08:45:01.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.459+0000 7fdb69e00640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 0x7fdb4807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.459+0000 7fdb7403f740 1 Processor -- start 2024-09-13T08:45:01.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.459+0000 7fdb7403f740 1 -- start start 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb69e00640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 0x7fdb4807b340 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0c39e0 tx=0x7fdb5c005f30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be69aa0 0x563e1be69e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 0x563e1be68580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563e1bda7430 con 0x563e1be25190 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563e1bd975d0 con 0x563e1be69aa0 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.460+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563e1bd86a00 con 0x563e1be24c10 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45220/0 (socket says 172.21.15.31:45220) 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 -- 172.21.15.31:0/3467928376 learned_addr learned my addr 172.21.15.31:0/3467928376 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb69e00640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be69aa0 0x563e1be69e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb61400640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 0x563e1be68580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 -- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 msgr2=0x563e1be68580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 0x563e1be68580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 -- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be69aa0 msgr2=0x563e1be69e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be69aa0 0x563e1be69e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 -- 172.21.15.31:0/3467928376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1bd4f520 con 0x563e1be25190 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb61400640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 0x563e1be68580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb6a800640 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdb64001f40 tx=0x7fdb640016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6e2df44d60264a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb5aa00640 1 -- 172.21.15.31:0/3467928376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400fe50 con 0x563e1be25190 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.461+0000 7fdb5aa00640 1 -- 172.21.15.31:0/3467928376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb64018070 con 0x563e1be25190 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb5aa00640 1 -- 172.21.15.31:0/3467928376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64003bd0 con 0x563e1be25190 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 -- 172.21.15.31:0/3467928376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 msgr2=0x563e1be68ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdb64001f40 tx=0x7fdb640016d0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 -- 172.21.15.31:0/3467928376 shutdown_connections 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be68ac0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be24c10 0x563e1be68580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 --2- 172.21.15.31:0/3467928376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be69aa0 0x563e1be69e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 -- 172.21.15.31:0/3467928376 >> 172.21.15.31:0/3467928376 conn(0x563e1be201e0 msgr2=0x563e1be23c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 -- 172.21.15.31:0/3467928376 shutdown_connections 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.462+0000 7fdb7403f740 1 -- 172.21.15.31:0/3467928376 wait complete. 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.463+0000 7fdb7403f740 1 -- start start 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.463+0000 7fdb7403f740 1 Processor -- start 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be4d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be22360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563e1bd2d330 con 0x563e1be25190 2024-09-13T08:45:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563e1bd8a810 con 0x563e1be21ee0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563e1baf7620 con 0x563e1be69aa0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb7403f740 10 client.0 _ll_register_callbacks cb 0x563e1bd32420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.464+0000 7fdb69e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45228/0 (socket says 172.21.15.31:45228) 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 -- 172.21.15.31:0/1256535188 learned_addr learned my addr 172.21.15.31:0/1256535188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb61400640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be4d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb6a800640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be22360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 -- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 msgr2=0x563e1be4d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be4d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 -- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 msgr2=0x563e1be22360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be22360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 -- 172.21.15.31:0/1256535188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e1bae8850 con 0x563e1be25190 2024-09-13T08:45:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb69e00640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c01ecc0 tx=0x7fdb5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4c9b5e01a9c7b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb4f400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c00e930 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.465+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fdb5c028b20 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb64018070 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdb5c028d20 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 10 client.8115 fetch_fsmap learned FSMap version 90 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x563e1bb5f6b0 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb64018070 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 90) ==== 3502+0+0 (secure 0 0 0) 0x7fdb5c02f050 con 0x563e1be25190 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 10 client.8115 fetch_fsmap finished waiting for FSMap version 90 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 10 client.8115 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 20 client.8115 populate_metadata read hostname 'smithi031' 2024-09-13T08:45:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be3b2d0 0x563e1be3b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 4 client.8115 mds_command: new command op to 7989 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:45:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.466+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 --> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e1bd99520 con 0x563e1be3b2d0 2024-09-13T08:45:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.467+0000 7fdb61400640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be3b2d0 0x563e1be3b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.467+0000 7fdb61400640 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be3b2d0 0x563e1be3b6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6869f91ce91d8cbb server_cookie=5510f312c34a5a59 in_seq=0 out_seq=0 2024-09-13T08:45:01.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.467+0000 7fdb4f400640 10 client.8115 ms_handle_connect on v2:172.21.15.31:6832/1614002559 2024-09-13T08:45:01.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb4f400640 1 -- 172.21.15.31:0/1256535188 <== mds.0 v2:172.21.15.31:6832/1614002559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563e1bd99520 con 0x563e1be3b2d0 2024-09-13T08:45:01.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb4f400640 10 client.8115 handle_command_reply: tid=0 2024-09-13T08:45:01.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 client.8115 shutdown 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 msgr2=0x563e1be65d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c01ecc0 tx=0x7fdb5c004730 comp rx=0 tx=0).stop 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 shutdown_connections 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be3b2d0 0x563e1be3b6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be22360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be4d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 --2- 172.21.15.31:0/1256535188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be25190 0x563e1be65d90 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb4f400640 0 client.8115 ms_handle_reset on v2:172.21.15.31:6832/1614002559 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.468+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 >> 172.21.15.31:0/1256535188 conn(0x563e1be24c10 msgr2=0x563e1be1edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.469+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 shutdown_connections 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.469+0000 7fdb7403f740 1 -- 172.21.15.31:0/1256535188 wait complete. 2024-09-13T08:45:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.469+0000 7fdb7403f740 20 client.8115 trim_cache size 0 max 16384 2024-09-13T08:45:01.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.488+0000 7fdb7403f740 1 Processor -- start 2024-09-13T08:45:01.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.488+0000 7fdb7403f740 1 -- start start 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be1ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 0x563e1be1dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c200190 con 0x563e1be69aa0 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1f4c30 con 0x563e1be21ee0 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6c1f5280 con 0x563e1be25190 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45230/0 (socket says 172.21.15.31:45230) 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 -- 172.21.15.31:0/2243056811 learned_addr learned my addr 172.21.15.31:0/2243056811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb61400640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 0x563e1be1dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb69e00640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be1ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 -- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 msgr2=0x563e1be1dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 0x563e1be1dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 -- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 msgr2=0x563e1be1ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be1ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb6a800640 1 -- 172.21.15.31:0/2243056811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1be6a5a0 con 0x563e1be69aa0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.489+0000 7fdb61400640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 0x563e1be1dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.490+0000 7fdb69e00640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be1ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.490+0000 7fdb6a800640 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fdb64015040 tx=0x7fdb640014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376d0074ccbfc3a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.490+0000 7fdb5a000640 1 -- 172.21.15.31:0/2243056811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64003db0 con 0x563e1be69aa0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.490+0000 7fdb5a000640 1 -- 172.21.15.31:0/2243056811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb64018070 con 0x563e1be69aa0 2024-09-13T08:45:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.490+0000 7fdb5a000640 1 -- 172.21.15.31:0/2243056811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64005330 con 0x563e1be69aa0 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- 172.21.15.31:0/2243056811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 msgr2=0x563e1be1e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fdb64015040 tx=0x7fdb640014a0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- 172.21.15.31:0/2243056811 shutdown_connections 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be69aa0 0x563e1be1e120 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be25190 0x563e1be1dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 --2- 172.21.15.31:0/2243056811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be21ee0 0x563e1be1ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- 172.21.15.31:0/2243056811 >> 172.21.15.31:0/2243056811 conn(0x563e1be24c10 msgr2=0x563e1be22cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- 172.21.15.31:0/2243056811 shutdown_connections 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- 172.21.15.31:0/2243056811 wait complete. 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.491+0000 7fdb7403f740 1 -- start start 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.492+0000 7fdb7403f740 1 Processor -- start 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be25190 0x563e1bee8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be68d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563e1be38db0 con 0x563e1be21ee0 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563e1be67870 con 0x563e1be25190 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563e1be68820 con 0x563e1be69aa0 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb7403f740 10 client.0 _ll_register_callbacks cb 0x7fdb5c01e910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45242/0 (socket says 172.21.15.31:45242) 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 -- 172.21.15.31:0/93069627 learned_addr learned my addr 172.21.15.31:0/93069627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb6a800640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be68d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb61400640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be25190 0x563e1bee8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 -- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 msgr2=0x563e1be68d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be68d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 -- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be25190 msgr2=0x563e1bee8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be25190 0x563e1bee8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.493+0000 7fdb69e00640 1 -- 172.21.15.31:0/93069627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e1be8ad40 con 0x563e1be21ee0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb69e00640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00ed80 tx=0x7fdb5c00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23e8a2e46cd2517 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb4f400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c004220 con 0x563e1be21ee0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fdb5c0043c0 con 0x563e1be21ee0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563e1be24420 con 0x563e1be21ee0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdb5c03f050 con 0x563e1be21ee0 2024-09-13T08:45:01.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb7403f740 10 client.8121 fetch_fsmap learned FSMap version 90 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x563e1bee8e70 con 0x563e1be21ee0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.494+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x563e1be24420 con 0x563e1be21ee0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 90) ==== 3502+0+0 (secure 0 0 0) 0x7fdb5c042020 con 0x563e1be21ee0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 10 client.8121 fetch_fsmap finished waiting for FSMap version 90 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 10 client.8121 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 20 client.8121 populate_metadata read hostname 'smithi031' 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be399b0 0x563e1bee94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 4 client.8121 mds_command: new command op to 7989 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 --> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] -- command(tid 0: {"prefix": "status"}) -- 0x563e1be4a830 con 0x563e1be399b0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb61400640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be399b0 0x563e1bee94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb61400640 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be399b0 0x563e1bee94f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=699f2509f7f6877a server_cookie=18b67f3e87de1e91 in_seq=0 out_seq=0 2024-09-13T08:45:01.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.495+0000 7fdb4f400640 10 client.8121 ms_handle_connect on v2:172.21.15.31:6832/1614002559 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb4f400640 1 -- 172.21.15.31:0/93069627 <== mds.0 v2:172.21.15.31:6832/1614002559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x563e1be4a830 con 0x563e1be399b0 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb4f400640 10 client.8121 handle_command_reply: tid=0 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 client.8121 shutdown 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 msgr2=0x563e1be621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00ed80 tx=0x7fdb5c00eba0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 shutdown_connections 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:6832/1614002559,v1:172.21.15.31:6834/1614002559] conn(0x563e1be399b0 0x563e1bee94f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563e1be69aa0 0x563e1be68d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e1be25190 0x563e1bee8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 --2- 172.21.15.31:0/93069627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e1be21ee0 0x563e1be621b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 >> 172.21.15.31:0/93069627 conn(0x563e1be24c10 msgr2=0x563e1be19850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb4f400640 0 client.8121 ms_handle_reset on v2:172.21.15.31:6832/1614002559 2024-09-13T08:45:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 shutdown_connections 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 1 -- 172.21.15.31:0/93069627 wait complete. 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.496+0000 7fdb7403f740 20 client.8121 trim_cache size 0 max 16384 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 7989, 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:45:01.503 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2348303280000001, 2024-09-13T08:45:01.504 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 90, 2024-09-13T08:45:01.504 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 139, 2024-09-13T08:45:01.504 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 139, 2024-09-13T08:45:01.504 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.539205089999999 2024-09-13T08:45:01.504 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 msgr2=0x7fdb4807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 0x7fdb4807b340 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0c39e0 tx=0x7fdb5c005f30 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 msgr2=0x7fdb6c0c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009850 tx=0x7fdb540038a0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 shutdown_connections 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdb48078e80 0x7fdb4807b340 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdb6c15bb70 0x7fdb6c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6c0ce4b0 0x7fdb6c0c8d30 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 --2- 172.21.15.31:0/873847179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb6c06e670 0x7fdb6c0c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 >> 172.21.15.31:0/873847179 conn(0x7fdb6c0cc7a0 msgr2=0x7fdb6c0725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 shutdown_connections 2024-09-13T08:45:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.497+0000 7fdb6b200640 1 -- 172.21.15.31:0/873847179 wait complete. 2024-09-13T08:45:01.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.674+0000 7fbbc0800640 1 -- 172.21.15.31:0/702719178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0004d40 msgr2=0x7fbba00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.674+0000 7fbbc0800640 1 --2- 172.21.15.31:0/702719178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0004d40 0x7fbba00051c0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009ed0 tx=0x7fbba0005830 comp rx=0 tx=0).stop 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.674+0000 7fbbbe800640 1 -- 172.21.15.31:0/702719178 reap_dead start 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.674+0000 7fbbc0800640 1 -- 172.21.15.31:0/702719178 shutdown_connections 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.674+0000 7fbbc0800640 1 -- 172.21.15.31:0/702719178 >> 172.21.15.31:0/702719178 conn(0x7fbbb80cc910 msgr2=0x7fbbb80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.675+0000 7fbbc0800640 1 -- 172.21.15.31:0/702719178 shutdown_connections 2024-09-13T08:45:01.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.675+0000 7fbbc0800640 1 -- 172.21.15.31:0/702719178 wait complete. 2024-09-13T08:45:01.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.676+0000 7fbbc0800640 1 Processor -- start 2024-09-13T08:45:01.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.676+0000 7fbbc0800640 1 -- start start 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbba0004d40 0x7fbbb815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 0x7fbbb80b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80cc690 con 0x7fbbb80b2d50 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80cc510 con 0x7fbbb80b4e00 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.677+0000 7fbbc0800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb8073780 con 0x7fbba0004d40 2024-09-13T08:45:01.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45266/0 (socket says 172.21.15.31:45266) 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 -- 172.21.15.31:0/2897526000 learned_addr learned my addr 172.21.15.31:0/2897526000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbde00640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbba0004d40 0x7fbbb815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbe800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 0x7fbbb80b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 -- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbba0004d40 msgr2=0x7fbbb815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbba0004d40 0x7fbbb815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 -- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 msgr2=0x7fbbb80b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 0x7fbbb80b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 -- 172.21.15.31:0/2897526000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80c9ea0 con 0x7fbbb80b2d50 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbe800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 0x7fbbb80b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.678+0000 7fbbbd400640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbba8006680 tx=0x7fbba80066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.679+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba8015070 con 0x7fbbb80b2d50 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.679+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbba8011900 con 0x7fbbb80b2d50 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.679+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba8011d50 con 0x7fbbb80b2d50 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.679+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb8161560 con 0x7fbbb80b2d50 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.679+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb80c9bc0 con 0x7fbbb80b2d50 2024-09-13T08:45:01.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.680+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb8073c90 con 0x7fbbb80b2d50 2024-09-13T08:45:01.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.681+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fbba8010420 con 0x7fbbb80b2d50 2024-09-13T08:45:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.682+0000 7fbbb2000640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 0x7fbb9c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.682+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fbba809c230 con 0x7fbbb80b2d50 2024-09-13T08:45:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.682+0000 7fbbbde00640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 0x7fbb9c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:01.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.683+0000 7fbbbde00640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 0x7fbb9c07b340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80b4bf0 tx=0x7fbbac006cb0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:01.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.686+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbbb80cbe40 con 0x7fbbb80b2d50 2024-09-13T08:45:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.920+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbbb80af5d0 con 0x7fbbb80b2d50 2024-09-13T08:45:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.921+0000 7fbbb2000640 1 -- 172.21.15.31:0/2897526000 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) ==== 53+0+73 (secure 0 0 0) 0x7fbba8063820 con 0x7fbbb80b2d50 2024-09-13T08:45:01.923 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.923+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 msgr2=0x7fbb9c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.923+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 0x7fbb9c07b340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80b4bf0 tx=0x7fbbac006cb0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.923+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 msgr2=0x7fbbb81f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.923+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbba8006680 tx=0x7fbba80066b0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.923+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 shutdown_connections 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbb9c078e80 0x7fbb9c07b340 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb80b4e00 0x7fbbb80b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb80b2d50 0x7fbbb81f5d40 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 --2- 172.21.15.31:0/2897526000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbba0004d40 0x7fbbb815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 >> 172.21.15.31:0/2897526000 conn(0x7fbbb80cc910 msgr2=0x7fbbb80aea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 shutdown_connections 2024-09-13T08:45:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:01.924+0000 7fbbc0800640 1 -- 172.21.15.31:0/2897526000 wait complete. 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:01.941 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:45:01.942 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:45:01.942 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:45:01.942 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:01.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:01.942 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:45:01.980 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:45:01.980 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:45:01.980 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:45:01.980 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:45:01.981 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:45:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42230sec preferred_lft 42230sec 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:01.983 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:45:01.984 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.985 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:01.985 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:45:01.985 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:45:01.985 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.985 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:01.986 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:45:01.986 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:45:01.986 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:01.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:01.987 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:45:02.074 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:45:02.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:02.078 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:45:02.117 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:45:02.117 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:45:02.117 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:45:02.117 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:45:02.117 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:45:02.118 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:45:02.118 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:45:02.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:45:02.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:45:02.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:45:02.119 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:45:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:45:02.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:02.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:02.316 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:45:02.351 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:45:02.351 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:45:02.352 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:02.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:02.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:45:02.447 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:02.447 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:02.482 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:02.483 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:02.483 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:02.558 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:45:02.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:02.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:02.686 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:02.686 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:45:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:45:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:45:02.715 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:45:02.715 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:45:02.715 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:45:02.715 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:45:02.715 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:45:02.718 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:45:02.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:02.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:02.775 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:45:02.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:45:02.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:02.872 INFO:teuthology.orchestra.run.smithi031.stdout:client6905 2024-09-13T08:45:02.873 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:02.873 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:02.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:45:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1052069804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 msgr2=0x7f2d84073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1052069804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84073fc0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009900 tx=0x7f2d7002a600 comp rx=0 tx=0).stop 2024-09-13T08:45:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1052069804 shutdown_connections 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1052069804 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 0x7f2d840b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1052069804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84073fc0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1052069804 >> 172.21.15.31:0/1052069804 conn(0x7f2d840cc910 msgr2=0x7f2d840cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1052069804 shutdown_connections 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.109+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1052069804 wait complete. 2024-09-13T08:45:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.111+0000 7f2d8ae00640 1 Processor -- start 2024-09-13T08:45:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.111+0000 7f2d8ae00640 1 -- start start 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 0x7f2d841f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 0x7f2d84153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840ca660 con 0x7f2d84072850 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840ca4e0 con 0x7f2d84152f10 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d8ae00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2d840b1e50 con 0x7f2d84074500 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45290/0 (socket says 172.21.15.31:45290) 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d83e00640 1 -- 172.21.15.31:0/1384291476 learned_addr learned my addr 172.21.15.31:0/1384291476 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d83400640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 0x7f2d841f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.112+0000 7f2d88e00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 0x7f2d84153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 -- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 msgr2=0x7f2d841f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 0x7f2d841f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 -- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 msgr2=0x7f2d84153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 0x7f2d84153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 -- 172.21.15.31:0/1384291476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d841f41d0 con 0x7f2d84072850 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d88e00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 0x7f2d84153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d83e00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400c3a0 tx=0x7f2d7400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74017070 con 0x7f2d84072850 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d740053c0 con 0x7f2d84072850 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74003850 con 0x7f2d84072850 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d84155d60 con 0x7f2d84072850 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.113+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d84153650 con 0x7f2d84072850 2024-09-13T08:45:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.114+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d84153470 con 0x7f2d84072850 2024-09-13T08:45:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.116+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f2d7401a070 con 0x7f2d84072850 2024-09-13T08:45:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.117+0000 7f2d7be00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 0x7f2d6407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.117+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f2d7409a8e0 con 0x7f2d84072850 2024-09-13T08:45:03.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.117+0000 7f2d83400640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 0x7f2d6407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:03.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.118+0000 7f2d83400640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 0x7f2d6407b340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2d70033000 tx=0x7f2d70004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:03.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.121+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2d84153470 con 0x7f2d84072850 2024-09-13T08:45:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.360+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2d840b23b0 con 0x7f2d84072850 2024-09-13T08:45:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.361+0000 7f2d7be00640 1 -- 172.21.15.31:0/1384291476 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) ==== 53+0+73 (secure 0 0 0) 0x7f2d841f41d0 con 0x7f2d84072850 2024-09-13T08:45:03.363 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:45:03.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 msgr2=0x7f2d6407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:03.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 0x7f2d6407b340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2d70033000 tx=0x7f2d70004730 comp rx=0 tx=0).stop 2024-09-13T08:45:03.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 msgr2=0x7f2d84152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:03.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400c3a0 tx=0x7f2d7400c860 comp rx=0 tx=0).stop 2024-09-13T08:45:03.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 shutdown_connections 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2d64078e80 0x7f2d6407b340 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d84152f10 0x7f2d84153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2d84074500 0x7f2d841f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 --2- 172.21.15.31:0/1384291476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d84072850 0x7f2d84152190 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 >> 172.21.15.31:0/1384291476 conn(0x7f2d840cc910 msgr2=0x7f2d840ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 shutdown_connections 2024-09-13T08:45:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:03.363+0000 7f2d8ae00640 1 -- 172.21.15.31:0/1384291476 wait complete. 2024-09-13T08:45:03.382 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:45:03.382 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:45:03.382 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:45:03.382 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:45:03.383 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:03.383 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:45:03.383 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:45:03.383 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:45:03.383 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:03.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:03.383 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:45:03.424 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:45:03.424 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:45:03.425 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42233sec preferred_lft 42233sec 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.426 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:03.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:03.428 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:45:03.523 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:45:03.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:03.527 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:45:03.565 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:45:03.565 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:45:03.565 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:45:03.565 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:45:03.566 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:45:03.566 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:45:03.566 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:45:03.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:45:03.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:45:03.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:45:03.567 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:45:03.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:45:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:03.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:03.764 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:45:03.799 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:45:03.799 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:45:03.800 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:03.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:03.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:45:03.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:03.897 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:03.932 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:03.932 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:03.932 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:04.006 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:45:04.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:04.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:04.135 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:04.135 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:04.163 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:04.164 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:04.164 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:45:04.164 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:45:04.164 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:45:04.166 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:45:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:45:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:45:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:45:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:45:04.169 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:45:04.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:04.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:04.225 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:45:04.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:45:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:04.316 INFO:teuthology.orchestra.run.smithi045.stdout:client7639 2024-09-13T08:45:04.317 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:04.317 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:04.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:45:04.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 -- 172.21.15.31:0/2180903837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484415bfc0 msgr2=0x7f4844072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 --2- 172.21.15.31:0/2180903837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484415bfc0 0x7f4844072560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4834009830 tx=0x7f483402a4a0 comp rx=0 tx=0).stop 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 -- 172.21.15.31:0/2180903837 shutdown_connections 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 --2- 172.21.15.31:0/2180903837 >> v2:172.21.15.45:3300/0 conn(0x7f484406fec0 0x7f48440b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 --2- 172.21.15.31:0/2180903837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484415bfc0 0x7f4844072560 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 -- 172.21.15.31:0/2180903837 >> 172.21.15.31:0/2180903837 conn(0x7f48440cfbb0 msgr2=0x7f48440cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 -- 172.21.15.31:0/2180903837 shutdown_connections 2024-09-13T08:45:04.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.529+0000 7f484a400640 1 -- 172.21.15.31:0/2180903837 wait complete. 2024-09-13T08:45:04.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.531+0000 7f484a400640 1 Processor -- start 2024-09-13T08:45:04.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.531+0000 7f484a400640 1 -- start start 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 0x7f48440c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484414e440 0x7f48441f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48440cbfd0 con 0x7f484415bfc0 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48440cbe50 con 0x7f484414e440 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f484a400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f48440b2f20 con 0x7f484406fa60 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f4842a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f4842a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45300/0 (socket says 172.21.15.31:45300) 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f4842a00640 1 -- 172.21.15.31:0/4126515649 learned_addr learned my addr 172.21.15.31:0/4126515649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f4843400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 0x7f48440c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.532+0000 7f4843e00640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484414e440 0x7f48441f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 -- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 msgr2=0x7f48440c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 0x7f48440c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 -- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484414e440 msgr2=0x7f48441f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484414e440 0x7f48441f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 -- 172.21.15.31:0/4126515649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48440c8e70 con 0x7f484415bfc0 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4843400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 0x7f48440c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:45:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f4842a00640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f483000c700 tx=0x7f483000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483000a840 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4830017070 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483000add0 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48441f8c60 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.533+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48441f5560 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.534+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48440b3370 con 0x7f484415bfc0 2024-09-13T08:45:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.536+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f483001c460 con 0x7f484415bfc0 2024-09-13T08:45:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.537+0000 7f483b400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 0x7f482407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.537+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f483009b380 con 0x7f484415bfc0 2024-09-13T08:45:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.537+0000 7f4843400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 0x7f482407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.538+0000 7f4843400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 0x7f482407b340 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4834034000 tx=0x7f4834004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.542+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4830062920 con 0x7f484415bfc0 2024-09-13T08:45:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:04.776+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f48440cb310 con 0x7f484415bfc0 2024-09-13T08:45:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.460+0000 7f483b400640 1 -- 172.21.15.31:0/4126515649 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v91) ==== 136+0+0 (secure 0 0 0) 0x7f4830067350 con 0x7f484415bfc0 2024-09-13T08:45:05.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 msgr2=0x7f482407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 0x7f482407b340 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4834034000 tx=0x7f4834004730 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 msgr2=0x7f48440c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f483000c700 tx=0x7f483000cbc0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 shutdown_connections 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4824078e80 0x7f482407b340 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484414e440 0x7f48441f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f484415bfc0 0x7f48440c9910 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 --2- 172.21.15.31:0/4126515649 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f484406fa60 0x7f48440c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.463+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 >> 172.21.15.31:0/4126515649 conn(0x7f48440cfbb0 msgr2=0x7f4844159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.464+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 shutdown_connections 2024-09-13T08:45:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.464+0000 7f484a400640 1 -- 172.21.15.31:0/4126515649 wait complete. 2024-09-13T08:45:05.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:45:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 -- 172.21.15.31:0/2321205669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa938158700 msgr2=0x7fa93806e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 --2- 172.21.15.31:0/2321205669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa938158700 0x7fa93806e660 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa92c005fd0 tx=0x7fa92c02a540 comp rx=0 tx=0).stop 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 -- 172.21.15.31:0/2321205669 shutdown_connections 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 --2- 172.21.15.31:0/2321205669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa93806eba0 0x7fa93806f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 --2- 172.21.15.31:0/2321205669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa938158700 0x7fa93806e660 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 -- 172.21.15.31:0/2321205669 >> 172.21.15.31:0/2321205669 conn(0x7fa9380b18f0 msgr2=0x7fa9380b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.648+0000 7fa940400640 1 -- 172.21.15.31:0/2321205669 shutdown_connections 2024-09-13T08:45:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.649+0000 7fa940400640 1 -- 172.21.15.31:0/2321205669 wait complete. 2024-09-13T08:45:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.650+0000 7fa940400640 1 Processor -- start 2024-09-13T08:45:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.650+0000 7fa940400640 1 -- start start 2024-09-13T08:45:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 0x7fa938073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa938158700 0x7fa938145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa9380b4b40 con 0x7fa93806eba0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9380b49c0 con 0x7fa93806f8c0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa940400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa9380ccd70 con 0x7fa938158700 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa93da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa93da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43182/0 (socket says 172.21.15.31:43182) 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.651+0000 7fa93da00640 1 -- 172.21.15.31:0/153765894 learned_addr learned my addr 172.21.15.31:0/153765894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93d000640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 0x7fa938073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 -- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa938158700 msgr2=0x7fa938145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa938158700 0x7fa938145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 -- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 msgr2=0x7fa938073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 0x7fa938073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 -- 172.21.15.31:0/153765894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa938074880 con 0x7fa93806eba0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93d000640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 0x7fa938073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa93da00640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fa92c034040 tx=0x7fa92c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c03e030 con 0x7fa93806eba0 2024-09-13T08:45:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa92c0047a0 con 0x7fa93806eba0 2024-09-13T08:45:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035040 con 0x7fa93806eba0 2024-09-13T08:45:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.652+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa938072940 con 0x7fa93806eba0 2024-09-13T08:45:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.653+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa938072f00 con 0x7fa93806eba0 2024-09-13T08:45:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.654+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9380cd2c0 con 0x7fa93806eba0 2024-09-13T08:45:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.655+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fa92c041030 con 0x7fa93806eba0 2024-09-13T08:45:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.656+0000 7fa935600640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 0x7fa91807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.656+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fa92c0bbbe0 con 0x7fa93806eba0 2024-09-13T08:45:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.656+0000 7fa93d000640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 0x7fa91807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.657+0000 7fa93d000640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 0x7fa91807b340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa938154f70 tx=0x7fa924005db0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.660+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa92c083900 con 0x7fa93806eba0 2024-09-13T08:45:05.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.878+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa938073960 con 0x7fa918078e80 2024-09-13T08:45:05.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.880+0000 7fa935600640 1 -- 172.21.15.31:0/153765894 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa92400b6b0 con 0x7fa918078e80 2024-09-13T08:45:05.882 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:45:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:45:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:05.882 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 msgr2=0x7fa91807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 0x7fa91807b340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa938154f70 tx=0x7fa924005db0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 msgr2=0x7fa9380c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fa92c034040 tx=0x7fa92c0039e0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 shutdown_connections 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa918078e80 0x7fa91807b340 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa938158700 0x7fa938145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa93806f8c0 0x7fa938073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 --2- 172.21.15.31:0/153765894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93806eba0 0x7fa9380c8f70 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:05.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.882+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 >> 172.21.15.31:0/153765894 conn(0x7fa9380b18f0 msgr2=0x7fa9380cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:05.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.883+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 shutdown_connections 2024-09-13T08:45:05.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:05.883+0000 7fa940400640 1 -- 172.21.15.31:0/153765894 wait complete. 2024-09-13T08:45:05.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:45:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 -- 172.21.15.31:0/265157993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 msgr2=0x7f91780cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 --2- 172.21.15.31:0/265157993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780cb740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9160009860 tx=0x7f916002a530 comp rx=0 tx=0).stop 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 -- 172.21.15.31:0/265157993 shutdown_connections 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 --2- 172.21.15.31:0/265157993 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780cb740 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 -- 172.21.15.31:0/265157993 >> 172.21.15.31:0/265157993 conn(0x7f91780cfbb0 msgr2=0x7f91780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 -- 172.21.15.31:0/265157993 shutdown_connections 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.062+0000 7f917f600640 1 -- 172.21.15.31:0/265157993 wait complete. 2024-09-13T08:45:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.064+0000 7f917f600640 1 Processor -- start 2024-09-13T08:45:06.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.064+0000 7f917f600640 1 -- start start 2024-09-13T08:45:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 0x7f9178157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f917806e740 con 0x7f917815bfc0 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f917806e5c0 con 0x7f91780cbc80 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917f600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f91780b30f0 con 0x7f9178072560 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43184/0 (socket says 172.21.15.31:43184) 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917d600640 1 -- 172.21.15.31:0/2051548592 learned_addr learned my addr 172.21.15.31:0/2051548592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f917cc00640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.065+0000 7f916fe00640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 0x7f9178157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 -- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 msgr2=0x7f91780c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 -- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 msgr2=0x7f9178157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 0x7f9178157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 -- 172.21.15.31:0/2051548592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91780c8940 con 0x7f917815bfc0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f916fe00640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 0x7f9178157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917d600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f9170001ed0 tx=0x7f917000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170016020 con 0x7f917815bfc0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9170012420 con 0x7f917815bfc0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170012810 con 0x7f917815bfc0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91780c4ba0 con 0x7f917815bfc0 2024-09-13T08:45:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.066+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91780c5160 con 0x7f917815bfc0 2024-09-13T08:45:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.067+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91780c8b10 con 0x7f917815bfc0 2024-09-13T08:45:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.069+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9170019020 con 0x7f917815bfc0 2024-09-13T08:45:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.070+0000 7f9175600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 0x7f915407b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.070+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f917009b9f0 con 0x7f917815bfc0 2024-09-13T08:45:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.070+0000 7f917cc00640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 0x7f915407b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.070+0000 7f917cc00640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 0x7f915407b370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9168001120 tx=0x7f9168009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.074+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9170020030 con 0x7f917815bfc0 2024-09-13T08:45:06.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.267+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f91780cb310 con 0x7f917815bfc0 2024-09-13T08:45:06.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.267+0000 7f9175600640 1 -- 172.21.15.31:0/2051548592 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f9170063630 con 0x7f917815bfc0 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 msgr2=0x7f915407b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 0x7f915407b370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9168001120 tx=0x7f9168009040 comp rx=0 tx=0).stop 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 msgr2=0x7f9178158410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f9170001ed0 tx=0x7f917000a430 comp rx=0 tx=0).stop 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 shutdown_connections 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9154078eb0 0x7f915407b370 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f917815bfc0 0x7f9178158410 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91780cbc80 0x7f9178157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 --2- 172.21.15.31:0/2051548592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9178072560 0x7f91780c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 >> 172.21.15.31:0/2051548592 conn(0x7f91780cfbb0 msgr2=0x7f917815ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:06.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 shutdown_connections 2024-09-13T08:45:06.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.270+0000 7f917f600640 1 -- 172.21.15.31:0/2051548592 wait complete. 2024-09-13T08:45:06.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create non_existent_subvolume subvolume_1 2024-09-13T08:45:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 -- 172.21.15.31:0/682816768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906406fec0 msgr2=0x7f90640cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 --2- 172.21.15.31:0/682816768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906406fec0 0x7f90640cbc80 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f9064069dd0 tx=0x7f905c02aa10 comp rx=0 tx=0).stop 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 -- 172.21.15.31:0/682816768 shutdown_connections 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 --2- 172.21.15.31:0/682816768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f906406fa60 0x7f90640cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 --2- 172.21.15.31:0/682816768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906406fec0 0x7f90640cbc80 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 --2- 172.21.15.31:0/682816768 >> v2:172.21.15.45:3300/0 conn(0x7f906415bfc0 0x7f9064072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 -- 172.21.15.31:0/682816768 >> 172.21.15.31:0/682816768 conn(0x7f90640cfbb0 msgr2=0x7f90640cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 -- 172.21.15.31:0/682816768 shutdown_connections 2024-09-13T08:45:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.451+0000 7f906ba00640 1 -- 172.21.15.31:0/682816768 wait complete. 2024-09-13T08:45:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.453+0000 7f906ba00640 1 Processor -- start 2024-09-13T08:45:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.453+0000 7f906ba00640 1 -- start start 2024-09-13T08:45:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f906406fa60 0x7f906414e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906415bfc0 0x7f90640c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640c3c20 0x7f906415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90640b2b60 con 0x7f906415bfc0 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90640b29e0 con 0x7f90640c3c20 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f906ba00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f90640b2ce0 con 0x7f906406fa60 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f905be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906415bfc0 0x7f90640c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f905be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906415bfc0 0x7f90640c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43190/0 (socket says 172.21.15.31:43190) 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f905be00640 1 -- 172.21.15.31:0/2620127592 learned_addr learned my addr 172.21.15.31:0/2620127592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f9069000640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f906406fa60 0x7f906414e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.454+0000 7f9069a00640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640c3c20 0x7f906415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 -- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f906406fa60 msgr2=0x7f906414e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f906406fa60 0x7f906414e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 -- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640c3c20 msgr2=0x7f906415ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640c3c20 0x7f906415ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 -- 172.21.15.31:0/2620127592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906415c690 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f905be00640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f906415bfc0 0x7f90640c36e0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f904c00d720 tx=0x7f904c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c017070 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f904c009d20 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c004280 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f906ba00640 1 -- 172.21.15.31:0/2620127592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90640c4250 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.455+0000 7f906ba00640 1 -- 172.21.15.31:0/2620127592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906415c930 con 0x7f906415bfc0 2024-09-13T08:45:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.456+0000 7f906ba00640 1 -- 172.21.15.31:0/2620127592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9064072560 con 0x7f906415bfc0 2024-09-13T08:45:06.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.458+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f904c005020 con 0x7f906415bfc0 2024-09-13T08:45:06.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.459+0000 7f9061600640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9040078e80 0x7f904007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.459+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f904c09b880 con 0x7f906415bfc0 2024-09-13T08:45:06.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.459+0000 7f9069000640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9040078e80 0x7f904007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.459+0000 7f9069000640 1 --2- 172.21.15.31:0/2620127592 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9040078e80 0x7f904007b340 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f905400bb50 tx=0x7f905400b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.463+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f904c020020 con 0x7f906415bfc0 2024-09-13T08:45:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.684+0000 7f906ba00640 1 -- 172.21.15.31:0/2620127592 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f90640c4770 con 0x7f9040078e80 2024-09-13T08:45:06.687 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:06.686+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.686+0000 7f9061600640 1 -- 172.21.15.31:0/2620127592 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f90640c4770 con 0x7f9040078e80 2024-09-13T08:45:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:06.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm non_existent_subvolume subvolume_1 2024-09-13T08:45:06.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 -- 172.21.15.31:0/2953595025 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff85c003c80 msgr2=0x7ff85c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 --2- 172.21.15.31:0/2953595025 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff85c003c80 0x7ff85c004080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff868009b50 tx=0x7ff85c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:06.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 -- 172.21.15.31:0/2953595025 shutdown_connections 2024-09-13T08:45:06.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 --2- 172.21.15.31:0/2953595025 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff85c003c80 0x7ff85c004080 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 -- 172.21.15.31:0/2953595025 >> 172.21.15.31:0/2953595025 conn(0x7ff8780cfbb0 msgr2=0x7ff8780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:06.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 -- 172.21.15.31:0/2953595025 shutdown_connections 2024-09-13T08:45:06.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.861+0000 7ff87e600640 1 -- 172.21.15.31:0/2953595025 wait complete. 2024-09-13T08:45:06.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.862+0000 7ff87e600640 1 Processor -- start 2024-09-13T08:45:06.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.863+0000 7ff87e600640 1 -- start start 2024-09-13T08:45:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.863+0000 7ff87e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff85c003c80 0x7ff87815f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.863+0000 7ff87e600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8780b5430 0x7ff87815bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff87e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87814adf0 0x7ff87814b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff87e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780cbdc0 con 0x7ff87814adf0 2024-09-13T08:45:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff87e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780cbc40 con 0x7ff85c003c80 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff87e600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff8780cbf40 con 0x7ff8780b5430 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87814adf0 0x7ff87814b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87814adf0 0x7ff87814b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43200/0 (socket says 172.21.15.31:43200) 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 -- 172.21.15.31:0/4123807942 learned_addr learned my addr 172.21.15.31:0/4123807942 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff876a00640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8780b5430 0x7ff87815bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877400640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff85c003c80 0x7ff87815f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 -- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8780b5430 msgr2=0x7ff87815bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8780b5430 0x7ff87815bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 -- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff85c003c80 msgr2=0x7ff87815f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff85c003c80 0x7ff87815f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.864+0000 7ff877e00640 1 -- 172.21.15.31:0/4123807942 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87815fd70 con 0x7ff87814adf0 2024-09-13T08:45:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff877e00640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87814adf0 0x7ff87814b250 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7ff878074e60 tx=0x7ff870006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870015070 con 0x7ff87814adf0 2024-09-13T08:45:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff870011900 con 0x7ff87814adf0 2024-09-13T08:45:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870011d70 con 0x7ff87814adf0 2024-09-13T08:45:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff87e600640 1 -- 172.21.15.31:0/4123807942 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff878160050 con 0x7ff87814adf0 2024-09-13T08:45:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.865+0000 7ff87e600640 1 -- 172.21.15.31:0/4123807942 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff878160610 con 0x7ff87814adf0 2024-09-13T08:45:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.866+0000 7ff87e600640 1 -- 172.21.15.31:0/4123807942 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8780cd940 con 0x7ff87814adf0 2024-09-13T08:45:06.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.867+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff870010420 con 0x7ff87814adf0 2024-09-13T08:45:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.868+0000 7ff86f400640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff858078eb0 0x7ff85807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.868+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7ff87009c170 con 0x7ff87814adf0 2024-09-13T08:45:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.869+0000 7ff877400640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff858078eb0 0x7ff85807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.869+0000 7ff877400640 1 --2- 172.21.15.31:0/4123807942 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff858078eb0 0x7ff85807b370 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff868035000 tx=0x7ff8680046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:06.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:06.873+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff870063820 con 0x7ff87814adf0 2024-09-13T08:45:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.095+0000 7ff87e600640 1 -- 172.21.15.31:0/4123807942 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7ff8780af1b0 con 0x7ff858078eb0 2024-09-13T08:45:07.098 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:07.097+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.097+0000 7ff86f400640 1 -- 172.21.15.31:0/4123807942 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7ff8780af1b0 con 0x7ff858078eb0 2024-09-13T08:45:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:07.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath non_existent_subvolume subvolume_1 2024-09-13T08:45:07.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:45:07.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.272+0000 7f2fd2800640 1 -- 172.21.15.31:0/175681260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc161480 msgr2=0x7f2fcc0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.272+0000 7f2fd2800640 1 --2- 172.21.15.31:0/175681260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc161480 0x7f2fcc0cd780 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009830 tx=0x7f2fbc02a520 comp rx=0 tx=0).stop 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.272+0000 7f2fd2800640 1 -- 172.21.15.31:0/175681260 shutdown_connections 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.272+0000 7f2fd2800640 1 --2- 172.21.15.31:0/175681260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc161480 0x7f2fcc0cd780 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.272+0000 7f2fd2800640 1 -- 172.21.15.31:0/175681260 >> 172.21.15.31:0/175681260 conn(0x7f2fcc0cfbb0 msgr2=0x7f2fcc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.273+0000 7f2fd2800640 1 -- 172.21.15.31:0/175681260 shutdown_connections 2024-09-13T08:45:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.273+0000 7f2fd2800640 1 -- 172.21.15.31:0/175681260 wait complete. 2024-09-13T08:45:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.274+0000 7f2fd2800640 1 Processor -- start 2024-09-13T08:45:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.274+0000 7f2fd2800640 1 -- start start 2024-09-13T08:45:07.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.275+0000 7f2fd2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc0cb380 0x7f2fcc155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.275+0000 7f2fd2800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fcc0cc190 0x7f2fcc1f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.275+0000 7f2fcb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc0cb380 0x7f2fcc155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc0cb380 0x7f2fcc155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43218/0 (socket says 172.21.15.31:43218) 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fc2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fcc0cc190 0x7f2fcc1f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fc2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fcc0cc190 0x7f2fcc1f87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:44796/0 (socket says 172.21.15.31:44796) 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fd2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fcc161480 0x7f2fcc0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc0cbdc0 con 0x7f2fcc0cb380 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 -- 172.21.15.31:0/3242434411 learned_addr learned my addr 172.21.15.31:0/3242434411 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc0cbc40 con 0x7f2fcc161480 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc0cbf40 con 0x7f2fcc0cc190 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcbe00640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fcc161480 0x7f2fcc0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 -- 172.21.15.31:0/3242434411 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fcc0cc190 msgr2=0x7f2fcc1f87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2fcc0cc190 0x7f2fcc1f87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 -- 172.21.15.31:0/3242434411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fcc161480 msgr2=0x7f2fcc0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fcc161480 0x7f2fcc0b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.276+0000 7f2fcb400640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc1f8d30 con 0x7f2fcc0cb380 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fcb400640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fcc0cb380 0x7f2fcc155080 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000c00 tx=0x7f2fbc033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc031d60 con 0x7f2fcc0cb380 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fbc033070 con 0x7f2fcc0cb380 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0334c0 con 0x7f2fcc0cb380 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fcc0b1130 con 0x7f2fcc0cb380 2024-09-13T08:45:07.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.277+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fcc0b16f0 con 0x7f2fcc0cb380 2024-09-13T08:45:07.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.279+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fcc0cde30 con 0x7f2fcc0cb380 2024-09-13T08:45:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.280+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f2fbc033660 con 0x7f2fcc0cb380 2024-09-13T08:45:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.280+0000 7f2fc3e00640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2fa8078e80 0x7f2fa807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.281+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f2fbc0bc7e0 con 0x7f2fcc0cb380 2024-09-13T08:45:07.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.281+0000 7f2fc2a00640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2fa8078e80 0x7f2fa807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.282+0000 7f2fc2a00640 1 --2- 172.21.15.31:0/3242434411 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2fa8078e80 0x7f2fa807b340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0c3130 tx=0x7f2fb4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:07.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.285+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2fcc1f8d30 con 0x7f2fcc0cb380 2024-09-13T08:45:07.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.515+0000 7f2fd2800640 1 -- 172.21.15.31:0/3242434411 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f2fcc0b18c0 con 0x7f2fa8078e80 2024-09-13T08:45:07.518 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:07.516+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.517+0000 7f2fc3e00640 1 -- 172.21.15.31:0/3242434411 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f2fcc0b18c0 con 0x7f2fa8078e80 2024-09-13T08:45:07.519 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:07.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info non_existent_subvolume subvolume_1 2024-09-13T08:45:07.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 -- 172.21.15.31:0/2621176837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 msgr2=0x7fe6f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 --2- 172.21.15.31:0/2621176837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 0x7fe6f4003dc0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fe700006f40 tx=0x7fe6f4005830 comp rx=0 tx=0).stop 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe70fe00640 1 -- 172.21.15.31:0/2621176837 reap_dead start 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 -- 172.21.15.31:0/2621176837 shutdown_connections 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 -- 172.21.15.31:0/2621176837 >> 172.21.15.31:0/2621176837 conn(0x7fe7100cfbb0 msgr2=0x7fe7100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 -- 172.21.15.31:0/2621176837 shutdown_connections 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.694+0000 7fe716800640 1 -- 172.21.15.31:0/2621176837 wait complete. 2024-09-13T08:45:07.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.696+0000 7fe716800640 1 Processor -- start 2024-09-13T08:45:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.696+0000 7fe716800640 1 -- start start 2024-09-13T08:45:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 0x7fe7100c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7101510e0 0x7fe710156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7101f6760 0x7fe7100c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe71006e740 con 0x7fe6f40039e0 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe71006e5c0 con 0x7fe7101f6760 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe716800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe7100b29e0 con 0x7fe7101510e0 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe70f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 0x7fe7100c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe70f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 0x7fe7100c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43230/0 (socket says 172.21.15.31:43230) 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe70f400640 1 -- 172.21.15.31:0/1874900515 learned_addr learned my addr 172.21.15.31:0/1874900515 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe70fe00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7101f6760 0x7fe7100c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.697+0000 7fe706a00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7101510e0 0x7fe710156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 -- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7101510e0 msgr2=0x7fe710156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7101510e0 0x7fe710156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 -- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7101f6760 msgr2=0x7fe7100c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7101f6760 0x7fe7100c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 -- 172.21.15.31:0/1874900515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7101f28e0 con 0x7fe6f40039e0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70fe00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7101f6760 0x7fe7100c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe70f400640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f40039e0 0x7fe7100c6c80 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c320 tx=0x7fe700001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700003a00 con 0x7fe6f40039e0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe700003c90 con 0x7fe6f40039e0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700038690 con 0x7fe6f40039e0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe716800640 1 -- 172.21.15.31:0/1874900515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe710156ad0 con 0x7fe6f40039e0 2024-09-13T08:45:07.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.698+0000 7fe716800640 1 -- 172.21.15.31:0/1874900515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7100c76d0 con 0x7fe6f40039e0 2024-09-13T08:45:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.699+0000 7fe716800640 1 -- 172.21.15.31:0/1874900515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7100747c0 con 0x7fe6f40039e0 2024-09-13T08:45:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.700+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fe700035020 con 0x7fe6f40039e0 2024-09-13T08:45:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.701+0000 7fe707e00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe6ec078e80 0x7fe6ec07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.701+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fe7000bb830 con 0x7fe6f40039e0 2024-09-13T08:45:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.702+0000 7fe706a00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe6ec078e80 0x7fe6ec07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.702+0000 7fe706a00640 1 --2- 172.21.15.31:0/1874900515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe6ec078e80 0x7fe6ec07b340 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe710142f00 tx=0x7fe6f8006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.706+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe7100747c0 con 0x7fe6f40039e0 2024-09-13T08:45:07.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.928+0000 7fe716800640 1 -- 172.21.15.31:0/1874900515 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7fe7100cb740 con 0x7fe6ec078e80 2024-09-13T08:45:07.931 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:07.929+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:07.929+0000 7fe707e00640 1 -- 172.21.15.31:0/1874900515 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fe7100cb740 con 0x7fe6ec078e80 2024-09-13T08:45:07.931 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:07.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize non_existent_subvolume subvolname_1 inf 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 -- 172.21.15.31:0/4085126853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38806fa60 msgr2=0x7ff3880cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 --2- 172.21.15.31:0/4085126853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38806fa60 0x7ff3880cb740 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff388069dd0 tx=0x7ff37402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 -- 172.21.15.31:0/4085126853 shutdown_connections 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 --2- 172.21.15.31:0/4085126853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38806fa60 0x7ff3880cb740 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 --2- 172.21.15.31:0/4085126853 >> v2:172.21.15.45:3300/0 conn(0x7ff38815bfc0 0x7ff388072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 -- 172.21.15.31:0/4085126853 >> 172.21.15.31:0/4085126853 conn(0x7ff3880cfbb0 msgr2=0x7ff3880cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 -- 172.21.15.31:0/4085126853 shutdown_connections 2024-09-13T08:45:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.105+0000 7ff38e600640 1 -- 172.21.15.31:0/4085126853 wait complete. 2024-09-13T08:45:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.107+0000 7ff38e600640 1 Processor -- start 2024-09-13T08:45:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.107+0000 7ff38e600640 1 -- start start 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff38806fa60 0x7ff388153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38815bfc0 0x7ff3880c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff38815d5e0 0x7ff3880c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880b3270 con 0x7ff38815bfc0 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880b30f0 con 0x7ff38815d5e0 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff38e600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff3880b33f0 con 0x7ff38806fa60 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff386a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38815bfc0 0x7ff3880c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff386a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38815bfc0 0x7ff3880c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43246/0 (socket says 172.21.15.31:43246) 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff386a00640 1 -- 172.21.15.31:0/3458564414 learned_addr learned my addr 172.21.15.31:0/3458564414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff387400640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff38806fa60 0x7ff388153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.108+0000 7ff387e00640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff38815d5e0 0x7ff3880c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 -- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff38806fa60 msgr2=0x7ff388153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff38806fa60 0x7ff388153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 -- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff38815d5e0 msgr2=0x7ff3880c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff38815d5e0 0x7ff3880c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 -- 172.21.15.31:0/3458564414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3880c5b10 con 0x7ff38815bfc0 2024-09-13T08:45:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff386a00640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38815bfc0 0x7ff3880c4a90 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff3740099c0 tx=0x7ff3740038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37403f070 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3740047a0 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff38e600640 1 -- 172.21.15.31:0/3458564414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3880c5df0 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374035030 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.109+0000 7ff38e600640 1 -- 172.21.15.31:0/3458564414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38815e9b0 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.111+0000 7ff38e600640 1 -- 172.21.15.31:0/3458564414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3880cbf20 con 0x7ff38815bfc0 2024-09-13T08:45:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.112+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7ff374042070 con 0x7ff38815bfc0 2024-09-13T08:45:08.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.113+0000 7ff37f400640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff368078e80 0x7ff36807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.113+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7ff3740bc1c0 con 0x7ff38815bfc0 2024-09-13T08:45:08.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.113+0000 7ff387400640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff368078e80 0x7ff36807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.114+0000 7ff387400640 1 --2- 172.21.15.31:0/3458564414 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff368078e80 0x7ff36807b340 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff378009ac0 tx=0x7ff378005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.117+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff374083850 con 0x7ff38815bfc0 2024-09-13T08:45:08.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.345+0000 7ff38e600640 1 -- 172.21.15.31:0/3458564414 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "new_size": "inf", "target": ["mon-mgr", ""]}) -- 0x7ff3880b2a70 con 0x7ff368078e80 2024-09-13T08:45:08.348 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:08.347+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.347+0000 7ff37f400640 1 -- 172.21.15.31:0/3458564414 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7ff3880b2a70 con 0x7ff368078e80 2024-09-13T08:45:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:08.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume pin non_existent_subvolume subvolname_1 export 1 2024-09-13T08:45:08.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 -- 172.21.15.31:0/588440579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18b0004d40 msgr2=0x7f18b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 --2- 172.21.15.31:0/588440579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18b0004d40 0x7f18b00051c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009e20 tx=0x7f18b0005830 comp rx=0 tx=0).stop 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cb400640 1 -- 172.21.15.31:0/588440579 reap_dead start 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 -- 172.21.15.31:0/588440579 shutdown_connections 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 -- 172.21.15.31:0/588440579 >> 172.21.15.31:0/588440579 conn(0x7f18cc0b1520 msgr2=0x7f18cc0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 -- 172.21.15.31:0/588440579 shutdown_connections 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.525+0000 7f18cbe00640 1 -- 172.21.15.31:0/588440579 wait complete. 2024-09-13T08:45:08.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.527+0000 7f18cbe00640 1 Processor -- start 2024-09-13T08:45:08.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.527+0000 7f18cbe00640 1 -- start start 2024-09-13T08:45:08.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18b0004d40 0x7f18cc0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc152060 0x7f18cc1518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18cc1576d0 0x7f18cc157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc0ccbc0 con 0x7f18cc1576d0 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc0cca40 con 0x7f18cc152060 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cbe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f18cc1527d0 con 0x7f18b0004d40 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18cc1576d0 0x7f18cc157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18cc1576d0 0x7f18cc157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43262/0 (socket says 172.21.15.31:43262) 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18cb400640 1 -- 172.21.15.31:0/3719170421 learned_addr learned my addr 172.21.15.31:0/3719170421 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18caa00640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18b0004d40 0x7f18cc0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.528+0000 7f18c2000640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc152060 0x7f18cc1518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 -- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18b0004d40 msgr2=0x7f18cc0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f18b0004d40 0x7f18cc0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 -- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc152060 msgr2=0x7f18cc1518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc152060 0x7f18cc1518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 -- 172.21.15.31:0/3719170421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc1f2b00 con 0x7f18cc1576d0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18c2000640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc152060 0x7f18cc1518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cb400640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18cc1576d0 0x7f18cc157b50 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009e20 tx=0x7f18c4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4038040 con 0x7f18cc1576d0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18c4042070 con 0x7f18cc1576d0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cbe00640 1 -- 172.21.15.31:0/3719170421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc1f2de0 con 0x7f18cc1576d0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4033a40 con 0x7f18cc1576d0 2024-09-13T08:45:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.529+0000 7f18cbe00640 1 -- 172.21.15.31:0/3719170421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc151df0 con 0x7f18cc1576d0 2024-09-13T08:45:08.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.530+0000 7f18cbe00640 1 -- 172.21.15.31:0/3719170421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc0b2820 con 0x7f18cc1576d0 2024-09-13T08:45:08.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.532+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f18c4033be0 con 0x7f18cc1576d0 2024-09-13T08:45:08.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.532+0000 7f18c3400640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f18a8078e80 0x7f18a807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.533+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f18c40bb6d0 con 0x7f18cc1576d0 2024-09-13T08:45:08.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.533+0000 7f18caa00640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f18a8078e80 0x7f18a807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.533+0000 7f18caa00640 1 --2- 172.21.15.31:0/3719170421 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f18a8078e80 0x7f18a807b340 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18bc001330 tx=0x7f18bc001280 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.537+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f18c4082dc0 con 0x7f18cc1576d0 2024-09-13T08:45:08.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.766+0000 7f18cbe00640 1 -- 172.21.15.31:0/3719170421 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume pin", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "pin_type": "export", "pin_setting": "1", "target": ["mon-mgr", ""]}) -- 0x7f18cc0c9520 con 0x7f18a8078e80 2024-09-13T08:45:08.769 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:08.767+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:08.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.767+0000 7f18c3400640 1 -- 172.21.15.31:0/3719170421 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f18bc016070 con 0x7f18a8078e80 2024-09-13T08:45:08.769 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:08.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls non_existent_subvolume 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.943+0000 7f9a5da00640 1 -- 172.21.15.31:0/1970826089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a3c003ad0 msgr2=0x7f9a3c003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.943+0000 7f9a5da00640 1 --2- 172.21.15.31:0/1970826089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a3c003ad0 0x7f9a3c003ef0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f9a480016c0 tx=0x7f9a3c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.943+0000 7f9a57400640 1 -- 172.21.15.31:0/1970826089 reap_dead start 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.944+0000 7f9a5da00640 1 -- 172.21.15.31:0/1970826089 shutdown_connections 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.944+0000 7f9a5da00640 1 -- 172.21.15.31:0/1970826089 >> 172.21.15.31:0/1970826089 conn(0x7f9a580cfb70 msgr2=0x7f9a580cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.944+0000 7f9a5da00640 1 -- 172.21.15.31:0/1970826089 shutdown_connections 2024-09-13T08:45:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.944+0000 7f9a5da00640 1 -- 172.21.15.31:0/1970826089 wait complete. 2024-09-13T08:45:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.945+0000 7f9a5da00640 1 Processor -- start 2024-09-13T08:45:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.945+0000 7f9a5da00640 1 -- start start 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a3c003ad0 0x7f9a5814fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a5815c4a0 0x7f9a58150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a58150e80 0x7f9a580b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a580cb7c0 con 0x7f9a58150e80 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a580cb640 con 0x7f9a3c003ad0 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.946+0000 7f9a5da00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9a580cb940 con 0x7f9a5815c4a0 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a58150e80 0x7f9a580b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a58150e80 0x7f9a580b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43278/0 (socket says 172.21.15.31:43278) 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 -- 172.21.15.31:0/935423041 learned_addr learned my addr 172.21.15.31:0/935423041 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a56a00640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a3c003ad0 0x7f9a5814fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a4e000640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a5815c4a0 0x7f9a58150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 -- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a5815c4a0 msgr2=0x7f9a58150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a5815c4a0 0x7f9a58150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 -- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a3c003ad0 msgr2=0x7f9a5814fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a3c003ad0 0x7f9a5814fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 -- 172.21.15.31:0/935423041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a580af570 con 0x7f9a58150e80 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a57400640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a58150e80 0x7f9a580b0610 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f9a580cc590 tx=0x7f9a5000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.947+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50003970 con 0x7f9a58150e80 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.948+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a50003b10 con 0x7f9a58150e80 2024-09-13T08:45:08.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.948+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004030 con 0x7f9a58150e80 2024-09-13T08:45:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.948+0000 7f9a5da00640 1 -- 172.21.15.31:0/935423041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a580af850 con 0x7f9a58150e80 2024-09-13T08:45:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.948+0000 7f9a5da00640 1 -- 172.21.15.31:0/935423041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a580afe10 con 0x7f9a58150e80 2024-09-13T08:45:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.949+0000 7f9a5da00640 1 -- 172.21.15.31:0/935423041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a580ca1c0 con 0x7f9a58150e80 2024-09-13T08:45:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.950+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9a500041d0 con 0x7f9a58150e80 2024-09-13T08:45:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.951+0000 7f9a4f400640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a34078e80 0x7f9a3407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.951+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f9a5009b9d0 con 0x7f9a58150e80 2024-09-13T08:45:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.951+0000 7f9a56a00640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a34078e80 0x7f9a3407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.952+0000 7f9a56a00640 1 --2- 172.21.15.31:0/935423041 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a34078e80 0x7f9a3407b340 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9a48032000 tx=0x7f9a48001190 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:08.955+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9a50062f30 con 0x7f9a58150e80 2024-09-13T08:45:09.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.176+0000 7f9a5da00640 1 -- 172.21.15.31:0/935423041 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7f9a580b38c0 con 0x7f9a34078e80 2024-09-13T08:45:09.180 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:09.178+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:09.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.178+0000 7f9a4f400640 1 -- 172.21.15.31:0/935423041 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f9a580b38c0 con 0x7f9a34078e80 2024-09-13T08:45:09.180 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:09.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create non_existent_subvolume subvolume_1 snapshot_1 2024-09-13T08:45:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 -- 172.21.15.31:0/696725994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a680cb380 msgr2=0x7f1a680cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 --2- 172.21.15.31:0/696725994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a680cb380 0x7f1a680cdcc0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f1a680767f0 tx=0x7f1a6002aa30 comp rx=0 tx=0).stop 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 -- 172.21.15.31:0/696725994 shutdown_connections 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 --2- 172.21.15.31:0/696725994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a680cb380 0x7f1a680cdcc0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 --2- 172.21.15.31:0/696725994 >> v2:172.21.15.45:3300/0 conn(0x7f1a68161480 0x7f1a680cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 -- 172.21.15.31:0/696725994 >> 172.21.15.31:0/696725994 conn(0x7f1a680cfbb0 msgr2=0x7f1a680cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 -- 172.21.15.31:0/696725994 shutdown_connections 2024-09-13T08:45:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.355+0000 7f1a6de00640 1 -- 172.21.15.31:0/696725994 wait complete. 2024-09-13T08:45:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.356+0000 7f1a6de00640 1 Processor -- start 2024-09-13T08:45:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.356+0000 7f1a6de00640 1 -- start start 2024-09-13T08:45:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680cb380 0x7f1a680b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a680cc190 0x7f1a6814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a68161480 0x7f1a680b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a680cbdc0 con 0x7f1a68161480 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a680cbc40 con 0x7f1a680cb380 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.357+0000 7f1a6de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1a680cbf40 con 0x7f1a680cc190 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680cb380 0x7f1a680b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a5e000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a680cc190 0x7f1a6814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680cb380 0x7f1a680b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:58706/0 (socket says 172.21.15.31:58706) 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 -- 172.21.15.31:0/1956806052 learned_addr learned my addr 172.21.15.31:0/1956806052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a67400640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a68161480 0x7f1a680b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 -- 172.21.15.31:0/1956806052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a680cc190 msgr2=0x7f1a6814df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a680cc190 0x7f1a6814df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 -- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a68161480 msgr2=0x7f1a680b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a68161480 0x7f1a680b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.358+0000 7f1a66a00640 1 -- 172.21.15.31:0/1956806052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a681572d0 con 0x7f1a680cb380 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a66a00640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a680cb380 0x7f1a680b34f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800d740 tx=0x7f1a5800dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58017070 con 0x7f1a680cb380 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a58009d20 con 0x7f1a680cb380 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a580106c0 con 0x7f1a680cb380 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a6de00640 1 -- 172.21.15.31:0/1956806052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a680b5650 con 0x7f1a680cb380 2024-09-13T08:45:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.359+0000 7f1a6de00640 1 -- 172.21.15.31:0/1956806052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a68158400 con 0x7f1a680cb380 2024-09-13T08:45:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.360+0000 7f1a6de00640 1 -- 172.21.15.31:0/1956806052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a681587e0 con 0x7f1a680cb380 2024-09-13T08:45:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.361+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f1a58041080 con 0x7f1a680cb380 2024-09-13T08:45:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.362+0000 7f1a5fe00640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a44078e80 0x7f1a4407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.362+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f1a5803f070 con 0x7f1a680cb380 2024-09-13T08:45:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.362+0000 7f1a5e000640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a44078e80 0x7f1a4407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.363+0000 7f1a5e000640 1 --2- 172.21.15.31:0/1956806052 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a44078e80 0x7f1a4407b340 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1a6815e7c0 tx=0x7f1a50009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.366+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1a681587e0 con 0x7f1a680cb380 2024-09-13T08:45:09.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.595+0000 7f1a6de00640 1 -- 172.21.15.31:0/1956806052 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f1a680b5930 con 0x7f1a44078e80 2024-09-13T08:45:09.598 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:09.597+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:09.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.597+0000 7f1a5fe00640 1 -- 172.21.15.31:0/1956806052 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f1a680b5930 con 0x7f1a44078e80 2024-09-13T08:45:09.599 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:09.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm non_existent_subvolume subvolume_1 snapshot_1 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.773+0000 7f5252600640 1 -- 172.21.15.31:0/2757438501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c0cbc80 msgr2=0x7f524c0b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.773+0000 7f5252600640 1 --2- 172.21.15.31:0/2757438501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c0cbc80 0x7f524c0b29e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5244005fd0 tx=0x7f524402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 -- 172.21.15.31:0/2757438501 shutdown_connections 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 --2- 172.21.15.31:0/2757438501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c0cbc80 0x7f524c0b29e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 --2- 172.21.15.31:0/2757438501 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f524c072560 0x7f524c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 -- 172.21.15.31:0/2757438501 >> 172.21.15.31:0/2757438501 conn(0x7f524c0cfbb0 msgr2=0x7f524c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 -- 172.21.15.31:0/2757438501 shutdown_connections 2024-09-13T08:45:09.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.774+0000 7f5252600640 1 -- 172.21.15.31:0/2757438501 wait complete. 2024-09-13T08:45:09.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.775+0000 7f5252600640 1 Processor -- start 2024-09-13T08:45:09.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.775+0000 7f5252600640 1 -- start start 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f524c072560 0x7f524c144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524c0cbc80 0x7f524c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c15bfc0 0x7f524c141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f524c06e740 con 0x7f524c15bfc0 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f524c06e5c0 con 0x7f524c0cbc80 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.776+0000 7f5252600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f524c0b2f20 con 0x7f524c072560 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c15bfc0 0x7f524c141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c15bfc0 0x7f524c141ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43308/0 (socket says 172.21.15.31:43308) 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 -- 172.21.15.31:0/1437602673 learned_addr learned my addr 172.21.15.31:0/1437602673 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f5242a00640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524c0cbc80 0x7f524c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524b400640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f524c072560 0x7f524c144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 -- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f524c072560 msgr2=0x7f524c144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f524c072560 0x7f524c144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 -- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524c0cbc80 msgr2=0x7f524c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524c0cbc80 0x7f524c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 -- 172.21.15.31:0/1437602673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c0c2f50 con 0x7f524c15bfc0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.777+0000 7f524be00640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f524c15bfc0 0x7f524c141ad0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f5244006010 tx=0x7f52440316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.778+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244040070 con 0x7f524c15bfc0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.778+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5244004040 con 0x7f524c15bfc0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.778+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244004450 con 0x7f524c15bfc0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.778+0000 7f5252600640 1 -- 172.21.15.31:0/1437602673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524c1533e0 con 0x7f524c15bfc0 2024-09-13T08:45:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.778+0000 7f5252600640 1 -- 172.21.15.31:0/1437602673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f524c15e130 con 0x7f524c15bfc0 2024-09-13T08:45:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.779+0000 7f5252600640 1 -- 172.21.15.31:0/1437602673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524c0b33b0 con 0x7f524c15bfc0 2024-09-13T08:45:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.780+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f5244039040 con 0x7f524c15bfc0 2024-09-13T08:45:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.781+0000 7f5243e00640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5228078e80 0x7f522807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.781+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f52440bca60 con 0x7f524c15bfc0 2024-09-13T08:45:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.781+0000 7f524b400640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5228078e80 0x7f522807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.782+0000 7f524b400640 1 --2- 172.21.15.31:0/1437602673 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5228078e80 0x7f522807b340 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f523c001610 tx=0x7f523c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:09.785+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f524c0c2f50 con 0x7f524c15bfc0 2024-09-13T08:45:10.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.013+0000 7f5252600640 1 -- 172.21.15.31:0/1437602673 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f524c153610 con 0x7f5228078e80 2024-09-13T08:45:10.016 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:10.014+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.014+0000 7f5243e00640 1 -- 172.21.15.31:0/1437602673 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f524c153610 con 0x7f5228078e80 2024-09-13T08:45:10.016 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:10.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info non_existent_subvolume subvolume_1 snapshot_1 2024-09-13T08:45:10.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 -- 172.21.15.31:0/1507247491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806f270 msgr2=0x7f00c8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 --2- 172.21.15.31:0/1507247491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806f270 0x7f00c8073f90 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f00c8072fa0 tx=0x7f00c002a9a0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 -- 172.21.15.31:0/1507247491 shutdown_connections 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 --2- 172.21.15.31:0/1507247491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806f270 0x7f00c8073f90 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 --2- 172.21.15.31:0/1507247491 >> v2:172.21.15.45:3300/0 conn(0x7f00c806ee30 0x7f00c8073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 --2- 172.21.15.31:0/1507247491 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 0x7f00c80cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 -- 172.21.15.31:0/1507247491 >> 172.21.15.31:0/1507247491 conn(0x7f00c80b1520 msgr2=0x7f00c80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 -- 172.21.15.31:0/1507247491 shutdown_connections 2024-09-13T08:45:10.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.191+0000 7f00ce000640 1 -- 172.21.15.31:0/1507247491 wait complete. 2024-09-13T08:45:10.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.193+0000 7f00ce000640 1 Processor -- start 2024-09-13T08:45:10.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.193+0000 7f00ce000640 1 -- start start 2024-09-13T08:45:10.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806ee30 0x7f00c814a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00c806f270 0x7f00c814cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 0x7f00c814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00c80b2bb0 con 0x7f00c806ee30 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00c80b2a30 con 0x7f00c806f270 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ce000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f00c80b2d30 con 0x7f00c815bae0 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ccc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806ee30 0x7f00c814a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ccc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806ee30 0x7f00c814a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43314/0 (socket says 172.21.15.31:43314) 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.194+0000 7f00ccc00640 1 -- 172.21.15.31:0/2463427304 learned_addr learned my addr 172.21.15.31:0/2463427304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00c7e00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00c806f270 0x7f00c814cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00cd600640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 0x7f00c814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 -- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 msgr2=0x7f00c814d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 0x7f00c814d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 -- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00c806f270 msgr2=0x7f00c814cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00c806f270 0x7f00c814cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 -- 172.21.15.31:0/2463427304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c815e330 con 0x7f00c806ee30 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00cd600640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f00c815bae0 0x7f00c814d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ccc00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00c806ee30 0x7f00c814a530 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00d6d0 tx=0x7f00bc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc017070 con 0x7f00c806ee30 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00bc005330 con 0x7f00c806ee30 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc009820 con 0x7f00c806ee30 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.195+0000 7f00ce000640 1 -- 172.21.15.31:0/2463427304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c815e610 con 0x7f00c806ee30 2024-09-13T08:45:10.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.196+0000 7f00ce000640 1 -- 172.21.15.31:0/2463427304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c814b1b0 con 0x7f00c806ee30 2024-09-13T08:45:10.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.196+0000 7f00ce000640 1 -- 172.21.15.31:0/2463427304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c814b3b0 con 0x7f00c806ee30 2024-09-13T08:45:10.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.198+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f00bc009bd0 con 0x7f00c806ee30 2024-09-13T08:45:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.199+0000 7f00c4c00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00a8078e80 0x7f00a807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.199+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f00bc09b6d0 con 0x7f00c806ee30 2024-09-13T08:45:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.199+0000 7f00c7e00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00a8078e80 0x7f00a807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.200+0000 7f00c7e00640 1 --2- 172.21.15.31:0/2463427304 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f00a8078e80 0x7f00a807b340 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f00c80b14f0 tx=0x7f00b400b410 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:10.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.203+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f00c814b3b0 con 0x7f00c806ee30 2024-09-13T08:45:10.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.434+0000 7f00ce000640 1 -- 172.21.15.31:0/2463427304 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f00c80b31e0 con 0x7f00a8078e80 2024-09-13T08:45:10.437 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:10.436+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.436+0000 7f00c4c00640 1 -- 172.21.15.31:0/2463427304 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f00c80b31e0 con 0x7f00a8078e80 2024-09-13T08:45:10.438 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:10.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot protect non_existent_subvolume subvolume_1 snapshot_1 2024-09-13T08:45:10.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 -- 172.21.15.31:0/3620695613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c15ad00 msgr2=0x7f1a2c15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 --2- 172.21.15.31:0/3620695613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c15ad00 0x7f1a2c15b0e0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0725d0 tx=0x7f1a1c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 -- 172.21.15.31:0/3620695613 shutdown_connections 2024-09-13T08:45:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 --2- 172.21.15.31:0/3620695613 >> v2:172.21.15.45:3300/0 conn(0x7f1a2c0cabb0 0x7f1a2c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 --2- 172.21.15.31:0/3620695613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c15ad00 0x7f1a2c15b0e0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 -- 172.21.15.31:0/3620695613 >> 172.21.15.31:0/3620695613 conn(0x7f1a2c0cfbb0 msgr2=0x7f1a2c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 -- 172.21.15.31:0/3620695613 shutdown_connections 2024-09-13T08:45:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.613+0000 7f1a31c00640 1 -- 172.21.15.31:0/3620695613 wait complete. 2024-09-13T08:45:10.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.615+0000 7f1a31c00640 1 Processor -- start 2024-09-13T08:45:10.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.615+0000 7f1a31c00640 1 -- start start 2024-09-13T08:45:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c0cabb0 0x7f1a2c14abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a2c15ad00 0x7f1a2c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a2c15b620 0x7f1a2c1f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c06eee0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c06ed60 con 0x7f1a2c15ad00 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a31c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1a2c06f060 con 0x7f1a2c15b620 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c0cabb0 0x7f1a2c14abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c0cabb0 0x7f1a2c14abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43330/0 (socket says 172.21.15.31:43330) 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a2aa00640 1 -- 172.21.15.31:0/2499122230 learned_addr learned my addr 172.21.15.31:0/2499122230 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a22000640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a2c15ad00 0x7f1a2c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.616+0000 7f1a2b400640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a2c15b620 0x7f1a2c1f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 -- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a2c15b620 msgr2=0x7f1a2c1f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a2c15b620 0x7f1a2c1f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 -- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a2c15ad00 msgr2=0x7f1a2c0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a2c15ad00 0x7f1a2c0b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 -- 172.21.15.31:0/2499122230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2c1593f0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2b400640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1a2c15b620 0x7f1a2c1f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a2aa00640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a2c0cabb0 0x7f1a2c14abb0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009850 tx=0x7f1a1c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c03f070 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a1c0047a0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c035030 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a31c00640 1 -- 172.21.15.31:0/2499122230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2c1596d0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.617+0000 7f1a31c00640 1 -- 172.21.15.31:0/2499122230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2c14b0f0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.619+0000 7f1a31c00640 1 -- 172.21.15.31:0/2499122230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a2c159bf0 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.620+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f1a1c042070 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.621+0000 7f1a23400640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a08078e80 0x7f1a0807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.621+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f1a1c0bc180 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.621+0000 7f1a22000640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a08078e80 0x7f1a0807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:10.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.621+0000 7f1a22000640 1 --2- 172.21.15.31:0/2499122230 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1a08078e80 0x7f1a0807b340 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c1f6040 tx=0x7f1a14009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.625+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1a1c083890 con 0x7f1a2c0cabb0 2024-09-13T08:45:10.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.856+0000 7f1a31c00640 1 -- 172.21.15.31:0/2499122230 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot protect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f1a2c0cdba0 con 0x7f1a08078e80 2024-09-13T08:45:10.859 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:10.857+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:10.857+0000 7f1a23400640 1 -- 172.21.15.31:0/2499122230 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f1a2c0cdba0 con 0x7f1a08078e80 2024-09-13T08:45:10.859 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:10.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot unprotect non_existent_subvolume subvolume_1 snapshot_1 2024-09-13T08:45:11.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 -- 172.21.15.31:0/1385754401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c0cbc80 msgr2=0x7fd21c0b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 --2- 172.21.15.31:0/1385754401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c0cbc80 0x7fd21c0b29e0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fd214005fd0 tx=0x7fd21402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 -- 172.21.15.31:0/1385754401 shutdown_connections 2024-09-13T08:45:11.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 --2- 172.21.15.31:0/1385754401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c0cbc80 0x7fd21c0b29e0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 --2- 172.21.15.31:0/1385754401 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd21c072560 0x7fd21c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 -- 172.21.15.31:0/1385754401 >> 172.21.15.31:0/1385754401 conn(0x7fd21c0cfbb0 msgr2=0x7fd21c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 -- 172.21.15.31:0/1385754401 shutdown_connections 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.034+0000 7fd223400640 1 -- 172.21.15.31:0/1385754401 wait complete. 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.036+0000 7fd223400640 1 Processor -- start 2024-09-13T08:45:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.036+0000 7fd223400640 1 -- start start 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c072560 0x7fd21c1f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd21c0cbc80 0x7fd21c15e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd21c15bfc0 0x7fd21c155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c06e740 con 0x7fd21c072560 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c06e5c0 con 0x7fd21c0cbc80 2024-09-13T08:45:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd223400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd21c0b2f20 con 0x7fd21c15bfc0 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd220a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c072560 0x7fd21c1f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd220a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c072560 0x7fd21c1f8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43336/0 (socket says 172.21.15.31:43336) 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd220a00640 1 -- 172.21.15.31:0/1826096348 learned_addr learned my addr 172.21.15.31:0/1826096348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd213e00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd21c0cbc80 0x7fd21c15e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.037+0000 7fd221400640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd21c15bfc0 0x7fd21c155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 -- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd21c15bfc0 msgr2=0x7fd21c155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd21c15bfc0 0x7fd21c155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 -- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd21c0cbc80 msgr2=0x7fd21c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd21c0cbc80 0x7fd21c15e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 -- 172.21.15.31:0/1826096348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21c1f8dc0 con 0x7fd21c072560 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd220a00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c072560 0x7fd21c1f8880 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00c3c0 tx=0x7fd20c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c017070 con 0x7fd21c072560 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd20c0035f0 con 0x7fd21c072560 2024-09-13T08:45:11.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c003a80 con 0x7fd21c072560 2024-09-13T08:45:11.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd223400640 1 -- 172.21.15.31:0/1826096348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21c15ecb0 con 0x7fd21c072560 2024-09-13T08:45:11.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.038+0000 7fd223400640 1 -- 172.21.15.31:0/1826096348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21c1f4d40 con 0x7fd21c072560 2024-09-13T08:45:11.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.040+0000 7fd223400640 1 -- 172.21.15.31:0/1826096348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21c0b33b0 con 0x7fd21c072560 2024-09-13T08:45:11.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.041+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fd20c01a070 con 0x7fd21c072560 2024-09-13T08:45:11.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.042+0000 7fd219600640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1f8078e80 0x7fd1f807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.042+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fd20c09a8f0 con 0x7fd21c072560 2024-09-13T08:45:11.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.042+0000 7fd213e00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1f8078e80 0x7fd1f807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.043+0000 7fd213e00640 1 --2- 172.21.15.31:0/1826096348 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1f8078e80 0x7fd1f807b340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd21c0c9e80 tx=0x7fd20400a2d0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.046+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd21c15ecb0 con 0x7fd21c072560 2024-09-13T08:45:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.277+0000 7fd223400640 1 -- 172.21.15.31:0/1826096348 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot unprotect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fd21c0cb310 con 0x7fd1f8078e80 2024-09-13T08:45:11.281 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:11.279+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:11.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.279+0000 7fd219600640 1 -- 172.21.15.31:0/1826096348 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fd21c0cb310 con 0x7fd1f8078e80 2024-09-13T08:45:11.281 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:11.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls non_existent_subvolume subvolume_1 2024-09-13T08:45:11.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.456+0000 7f9c62000640 1 -- 172.21.15.31:0/3343246978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 msgr2=0x7f9c5c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.456+0000 7f9c62000640 1 --2- 172.21.15.31:0/3343246978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c0cb740 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c069dd0 tx=0x7f9c4402a560 comp rx=0 tx=0).stop 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 -- 172.21.15.31:0/3343246978 shutdown_connections 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 --2- 172.21.15.31:0/3343246978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c0cb740 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 --2- 172.21.15.31:0/3343246978 >> v2:172.21.15.45:3300/0 conn(0x7f9c5c15bfc0 0x7f9c5c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 -- 172.21.15.31:0/3343246978 >> 172.21.15.31:0/3343246978 conn(0x7f9c5c0cfbb0 msgr2=0x7f9c5c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 -- 172.21.15.31:0/3343246978 shutdown_connections 2024-09-13T08:45:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.457+0000 7f9c62000640 1 -- 172.21.15.31:0/3343246978 wait complete. 2024-09-13T08:45:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.458+0000 7f9c62000640 1 Processor -- start 2024-09-13T08:45:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.458+0000 7f9c62000640 1 -- start start 2024-09-13T08:45:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c5c15bfc0 0x7f9c5c1f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c5c1f8920 0x7f9c5c0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5c0b3270 con 0x7f9c5c06fa60 2024-09-13T08:45:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5c0b30f0 con 0x7f9c5c1f8920 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c62000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9c5c0b33f0 con 0x7f9c5c15bfc0 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c5b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c5b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43348/0 (socket says 172.21.15.31:43348) 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.459+0000 7f9c5b400640 1 -- 172.21.15.31:0/1455140546 learned_addr learned my addr 172.21.15.31:0/1455140546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5be00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c5c1f8920 0x7f9c5c0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c52a00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c5c15bfc0 0x7f9c5c1f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 -- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c5c15bfc0 msgr2=0x7f9c5c1f83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c5c15bfc0 0x7f9c5c1f83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 -- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c5c1f8920 msgr2=0x7f9c5c0c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c5c1f8920 0x7f9c5c0c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 -- 172.21.15.31:0/1455140546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5c1536d0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c52a00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c5c15bfc0 0x7f9c5c1f83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:11.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c5b400640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c5c06fa60 0x7f9c5c153190 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00d6d0 tx=0x7f9c4c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c017070 con 0x7f9c5c06fa60 2024-09-13T08:45:11.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.460+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c4c005330 con 0x7f9c5c06fa60 2024-09-13T08:45:11.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.461+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c009820 con 0x7f9c5c06fa60 2024-09-13T08:45:11.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.461+0000 7f9c62000640 1 -- 172.21.15.31:0/1455140546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c5c1f8ea0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.461+0000 7f9c62000640 1 -- 172.21.15.31:0/1455140546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c5c153cd0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.462+0000 7f9c62000640 1 -- 172.21.15.31:0/1455140546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c5c153ac0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.463+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9c4c009bd0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.464+0000 7f9c53e00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c38078e80 0x7f9c3807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.464+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f9c4c09b6d0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.464+0000 7f9c52a00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c38078e80 0x7f9c3807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.465+0000 7f9c52a00640 1 --2- 172.21.15.31:0/1455140546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c38078e80 0x7f9c3807b340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9c44033000 tx=0x7f9c44004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.468+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9c4c062dc0 con 0x7f9c5c06fa60 2024-09-13T08:45:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.695+0000 7f9c62000640 1 -- 172.21.15.31:0/1455140546 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f9c5c0cbf40 con 0x7f9c38078e80 2024-09-13T08:45:11.698 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:11.696+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.697+0000 7f9c53e00640 1 -- 172.21.15.31:0/1455140546 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f9c5c0cbf40 con 0x7f9c38078e80 2024-09-13T08:45:11.699 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:11.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone non_existent_subvolume subvolume_1 snapshot_1 clone_1 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 -- 172.21.15.31:0/2404836778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 msgr2=0x7fc2f80ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 --2- 172.21.15.31:0/2404836778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f80ca020 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80b22f0 tx=0x7fc2f002a980 comp rx=0 tx=0).stop 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 -- 172.21.15.31:0/2404836778 shutdown_connections 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 --2- 172.21.15.31:0/2404836778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f80ca020 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 --2- 172.21.15.31:0/2404836778 >> v2:172.21.15.45:3300/0 conn(0x7fc2f80b5300 0x7fc2f80cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 --2- 172.21.15.31:0/2404836778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2f815bb70 0x7fc2f806fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 -- 172.21.15.31:0/2404836778 >> 172.21.15.31:0/2404836778 conn(0x7fc2f80b1520 msgr2=0x7fc2f80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 -- 172.21.15.31:0/2404836778 shutdown_connections 2024-09-13T08:45:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.874+0000 7fc2fca00640 1 -- 172.21.15.31:0/2404836778 wait complete. 2024-09-13T08:45:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.876+0000 7fc2fca00640 1 Processor -- start 2024-09-13T08:45:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.876+0000 7fc2fca00640 1 -- start start 2024-09-13T08:45:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2f80b5300 0x7fc2f815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2f815bb70 0x7fc2f81e86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80ca560 con 0x7fc2f80cc920 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f815c090 con 0x7fc2f80b5300 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2fca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80ca6e0 con 0x7fc2f815bb70 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43360/0 (socket says 172.21.15.31:43360) 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2eea00640 1 -- 172.21.15.31:0/2052876161 learned_addr learned my addr 172.21.15.31:0/2052876161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.877+0000 7fc2f7400640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2f80b5300 0x7fc2f815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2f7e00640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2f815bb70 0x7fc2f81e86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 -- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2f815bb70 msgr2=0x7fc2f81e86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2f815bb70 0x7fc2f81e86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 -- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2f80b5300 msgr2=0x7fc2f815a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2f80b5300 0x7fc2f815a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 -- 172.21.15.31:0/2052876161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f815cbd0 con 0x7fc2f80cc920 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2eea00640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f80cc920 0x7fc2f815c260 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000d950 tx=0x7fc2e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0017020 con 0x7fc2f80cc920 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e0009d20 con 0x7fc2f80cc920 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2fca00640 1 -- 172.21.15.31:0/2052876161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f815ceb0 con 0x7fc2f80cc920 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00106c0 con 0x7fc2f80cc920 2024-09-13T08:45:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.878+0000 7fc2fca00640 1 -- 172.21.15.31:0/2052876161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f815d470 con 0x7fc2f80cc920 2024-09-13T08:45:11.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.879+0000 7fc2fca00640 1 -- 172.21.15.31:0/2052876161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f80cabb0 con 0x7fc2f80cc920 2024-09-13T08:45:11.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.881+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fc2e001a020 con 0x7fc2f80cc920 2024-09-13T08:45:11.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.882+0000 7fc2efe00640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2d4078eb0 0x7fc2d407b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:11.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.882+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fc2e009bba0 con 0x7fc2f80cc920 2024-09-13T08:45:11.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.882+0000 7fc2f7400640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2d4078eb0 0x7fc2d407b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:11.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.882+0000 7fc2f7400640 1 --2- 172.21.15.31:0/2052876161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2d4078eb0 0x7fc2d407b370 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800bad0 tx=0x7fc2e800b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:11.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:11.886+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc2e009de20 con 0x7fc2f80cc920 2024-09-13T08:45:12.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.116+0000 7fc2fca00640 1 -- 172.21.15.31:0/2052876161 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target_sub_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7fc2f8074080 con 0x7fc2d4078eb0 2024-09-13T08:45:12.125 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.123+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:12.125 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.123+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:12.125 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.123+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:12.125 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.123+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:12.125 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.123+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:12.131 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.129+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.130+0000 7fc2efe00640 1 -- 172.21.15.31:0/2052876161 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc2f8074080 con 0x7fc2d4078eb0 2024-09-13T08:45:12.132 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:12.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status non_existent_subvolume clone_1 2024-09-13T08:45:12.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 -- 172.21.15.31:0/75457837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b9415bb70 msgr2=0x7f9b9406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 --2- 172.21.15.31:0/75457837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b9415bb70 0x7f9b9406fa60 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009830 tx=0x7f9b8402a530 comp rx=0 tx=0).stop 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 -- 172.21.15.31:0/75457837 shutdown_connections 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 --2- 172.21.15.31:0/75457837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b9415bb70 0x7f9b9406fa60 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 -- 172.21.15.31:0/75457837 >> 172.21.15.31:0/75457837 conn(0x7f9b940b1520 msgr2=0x7f9b940b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 -- 172.21.15.31:0/75457837 shutdown_connections 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.306+0000 7f9b98e00640 1 -- 172.21.15.31:0/75457837 wait complete. 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.308+0000 7f9b98e00640 1 Processor -- start 2024-09-13T08:45:12.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.308+0000 7f9b98e00640 1 -- start start 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b940b5300 0x7f9b9414b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b940ccaf0 0x7f9b9414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b9415bb70 0x7f9b941592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9406eef0 con 0x7f9b940b5300 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9406ed70 con 0x7f9b940ccaf0 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b98e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9406f070 con 0x7f9b9415bb70 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b940b5300 0x7f9b9414b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b940b5300 0x7f9b9414b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43392/0 (socket says 172.21.15.31:43392) 2024-09-13T08:45:12.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 -- 172.21.15.31:0/1561896331 learned_addr learned my addr 172.21.15.31:0/1561896331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b92a00640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b940ccaf0 0x7f9b9414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93e00640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b9415bb70 0x7f9b941592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 -- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b9415bb70 msgr2=0x7f9b941592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b9415bb70 0x7f9b941592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 -- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b940ccaf0 msgr2=0x7f9b9414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b940ccaf0 0x7f9b9414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.309+0000 7f9b93400640 1 -- 172.21.15.31:0/1561896331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b941f8030 con 0x7f9b940b5300 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b93400640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b940b5300 0x7f9b9414b540 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009950 tx=0x7f9b84033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84031b50 con 0x7f9b940b5300 2024-09-13T08:45:12.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b84031de0 con 0x7f9b940b5300 2024-09-13T08:45:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b98e00640 1 -- 172.21.15.31:0/1561896331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b941f8310 con 0x7f9b940b5300 2024-09-13T08:45:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84005280 con 0x7f9b940b5300 2024-09-13T08:45:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.310+0000 7f9b98e00640 1 -- 172.21.15.31:0/1561896331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b941f88d0 con 0x7f9b940b5300 2024-09-13T08:45:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.311+0000 7f9b98e00640 1 -- 172.21.15.31:0/1561896331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b9406f550 con 0x7f9b940b5300 2024-09-13T08:45:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.312+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f9b84005420 con 0x7f9b940b5300 2024-09-13T08:45:12.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.313+0000 7f9b8b400640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b74078e80 0x7f9b7407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.313+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f9b840bc3b0 con 0x7f9b940b5300 2024-09-13T08:45:12.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.313+0000 7f9b92a00640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b74078e80 0x7f9b7407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.314+0000 7f9b92a00640 1 --2- 172.21.15.31:0/1561896331 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b74078e80 0x7f9b7407b340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9b940cee50 tx=0x7f9b80009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:12.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.317+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9b8403f030 con 0x7f9b940b5300 2024-09-13T08:45:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.552+0000 7f9b98e00640 1 -- 172.21.15.31:0/1561896331 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "non_existent_subvolume", "clone_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7f9b940ae8b0 con 0x7f9b74078e80 2024-09-13T08:45:12.556 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.554+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.554+0000 7f9b8b400640 1 -- 172.21.15.31:0/1561896331 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f9b940ae8b0 con 0x7f9b74078e80 2024-09-13T08:45:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:12.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create non_existent_subvolume group_1 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 -- 172.21.15.31:0/812878446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 msgr2=0x7f70800cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 --2- 172.21.15.31:0/812878446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70800cc820 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f70800ccee0 tx=0x7f707802a9e0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 -- 172.21.15.31:0/812878446 shutdown_connections 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 --2- 172.21.15.31:0/812878446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70800cc820 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 --2- 172.21.15.31:0/812878446 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f708015bae0 0x7f7080073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 -- 172.21.15.31:0/812878446 >> 172.21.15.31:0/812878446 conn(0x7f70800b1520 msgr2=0x7f70800b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 -- 172.21.15.31:0/812878446 shutdown_connections 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.729+0000 7f7085600640 1 -- 172.21.15.31:0/812878446 wait complete. 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.731+0000 7f7085600640 1 Processor -- start 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.731+0000 7f7085600640 1 -- start start 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800b2820 0x7f70800c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70801586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f708015bae0 0x7f70801e8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70800cac10 con 0x7f70800cc390 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70800caa90 con 0x7f70800b2820 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7085600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f70800cad90 con 0x7f708015bae0 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7077e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70801586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7077e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70801586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43418/0 (socket says 172.21.15.31:43418) 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7077e00640 1 -- 172.21.15.31:0/1254167561 learned_addr learned my addr 172.21.15.31:0/1254167561 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f707fe00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800b2820 0x7f70800c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.732+0000 7f7084c00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f708015bae0 0x7f70801e8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 -- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f708015bae0 msgr2=0x7f70801e8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f708015bae0 0x7f70801e8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 -- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800b2820 msgr2=0x7f70800c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70800b2820 0x7f70800c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 -- 172.21.15.31:0/1254167561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7080158c10 con 0x7f70800cc390 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7077e00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70800cc390 0x7f70801586d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f706800c400 tx=0x7f706800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068011790 con 0x7f70800cc390 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7068017070 con 0x7f70800cc390 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068011d00 con 0x7f70800cc390 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7085600640 1 -- 172.21.15.31:0/1254167561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801f4b50 con 0x7f70800cc390 2024-09-13T08:45:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.733+0000 7f7085600640 1 -- 172.21.15.31:0/1254167561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801f5110 con 0x7f70800cc390 2024-09-13T08:45:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.734+0000 7f7085600640 1 -- 172.21.15.31:0/1254167561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70800cc820 con 0x7f70800cc390 2024-09-13T08:45:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.736+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f706801c460 con 0x7f70800cc390 2024-09-13T08:45:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.737+0000 7f707cc00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f705c078e80 0x7f705c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.737+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f706809b360 con 0x7f70800cc390 2024-09-13T08:45:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.737+0000 7f707fe00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f705c078e80 0x7f705c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.737+0000 7f707fe00640 1 --2- 172.21.15.31:0/1254167561 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f705c078e80 0x7f705c07b340 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7070005fd0 tx=0x7f707000a250 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:12.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.741+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7080158c10 con 0x7f70800cc390 2024-09-13T08:45:12.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.964+0000 7f7085600640 1 -- 172.21.15.31:0/1254167561 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f70800cd5a0 con 0x7f705c078e80 2024-09-13T08:45:12.967 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:12.965+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:12.966+0000 7f707cc00640 1 -- 172.21.15.31:0/1254167561 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f70800cd5a0 con 0x7f705c078e80 2024-09-13T08:45:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:12.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm non_existent_subvolume group_1 2024-09-13T08:45:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 -- 172.21.15.31:0/4190963545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc86015bfc0 msgr2=0x7fc860072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 --2- 172.21.15.31:0/4190963545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc86015bfc0 0x7fc860072560 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fc86006e410 tx=0x7fc85002a560 comp rx=0 tx=0).stop 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 -- 172.21.15.31:0/4190963545 shutdown_connections 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 --2- 172.21.15.31:0/4190963545 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86006fa60 0x7fc8600cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 --2- 172.21.15.31:0/4190963545 >> v2:172.21.15.45:3300/0 conn(0x7fc86006fec0 0x7fc8600cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 --2- 172.21.15.31:0/4190963545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc86015bfc0 0x7fc860072560 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.141+0000 7fc865c00640 1 -- 172.21.15.31:0/4190963545 >> 172.21.15.31:0/4190963545 conn(0x7fc8600cfbb0 msgr2=0x7fc8600cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.142+0000 7fc865c00640 1 -- 172.21.15.31:0/4190963545 shutdown_connections 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.142+0000 7fc865c00640 1 -- 172.21.15.31:0/4190963545 wait complete. 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.143+0000 7fc865c00640 1 Processor -- start 2024-09-13T08:45:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.143+0000 7fc865c00640 1 -- start start 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc86006fa60 0x7fc8601f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86015bfc0 0x7fc8600cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8600c5660 0x7fc8600c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600b2b60 con 0x7fc8600c5660 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600b29e0 con 0x7fc86006fa60 2024-09-13T08:45:13.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc865c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600b2ce0 con 0x7fc86015bfc0 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.144+0000 7fc85f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8600c5660 0x7fc8600c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8600c5660 0x7fc8600c5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43450/0 (socket says 172.21.15.31:43450) 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 -- 172.21.15.31:0/3613186464 learned_addr learned my addr 172.21.15.31:0/3613186464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85ea00640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc86006fa60 0x7fc8601f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc856000640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86015bfc0 0x7fc8600cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 -- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86015bfc0 msgr2=0x7fc8600cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86015bfc0 0x7fc8600cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 -- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc86006fa60 msgr2=0x7fc8601f8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc86006fa60 0x7fc8601f8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 -- 172.21.15.31:0/3613186464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8601f9120 con 0x7fc8600c5660 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc856000640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc86015bfc0 0x7fc8600cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:13.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc85f400640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8600c5660 0x7fc8600c5af0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fc85800ead0 tx=0x7fc85800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85800cde0 con 0x7fc8600c5660 2024-09-13T08:45:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.145+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc858004c10 con 0x7fc8600c5660 2024-09-13T08:45:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.146+0000 7fc865c00640 1 -- 172.21.15.31:0/3613186464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8600c61e0 con 0x7fc8600c5660 2024-09-13T08:45:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.146+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858005020 con 0x7fc8600c5660 2024-09-13T08:45:13.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.146+0000 7fc865c00640 1 -- 172.21.15.31:0/3613186464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8601f8400 con 0x7fc8600c5660 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.147+0000 7fc865c00640 1 -- 172.21.15.31:0/3613186464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8601f8230 con 0x7fc8600c5660 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.148+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fc8580051c0 con 0x7fc8600c5660 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.149+0000 7fc857400640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc83c078e80 0x7fc83c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.149+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fc85809bee0 con 0x7fc8600c5660 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.149+0000 7fc85ea00640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc83c078e80 0x7fc83c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.150+0000 7fc85ea00640 1 --2- 172.21.15.31:0/3613186464 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc83c078e80 0x7fc83c07b340 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc850033000 tx=0x7fc850004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.153+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc8601f9120 con 0x7fc8600c5660 2024-09-13T08:45:13.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.373+0000 7fc865c00640 1 -- 172.21.15.31:0/3613186464 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7fc860072560 con 0x7fc83c078e80 2024-09-13T08:45:13.376 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:13.374+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:13.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.374+0000 7fc857400640 1 -- 172.21.15.31:0/3613186464 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc860072560 con 0x7fc83c078e80 2024-09-13T08:45:13.376 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:13.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup getpath non_existent_subvolume group_1 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 -- 172.21.15.31:0/528402215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec070140 msgr2=0x7fa5ec0688b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 --2- 172.21.15.31:0/528402215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec070140 0x7fa5ec0688b0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4005fd0 tx=0x7fa5d402a540 comp rx=0 tx=0).stop 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 -- 172.21.15.31:0/528402215 shutdown_connections 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 --2- 172.21.15.31:0/528402215 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 0x7fa5ec06ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 --2- 172.21.15.31:0/528402215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec070140 0x7fa5ec0688b0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 -- 172.21.15.31:0/528402215 >> 172.21.15.31:0/528402215 conn(0x7fa5ec0ccd00 msgr2=0x7fa5ec0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 -- 172.21.15.31:0/528402215 shutdown_connections 2024-09-13T08:45:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.550+0000 7fa5f1600640 1 -- 172.21.15.31:0/528402215 wait complete. 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.552+0000 7fa5f1600640 1 Processor -- start 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.552+0000 7fa5f1600640 1 -- start start 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ec070140 0x7fa5ec1dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec0b1f70 0x7fa5ec1f8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 0x7fa5ec0c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ec0b24d0 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ec0b2350 con 0x7fa5ec070140 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5f1600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ec0b2650 con 0x7fa5ec0b40c0 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5e3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec0b1f70 0x7fa5ec1f8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5e3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec0b1f70 0x7fa5ec1f8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43466/0 (socket says 172.21.15.31:43466) 2024-09-13T08:45:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.553+0000 7fa5e3400640 1 -- 172.21.15.31:0/864219531 learned_addr learned my addr 172.21.15.31:0/864219531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5ebe00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ec070140 0x7fa5ec1dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5f0c00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 0x7fa5ec0c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 -- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 msgr2=0x7fa5ec0c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 0x7fa5ec0c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 -- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ec070140 msgr2=0x7fa5ec1dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ec070140 0x7fa5ec1dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 -- 172.21.15.31:0/864219531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ec0c9310 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5f0c00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa5ec0b40c0 0x7fa5ec0c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e3400640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5ec0b1f70 0x7fa5ec1f8ce0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4034040 tx=0x7fa5d4004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.554+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d403e030 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.555+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5d4005060 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.555+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4035040 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.555+0000 7fa5f1600640 1 -- 172.21.15.31:0/864219531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ec0c95f0 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.555+0000 7fa5f1600640 1 -- 172.21.15.31:0/864219531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ec0c8f70 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.556+0000 7fa5f1600640 1 -- 172.21.15.31:0/864219531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ec0688b0 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.557+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fa5d4041030 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.558+0000 7fa5e8c00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5c8078e80 0x7fa5c807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.558+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fa5d40bba50 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.558+0000 7fa5ebe00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5c8078e80 0x7fa5c807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.559+0000 7fa5ebe00640 1 --2- 172.21.15.31:0/864219531 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa5c8078e80 0x7fa5c807b340 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc001760 tx=0x7fa5dc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.562+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa5d40831b0 con 0x7fa5ec0b1f70 2024-09-13T08:45:13.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.785+0000 7fa5f1600640 1 -- 172.21.15.31:0/864219531 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7fa5ec0b2ae0 con 0x7fa5c8078e80 2024-09-13T08:45:13.788 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:13.787+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.787+0000 7fa5e8c00640 1 -- 172.21.15.31:0/864219531 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fa5ec0b2ae0 con 0x7fa5c8078e80 2024-09-13T08:45:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:13.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup pin non_existent_subvolume group_1 export 0 2024-09-13T08:45:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 -- 172.21.15.31:0/611937235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b4005230 msgr2=0x7fc2b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 --2- 172.21.15.31:0/611937235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b4005230 0x7fc2b400b9c0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009ed0 tx=0x7fc2b4005830 comp rx=0 tx=0).stop 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 -- 172.21.15.31:0/611937235 shutdown_connections 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 --2- 172.21.15.31:0/611937235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b4005230 0x7fc2b400b9c0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 -- 172.21.15.31:0/611937235 >> 172.21.15.31:0/611937235 conn(0x7fc2d00d0440 msgr2=0x7fc2d00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 -- 172.21.15.31:0/611937235 shutdown_connections 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.967+0000 7fc2d6e00640 1 -- 172.21.15.31:0/611937235 wait complete. 2024-09-13T08:45:13.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.968+0000 7fc2d6e00640 1 Processor -- start 2024-09-13T08:45:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.969+0000 7fc2d6e00640 1 -- start start 2024-09-13T08:45:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2b4005230 0x7fc2d00b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2d00c9e60 0x7fc2d00b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2d00c6620 0x7fc2d00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d015c4d0 con 0x7fc2d00c6620 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0159e90 con 0x7fc2d00c9e60 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d6e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d00cd230 con 0x7fc2b4005230 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2d00c6620 0x7fc2d00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2d00c6620 0x7fc2d00c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43472/0 (socket says 172.21.15.31:43472) 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 -- 172.21.15.31:0/1507211546 learned_addr learned my addr 172.21.15.31:0/1507211546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2cfe00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2b4005230 0x7fc2d00b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2c7400640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2d00c9e60 0x7fc2d00b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 -- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2b4005230 msgr2=0x7fc2d00b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2b4005230 0x7fc2d00b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 -- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2d00c9e60 msgr2=0x7fc2d00b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2d00c9e60 0x7fc2d00b4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.970+0000 7fc2d4e00640 1 -- 172.21.15.31:0/1507211546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d00ca2f0 con 0x7fc2d00c6620 2024-09-13T08:45:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2d4e00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2d00c6620 0x7fc2d00c6ad0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009ed0 tx=0x7fc2c80042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c80046d0 con 0x7fc2d00c6620 2024-09-13T08:45:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2c803c480 con 0x7fc2d00c6620 2024-09-13T08:45:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c803c890 con 0x7fc2d00c6620 2024-09-13T08:45:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2d6e00640 1 -- 172.21.15.31:0/1507211546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d00b57e0 con 0x7fc2d00c6620 2024-09-13T08:45:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.971+0000 7fc2d6e00640 1 -- 172.21.15.31:0/1507211546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d00b5500 con 0x7fc2d00c6620 2024-09-13T08:45:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.972+0000 7fc2d6e00640 1 -- 172.21.15.31:0/1507211546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d0074220 con 0x7fc2d00c6620 2024-09-13T08:45:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.973+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7fc2c8044020 con 0x7fc2d00c6620 2024-09-13T08:45:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.974+0000 7fc2ccc00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2ac078e80 0x7fc2ac07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.974+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fc2c80bc3b0 con 0x7fc2d00c6620 2024-09-13T08:45:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.974+0000 7fc2cfe00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2ac078e80 0x7fc2ac07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:13.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.975+0000 7fc2cfe00640 1 --2- 172.21.15.31:0/1507211546 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2ac078e80 0x7fc2ac07b340 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0001eb0 tx=0x7fc2c0001de0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:13.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:13.979+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc2c804b020 con 0x7fc2d00c6620 2024-09-13T08:45:14.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.200+0000 7fc2d6e00640 1 -- 172.21.15.31:0/1507211546 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "non_existent_subvolume", "group_name": "group_1", "pin_type": "export", "pin_setting": "0", "target": ["mon-mgr", ""]}) -- 0x7fc2d0075000 con 0x7fc2ac078e80 2024-09-13T08:45:14.203 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:14.201+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:14.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.201+0000 7fc2ccc00640 1 -- 172.21.15.31:0/1507211546 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc2d0075000 con 0x7fc2ac078e80 2024-09-13T08:45:14.203 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:14.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup ls non_existent_subvolume 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 -- 172.21.15.31:0/134443894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cde50 msgr2=0x7f417c0ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 --2- 172.21.15.31:0/134443894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cde50 0x7f417c0ca1c0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f4164005fd0 tx=0x7f416402a540 comp rx=0 tx=0).stop 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 -- 172.21.15.31:0/134443894 shutdown_connections 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 --2- 172.21.15.31:0/134443894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f417c0cc0e0 0x7f417c0ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 --2- 172.21.15.31:0/134443894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cde50 0x7f417c0ca1c0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 -- 172.21.15.31:0/134443894 >> 172.21.15.31:0/134443894 conn(0x7f417c0cfb70 msgr2=0x7f417c0cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 -- 172.21.15.31:0/134443894 shutdown_connections 2024-09-13T08:45:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.375+0000 7f4180a00640 1 -- 172.21.15.31:0/134443894 wait complete. 2024-09-13T08:45:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.377+0000 7f4180a00640 1 Processor -- start 2024-09-13T08:45:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.377+0000 7f4180a00640 1 -- start start 2024-09-13T08:45:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cc0e0 0x7f417c1f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f417c0cde50 0x7f417c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c15df30 0x7f417c14d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f417c0cb7c0 con 0x7f417c0cc0e0 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f417c0cb640 con 0x7f417c15df30 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4180a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f417c0cb940 con 0x7f417c0cde50 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.378+0000 7f4179e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cc0e0 0x7f417c1f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cc0e0 0x7f417c1f8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43488/0 (socket says 172.21.15.31:43488) 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 -- 172.21.15.31:0/2715601281 learned_addr learned my addr 172.21.15.31:0/2715601281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4171400640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f417c0cde50 0x7f417c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f417a800640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c15df30 0x7f417c14d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 -- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f417c0cde50 msgr2=0x7f417c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f417c0cde50 0x7f417c15e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 -- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c15df30 msgr2=0x7f417c14d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c15df30 0x7f417c14d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 -- 172.21.15.31:0/2715601281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417c0afb60 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.379+0000 7f4179e00640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f417c0cc0e0 0x7f417c1f8520 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f416c00b3f0 tx=0x7f416c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.380+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c015020 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.380+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f416c011920 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.380+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c011d70 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.380+0000 7f4180a00640 1 -- 172.21.15.31:0/2715601281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417c0afe40 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.380+0000 7f4180a00640 1 -- 172.21.15.31:0/2715601281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f417c0b0400 con 0x7f417c0cc0e0 2024-09-13T08:45:14.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.381+0000 7f4180a00640 1 -- 172.21.15.31:0/2715601281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f417c0caf30 con 0x7f417c0cc0e0 2024-09-13T08:45:14.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.382+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 21) ==== 101848+0+0 (secure 0 0 0) 0x7f416c018020 con 0x7f417c0cc0e0 2024-09-13T08:45:14.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.383+0000 7f4172a00640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4158078e80 0x7f415807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.383+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f416c09b0f0 con 0x7f417c0cc0e0 2024-09-13T08:45:14.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.383+0000 7f4171400640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4158078e80 0x7f415807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.384+0000 7f4171400640 1 --2- 172.21.15.31:0/2715601281 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4158078e80 0x7f415807b340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4164033000 tx=0x7f4164004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:14.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.387+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f416c01f030 con 0x7f417c0cc0e0 2024-09-13T08:45:14.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.503+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f416c05e950 con 0x7f417c0cc0e0 2024-09-13T08:45:14.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.613+0000 7f4180a00640 1 -- 172.21.15.31:0/2715601281 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7f417c0b5af0 con 0x7f4158078e80 2024-09-13T08:45:14.616 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:14.615+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:14.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.615+0000 7f4172a00640 1 -- 172.21.15.31:0/2715601281 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f417c0b5af0 con 0x7f4158078e80 2024-09-13T08:45:14.617 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:14.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot create non_existent_subvolume group_1 snapshot_1 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 -- 172.21.15.31:0/3483739676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb006fec0 msgr2=0x7f7bb00b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 --2- 172.21.15.31:0/3483739676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb006fec0 0x7f7bb00b29e0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0069dd0 tx=0x7f7ba802aac0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 -- 172.21.15.31:0/3483739676 shutdown_connections 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 --2- 172.21.15.31:0/3483739676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb006fec0 0x7f7bb00b29e0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 --2- 172.21.15.31:0/3483739676 >> v2:172.21.15.45:3300/0 conn(0x7f7bb015bfc0 0x7f7bb0072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 -- 172.21.15.31:0/3483739676 >> 172.21.15.31:0/3483739676 conn(0x7f7bb00cfbb0 msgr2=0x7f7bb00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:14.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 -- 172.21.15.31:0/3483739676 shutdown_connections 2024-09-13T08:45:14.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.792+0000 7f7bb5a00640 1 -- 172.21.15.31:0/3483739676 wait complete. 2024-09-13T08:45:14.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.794+0000 7f7bb5a00640 1 Processor -- start 2024-09-13T08:45:14.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.794+0000 7f7bb5a00640 1 -- start start 2024-09-13T08:45:14.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb006fa60 0x7f7bb00c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb015bfc0 0x7f7bb00c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb00c96f0 0x7f7bb01f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb00cbfd0 con 0x7f7bb00c96f0 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb00cbe50 con 0x7f7bb006fa60 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7bb5a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb00b2f20 con 0x7f7bb015bfc0 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7baf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb00c96f0 0x7f7bb01f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7baf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb00c96f0 0x7f7bb01f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43504/0 (socket says 172.21.15.31:43504) 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7baf400640 1 -- 172.21.15.31:0/2229868449 learned_addr learned my addr 172.21.15.31:0/2229868449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7ba6000640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb015bfc0 0x7f7bb00c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.795+0000 7f7baea00640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb006fa60 0x7f7bb00c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 -- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb015bfc0 msgr2=0x7f7bb00c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb015bfc0 0x7f7bb00c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 -- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb006fa60 msgr2=0x7f7bb00c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb006fa60 0x7f7bb00c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 -- 172.21.15.31:0/2229868449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb00c8650 con 0x7f7bb00c96f0 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.796+0000 7f7baf400640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb00c96f0 0x7f7bb01f60b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009d60 tx=0x7f7ba8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.797+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8044070 con 0x7f7bb00c96f0 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.797+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ba8004250 con 0x7f7bb00c96f0 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.797+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8004680 con 0x7f7bb00c96f0 2024-09-13T08:45:14.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.797+0000 7f7bb5a00640 1 -- 172.21.15.31:0/2229868449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb00c8930 con 0x7f7bb00c96f0 2024-09-13T08:45:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.797+0000 7f7bb5a00640 1 -- 172.21.15.31:0/2229868449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb00c8ef0 con 0x7f7bb00c96f0 2024-09-13T08:45:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.798+0000 7f7bb5a00640 1 -- 172.21.15.31:0/2229868449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb00b33e0 con 0x7f7bb00c96f0 2024-09-13T08:45:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.799+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f7ba802c840 con 0x7f7bb00c96f0 2024-09-13T08:45:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.800+0000 7f7ba7400640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b8c078f40 0x7f7b8c07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.801+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f7ba80bbc00 con 0x7f7bb00c96f0 2024-09-13T08:45:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.801+0000 7f7baea00640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b8c078f40 0x7f7b8c07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.801+0000 7f7baea00640 1 --2- 172.21.15.31:0/2229868449 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b8c078f40 0x7f7b8c07b400 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0156b30 tx=0x7f7ba000b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:14.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:14.805+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7ba8032030 con 0x7f7bb00c96f0 2024-09-13T08:45:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.027+0000 7f7bb5a00640 1 -- 172.21.15.31:0/2229868449 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f7bb00cb310 con 0x7f7b8c078f40 2024-09-13T08:45:15.030 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:15.028+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.029+0000 7f7ba7400640 1 -- 172.21.15.31:0/2229868449 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f7bb00cb310 con 0x7f7b8c078f40 2024-09-13T08:45:15.031 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:15.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot rm non_existent_subvolume group_1 snapshot_1 2024-09-13T08:45:15.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 -- 172.21.15.31:0/3219608395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fb4004800 msgr2=0x7f3fb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 --2- 172.21.15.31:0/3219608395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c310 tx=0x7f3fb4005830 comp rx=0 tx=0).stop 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 -- 172.21.15.31:0/3219608395 shutdown_connections 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 --2- 172.21.15.31:0/3219608395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 -- 172.21.15.31:0/3219608395 >> 172.21.15.31:0/3219608395 conn(0x7f3fd00ccd00 msgr2=0x7f3fd00cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 -- 172.21.15.31:0/3219608395 shutdown_connections 2024-09-13T08:45:15.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.209+0000 7f3fd6400640 1 -- 172.21.15.31:0/3219608395 wait complete. 2024-09-13T08:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.211+0000 7f3fd6400640 1 Processor -- start 2024-09-13T08:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.211+0000 7f3fd6400640 1 -- start start 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fb4004800 0x7f3fd00c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd00c92f0 0x7f3fd015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fd01f7a40 0x7f3fd01f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd00ce510 con 0x7f3fd00c92f0 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd00ce390 con 0x7f3fd01f7a40 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd6400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd00ce690 con 0x7f3fb4004800 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd00c92f0 0x7f3fd015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd00c92f0 0x7f3fd015cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60240/0 (socket says 172.21.15.31:60240) 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fc7e00640 1 -- 172.21.15.31:0/2816916532 learned_addr learned my addr 172.21.15.31:0/2816916532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.212+0000 7f3fd5000640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fb4004800 0x7f3fd00c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fd5a00640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fd01f7a40 0x7f3fd01f7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 -- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fb4004800 msgr2=0x7f3fd00c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3fb4004800 0x7f3fd00c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 -- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fd01f7a40 msgr2=0x7f3fd01f7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fd01f7a40 0x7f3fd01f7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 -- 172.21.15.31:0/2816916532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd01f8810 con 0x7f3fd00c92f0 2024-09-13T08:45:15.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fc7e00640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd00c92f0 0x7f3fd015cd80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8004bc0 tx=0x7f3fb8004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:15.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8031040 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fb8004e10 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8006c70 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.213+0000 7f3fd6400640 1 -- 172.21.15.31:0/2816916532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd00c44d0 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.214+0000 7f3fd6400640 1 -- 172.21.15.31:0/2816916532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd00c4a90 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.215+0000 7f3fd6400640 1 -- 172.21.15.31:0/2816916532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd00b38f0 con 0x7f3fd00c92f0 2024-09-13T08:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.216+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f3fb8034070 con 0x7f3fd00c92f0 2024-09-13T08:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.217+0000 7f3fcd600640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3fac078f10 0x7f3fac07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.217+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f3fb80bb5b0 con 0x7f3fd00c92f0 2024-09-13T08:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.217+0000 7f3fd5000640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3fac078f10 0x7f3fac07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.218+0000 7f3fd5000640 1 --2- 172.21.15.31:0/2816916532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3fac078f10 0x7f3fac07b3d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0052890 tx=0x7f3fc0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:15.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.221+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3fb8083210 con 0x7f3fd00c92f0 2024-09-13T08:45:15.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.447+0000 7f3fd6400640 1 -- 172.21.15.31:0/2816916532 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f3fd00c9860 con 0x7f3fac078f10 2024-09-13T08:45:15.450 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:15.449+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.449+0000 7f3fcd600640 1 -- 172.21.15.31:0/2816916532 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f3fc0009d20 con 0x7f3fac078f10 2024-09-13T08:45:15.451 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:15.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot ls non_existent_subvolume group_1 2024-09-13T08:45:15.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 -- 172.21.15.31:0/3609264769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d2710 msgr2=0x7fad280b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 --2- 172.21.15.31:0/3609264769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d2710 0x7fad280b3460 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fad2806cfe0 tx=0x7fad2002a420 comp rx=0 tx=0).stop 2024-09-13T08:45:15.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 -- 172.21.15.31:0/3609264769 shutdown_connections 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 --2- 172.21.15.31:0/3609264769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d2710 0x7fad280b3460 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 --2- 172.21.15.31:0/3609264769 >> v2:172.21.15.45:3300/0 conn(0x7fad280d1d90 0x7fad280d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 --2- 172.21.15.31:0/3609264769 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad28161470 0x7fad280b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 -- 172.21.15.31:0/3609264769 >> 172.21.15.31:0/3609264769 conn(0x7fad280d3e90 msgr2=0x7fad280d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 -- 172.21.15.31:0/3609264769 shutdown_connections 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.625+0000 7fad2ca00640 1 -- 172.21.15.31:0/3609264769 wait complete. 2024-09-13T08:45:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.628+0000 7fad2ca00640 1 Processor -- start 2024-09-13T08:45:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.628+0000 7fad2ca00640 1 -- start start 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d1d90 0x7fad280cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad280d2710 0x7fad280b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad28161470 0x7fad280b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad280b3b20 con 0x7fad280d1d90 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fad280b39a0 con 0x7fad28161470 2024-09-13T08:45:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fad280b3ca0 con 0x7fad280d2710 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d1d90 0x7fad280cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d1d90 0x7fad280cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60262/0 (socket says 172.21.15.31:60262) 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 -- 172.21.15.31:0/812156675 learned_addr learned my addr 172.21.15.31:0/812156675 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad27200640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad28161470 0x7fad280b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad25e00640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad280d2710 0x7fad280b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 -- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad280d2710 msgr2=0x7fad280b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fad280d2710 0x7fad280b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 -- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad28161470 msgr2=0x7fad280b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad28161470 0x7fad280b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 -- 172.21.15.31:0/812156675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad281f4cc0 con 0x7fad280d1d90 2024-09-13T08:45:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad27200640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad28161470 0x7fad280b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad26800640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad280d1d90 0x7fad280cd8f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fad1800ba50 tx=0x7fad1800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18019070 con 0x7fad280d1d90 2024-09-13T08:45:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad1800ccf0 con 0x7fad280d1d90 2024-09-13T08:45:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad180146a0 con 0x7fad280d1d90 2024-09-13T08:45:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 -- 172.21.15.31:0/812156675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad281f4fa0 con 0x7fad280d1d90 2024-09-13T08:45:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.629+0000 7fad2ca00640 1 -- 172.21.15.31:0/812156675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad281f5560 con 0x7fad280d1d90 2024-09-13T08:45:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.630+0000 7fad2ca00640 1 -- 172.21.15.31:0/812156675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad280b4150 con 0x7fad280d1d90 2024-09-13T08:45:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.631+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fad1801c070 con 0x7fad280d1d90 2024-09-13T08:45:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.632+0000 7fad1ea00640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fad08078f10 0x7fad0807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.632+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fad1809b280 con 0x7fad280d1d90 2024-09-13T08:45:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.632+0000 7fad25e00640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fad08078f10 0x7fad0807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.633+0000 7fad25e00640 1 --2- 172.21.15.31:0/812156675 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fad08078f10 0x7fad0807b3d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fad140058c0 tx=0x7fad14005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.637+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fad18062f70 con 0x7fad280d1d90 2024-09-13T08:45:15.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.861+0000 7fad2ca00640 1 -- 172.21.15.31:0/812156675 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot ls", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7fad280d4550 con 0x7fad08078f10 2024-09-13T08:45:15.864 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:15.863+0000 7fe16b200640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:15.863+0000 7fad1ea00640 1 -- 172.21.15.31:0/812156675 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fad280d4550 con 0x7fad08078f10 2024-09-13T08:45:15.865 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-13T08:45:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T08:45:15.875 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:45:15.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:15.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:16.060 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:45:16.061 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:16.061 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:16.061 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:16.097 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:16.097 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:16.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.374+0000 7fc04f200640 1 -- 172.21.15.31:0/4000479055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc05015e160 msgr2=0x7fc0500d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.374+0000 7fc04f200640 1 --2- 172.21.15.31:0/4000479055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc05015e160 0x7fc0500d0790 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc040009900 tx=0x7fc04002a610 comp rx=0 tx=0).stop 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 -- 172.21.15.31:0/4000479055 shutdown_connections 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 --2- 172.21.15.31:0/4000479055 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc0500d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 --2- 172.21.15.31:0/4000479055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc05015e160 0x7fc0500d0790 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 -- 172.21.15.31:0/4000479055 >> 172.21.15.31:0/4000479055 conn(0x7fc0500d3f50 msgr2=0x7fc0500d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 -- 172.21.15.31:0/4000479055 shutdown_connections 2024-09-13T08:45:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 -- 172.21.15.31:0/4000479055 wait complete. 2024-09-13T08:45:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 Processor -- start 2024-09-13T08:45:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.375+0000 7fc04f200640 1 -- start start 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0500c8340 0x7fc0501f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc050157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc0500cfe00 con 0x7fc05015e160 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0500cfc80 con 0x7fc0500c8340 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc0500c8f00 con 0x7fc0500d0cd0 2024-09-13T08:45:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60274/0 (socket says 172.21.15.31:60274) 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 -- 172.21.15.31:0/2572839813 learned_addr learned my addr 172.21.15.31:0/2572839813 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04de00640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0500c8340 0x7fc0501f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc045400640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc050157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 -- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 msgr2=0x7fc050157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc050157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 -- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0500c8340 msgr2=0x7fc0501f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0500c8340 0x7fc0501f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc04e800640 1 -- 172.21.15.31:0/2572839813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0501f8fc0 con 0x7fc05015e160 2024-09-13T08:45:16.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.377+0000 7fc045400640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc050157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc04e800640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c7d0 tx=0x7fc04800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048003b70 con 0x7fc05015e160 2024-09-13T08:45:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc048003d30 con 0x7fc05015e160 2024-09-13T08:45:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04800aab0 con 0x7fc05015e160 2024-09-13T08:45:16.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0500b9010 con 0x7fc05015e160 2024-09-13T08:45:16.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.378+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0501e35b0 con 0x7fc05015e160 2024-09-13T08:45:16.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.380+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0500c93b0 con 0x7fc05015e160 2024-09-13T08:45:16.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.380+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fc04800ac50 con 0x7fc05015e160 2024-09-13T08:45:16.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.381+0000 7fc046a00640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 0x7fc02c07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:16.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.381+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fc04809b320 con 0x7fc05015e160 2024-09-13T08:45:16.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.382+0000 7fc04de00640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 0x7fc02c07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:16.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.382+0000 7fc04de00640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 0x7fc02c07b400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc040033000 tx=0x7fc040004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:16.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.387+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc0501f8fc0 con 0x7fc05015e160 2024-09-13T08:45:16.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.599+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0500d30e0 con 0x7fc05015e160 2024-09-13T08:45:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.599+0000 7fc046a00640 1 -- 172.21.15.31:0/2572839813 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v139) ==== 80+0+0 (secure 0 0 0) 0x7fc05015e540 con 0x7fc05015e160 2024-09-13T08:45:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 msgr2=0x7fc02c07b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 0x7fc02c07b400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc040033000 tx=0x7fc040004730 comp rx=0 tx=0).stop 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 msgr2=0x7fc0500b6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c7d0 tx=0x7fc04800cc90 comp rx=0 tx=0).stop 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 shutdown_connections 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc02c078f40 0x7fc02c07b400 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05015e160 0x7fc0500b6d80 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc0500d0cd0 0x7fc050157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 --2- 172.21.15.31:0/2572839813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0500c8340 0x7fc0501f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 >> 172.21.15.31:0/2572839813 conn(0x7fc0500d3f50 msgr2=0x7fc05006e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 shutdown_connections 2024-09-13T08:45:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:16.602+0000 7fc04f200640 1 -- 172.21.15.31:0/2572839813 wait complete. 2024-09-13T08:45:16.619 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:45:16.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:16.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:16.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:45:16.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:16.683 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:45:16.719 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:16.719 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:16.720 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:16.760 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:16.760 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:16.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:45:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 -- 172.21.15.31:0/2034691093 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c0ced20 msgr2=0x7fce3c0ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 --2- 172.21.15.31:0/2034691093 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c0ced20 0x7fce3c0ce290 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce24009860 tx=0x7fce2402a440 comp rx=0 tx=0).stop 2024-09-13T08:45:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 -- 172.21.15.31:0/2034691093 shutdown_connections 2024-09-13T08:45:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 --2- 172.21.15.31:0/2034691093 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c0ced20 0x7fce3c0ce290 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 -- 172.21.15.31:0/2034691093 >> 172.21.15.31:0/2034691093 conn(0x7fce3c0b18f0 msgr2=0x7fce3c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 -- 172.21.15.31:0/2034691093 shutdown_connections 2024-09-13T08:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.002+0000 7fce43400640 1 -- 172.21.15.31:0/2034691093 wait complete. 2024-09-13T08:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.003+0000 7fce43400640 1 Processor -- start 2024-09-13T08:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.004+0000 7fce43400640 1 -- start start 2024-09-13T08:45:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.004+0000 7fce43400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c073860 0x7fce3c1f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.004+0000 7fce43400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce3c0ced20 0x7fce3c14c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce43400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce43400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c074400 con 0x7fce3c15be30 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce43400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c0ce9e0 con 0x7fce3c0ced20 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce43400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fce3c074580 con 0x7fce3c073860 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60276/0 (socket says 172.21.15.31:60276) 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 -- 172.21.15.31:0/773088495 learned_addr learned my addr 172.21.15.31:0/773088495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce40a00640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c073860 0x7fce3c1f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce33e00640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce3c0ced20 0x7fce3c14c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 -- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c073860 msgr2=0x7fce3c1f5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c073860 0x7fce3c1f5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 -- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce3c0ced20 msgr2=0x7fce3c14c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce3c0ced20 0x7fce3c14c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.005+0000 7fce41400640 1 -- 172.21.15.31:0/773088495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3c0c8f70 con 0x7fce3c15be30 2024-09-13T08:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce41400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c7f0 tx=0x7fce3400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34017070 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce34005070 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340054c0 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce3c14cc80 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.006+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce3c1f5bb0 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.007+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce3c0c9140 con 0x7fce3c15be30 2024-09-13T08:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.008+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fce3401a070 con 0x7fce3c15be30 2024-09-13T08:45:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.009+0000 7fce39600640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 0x7fce1807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.009+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fce3409ad80 con 0x7fce3c15be30 2024-09-13T08:45:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.010+0000 7fce40a00640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 0x7fce1807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.010+0000 7fce40a00640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 0x7fce1807b3d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fce2c001890 tx=0x7fce2c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:17.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.013+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fce34062290 con 0x7fce3c15be30 2024-09-13T08:45:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.221+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fce3c0af380 con 0x7fce3c15be30 2024-09-13T08:45:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.221+0000 7fce39600640 1 -- 172.21.15.31:0/773088495 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v139) ==== 80+0+0 (secure 0 0 0) 0x7fce3c0af380 con 0x7fce3c15be30 2024-09-13T08:45:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 msgr2=0x7fce1807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 0x7fce1807b3d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fce2c001890 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 msgr2=0x7fce3c09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c7f0 tx=0x7fce3400ccb0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 shutdown_connections 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fce18078f10 0x7fce1807b3d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce3c15be30 0x7fce3c09aa10 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce3c0ced20 0x7fce3c14c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 --2- 172.21.15.31:0/773088495 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fce3c073860 0x7fce3c1f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.223+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 >> 172.21.15.31:0/773088495 conn(0x7fce3c0b18f0 msgr2=0x7fce3c0b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.224+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 shutdown_connections 2024-09-13T08:45:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.224+0000 7fce43400640 1 -- 172.21.15.31:0/773088495 wait complete. 2024-09-13T08:45:17.240 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:45:17.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:17.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:17.309 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:45:17.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:17.310 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:45:17.350 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:17.350 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:17.350 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:17.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:45:17.386 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:17.386 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:17.386 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:17.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:45:17.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 -- 172.21.15.31:0/1280047228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee040cf5b0 msgr2=0x7fee0406f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 --2- 172.21.15.31:0/1280047228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee040cf5b0 0x7fee0406f4e0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fedfc005fd0 tx=0x7fedfc02aa10 comp rx=0 tx=0).stop 2024-09-13T08:45:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 -- 172.21.15.31:0/1280047228 shutdown_connections 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 --2- 172.21.15.31:0/1280047228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee040cf5b0 0x7fee0406f4e0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 --2- 172.21.15.31:0/1280047228 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415ad60 0x7fee0415f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 -- 172.21.15.31:0/1280047228 >> 172.21.15.31:0/1280047228 conn(0x7fee040cc910 msgr2=0x7fee040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 -- 172.21.15.31:0/1280047228 shutdown_connections 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.586+0000 7fee08e00640 1 -- 172.21.15.31:0/1280047228 wait complete. 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.588+0000 7fee08e00640 1 Processor -- start 2024-09-13T08:45:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.588+0000 7fee08e00640 1 -- start start 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee040cf5b0 0x7fee04150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415b140 0x7fee041e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee0406fea0 con 0x7fee0415ad60 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fee0406fd20 con 0x7fee040cf5b0 2024-09-13T08:45:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee08e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fee040ca4e0 con 0x7fee0415b140 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60298/0 (socket says 172.21.15.31:60298) 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 -- 172.21.15.31:0/1345588597 learned_addr learned my addr 172.21.15.31:0/1345588597 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee02800640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415b140 0x7fee041e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fee01e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee040cf5b0 0x7fee04150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 -- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415b140 msgr2=0x7fee041e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415b140 0x7fee041e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 -- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee040cf5b0 msgr2=0x7fee04150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee040cf5b0 0x7fee04150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.589+0000 7fedf9400640 1 -- 172.21.15.31:0/1345588597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0415dd40 con 0x7fee0415ad60 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fedf9400640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c430 tx=0x7fedec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:17.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec017070 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedec0053c0 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee040afa00 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec003850 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.590+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0415da60 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.591+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee040743d0 con 0x7fee0415ad60 2024-09-13T08:45:17.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.592+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fedec01a070 con 0x7fee0415ad60 2024-09-13T08:45:17.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.593+0000 7fedfaa00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 0x7fede007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:17.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.594+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fedec09aa00 con 0x7fee0415ad60 2024-09-13T08:45:17.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.594+0000 7fee01e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 0x7fede007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:17.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.594+0000 7fee01e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 0x7fede007b3d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fee040b5d10 tx=0x7fedf400b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:17.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.597+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fee040743d0 con 0x7fee0415ad60 2024-09-13T08:45:17.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.834+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee040c33c0 con 0x7fee0415ad60 2024-09-13T08:45:17.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.835+0000 7fedfaa00640 1 -- 172.21.15.31:0/1345588597 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5221 (secure 0 0 0) 0x7fee040c33c0 con 0x7fee0415ad60 2024-09-13T08:45:17.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:17.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"btime":"2024-09-13T08:45:05:455477+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:45:17.836 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2024-09-13T08:45:17.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 msgr2=0x7fede007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 0x7fede007b3d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fee040b5d10 tx=0x7fedf400b390 comp rx=0 tx=0).stop 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 msgr2=0x7fee0415cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c430 tx=0x7fedec00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 shutdown_connections 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fede0078f10 0x7fede007b3d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fee0415b140 0x7fee041e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee0415ad60 0x7fee0415cda0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 --2- 172.21.15.31:0/1345588597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee040cf5b0 0x7fee04150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 >> 172.21.15.31:0/1345588597 conn(0x7fee040cc910 msgr2=0x7fee04072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 shutdown_connections 2024-09-13T08:45:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:17.837+0000 7fee08e00640 1 -- 172.21.15.31:0/1345588597 wait complete. 2024-09-13T08:45:17.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 -- 172.21.15.31:0/2323048443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 msgr2=0x7f76880cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 --2- 172.21.15.31:0/2323048443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f76880cbc50 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f7680005fd0 tx=0x7f768002aa90 comp rx=0 tx=0).stop 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 -- 172.21.15.31:0/2323048443 shutdown_connections 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 --2- 172.21.15.31:0/2323048443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f76880cbc50 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 --2- 172.21.15.31:0/2323048443 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f768815bd70 0x7f76880cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 -- 172.21.15.31:0/2323048443 >> 172.21.15.31:0/2323048443 conn(0x7f76880cc910 msgr2=0x7f76880cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 -- 172.21.15.31:0/2323048443 shutdown_connections 2024-09-13T08:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.015+0000 7f768e800640 1 -- 172.21.15.31:0/2323048443 wait complete. 2024-09-13T08:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.016+0000 7f768e800640 1 Processor -- start 2024-09-13T08:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 -- start start 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f768815bd70 0x7f7688154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 0x7f7688151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76880cec50 con 0x7f768815fe20 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f76880cead0 con 0x7f768815f9e0 2024-09-13T08:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.017+0000 7f768e800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f76880cb850 con 0x7f768815bd70 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60324/0 (socket says 172.21.15.31:60324) 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 -- 172.21.15.31:0/51708584 learned_addr learned my addr 172.21.15.31:0/51708584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 -- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f768815bd70 msgr2=0x7f7688154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f768815bd70 0x7f7688154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 -- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 msgr2=0x7f7688151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7686a00640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 0x7f7688151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 0x7f7688151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 -- 172.21.15.31:0/51708584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76880c51e0 con 0x7f768815fe20 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7686a00640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 0x7f7688151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.018+0000 7f7687e00640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f7680006010 tx=0x7f7680003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.019+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680044070 con 0x7f768815fe20 2024-09-13T08:45:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.019+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7680004040 con 0x7f768815fe20 2024-09-13T08:45:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.019+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680004430 con 0x7f768815fe20 2024-09-13T08:45:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.019+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76880c54c0 con 0x7f768815fe20 2024-09-13T08:45:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.019+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881524f0 con 0x7f768815fe20 2024-09-13T08:45:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.020+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76880cf850 con 0x7f768815fe20 2024-09-13T08:45:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.021+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f7680047070 con 0x7f768815fe20 2024-09-13T08:45:18.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.022+0000 7f767f400640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 0x7f766807b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.022+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f76800bc380 con 0x7f768815fe20 2024-09-13T08:45:18.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.022+0000 7f7687400640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 0x7f766807b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.023+0000 7f7687400640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 0x7f766807b400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f76780058c0 tx=0x7f7678005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.026+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7680032030 con 0x7f768815fe20 2024-09-13T08:45:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.260+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76880cb1b0 con 0x7f768815fe20 2024-09-13T08:45:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.260+0000 7f767f400640 1 -- 172.21.15.31:0/51708584 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5221 (secure 0 0 0) 0x7f7680083ee0 con 0x7f768815fe20 2024-09-13T08:45:18.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:18.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"btime":"2024-09-13T08:45:05:455477+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:45:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 msgr2=0x7f766807b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 0x7f766807b400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f76780058c0 tx=0x7f7678005850 comp rx=0 tx=0).stop 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 msgr2=0x7f7688153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f7680006010 tx=0x7f7680003ce0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 shutdown_connections 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7668078f40 0x7f766807b400 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f768815fe20 0x7f7688153a00 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768815f9e0 0x7f7688151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 --2- 172.21.15.31:0/51708584 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f768815bd70 0x7f7688154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 >> 172.21.15.31:0/51708584 conn(0x7f76880cc910 msgr2=0x7f76880cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 shutdown_connections 2024-09-13T08:45:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.263+0000 7f768e800640 1 -- 172.21.15.31:0/51708584 wait complete. 2024-09-13T08:45:18.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 -- 172.21.15.31:0/2307317134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c072850 msgr2=0x7f391c073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 --2- 172.21.15.31:0/2307317134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c072850 0x7f391c073fc0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f3908009830 tx=0x7f390802a560 comp rx=0 tx=0).stop 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 -- 172.21.15.31:0/2307317134 shutdown_connections 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 --2- 172.21.15.31:0/2307317134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c072850 0x7f391c073fc0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 --2- 172.21.15.31:0/2307317134 >> v2:172.21.15.45:3300/0 conn(0x7f391c152f10 0x7f391c0cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 -- 172.21.15.31:0/2307317134 >> 172.21.15.31:0/2307317134 conn(0x7f391c0cc910 msgr2=0x7f391c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 -- 172.21.15.31:0/2307317134 shutdown_connections 2024-09-13T08:45:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.440+0000 7f3921600640 1 -- 172.21.15.31:0/2307317134 wait complete. 2024-09-13T08:45:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.441+0000 7f3921600640 1 Processor -- start 2024-09-13T08:45:18.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.442+0000 7f3921600640 1 -- start start 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.442+0000 7f3921600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f391c072850 0x7f391c0c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.442+0000 7f3921600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f391c074500 0x7f391c14da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.442+0000 7f3921600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f3921600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0ca4e0 con 0x7f391c152f10 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f3921600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0cc6b0 con 0x7f391c072850 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f3921600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f391c0ca660 con 0x7f391c074500 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60346/0 (socket says 172.21.15.31:60346) 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 -- 172.21.15.31:0/280315583 learned_addr learned my addr 172.21.15.31:0/280315583 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391a000640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f391c074500 0x7f391c14da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391aa00640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f391c072850 0x7f391c0c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 -- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f391c074500 msgr2=0x7f391c14da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f391c074500 0x7f391c14da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 -- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f391c072850 msgr2=0x7f391c0c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f391c072850 0x7f391c0c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.443+0000 7f391b400640 1 -- 172.21.15.31:0/280315583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c070340 con 0x7f391c152f10 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f391b400640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f391400cbc0 tx=0x7f391400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914016020 con 0x7f391c152f10 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f391400ae30 con 0x7f391c152f10 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c0c4c60 con 0x7f391c152f10 2024-09-13T08:45:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914012660 con 0x7f391c152f10 2024-09-13T08:45:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.444+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c14e9e0 con 0x7f391c152f10 2024-09-13T08:45:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.445+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391c0cfb10 con 0x7f391c152f10 2024-09-13T08:45:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.446+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f3914019020 con 0x7f391c152f10 2024-09-13T08:45:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.447+0000 7f3912a00640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 0x7f38fc07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.447+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7f391409bd60 con 0x7f391c152f10 2024-09-13T08:45:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.448+0000 7f391aa00640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 0x7f38fc07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.448+0000 7f391aa00640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 0x7f38fc07b400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f390c005fd0 tx=0x7f390c005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.451+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f391c0cfb10 con 0x7f391c152f10 2024-09-13T08:45:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.650+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f391c0b2250 con 0x7f391c152f10 2024-09-13T08:45:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.651+0000 7f3912a00640 1 -- 172.21.15.31:0/280315583 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16758 (secure 0 0 0) 0x7f3914020020 con 0x7f391c152f10 2024-09-13T08:45:18.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:18.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":139,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:56.260431+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:52.478415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:53.653411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 msgr2=0x7f38fc07b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 0x7f38fc07b400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f390c005fd0 tx=0x7f390c005850 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 msgr2=0x7f391c1e8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f391400cbc0 tx=0x7f391400a430 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 shutdown_connections 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f38fc078f40 0x7f38fc07b400 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c152f10 0x7f391c1e8a80 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f391c074500 0x7f391c14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 --2- 172.21.15.31:0/280315583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f391c072850 0x7f391c0c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.654+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 >> 172.21.15.31:0/280315583 conn(0x7f391c0cc910 msgr2=0x7f391c0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.655+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 shutdown_connections 2024-09-13T08:45:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.655+0000 7f3921600640 1 -- 172.21.15.31:0/280315583 wait complete. 2024-09-13T08:45:18.671 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:45:18.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:18.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 -- 172.21.15.31:0/2483841349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 msgr2=0x7efd0c0b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 --2- 172.21.15.31:0/2483841349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0b1910 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0cd9b0 tx=0x7efd0402a980 comp rx=0 tx=0).stop 2024-09-13T08:45:18.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 -- 172.21.15.31:0/2483841349 shutdown_connections 2024-09-13T08:45:18.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 --2- 172.21.15.31:0/2483841349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0b1910 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 --2- 172.21.15.31:0/2483841349 >> v2:172.21.15.45:3300/0 conn(0x7efd0c072850 0x7efd0c073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 --2- 172.21.15.31:0/2483841349 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 0x7efd0c0cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 -- 172.21.15.31:0/2483841349 >> 172.21.15.31:0/2483841349 conn(0x7efd0c0cc910 msgr2=0x7efd0c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 -- 172.21.15.31:0/2483841349 shutdown_connections 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.832+0000 7efd10a00640 1 -- 172.21.15.31:0/2483841349 wait complete. 2024-09-13T08:45:18.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.833+0000 7efd10a00640 1 Processor -- start 2024-09-13T08:45:18.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 -- start start 2024-09-13T08:45:18.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c072850 0x7efd0c1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 0x7efd0c0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0ca4e0 con 0x7efd0c074500 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0cc6b0 con 0x7efd0c072850 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.834+0000 7efd10a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c0ca660 con 0x7efd0c152f10 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60362/0 (socket says 172.21.15.31:60362) 2024-09-13T08:45:18.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 -- 172.21.15.31:0/1128878483 learned_addr learned my addr 172.21.15.31:0/1128878483 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd09e00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c072850 0x7efd0c1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd0a800640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 0x7efd0c0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 -- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 msgr2=0x7efd0c0b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 0x7efd0c0b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 -- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c072850 msgr2=0x7efd0c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c072850 0x7efd0c1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.835+0000 7efd01400640 1 -- 172.21.15.31:0/1128878483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c0b39f0 con 0x7efd0c074500 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd01400640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7efcf400b4a0 tx=0x7efcf400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4014070 con 0x7efd0c074500 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf4010700 con 0x7efd0c074500 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4010b50 con 0x7efd0c074500 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0c0b4690 con 0x7efd0c074500 2024-09-13T08:45:18.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.836+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0c152700 con 0x7efd0c074500 2024-09-13T08:45:18.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.837+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0c0b1910 con 0x7efd0c074500 2024-09-13T08:45:18.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.838+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7efcf4017070 con 0x7efd0c074500 2024-09-13T08:45:18.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.839+0000 7efd02a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 0x7efce807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:18.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.839+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7efcf409b0b0 con 0x7efd0c074500 2024-09-13T08:45:18.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.839+0000 7efd09e00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 0x7efce807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:18.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.840+0000 7efd09e00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 0x7efce807b3d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0c09d4d0 tx=0x7efcfc008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:18.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:18.844+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efcf4062c50 con 0x7efd0c074500 2024-09-13T08:45:19.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.077+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd0c0cfa70 con 0x7efd0c074500 2024-09-13T08:45:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.078+0000 7efd02a00640 1 -- 172.21.15.31:0/1128878483 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5221 (secure 0 0 0) 0x7efcf4062550 con 0x7efd0c074500 2024-09-13T08:45:19.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:19.079 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"btime":"2024-09-13T08:45:05:455477+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:45:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 msgr2=0x7efce807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 0x7efce807b3d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0c09d4d0 tx=0x7efcfc008040 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 msgr2=0x7efd0c0c3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7efcf400b4a0 tx=0x7efcf400b960 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 shutdown_connections 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efce8078f10 0x7efce807b3d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efd0c152f10 0x7efd0c0b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd0c074500 0x7efd0c0c3530 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.080+0000 7efd10a00640 1 --2- 172.21.15.31:0/1128878483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd0c072850 0x7efd0c1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.081+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 >> 172.21.15.31:0/1128878483 conn(0x7efd0c0cc910 msgr2=0x7efd0c0c66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.081+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 shutdown_connections 2024-09-13T08:45:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.081+0000 7efd10a00640 1 -- 172.21.15.31:0/1128878483 wait complete. 2024-09-13T08:45:19.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:19.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 -- 172.21.15.31:0/2105562757 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15f530 msgr2=0x7fef3c15fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 --2- 172.21.15.31:0/2105562757 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15f530 0x7fef3c15fa70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fef24005fd0 tx=0x7fef2402a4b0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 -- 172.21.15.31:0/2105562757 shutdown_connections 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 --2- 172.21.15.31:0/2105562757 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15f530 0x7fef3c15fa70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 --2- 172.21.15.31:0/2105562757 >> v2:172.21.15.45:3300/0 conn(0x7fef3c15f150 0x7fef3c0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 -- 172.21.15.31:0/2105562757 >> 172.21.15.31:0/2105562757 conn(0x7fef3c0d1c70 msgr2=0x7fef3c0d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 -- 172.21.15.31:0/2105562757 shutdown_connections 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.258+0000 7fef41e00640 1 -- 172.21.15.31:0/2105562757 wait complete. 2024-09-13T08:45:19.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.260+0000 7fef41e00640 1 Processor -- start 2024-09-13T08:45:19.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.260+0000 7fef41e00640 1 -- start start 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef3c15f530 0x7fef3c151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15ffb0 0x7fef3c071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c1607f0 con 0x7fef3c15f150 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c160670 con 0x7fef3c15f530 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fef3c0d0280 con 0x7fef3c15ffb0 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef40a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef40a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60376/0 (socket says 172.21.15.31:60376) 2024-09-13T08:45:19.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef40a00640 1 -- 172.21.15.31:0/3922106031 learned_addr learned my addr 172.21.15.31:0/3922106031 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:19.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef33e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef3c15f530 0x7fef3c151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.261+0000 7fef41400640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15ffb0 0x7fef3c071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 -- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15ffb0 msgr2=0x7fef3c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15ffb0 0x7fef3c071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 -- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef3c15f530 msgr2=0x7fef3c151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef3c15f530 0x7fef3c151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 -- 172.21.15.31:0/3922106031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c0cbd20 con 0x7fef3c15f150 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef40a00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea20 tx=0x7fef2c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00cd80 con 0x7fef3c15f150 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef2c010430 con 0x7fef3c15f150 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c010840 con 0x7fef3c15f150 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.262+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3c0cc000 con 0x7fef3c15f150 2024-09-13T08:45:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.263+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3c0cdc20 con 0x7fef3c15f150 2024-09-13T08:45:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.264+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3c15fa70 con 0x7fef3c15f150 2024-09-13T08:45:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.265+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fef2c0109e0 con 0x7fef3c15f150 2024-09-13T08:45:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.266+0000 7fef39600640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 0x7fef1807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.267+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fef2c09c100 con 0x7fef3c15f150 2024-09-13T08:45:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.267+0000 7fef33e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 0x7fef1807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.267+0000 7fef33e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 0x7fef1807b3d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef24033000 tx=0x7fef24004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:19.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.271+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fef2c063610 con 0x7fef3c15f150 2024-09-13T08:45:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.506+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef3c074940 con 0x7fef3c15f150 2024-09-13T08:45:19.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.507+0000 7fef39600640 1 -- 172.21.15.31:0/3922106031 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5221 (secure 0 0 0) 0x7fef2c068040 con 0x7fef3c15f150 2024-09-13T08:45:19.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:19.508 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"btime":"2024-09-13T08:45:05:455477+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:44:54.747650+0000","modified":"2024-09-13T08:44:57.717583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/1614002559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1614002559},{"type":"v1","addr":"172.21.15.31:6834","nonce":1614002559}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":10}]} 2024-09-13T08:45:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 91 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.509+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 msgr2=0x7fef1807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.509+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 0x7fef1807b3d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef24033000 tx=0x7fef24004730 comp rx=0 tx=0).stop 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 msgr2=0x7fef3c1f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea20 tx=0x7fef2c00eee0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 shutdown_connections 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fef18078f10 0x7fef1807b3d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fef3c15ffb0 0x7fef3c071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef3c15f530 0x7fef3c151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 --2- 172.21.15.31:0/3922106031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef3c15f150 0x7fef3c1f2dc0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 >> 172.21.15.31:0/3922106031 conn(0x7fef3c0d1c70 msgr2=0x7fef3c0d2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 shutdown_connections 2024-09-13T08:45:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.510+0000 7fef41e00640 1 -- 172.21.15.31:0/3922106031 wait complete. 2024-09-13T08:45:19.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.685+0000 7fb53da00640 1 -- 172.21.15.31:0/3164555079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380d08a0 msgr2=0x7fb5380d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.685+0000 7fb53da00640 1 --2- 172.21.15.31:0/3164555079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380d08a0 0x7fb5380d0c80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fb528005fd0 tx=0x7fb52802a530 comp rx=0 tx=0).stop 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 -- 172.21.15.31:0/3164555079 shutdown_connections 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 --2- 172.21.15.31:0/3164555079 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 --2- 172.21.15.31:0/3164555079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380d08a0 0x7fb5380d0c80 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 -- 172.21.15.31:0/3164555079 >> 172.21.15.31:0/3164555079 conn(0x7fb5380d3e90 msgr2=0x7fb5380d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 -- 172.21.15.31:0/3164555079 shutdown_connections 2024-09-13T08:45:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.686+0000 7fb53da00640 1 -- 172.21.15.31:0/3164555079 wait complete. 2024-09-13T08:45:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.687+0000 7fb53da00640 1 Processor -- start 2024-09-13T08:45:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.687+0000 7fb53da00640 1 -- start start 2024-09-13T08:45:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5380d08a0 0x7fb5380ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb5380b4660 con 0x7fb5380cd4b0 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5380b44e0 con 0x7fb5380d08a0 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.688+0000 7fb53da00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb5380b47e0 con 0x7fb5380b2f80 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60380/0 (socket says 172.21.15.31:60380) 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 -- 172.21.15.31:0/3303027206 learned_addr learned my addr 172.21.15.31:0/3303027206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb537e00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb52fe00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5380d08a0 0x7fb5380ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 -- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 msgr2=0x7fb5380cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 -- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5380d08a0 msgr2=0x7fb5380ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5380d08a0 0x7fb5380ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 -- 172.21.15.31:0/3303027206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5381f56f0 con 0x7fb5380cd4b0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb537e00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.689+0000 7fb53d000640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fb53000cce0 tx=0x7fb53000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.690+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000f450 con 0x7fb5380cd4b0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.690+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb53000f700 con 0x7fb5380cd4b0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.690+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000fb50 con 0x7fb5380cd4b0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.690+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5380c8f30 con 0x7fb5380cd4b0 2024-09-13T08:45:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.690+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5380c94f0 con 0x7fb5380cd4b0 2024-09-13T08:45:19.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.691+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5380d28b0 con 0x7fb5380cd4b0 2024-09-13T08:45:19.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.692+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fb53000fcf0 con 0x7fb5380cd4b0 2024-09-13T08:45:19.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.693+0000 7fb534c00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 0x7fb51407b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:19.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.693+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7fb53009c370 con 0x7fb5380cd4b0 2024-09-13T08:45:19.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.693+0000 7fb537e00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 0x7fb51407b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.694+0000 7fb537e00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 0x7fb51407b3d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb528033000 tx=0x7fb528004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:19.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.697+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb530014030 con 0x7fb5380cd4b0 2024-09-13T08:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.891+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5380b9cf0 con 0x7fb5380cd4b0 2024-09-13T08:45:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.893+0000 7fb534c00640 1 -- 172.21.15.31:0/3303027206 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16758 (secure 0 0 0) 0x7fb530063880 con 0x7fb5380cd4b0 2024-09-13T08:45:19.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:19.894 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":139,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:44:56.260431+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:44:52.478415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T08:44:53.653411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 msgr2=0x7fb51407b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 0x7fb51407b3d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb528033000 tx=0x7fb528004730 comp rx=0 tx=0).stop 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 msgr2=0x7fb5381f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fb53000cce0 tx=0x7fb53000a620 comp rx=0 tx=0).stop 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 shutdown_connections 2024-09-13T08:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb514078f10 0x7fb51407b3d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5380cd4b0 0x7fb5381f4970 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5380d08a0 0x7fb5380ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 --2- 172.21.15.31:0/3303027206 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb5380b2f80 0x7fb5380cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 >> 172.21.15.31:0/3303027206 conn(0x7fb5380d3e90 msgr2=0x7fb53815da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 shutdown_connections 2024-09-13T08:45:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:19.895+0000 7fb53da00640 1 -- 172.21.15.31:0/3303027206 wait complete. 2024-09-13T08:45:19.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:45:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 -- 172.21.15.31:0/2483998300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 msgr2=0x7eff80004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 --2- 172.21.15.31:0/2483998300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff80004010 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006fb0 tx=0x7eff80005830 comp rx=0 tx=0).stop 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7eff9b200640 1 -- 172.21.15.31:0/2483998300 reap_dead start 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 -- 172.21.15.31:0/2483998300 shutdown_connections 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 -- 172.21.15.31:0/2483998300 >> 172.21.15.31:0/2483998300 conn(0x7eff9c0b18f0 msgr2=0x7eff9c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 -- 172.21.15.31:0/2483998300 shutdown_connections 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.073+0000 7effa1400640 1 -- 172.21.15.31:0/2483998300 wait complete. 2024-09-13T08:45:20.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.075+0000 7effa1400640 1 Processor -- start 2024-09-13T08:45:20.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.075+0000 7effa1400640 1 -- start start 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 0x7eff9c073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c1f8960 0x7eff9c1f8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c06f710 con 0x7eff80003c10 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c06f590 con 0x7eff9c073720 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7effa1400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c0b40c0 con 0x7eff9c1f8960 2024-09-13T08:45:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7eff9a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7eff9a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60396/0 (socket says 172.21.15.31:60396) 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7eff9a800640 1 -- 172.21.15.31:0/2726837955 learned_addr learned my addr 172.21.15.31:0/2726837955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7eff99e00640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 0x7eff9c073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.076+0000 7eff9b200640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c1f8960 0x7eff9c1f8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 -- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c1f8960 msgr2=0x7eff9c1f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c1f8960 0x7eff9c1f8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 -- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 msgr2=0x7eff9c073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 0x7eff9c073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 -- 172.21.15.31:0/2726837955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c074250 con 0x7eff80003c10 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff99e00640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 0x7eff9c073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff9a800640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02b0a0 tx=0x7eff8c006bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c032030 con 0x7eff80003c10 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff8c003af0 con 0x7eff80003c10 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c006580 con 0x7eff80003c10 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c074530 con 0x7eff80003c10 2024-09-13T08:45:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.077+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c1f97b0 con 0x7eff80003c10 2024-09-13T08:45:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.078+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c0b39b0 con 0x7eff80003c10 2024-09-13T08:45:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.079+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7eff8c035020 con 0x7eff80003c10 2024-09-13T08:45:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.080+0000 7eff92a00640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 0x7eff7c07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.080+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5753+0+0 (secure 0 0 0) 0x7eff8c0bbf20 con 0x7eff80003c10 2024-09-13T08:45:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.081+0000 7eff99e00640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 0x7eff7c07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.081+0000 7eff99e00640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 0x7eff7c07b400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0736b0 tx=0x7eff88005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:20.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.084+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7eff8c047020 con 0x7eff80003c10 2024-09-13T08:45:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.318+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7eff9c160fa0 con 0x7eff80003c10 2024-09-13T08:45:20.320 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:45:20.319+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:45:20.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.321+0000 7eff92a00640 1 -- 172.21.15.31:0/2726837955 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7eff8c0839f0 con 0x7eff80003c10 2024-09-13T08:45:20.322 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:45:20.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 msgr2=0x7eff7c07b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 0x7eff7c07b400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0736b0 tx=0x7eff88005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:20.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 msgr2=0x7eff9c13d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02b0a0 tx=0x7eff8c006bd0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 shutdown_connections 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c078f40 0x7eff7c07b400 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c1f8960 0x7eff9c1f8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff9c073720 0x7eff9c073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 --2- 172.21.15.31:0/2726837955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003c10 0x7eff9c13d890 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 >> 172.21.15.31:0/2726837955 conn(0x7eff9c0b18f0 msgr2=0x7eff9c0cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.324+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 shutdown_connections 2024-09-13T08:45:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.325+0000 7effa1400640 1 -- 172.21.15.31:0/2726837955 wait complete. 2024-09-13T08:45:20.331 INFO:tasks.ceph.mds.c.smithi031.stderr: -2975> 2024-09-13T08:45:00.219+0000 7fd5c1400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:45:20.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:45:20.382 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:45:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 -- 172.21.15.31:0/3817650623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe250073ac0 msgr2=0x7fe2500b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 --2- 172.21.15.31:0/3817650623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe250073ac0 0x7fe2500b2820 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe2500c9fb0 tx=0x7fe23c02a560 comp rx=0 tx=0).stop 2024-09-13T08:45:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 -- 172.21.15.31:0/3817650623 shutdown_connections 2024-09-13T08:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 --2- 172.21.15.31:0/3817650623 >> v2:172.21.15.45:3300/0 conn(0x7fe2500cc3e0 0x7fe2500cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 --2- 172.21.15.31:0/3817650623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe250073ac0 0x7fe2500b2820 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 -- 172.21.15.31:0/3817650623 >> 172.21.15.31:0/3817650623 conn(0x7fe2500b1520 msgr2=0x7fe2500b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 -- 172.21.15.31:0/3817650623 shutdown_connections 2024-09-13T08:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.509+0000 7fe24fe00640 1 -- 172.21.15.31:0/3817650623 wait complete. 2024-09-13T08:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.510+0000 7fe24fe00640 1 Processor -- start 2024-09-13T08:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.511+0000 7fe24fe00640 1 -- start start 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.511+0000 7fe24fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe250073ac0 0x7fe25015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.511+0000 7fe24fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe2500cc3e0 0x7fe2501e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.511+0000 7fe24fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2500cad20 con 0x7fe25015bc00 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2500caba0 con 0x7fe250073ac0 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe2500caea0 con 0x7fe2500cc3e0 2024-09-13T08:45:20.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe250073ac0 0x7fe25015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60414/0 (socket says 172.21.15.31:60414) 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 -- 172.21.15.31:0/2045951612 learned_addr learned my addr 172.21.15.31:0/2045951612 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24e000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe2500cc3e0 0x7fe2501e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 -- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe2500cc3e0 msgr2=0x7fe2501e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe2500cc3e0 0x7fe2501e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 -- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe250073ac0 msgr2=0x7fe25015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe250073ac0 0x7fe25015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.512+0000 7fe24f400640 1 -- 172.21.15.31:0/2045951612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25015d8f0 con 0x7fe25015bc00 2024-09-13T08:45:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe24f400640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe24800eb10 tx=0x7fe24800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248004b20 con 0x7fe25015bc00 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe248004ce0 con 0x7fe25015bc00 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248005280 con 0x7fe25015bc00 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25015dbd0 con 0x7fe25015bc00 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.513+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25015e4b0 con 0x7fe25015bc00 2024-09-13T08:45:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.514+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2500cb430 con 0x7fe25015bc00 2024-09-13T08:45:20.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.515+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fe248005420 con 0x7fe25015bc00 2024-09-13T08:45:20.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.516+0000 7fe246a00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 0x7fe23007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:20.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.516+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5839+0+0 (secure 0 0 0) 0x7fe24809c1d0 con 0x7fe25015bc00 2024-09-13T08:45:20.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.517+0000 7fe24ea00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 0x7fe23007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:20.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.517+0000 7fe24ea00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 0x7fe23007b3d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe240001720 tx=0x7fe240009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.521+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe248063790 con 0x7fe25015bc00 2024-09-13T08:45:20.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:20.757+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe2500c2f80 con 0x7fe25015bc00 2024-09-13T08:45:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.323+0000 7fe246a00640 1 -- 172.21.15.31:0/2045951612 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) ==== 104+0+0 (secure 0 0 0) 0x7fe2480681c0 con 0x7fe25015bc00 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 msgr2=0x7fe23007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 0x7fe23007b3d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe240001720 tx=0x7fe240009040 comp rx=0 tx=0).stop 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 msgr2=0x7fe25015e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe24800eb10 tx=0x7fe24800c2d0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 shutdown_connections 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe230078f10 0x7fe23007b3d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe25015bc00 0x7fe25015e9b0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe2500cc3e0 0x7fe2501e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 --2- 172.21.15.31:0/2045951612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe250073ac0 0x7fe25015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 >> 172.21.15.31:0/2045951612 conn(0x7fe2500b1520 msgr2=0x7fe2500b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 shutdown_connections 2024-09-13T08:45:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.326+0000 7fe24fe00640 1 -- 172.21.15.31:0/2045951612 wait complete. 2024-09-13T08:45:21.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:45:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 -- 172.21.15.31:0/1919576779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbec8004d40 msgr2=0x7fbec80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 --2- 172.21.15.31:0/1919576779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbec8004d40 0x7fbec80051c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009ed0 tx=0x7fbec8005830 comp rx=0 tx=0).stop 2024-09-13T08:45:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbee9800640 1 -- 172.21.15.31:0/1919576779 reap_dead start 2024-09-13T08:45:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 -- 172.21.15.31:0/1919576779 shutdown_connections 2024-09-13T08:45:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 -- 172.21.15.31:0/1919576779 >> 172.21.15.31:0/1919576779 conn(0x7fbee40cfbb0 msgr2=0x7fbee40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 -- 172.21.15.31:0/1919576779 shutdown_connections 2024-09-13T08:45:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.552+0000 7fbeeb800640 1 -- 172.21.15.31:0/1919576779 wait complete. 2024-09-13T08:45:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.554+0000 7fbeeb800640 1 Processor -- start 2024-09-13T08:45:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.554+0000 7fbeeb800640 1 -- start start 2024-09-13T08:45:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec8004d40 0x7fbee41e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:21.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbee40c9390 0x7fbee41f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbee40cd900 con 0x7fbee40b1f60 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbee40cd780 con 0x7fbec8004d40 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbeeb800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbee40cda80 con 0x7fbee40c9390 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbedbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbedbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60438/0 (socket says 172.21.15.31:60438) 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbedbe00640 1 -- 172.21.15.31:0/2663565332 learned_addr learned my addr 172.21.15.31:0/2663565332 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbee8e00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec8004d40 0x7fbee41e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.555+0000 7fbee9800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbee40c9390 0x7fbee41f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 -- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbee40c9390 msgr2=0x7fbee41f7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbee40c9390 0x7fbee41f7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 -- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec8004d40 msgr2=0x7fbee41e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec8004d40 0x7fbee41e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 -- 172.21.15.31:0/2663565332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee41e4bf0 con 0x7fbee40b1f60 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbedbe00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbecc011860 tx=0x7fbecc011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc004460 con 0x7fbee40b1f60 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbecc016070 con 0x7fbee40b1f60 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00fba0 con 0x7fbee40b1f60 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.556+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee4147c50 con 0x7fbee40b1f60 2024-09-13T08:45:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.557+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee4148210 con 0x7fbee40b1f60 2024-09-13T08:45:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.558+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee415e820 con 0x7fbee40b1f60 2024-09-13T08:45:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.559+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fbecc00fd40 con 0x7fbee40b1f60 2024-09-13T08:45:21.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.560+0000 7fbee1600640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 0x7fbec007b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:21.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.560+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5839+0+0 (secure 0 0 0) 0x7fbecc09c200 con 0x7fbee40b1f60 2024-09-13T08:45:21.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.560+0000 7fbee8e00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 0x7fbec007b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:21.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.561+0000 7fbee8e00640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 0x7fbec007b400 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbed4001d40 tx=0x7fbed4006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.564+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbecc0636c0 con 0x7fbee40b1f60 2024-09-13T08:45:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:21.759+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbee415b4f0 con 0x7fbee40b1f60 2024-09-13T08:45:22.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.536+0000 7fbee1600640 1 -- 172.21.15.31:0/2663565332 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v141) ==== 181+0+0 (secure 0 0 0) 0x7fbecc0680f0 con 0x7fbee40b1f60 2024-09-13T08:45:22.537 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:45:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 msgr2=0x7fbec007b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 0x7fbec007b400 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbed4001d40 tx=0x7fbed4006040 comp rx=0 tx=0).stop 2024-09-13T08:45:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 msgr2=0x7fbee40c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbecc011860 tx=0x7fbecc011d20 comp rx=0 tx=0).stop 2024-09-13T08:45:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 shutdown_connections 2024-09-13T08:45:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbec0078f40 0x7fbec007b400 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbee40c9390 0x7fbee41f7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee40b1f60 0x7fbee40c8e50 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 --2- 172.21.15.31:0/2663565332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec8004d40 0x7fbee41e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.540+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 >> 172.21.15.31:0/2663565332 conn(0x7fbee40cfbb0 msgr2=0x7fbee40b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.541+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 shutdown_connections 2024-09-13T08:45:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.541+0000 7fbeeb800640 1 -- 172.21.15.31:0/2663565332 wait complete. 2024-09-13T08:45:22.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 -- 172.21.15.31:0/2147389420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc0cde90 msgr2=0x7f0ccc0ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 --2- 172.21.15.31:0/2147389420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc0cde90 0x7f0ccc0ca370 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4005fd0 tx=0x7f0cb402a540 comp rx=0 tx=0).stop 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 -- 172.21.15.31:0/2147389420 shutdown_connections 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 --2- 172.21.15.31:0/2147389420 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cc190 0x7f0ccc0ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 --2- 172.21.15.31:0/2147389420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc0cde90 0x7f0ccc0ca370 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 -- 172.21.15.31:0/2147389420 >> 172.21.15.31:0/2147389420 conn(0x7f0ccc0cfbb0 msgr2=0x7f0ccc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 -- 172.21.15.31:0/2147389420 shutdown_connections 2024-09-13T08:45:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.732+0000 7f0cd1400640 1 -- 172.21.15.31:0/2147389420 wait complete. 2024-09-13T08:45:22.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.734+0000 7f0cd1400640 1 Processor -- start 2024-09-13T08:45:22.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.734+0000 7f0cd1400640 1 -- start start 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ccc0cc190 0x7f0ccc0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cde90 0x7f0ccc0af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc0cbfd0 con 0x7f0ccc161480 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc0cbe50 con 0x7f0ccc0cc190 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cd1400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc0cadf0 con 0x7f0ccc0cde90 2024-09-13T08:45:22.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0ccb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0ccb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60462/0 (socket says 172.21.15.31:60462) 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0ccb200640 1 -- 172.21.15.31:0/45638982 learned_addr learned my addr 172.21.15.31:0/45638982 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cc1e00640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cde90 0x7f0ccc0af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.735+0000 7f0cca800640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ccc0cc190 0x7f0ccc0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 -- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cde90 msgr2=0x7f0ccc0af5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cde90 0x7f0ccc0af5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 -- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ccc0cc190 msgr2=0x7f0ccc0b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ccc0cc190 0x7f0ccc0b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 -- 172.21.15.31:0/45638982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc0b0270 con 0x7f0ccc161480 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0ccb200640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400cce0 tx=0x7f0cc400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:22.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400f450 con 0x7f0ccc161480 2024-09-13T08:45:22.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cc400f700 con 0x7f0ccc161480 2024-09-13T08:45:22.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc0b0550 con 0x7f0ccc161480 2024-09-13T08:45:22.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400fb50 con 0x7f0ccc161480 2024-09-13T08:45:22.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.736+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc0b0b10 con 0x7f0ccc161480 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.737+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ccc0cb2a0 con 0x7f0ccc161480 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.739+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f0cc400fcf0 con 0x7f0ccc161480 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.739+0000 7f0cc3400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 0x7f0ca807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.740+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5401+0+0 (secure 0 0 0) 0x7f0cc409c0d0 con 0x7f0ccc161480 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.740+0000 7f0cca800640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 0x7f0ca807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:22.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.740+0000 7f0cca800640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 0x7f0ca807b3d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc000f60 tx=0x7f0cbc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:22.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.744+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0cc4014030 con 0x7f0ccc161480 2024-09-13T08:45:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:22.940+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0ccc0b5b30 con 0x7f0ccc161480 2024-09-13T08:45:23.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.542+0000 7f0cc3400640 1 -- 172.21.15.31:0/45638982 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v142) ==== 169+0+0 (secure 0 0 0) 0x7f0cc4063660 con 0x7f0ccc161480 2024-09-13T08:45:23.543 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:45:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.546+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 msgr2=0x7f0ca807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.546+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 0x7f0ca807b3d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc000f60 tx=0x7f0cbc009040 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.546+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 msgr2=0x7f0ccc0afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.546+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400cce0 tx=0x7f0cc400a620 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 shutdown_connections 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ca8078f10 0x7f0ca807b3d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ccc161480 0x7f0ccc0afb30 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0ccc0cde90 0x7f0ccc0af5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 --2- 172.21.15.31:0/45638982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ccc0cc190 0x7f0ccc0b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 >> 172.21.15.31:0/45638982 conn(0x7f0ccc0cfbb0 msgr2=0x7f0ccc0ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 shutdown_connections 2024-09-13T08:45:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.547+0000 7f0cd1400640 1 -- 172.21.15.31:0/45638982 wait complete. 2024-09-13T08:45:23.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.730+0000 7f507de00640 1 -- 172.21.15.31:0/209130918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5060004800 msgr2=0x7f5060004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.730+0000 7f507de00640 1 --2- 172.21.15.31:0/209130918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5060004800 0x7f5060004c80 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f5064001690 tx=0x7f5060005830 comp rx=0 tx=0).stop 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.730+0000 7f507d400640 1 -- 172.21.15.31:0/209130918 reap_dead start 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.730+0000 7f507de00640 1 -- 172.21.15.31:0/209130918 shutdown_connections 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.730+0000 7f507de00640 1 -- 172.21.15.31:0/209130918 >> 172.21.15.31:0/209130918 conn(0x7f50780d3f50 msgr2=0x7f50780d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.731+0000 7f507de00640 1 -- 172.21.15.31:0/209130918 shutdown_connections 2024-09-13T08:45:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.731+0000 7f507de00640 1 -- 172.21.15.31:0/209130918 wait complete. 2024-09-13T08:45:23.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.732+0000 7f507de00640 1 Processor -- start 2024-09-13T08:45:23.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.732+0000 7f507de00640 1 -- start start 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5060004800 0x7f50781579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50781f8370 0x7f5078154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50780d0b20 con 0x7f50780b0c90 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50780d09a0 con 0x7f5060004800 2024-09-13T08:45:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.733+0000 7f507de00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f50780d1110 con 0x7f50781f8370 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60484/0 (socket says 172.21.15.31:60484) 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 -- 172.21.15.31:0/1667509698 learned_addr learned my addr 172.21.15.31:0/1667509698 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f5077e00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50781f8370 0x7f5078154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507ca00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5060004800 0x7f50781579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 -- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50781f8370 msgr2=0x7f5078154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50781f8370 0x7f5078154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 -- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5060004800 msgr2=0x7f50781579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5060004800 0x7f50781579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 -- 172.21.15.31:0/1667509698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50781f8870 con 0x7f50780b0c90 2024-09-13T08:45:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.734+0000 7f507d400640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f50780a1e50 tx=0x7f5070006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.735+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507000fa80 con 0x7f50780b0c90 2024-09-13T08:45:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.735+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5070019070 con 0x7f50780b0c90 2024-09-13T08:45:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.735+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070011430 con 0x7f50780b0c90 2024-09-13T08:45:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.735+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50781f8b50 con 0x7f50780b0c90 2024-09-13T08:45:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.735+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50781f9110 con 0x7f50780b0c90 2024-09-13T08:45:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.736+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50780cea60 con 0x7f50780b0c90 2024-09-13T08:45:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.737+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f50700115d0 con 0x7f50780b0c90 2024-09-13T08:45:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.738+0000 7f5074c00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 0x7f505807b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.738+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f507009b3a0 con 0x7f50780b0c90 2024-09-13T08:45:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.738+0000 7f507ca00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 0x7f505807b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.739+0000 7f507ca00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 0x7f505807b400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f50780c75a0 tx=0x7f506c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:23.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.742+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f507815e540 con 0x7f50780b0c90 2024-09-13T08:45:23.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.951+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f50780ce2f0 con 0x7f50780b0c90 2024-09-13T08:45:23.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.955+0000 7f5074c00640 1 -- 172.21.15.31:0/1667509698 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7f5070063440 con 0x7f50780b0c90 2024-09-13T08:45:23.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 msgr2=0x7f505807b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 0x7f505807b400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f50780c75a0 tx=0x7f506c005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:23.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 msgr2=0x7f50780b1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f50780a1e50 tx=0x7f5070006720 comp rx=0 tx=0).stop 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 shutdown_connections 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5058078f40 0x7f505807b400 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50780b0c90 0x7f50780b1140 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f50781f8370 0x7f5078154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 --2- 172.21.15.31:0/1667509698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5060004800 0x7f50781579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 >> 172.21.15.31:0/1667509698 conn(0x7f50780d3f50 msgr2=0x7f50780c99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 shutdown_connections 2024-09-13T08:45:23.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:23.958+0000 7f507de00640 1 -- 172.21.15.31:0/1667509698 wait complete. 2024-09-13T08:45:23.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-13T08:45:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1651851442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 msgr2=0x7f5ba8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1651851442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5ba8004800 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009b20 tx=0x7f5ba8005830 comp rx=0 tx=0).stop 2024-09-13T08:45:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc3e00640 1 -- 172.21.15.31:0/1651851442 reap_dead start 2024-09-13T08:45:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1651851442 shutdown_connections 2024-09-13T08:45:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1651851442 >> 172.21.15.31:0/1651851442 conn(0x7f5bc409d190 msgr2=0x7f5bc40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1651851442 shutdown_connections 2024-09-13T08:45:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.138+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1651851442 wait complete. 2024-09-13T08:45:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.140+0000 7f5bc8e00640 1 Processor -- start 2024-09-13T08:45:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.140+0000 7f5bc8e00640 1 -- start start 2024-09-13T08:45:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc40cde50 0x7f5bc40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5bc40cd1d0 0x7f5bc40cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40d1cb0 con 0x7f5ba8003fd0 2024-09-13T08:45:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40d1b30 con 0x7f5bc40cde50 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc8e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc40d1e30 con 0x7f5bc40cd1d0 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60498/0 (socket says 172.21.15.31:60498) 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc3400640 1 -- 172.21.15.31:0/1539016514 learned_addr learned my addr 172.21.15.31:0/1539016514 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc2a00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc40cde50 0x7f5bc40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.141+0000 7f5bc3e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5bc40cd1d0 0x7f5bc40cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 -- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5bc40cd1d0 msgr2=0x7f5bc40cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5bc40cd1d0 0x7f5bc40cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 -- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc40cde50 msgr2=0x7f5bc40ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc40cde50 0x7f5bc40ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 -- 172.21.15.31:0/1539016514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc40c7ce0 con 0x7f5ba8003fd0 2024-09-13T08:45:24.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc3400640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011ae0 tx=0x7f5bb4011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb400fc40 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bb400fde0 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4004280 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc40c7fc0 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.142+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc40c8720 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.143+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc40d2b90 con 0x7f5ba8003fd0 2024-09-13T08:45:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.145+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f5bb4020020 con 0x7f5ba8003fd0 2024-09-13T08:45:24.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.146+0000 7f5bbb400640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 0x7f5ba407b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.146+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f5bb409bba0 con 0x7f5ba8003fd0 2024-09-13T08:45:24.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.146+0000 7f5bc2a00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 0x7f5ba407b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.147+0000 7f5bc2a00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 0x7f5ba407b3d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0035000 tx=0x7f5bb002d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:24.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.151+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5bb409de20 con 0x7f5ba8003fd0 2024-09-13T08:45:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.421+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7f5bc40d0d30 con 0x7f5ba8003fd0 2024-09-13T08:45:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.544+0000 7f5bbb400640 1 -- 172.21.15.31:0/1539016514 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v453) ==== 151+0+0 (secure 0 0 0) 0x7f5bb4063970 con 0x7f5ba8003fd0 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 msgr2=0x7f5ba407b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 0x7f5ba407b3d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0035000 tx=0x7f5bb002d040 comp rx=0 tx=0).stop 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 msgr2=0x7f5bc41f5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011ae0 tx=0x7f5bb4011ee0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 shutdown_connections 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5ba4078f10 0x7f5ba407b3d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5bc40cd1d0 0x7f5bc40cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc40cde50 0x7f5bc40ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 --2- 172.21.15.31:0/1539016514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ba8003fd0 0x7f5bc41f5780 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 >> 172.21.15.31:0/1539016514 conn(0x7f5bc409d190 msgr2=0x7f5bc40b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 shutdown_connections 2024-09-13T08:45:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.547+0000 7f5bc8e00640 1 -- 172.21.15.31:0/1539016514 wait complete. 2024-09-13T08:45:24.564 INFO:tasks.cephfs_test_runner:test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-13T08:45:24.564 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- rather than going through a full upgrade cycle, 2024-09-13T08:45:24.564 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:45:24.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.725+0000 7f2995000640 1 -- 172.21.15.31:0/525028962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2974003fd0 msgr2=0x7f2974004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.725+0000 7f2995000640 1 --2- 172.21.15.31:0/525028962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2974003fd0 0x7f2974004800 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f297c009b50 tx=0x7f2974005830 comp rx=0 tx=0).stop 2024-09-13T08:45:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.726+0000 7f298e800640 1 -- 172.21.15.31:0/525028962 reap_dead start 2024-09-13T08:45:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.726+0000 7f2995000640 1 -- 172.21.15.31:0/525028962 shutdown_connections 2024-09-13T08:45:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.726+0000 7f2995000640 1 -- 172.21.15.31:0/525028962 >> 172.21.15.31:0/525028962 conn(0x7f2990074260 msgr2=0x7f2990074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.726+0000 7f2995000640 1 -- 172.21.15.31:0/525028962 shutdown_connections 2024-09-13T08:45:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.726+0000 7f2995000640 1 -- 172.21.15.31:0/525028962 wait complete. 2024-09-13T08:45:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.727+0000 7f2995000640 1 Processor -- start 2024-09-13T08:45:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.727+0000 7f2995000640 1 -- start start 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2974003fd0 0x7f2990156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 0x7f29900b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29900b1880 con 0x7f29900b3de0 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29900b1700 con 0x7f2990159a50 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.728+0000 7f2995000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f29900b1a00 con 0x7f2974003fd0 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60506/0 (socket says 172.21.15.31:60506) 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 -- 172.21.15.31:0/3087395656 learned_addr learned my addr 172.21.15.31:0/3087395656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298d400640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 0x7f29900b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298de00640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2974003fd0 0x7f2990156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 -- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2974003fd0 msgr2=0x7f2990156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2974003fd0 0x7f2990156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 -- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 msgr2=0x7f29900b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 0x7f29900b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 -- 172.21.15.31:0/3087395656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29900b4280 con 0x7f29900b3de0 2024-09-13T08:45:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298d400640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 0x7f29900b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.729+0000 7f298e800640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f29900b1080 tx=0x7f2988011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.730+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298800fd40 con 0x7f29900b3de0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.730+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2988004b20 con 0x7f29900b3de0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.730+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988003970 con 0x7f29900b3de0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.730+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2990158f70 con 0x7f29900b3de0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.730+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2990159530 con 0x7f29900b3de0 2024-09-13T08:45:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.731+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2990159700 con 0x7f29900b3de0 2024-09-13T08:45:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.732+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f2988020020 con 0x7f29900b3de0 2024-09-13T08:45:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.733+0000 7f2986000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 0x7f297007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.733+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f298809bb40 con 0x7f29900b3de0 2024-09-13T08:45:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.733+0000 7f298de00640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 0x7f297007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.734+0000 7f298de00640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 0x7f297007b3d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2980006b90 tx=0x7f29800067a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:24.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.737+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f29880639a0 con 0x7f29900b3de0 2024-09-13T08:45:24.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.939+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29900c7420 con 0x7f29900b3de0 2024-09-13T08:45:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.940+0000 7f2986000640 1 -- 172.21.15.31:0/3087395656 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+13061 (secure 0 0 0) 0x7f29880632a0 con 0x7f29900b3de0 2024-09-13T08:45:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:24.979 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:23.537388+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6834/1614002559":"2024-09-14T08:45:20.319784+0000","172.21.15.31:6832/1614002559":"2024-09-14T08:45:20.319784+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.942+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 msgr2=0x7f297007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.942+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 0x7f297007b3d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2980006b90 tx=0x7f29800067a0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.942+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 msgr2=0x7f29900d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.942+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f29900b1080 tx=0x7f2988011f90 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 shutdown_connections 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2970078f10 0x7f297007b3d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29900b3de0 0x7f29900d0a80 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2990159a50 0x7f29900b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 --2- 172.21.15.31:0/3087395656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2974003fd0 0x7f2990156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 >> 172.21.15.31:0/3087395656 conn(0x7f2990074260 msgr2=0x7f299015b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 shutdown_connections 2024-09-13T08:45:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:24.943+0000 7f2995000640 1 -- 172.21.15.31:0/3087395656 wait complete. 2024-09-13T08:45:24.982 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:45:24.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 -- 172.21.15.31:0/520001243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec161480 msgr2=0x7f59ec0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 --2- 172.21.15.31:0/520001243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec161480 0x7f59ec0cd780 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0cfb50 tx=0x7f59dc02a4d0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 -- 172.21.15.31:0/520001243 shutdown_connections 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 --2- 172.21.15.31:0/520001243 >> v2:172.21.15.45:3300/0 conn(0x7f59ec0cb380 0x7f59ec0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 --2- 172.21.15.31:0/520001243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec161480 0x7f59ec0cd780 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.145+0000 7f59f1800640 1 -- 172.21.15.31:0/520001243 >> 172.21.15.31:0/520001243 conn(0x7f59ec0cfbb0 msgr2=0x7f59ec0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.146+0000 7f59f1800640 1 -- 172.21.15.31:0/520001243 shutdown_connections 2024-09-13T08:45:25.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.146+0000 7f59f1800640 1 -- 172.21.15.31:0/520001243 wait complete. 2024-09-13T08:45:25.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.147+0000 7f59f1800640 1 Processor -- start 2024-09-13T08:45:25.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.147+0000 7f59f1800640 1 -- start start 2024-09-13T08:45:25.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 0x7f59ec154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f59ec161480 0x7f59ec0b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec0cbdc0 con 0x7f59ec0cb380 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec0cbc40 con 0x7f59ec0cc190 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59f1800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f59ec0cbf40 con 0x7f59ec161480 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59eaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59eaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47510/0 (socket says 172.21.15.31:47510) 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59eaa00640 1 -- 172.21.15.31:0/4157450141 learned_addr learned my addr 172.21.15.31:0/4157450141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59eb400640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f59ec161480 0x7f59ec0b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.148+0000 7f59e2000640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 0x7f59ec154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 -- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f59ec161480 msgr2=0x7f59ec0b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f59ec161480 0x7f59ec0b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 -- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 msgr2=0x7f59ec154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 0x7f59ec154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 -- 172.21.15.31:0/4157450141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec0b3290 con 0x7f59ec0cb380 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59e2000640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 0x7f59ec154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59eaa00640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f59dc034040 tx=0x7f59dc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc03e030 con 0x7f59ec0cb380 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59dc0047a0 con 0x7f59ec0cb380 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec0b3570 con 0x7f59ec0cb380 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.149+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc035040 con 0x7f59ec0cb380 2024-09-13T08:45:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.150+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ec1554b0 con 0x7f59ec0cb380 2024-09-13T08:45:25.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.151+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ec0c3c40 con 0x7f59ec0cb380 2024-09-13T08:45:25.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.152+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f59dc041030 con 0x7f59ec0cb380 2024-09-13T08:45:25.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.153+0000 7f59e3400640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 0x7f59c807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.153+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f59dc0bba80 con 0x7f59ec0cb380 2024-09-13T08:45:25.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.153+0000 7f59e2000640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 0x7f59c807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.154+0000 7f59e2000640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 0x7f59c807b3d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f59ec1f7c70 tx=0x7f59d4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.157+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f59dc083200 con 0x7f59ec0cb380 2024-09-13T08:45:25.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.354+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59ec0b5b30 con 0x7f59ec0cb380 2024-09-13T08:45:25.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.355+0000 7f59e3400640 1 -- 172.21.15.31:0/4157450141 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v142) ==== 93+0+10 (secure 0 0 0) 0x7f59dc087c30 con 0x7f59ec0cb380 2024-09-13T08:45:25.356 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:45:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 msgr2=0x7f59c807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 0x7f59c807b3d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f59ec1f7c70 tx=0x7f59d4009040 comp rx=0 tx=0).stop 2024-09-13T08:45:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 msgr2=0x7f59ec1f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f59dc034040 tx=0x7f59dc0039e0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 shutdown_connections 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f59c8078f10 0x7f59c807b3d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f59ec161480 0x7f59ec0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec0cc190 0x7f59ec154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 --2- 172.21.15.31:0/4157450141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59ec0cb380 0x7f59ec1f4720 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 >> 172.21.15.31:0/4157450141 conn(0x7f59ec0cfbb0 msgr2=0x7f59ec0ad890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 shutdown_connections 2024-09-13T08:45:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.357+0000 7f59f1800640 1 -- 172.21.15.31:0/4157450141 wait complete. 2024-09-13T08:45:25.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-13T08:45:25.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 -- 172.21.15.31:0/2416878325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c0cbc50 msgr2=0x7ff63c0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 --2- 172.21.15.31:0/2416878325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c0cbc50 0x7ff63c0cc060 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0cfe60 tx=0x7ff63402aa90 comp rx=0 tx=0).stop 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 -- 172.21.15.31:0/2416878325 shutdown_connections 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 --2- 172.21.15.31:0/2416878325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c0cbc50 0x7ff63c0cc060 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 -- 172.21.15.31:0/2416878325 >> 172.21.15.31:0/2416878325 conn(0x7ff63c0cc910 msgr2=0x7ff63c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 -- 172.21.15.31:0/2416878325 shutdown_connections 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.534+0000 7ff642c00640 1 -- 172.21.15.31:0/2416878325 wait complete. 2024-09-13T08:45:25.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.536+0000 7ff642c00640 1 Processor -- start 2024-09-13T08:45:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.536+0000 7ff642c00640 1 -- start start 2024-09-13T08:45:25.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff63c0cbc50 0x7ff63c15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff63c0cf850 0x7ff63c0b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1600f0 con 0x7ff63c15bd70 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c0b19c0 con 0x7ff63c0cf850 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff642c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff63c09c650 con 0x7ff63c0cbc50 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff640c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff640c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47524/0 (socket says 172.21.15.31:47524) 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff640c00640 1 -- 172.21.15.31:0/2482831876 learned_addr learned my addr 172.21.15.31:0/2482831876 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff63be00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff63c0cbc50 0x7ff63c15a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.537+0000 7ff63b400640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff63c0cf850 0x7ff63c0b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 -- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff63c0cbc50 msgr2=0x7ff63c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff63c0cbc50 0x7ff63c15a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 -- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff63c0cf850 msgr2=0x7ff63c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff63c0cf850 0x7ff63c0b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 -- 172.21.15.31:0/2482831876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c09ba60 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff640c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7ff634009d60 tx=0x7ff634003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634046070 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff634004250 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6340046a0 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63c09bd20 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.538+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63c09c2e0 con 0x7ff63c15bd70 2024-09-13T08:45:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.539+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63c0cb870 con 0x7ff63c15bd70 2024-09-13T08:45:25.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.541+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7ff634049070 con 0x7ff63c15bd70 2024-09-13T08:45:25.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.541+0000 7ff633e00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 0x7ff61c07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:25.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.542+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7ff6340bc2e0 con 0x7ff63c15bd70 2024-09-13T08:45:25.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.542+0000 7ff63be00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 0x7ff61c07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:25.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.542+0000 7ff63be00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 0x7ff61c07b400 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c009060 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:25.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.546+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff634032030 con 0x7ff63c15bd70 2024-09-13T08:45:25.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:25.814+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7ff63c09c7d0 con 0x7ff63c15bd70 2024-09-13T08:45:26.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.553+0000 7ff633e00640 1 -- 172.21.15.31:0/2482831876 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v455) ==== 152+0+0 (secure 0 0 0) 0x7ff63c0cc5a0 con 0x7ff63c15bd70 2024-09-13T08:45:26.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 msgr2=0x7ff61c07b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 0x7ff61c07b400 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c009060 comp rx=0 tx=0).stop 2024-09-13T08:45:26.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 msgr2=0x7ff63c0b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7ff634009d60 tx=0x7ff634003f10 comp rx=0 tx=0).stop 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 shutdown_connections 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff61c078f40 0x7ff61c07b400 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff63c15bd70 0x7ff63c0b0fb0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff63c0cf850 0x7ff63c0b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 --2- 172.21.15.31:0/2482831876 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff63c0cbc50 0x7ff63c15a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 >> 172.21.15.31:0/2482831876 conn(0x7ff63c0cc910 msgr2=0x7ff63c15ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 shutdown_connections 2024-09-13T08:45:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.556+0000 7ff642c00640 1 -- 172.21.15.31:0/2482831876 wait complete. 2024-09-13T08:45:26.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:45:26.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 -- 172.21.15.31:0/747117563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 msgr2=0x7f132c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 --2- 172.21.15.31:0/747117563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c0cf850 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f131c005fd0 tx=0x7f131c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 -- 172.21.15.31:0/747117563 shutdown_connections 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 --2- 172.21.15.31:0/747117563 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 0x7f132c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 --2- 172.21.15.31:0/747117563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c0cf850 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 -- 172.21.15.31:0/747117563 >> 172.21.15.31:0/747117563 conn(0x7f132c0cc910 msgr2=0x7f132c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 -- 172.21.15.31:0/747117563 shutdown_connections 2024-09-13T08:45:26.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.740+0000 7f1331800640 1 -- 172.21.15.31:0/747117563 wait complete. 2024-09-13T08:45:26.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.742+0000 7f1331800640 1 Processor -- start 2024-09-13T08:45:26.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.742+0000 7f1331800640 1 -- start start 2024-09-13T08:45:26.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:26.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:26.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 0x7f132c0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:26.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0cec50 con 0x7f132c15f9e0 2024-09-13T08:45:26.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0cead0 con 0x7f132c15bd70 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1331800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f132c0cb850 con 0x7f132c15fe20 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1322000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1322000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47530/0 (socket says 172.21.15.31:47530) 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f1322000640 1 -- 172.21.15.31:0/327658154 learned_addr learned my addr 172.21.15.31:0/327658154 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.743+0000 7f132aa00640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:26.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f132b400640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 0x7f132c0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 -- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 msgr2=0x7f132c0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 0x7f132c0c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 -- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 msgr2=0x7f132c15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c15ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 -- 172.21.15.31:0/327658154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0b3f70 con 0x7f132c15f9e0 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1322000640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f131400c450 tx=0x7f131400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.744+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13140052d0 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.745+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1314005470 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.745+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c0b5060 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.745+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1314012870 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.745+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c0c48d0 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.746+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c0cf850 con 0x7f132c15f9e0 2024-09-13T08:45:26.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.747+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f13140035f0 con 0x7f132c15f9e0 2024-09-13T08:45:26.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.748+0000 7f1323400640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 0x7f130807b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:26.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.748+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f131409acb0 con 0x7f132c15f9e0 2024-09-13T08:45:26.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.748+0000 7f132aa00640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 0x7f130807b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:26.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.749+0000 7f132aa00640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 0x7f130807b400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f131c033000 tx=0x7f131c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:26.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.752+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f13140625c0 con 0x7f132c15f9e0 2024-09-13T08:45:26.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.949+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f132c0ce610 con 0x7f132c15f9e0 2024-09-13T08:45:26.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.949+0000 7f1323400640 1 -- 172.21.15.31:0/327658154 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+1998 (secure 0 0 0) 0x7f1314066ff0 con 0x7f132c15f9e0 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:45:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:45:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:45:26.953 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:45:26.953 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 msgr2=0x7f130807b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 0x7f130807b400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f131c033000 tx=0x7f131c004730 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 msgr2=0x7f132c0c4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f131400c450 tx=0x7f131400c910 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 shutdown_connections 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1308078f40 0x7f130807b400 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f132c15fe20 0x7f132c0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f132c15f9e0 0x7f132c0c4390 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 --2- 172.21.15.31:0/327658154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132c15bd70 0x7f132c15ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 >> 172.21.15.31:0/327658154 conn(0x7f132c0cc910 msgr2=0x7f132c0ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 shutdown_connections 2024-09-13T08:45:26.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:26.952+0000 7f1331800640 1 -- 172.21.15.31:0/327658154 wait complete. 2024-09-13T08:45:26.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:45:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 -- 172.21.15.31:0/356000104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cc190 msgr2=0x7fb6e40ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 --2- 172.21.15.31:0/356000104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cc190 0x7fb6e40ca8b0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40cfb50 tx=0x7fb6dc02aa30 comp rx=0 tx=0).stop 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 -- 172.21.15.31:0/356000104 shutdown_connections 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 --2- 172.21.15.31:0/356000104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e40cc190 0x7fb6e40ca8b0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 --2- 172.21.15.31:0/356000104 >> v2:172.21.15.45:3300/0 conn(0x7fb6e40cde90 0x7fb6e40ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 -- 172.21.15.31:0/356000104 >> 172.21.15.31:0/356000104 conn(0x7fb6e40cfbb0 msgr2=0x7fb6e40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 -- 172.21.15.31:0/356000104 shutdown_connections 2024-09-13T08:45:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.133+0000 7fb6ea400640 1 -- 172.21.15.31:0/356000104 wait complete. 2024-09-13T08:45:27.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.135+0000 7fb6ea400640 1 Processor -- start 2024-09-13T08:45:27.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.135+0000 7fb6ea400640 1 -- start start 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cc190 0x7fb6e414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cde90 0x7fb6e40c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40cbfd0 con 0x7fb6e4161480 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40cbe50 con 0x7fb6e40cde90 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6ea400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e40cadf0 con 0x7fb6e40cc190 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47544/0 (socket says 172.21.15.31:47544) 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6e3e00640 1 -- 172.21.15.31:0/3807863671 learned_addr learned my addr 172.21.15.31:0/3807863671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6e3400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cc190 0x7fb6e414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.136+0000 7fb6daa00640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cde90 0x7fb6e40c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 -- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cc190 msgr2=0x7fb6e414d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cc190 0x7fb6e414d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 -- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cde90 msgr2=0x7fb6e40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cde90 0x7fb6e40c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 -- 172.21.15.31:0/3807863671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e414d560 con 0x7fb6e4161480 2024-09-13T08:45:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6e3e00640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc005fd0 tx=0x7fb6dc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc046070 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6dc02f690 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc033050 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.137+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e40c52f0 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.138+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e40c58b0 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.138+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e40cb2a0 con 0x7fb6e4161480 2024-09-13T08:45:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.140+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fb6dc049070 con 0x7fb6e4161480 2024-09-13T08:45:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.141+0000 7fb6dbe00640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 0x7fb6c007b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.141+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7fb6dc0bc190 con 0x7fb6e4161480 2024-09-13T08:45:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.141+0000 7fb6e3400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 0x7fb6c007b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.142+0000 7fb6e3400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 0x7fb6c007b400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4001720 tx=0x7fb6d4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.145+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb6e414d560 con 0x7fb6e4161480 2024-09-13T08:45:27.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.338+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb6e4074820 con 0x7fb6e4161480 2024-09-13T08:45:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.342+0000 7fb6dbe00640 1 -- 172.21.15.31:0/3807863671 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7fb6dc0839a0 con 0x7fb6e4161480 2024-09-13T08:45:27.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 msgr2=0x7fb6c007b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 0x7fb6c007b400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4001720 tx=0x7fb6d4009040 comp rx=0 tx=0).stop 2024-09-13T08:45:27.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 msgr2=0x7fb6e40c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc005fd0 tx=0x7fb6dc003f10 comp rx=0 tx=0).stop 2024-09-13T08:45:27.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 shutdown_connections 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb6c0078f40 0x7fb6c007b400 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6e4161480 0x7fb6e40c4c70 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6e40cde90 0x7fb6e40c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 --2- 172.21.15.31:0/3807863671 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb6e40cc190 0x7fb6e414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 >> 172.21.15.31:0/3807863671 conn(0x7fb6e40cfbb0 msgr2=0x7fb6e40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 shutdown_connections 2024-09-13T08:45:27.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.345+0000 7fb6ea400640 1 -- 172.21.15.31:0/3807863671 wait complete. 2024-09-13T08:45:27.363 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:45:27.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:27.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:27.558 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:45:27.558 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:27.558 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:27.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:45:27.595 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:27.595 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:27.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:45:27.632 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:27.632 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:27.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:45:27.672 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:27.672 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:27.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:45:27.709 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:45:27.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 -- 172.21.15.31:0/1379527612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7464003a50 msgr2=0x7f7464003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 --2- 172.21.15.31:0/1379527612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7464003a50 0x7f7464003e70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f747000c610 tx=0x7f7464005830 comp rx=0 tx=0).stop 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 -- 172.21.15.31:0/1379527612 shutdown_connections 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 --2- 172.21.15.31:0/1379527612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7464003a50 0x7f7464003e70 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 -- 172.21.15.31:0/1379527612 >> 172.21.15.31:0/1379527612 conn(0x7f74800cc910 msgr2=0x7f74800cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 -- 172.21.15.31:0/1379527612 shutdown_connections 2024-09-13T08:45:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.872+0000 7f7484e00640 1 -- 172.21.15.31:0/1379527612 wait complete. 2024-09-13T08:45:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.874+0000 7f7484e00640 1 Processor -- start 2024-09-13T08:45:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.874+0000 7f7484e00640 1 -- start start 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7464003a50 0x7f748009d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7480073cb0 0x7f7480074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74800b1d60 con 0x7f74800b4ce0 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f74800cc620 con 0x7f7480073cb0 2024-09-13T08:45:27.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7484e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f74800b1ee0 con 0x7f7464003a50 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f747e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f747e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47562/0 (socket says 172.21.15.31:47562) 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f747e800640 1 -- 172.21.15.31:0/496056147 learned_addr learned my addr 172.21.15.31:0/496056147 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f7475400640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7480073cb0 0x7f7480074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.875+0000 7f747de00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7464003a50 0x7f748009d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 -- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7464003a50 msgr2=0x7f748009d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7464003a50 0x7f748009d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 -- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7480073cb0 msgr2=0x7f7480074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7480073cb0 0x7f7480074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 -- 172.21.15.31:0/496056147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74801f98d0 con 0x7f74800b4ce0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f747e800640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f74800af880 tx=0x7f74780019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478011870 con 0x7f74800b4ce0 2024-09-13T08:45:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7478017070 con 0x7f74800b4ce0 2024-09-13T08:45:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478011da0 con 0x7f74800b4ce0 2024-09-13T08:45:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74801f9b90 con 0x7f74800b4ce0 2024-09-13T08:45:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.876+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74801fa150 con 0x7f74800b4ce0 2024-09-13T08:45:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.877+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74800ce600 con 0x7f74800b4ce0 2024-09-13T08:45:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.879+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f747801c460 con 0x7f74800b4ce0 2024-09-13T08:45:27.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.880+0000 7f7476a00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 0x7f745c07b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:27.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.880+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f747809b8c0 con 0x7f74800b4ce0 2024-09-13T08:45:27.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.880+0000 7f747de00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 0x7f745c07b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:27.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.881+0000 7f747de00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 0x7f745c07b400 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7470033000 tx=0x7f7470001bc0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:27.884+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7478063820 con 0x7f74800b4ce0 2024-09-13T08:45:28.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.116+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74800b16f0 con 0x7f74800b4ce0 2024-09-13T08:45:28.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.117+0000 7f7476a00640 1 -- 172.21.15.31:0/496056147 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+3902 (secure 0 0 0) 0x7f7478063120 con 0x7f74800b4ce0 2024-09-13T08:45:28.118 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:28.118 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"btime":"2024-09-13T08:45:21:321907+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/567256446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":567256446},{"type":"v1","addr":"172.21.15.31:6835","nonce":567256446}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4218886524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4218886524},{"type":"v1","addr":"172.21.15.45:6801","nonce":4218886524}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":7525,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/215234136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":215234136},{"type":"v1","addr":"172.21.15.45:6835","nonce":215234136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/324483072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":324483072},{"type":"v1","addr":"172.21.15.31:6834","nonce":324483072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[]} 2024-09-13T08:45:28.118 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2024-09-13T08:45:28.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 msgr2=0x7f745c07b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 0x7f745c07b400 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7470033000 tx=0x7f7470001bc0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 msgr2=0x7f74800b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f74800af880 tx=0x7f74780019b0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 shutdown_connections 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f745c078f40 0x7f745c07b400 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74800b4ce0 0x7f74800b5140 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7480073cb0 0x7f7480074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 --2- 172.21.15.31:0/496056147 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7464003a50 0x7f748009d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 >> 172.21.15.31:0/496056147 conn(0x7f74800cc910 msgr2=0x7f74800d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 shutdown_connections 2024-09-13T08:45:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.119+0000 7f7484e00640 1 -- 172.21.15.31:0/496056147 wait complete. 2024-09-13T08:45:28.137 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:45:28.137 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:45:28.138 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:45:28.138 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:45:28.138 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:45:28.138 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:45:28.138 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:45:28.138 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:45:28.138 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:45:28.139 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:45:28.139 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:45:28.139 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:45:28.140 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:45:28.140 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:45:28.140 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:45:28.140 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:45:28.154 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:45:28.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:45:28.155 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:45:28.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:45:28.161 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:45:28.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:45:28.162 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:45:28.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:45:28.199 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:45:28.200 DEBUG:teuthology.parallel:result is None 2024-09-13T08:45:28.202 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:45:28.202 DEBUG:teuthology.parallel:result is None 2024-09-13T08:45:28.206 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:45:28.206 DEBUG:teuthology.parallel:result is None 2024-09-13T08:45:28.209 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:45:28.209 DEBUG:teuthology.parallel:result is None 2024-09-13T08:45:28.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:45:28.333 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:45:28.334 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:45:28.345 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:45:28.345 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 -- 172.21.15.31:0/1846096078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c15e3a0 msgr2=0x7f825c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 --2- 172.21.15.31:0/1846096078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c15e3a0 0x7f825c0cf900 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8244009860 tx=0x7f824402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 -- 172.21.15.31:0/1846096078 shutdown_connections 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 --2- 172.21.15.31:0/1846096078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c15e3a0 0x7f825c0cf900 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 -- 172.21.15.31:0/1846096078 >> 172.21.15.31:0/1846096078 conn(0x7f825c0d1d30 msgr2=0x7f825c0d2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 -- 172.21.15.31:0/1846096078 shutdown_connections 2024-09-13T08:45:28.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.377+0000 7f825b200640 1 -- 172.21.15.31:0/1846096078 wait complete. 2024-09-13T08:45:28.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.379+0000 7f825b200640 1 Processor -- start 2024-09-13T08:45:28.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.379+0000 7f825b200640 1 -- start start 2024-09-13T08:45:28.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c0ce560 0x7f825c1f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f825c0cfe40 0x7f825c1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0cca80 con 0x7f825c15e3a0 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0d09a0 con 0x7f825c0ce560 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f825c0ccc00 con 0x7f825c0cfe40 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47602/0 (socket says 172.21.15.31:47602) 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f825a800640 1 -- 172.21.15.31:0/3239519142 learned_addr learned my addr 172.21.15.31:0/3239519142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f8259e00640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c0ce560 0x7f825c1f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.380+0000 7f8251400640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f825c0cfe40 0x7f825c1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 -- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f825c0cfe40 msgr2=0x7f825c1f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:28.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f825c0cfe40 0x7f825c1f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 -- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c0ce560 msgr2=0x7f825c1f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c0ce560 0x7f825c1f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 -- 172.21.15.31:0/3239519142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c0b7860 con 0x7f825c15e3a0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825a800640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f825400c7a0 tx=0x7f825400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254003b70 con 0x7f825c15e3a0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8254003d10 con 0x7f825c15e3a0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825400aab0 con 0x7f825c15e3a0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.381+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c15fbc0 con 0x7f825c15e3a0 2024-09-13T08:45:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.382+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1620a0 con 0x7f825c15e3a0 2024-09-13T08:45:28.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.383+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825c0cf900 con 0x7f825c15e3a0 2024-09-13T08:45:28.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.384+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f825400ac50 con 0x7f825c15e3a0 2024-09-13T08:45:28.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.385+0000 7f8252a00640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 0x7f823807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:28.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.386+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4987+0+0 (secure 0 0 0) 0x7f825409afb0 con 0x7f825c15e3a0 2024-09-13T08:45:28.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.386+0000 7f8259e00640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 0x7f823807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:28.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.386+0000 7f8259e00640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 0x7f823807b3d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f824c007fd0 tx=0x7f824c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:28.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.390+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f82540628c0 con 0x7f825c15e3a0 2024-09-13T08:45:28.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:28.606+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f825c0d2c10 con 0x7f825c15e3a0 2024-09-13T08:45:29.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.584+0000 7f8252a00640 1 -- 172.21.15.31:0/3239519142 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v147) ==== 97+0+0 (secure 0 0 0) 0x7f82540672f0 con 0x7f825c15e3a0 2024-09-13T08:45:29.585 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.587+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 msgr2=0x7f823807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.587+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 0x7f823807b3d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f824c007fd0 tx=0x7f824c009040 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.587+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 msgr2=0x7f825c161330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f825400c7a0 tx=0x7f825400cc60 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 shutdown_connections 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8238078f10 0x7f823807b3d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f825c15e3a0 0x7f825c161330 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f825c0cfe40 0x7f825c1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 --2- 172.21.15.31:0/3239519142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f825c0ce560 0x7f825c1f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 >> 172.21.15.31:0/3239519142 conn(0x7f825c0d1d30 msgr2=0x7f825c15c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 shutdown_connections 2024-09-13T08:45:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.588+0000 7f825b200640 1 -- 172.21.15.31:0/3239519142 wait complete. 2024-09-13T08:45:29.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 -- 172.21.15.31:0/603544112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc15dfc0 msgr2=0x7fb4dc0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 --2- 172.21.15.31:0/603544112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0cc300 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc070b80 tx=0x7fb4cc02a560 comp rx=0 tx=0).stop 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 -- 172.21.15.31:0/603544112 shutdown_connections 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 --2- 172.21.15.31:0/603544112 >> v2:172.21.15.45:3300/0 conn(0x7fb4dc0ca5c0 0x7fb4dc0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 --2- 172.21.15.31:0/603544112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0cc300 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 -- 172.21.15.31:0/603544112 >> 172.21.15.31:0/603544112 conn(0x7fb4dc0cc910 msgr2=0x7fb4dc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 -- 172.21.15.31:0/603544112 shutdown_connections 2024-09-13T08:45:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.773+0000 7fb4e1600640 1 -- 172.21.15.31:0/603544112 wait complete. 2024-09-13T08:45:29.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.775+0000 7fb4e1600640 1 Processor -- start 2024-09-13T08:45:29.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.775+0000 7fb4e1600640 1 -- start start 2024-09-13T08:45:29.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4dc0b1910 0x7fb4dc146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc074010 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc073e90 con 0x7fb4dc15dfc0 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4e1600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb4dc15a9a0 con 0x7fb4dc0b1910 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4d2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4d2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47622/0 (socket says 172.21.15.31:47622) 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.776+0000 7fb4d2000640 1 -- 172.21.15.31:0/3753542708 learned_addr learned my addr 172.21.15.31:0/3753542708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4daa00640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4dc0b1910 0x7fb4dc146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4db400640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 -- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4dc0b1910 msgr2=0x7fb4dc146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:29.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4dc0b1910 0x7fb4dc146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 -- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 msgr2=0x7fb4dc0c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 -- 172.21.15.31:0/3753542708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4dc14a1e0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4db400640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d2000640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d7a0 tx=0x7fb4c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4017070 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.777+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4c4009cc0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.778+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4005280 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.778+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4dc14a4c0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.778+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4dc0c91c0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.779+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4dc0b20f0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.780+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fb4c4005420 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.781+0000 7fb4d3400640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 0x7fb4b807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:29.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.781+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4901+0+0 (secure 0 0 0) 0x7fb4c409b370 con 0x7fb4dc0ca5c0 2024-09-13T08:45:29.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.781+0000 7fb4daa00640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 0x7fb4b807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:29.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.782+0000 7fb4daa00640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 0x7fb4b807b3d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc033000 tx=0x7fb4cc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:29.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:29.785+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb4c4062c50 con 0x7fb4dc0ca5c0 2024-09-13T08:45:30.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.053+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb4dc0af5d0 con 0x7fb4dc0ca5c0 2024-09-13T08:45:30.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.053+0000 7fb4d3400640 1 -- 172.21.15.31:0/3753542708 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5304 (secure 0 0 0) 0x7fb4c4067680 con 0x7fb4dc0ca5c0 2024-09-13T08:45:30.054 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:45:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:45:30.059 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:45:30.061 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.062 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:45:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.066 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:45:30.067 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:45:30.068 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:45:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:45:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 msgr2=0x7fb4b807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 0x7fb4b807b3d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc033000 tx=0x7fb4cc004730 comp rx=0 tx=0).stop 2024-09-13T08:45:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 msgr2=0x7fb4dc147570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d7a0 tx=0x7fb4c400dc60 comp rx=0 tx=0).stop 2024-09-13T08:45:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 shutdown_connections 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb4b8078f10 0x7fb4b807b3d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4dc15dfc0 0x7fb4dc0c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4dc0ca5c0 0x7fb4dc147570 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 --2- 172.21.15.31:0/3753542708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb4dc0b1910 0x7fb4dc146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 >> 172.21.15.31:0/3753542708 conn(0x7fb4dc0cc910 msgr2=0x7fb4dc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 shutdown_connections 2024-09-13T08:45:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.056+0000 7fb4e1600640 1 -- 172.21.15.31:0/3753542708 wait complete. 2024-09-13T08:45:30.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:45:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 -- 172.21.15.31:0/233620565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f50003c30 msgr2=0x7f3f50004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/233620565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f50003c30 0x7f3f50004030 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009b50 tx=0x7f3f50005830 comp rx=0 tx=0).stop 2024-09-13T08:45:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 -- 172.21.15.31:0/233620565 shutdown_connections 2024-09-13T08:45:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/233620565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f50003c30 0x7f3f50004030 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 -- 172.21.15.31:0/233620565 >> 172.21.15.31:0/233620565 conn(0x7f3f6806cb00 msgr2=0x7f3f6806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 -- 172.21.15.31:0/233620565 shutdown_connections 2024-09-13T08:45:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.234+0000 7f3f6ee00640 1 -- 172.21.15.31:0/233620565 wait complete. 2024-09-13T08:45:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.235+0000 7f3f6ee00640 1 Processor -- start 2024-09-13T08:45:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.236+0000 7f3f6ee00640 1 -- start start 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.236+0000 7f3f6ee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3f50003c30 0x7f3f680c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.236+0000 7f3f6ee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f68152e90 0x7f3f68153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6806f9b0 con 0x7f3f680af090 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6ee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680703b0 con 0x7f3f68152e90 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6ee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6806fb30 con 0x7f3f50003c30 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47642/0 (socket says 172.21.15.31:47642) 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 -- 172.21.15.31:0/478491515 learned_addr learned my addr 172.21.15.31:0/478491515 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6d000640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f68152e90 0x7f3f68153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6da00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3f50003c30 0x7f3f680c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 -- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3f50003c30 msgr2=0x7f3f680c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3f50003c30 0x7f3f680c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 -- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f68152e90 msgr2=0x7f3f68153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f68152e90 0x7f3f68153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.237+0000 7f3f6e400640 1 -- 172.21.15.31:0/478491515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f680ada30 con 0x7f3f680af090 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f6e400640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3f680b2640 tx=0x7f3f60001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60011870 con 0x7f3f680af090 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f60017070 con 0x7f3f680af090 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60011da0 con 0x7f3f680af090 2024-09-13T08:45:30.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f680add10 con 0x7f3f680af090 2024-09-13T08:45:30.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.238+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f680ae2d0 con 0x7f3f680af090 2024-09-13T08:45:30.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.239+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f680b4bc0 con 0x7f3f680af090 2024-09-13T08:45:30.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.240+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f3f6001c460 con 0x7f3f680af090 2024-09-13T08:45:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.241+0000 7f3f65600640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 0x7f3f4807b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.241+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4901+0+0 (secure 0 0 0) 0x7f3f6009ba10 con 0x7f3f680af090 2024-09-13T08:45:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.241+0000 7f3f6da00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 0x7f3f4807b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.242+0000 7f3f6da00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 0x7f3f4807b400 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c035000 tx=0x7f3f5c0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.245+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3f600639c0 con 0x7f3f680af090 2024-09-13T08:45:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.444+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3f680ae4a0 con 0x7f3f680af090 2024-09-13T08:45:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.444+0000 7f3f65600640 1 -- 172.21.15.31:0/478491515 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) ==== 84+0+65 (secure 0 0 0) 0x7f3f600632c0 con 0x7f3f680af090 2024-09-13T08:45:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:30.445 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:45:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:45:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:45:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 msgr2=0x7f3f4807b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 0x7f3f4807b400 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c035000 tx=0x7f3f5c0046c0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 msgr2=0x7f3f680af4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3f680b2640 tx=0x7f3f60001b80 comp rx=0 tx=0).stop 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 shutdown_connections 2024-09-13T08:45:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3f48078f40 0x7f3f4807b400 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f680af090 0x7f3f680af4f0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f68152e90 0x7f3f68153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 --2- 172.21.15.31:0/478491515 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3f50003c30 0x7f3f680c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 >> 172.21.15.31:0/478491515 conn(0x7f3f6806cb00 msgr2=0x7f3f680c98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 shutdown_connections 2024-09-13T08:45:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.447+0000 7f3f6ee00640 1 -- 172.21.15.31:0/478491515 wait complete. 2024-09-13T08:45:30.464 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:45:30.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:45:30.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 -- 172.21.15.31:0/3521405098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f454006fec0 msgr2=0x7f45400b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 --2- 172.21.15.31:0/3521405098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f454006fec0 0x7f45400b29e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f4540069dd0 tx=0x7f453802a530 comp rx=0 tx=0).stop 2024-09-13T08:45:30.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 -- 172.21.15.31:0/3521405098 shutdown_connections 2024-09-13T08:45:30.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 --2- 172.21.15.31:0/3521405098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f454006fec0 0x7f45400b29e0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 --2- 172.21.15.31:0/3521405098 >> v2:172.21.15.45:3300/0 conn(0x7f454015bfc0 0x7f4540072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 -- 172.21.15.31:0/3521405098 >> 172.21.15.31:0/3521405098 conn(0x7f45400cfbb0 msgr2=0x7f45400cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:30.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 -- 172.21.15.31:0/3521405098 shutdown_connections 2024-09-13T08:45:30.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.629+0000 7f4545a00640 1 -- 172.21.15.31:0/3521405098 wait complete. 2024-09-13T08:45:30.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.630+0000 7f4545a00640 1 Processor -- start 2024-09-13T08:45:30.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.630+0000 7f4545a00640 1 -- start start 2024-09-13T08:45:30.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454006fa60 0x7f45401f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f454015bfc0 0x7f4540151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45400cbfd0 con 0x7f45400c7ea0 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f45400cbe50 con 0x7f454006fa60 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.631+0000 7f4545a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f45400b2f20 con 0x7f454015bfc0 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47668/0 (socket says 172.21.15.31:47668) 2024-09-13T08:45:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 -- 172.21.15.31:0/2453096161 learned_addr learned my addr 172.21.15.31:0/2453096161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453ea00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454006fa60 0x7f45401f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453e000640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f454015bfc0 0x7f4540151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 -- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f454015bfc0 msgr2=0x7f4540151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f454015bfc0 0x7f4540151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 -- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454006fa60 msgr2=0x7f45401f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454006fa60 0x7f45401f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 -- 172.21.15.31:0/2453096161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45400c8cb0 con 0x7f45400c7ea0 2024-09-13T08:45:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.632+0000 7f453f400640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f45380098f0 tx=0x7f4538032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.633+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538040070 con 0x7f45400c7ea0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.633+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4538005310 con 0x7f45400c7ea0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.633+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45400c8f90 con 0x7f45400c7ea0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.633+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538005c90 con 0x7f45400c7ea0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.633+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45400c9550 con 0x7f45400c7ea0 2024-09-13T08:45:30.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.634+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45400b33d0 con 0x7f45400c7ea0 2024-09-13T08:45:30.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.635+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f4538039040 con 0x7f45400c7ea0 2024-09-13T08:45:30.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.636+0000 7f4536a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 0x7f452007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:30.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.636+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4901+0+0 (secure 0 0 0) 0x7f45380bc590 con 0x7f45400c7ea0 2024-09-13T08:45:30.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.636+0000 7f453ea00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 0x7f452007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:30.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.637+0000 7f453ea00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 0x7f452007b3d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4530005fd0 tx=0x7f4530009500 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:30.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.641+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4538084480 con 0x7f45400c7ea0 2024-09-13T08:45:30.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:30.836+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f45400cb310 con 0x7f45400c7ea0 2024-09-13T08:45:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.587+0000 7f4536a00640 1 -- 172.21.15.31:0/2453096161 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v148) ==== 138+0+0 (secure 0 0 0) 0x7f4538083d80 con 0x7f45400c7ea0 2024-09-13T08:45:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:45:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.590+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 msgr2=0x7f452007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.590+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 0x7f452007b3d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4530005fd0 tx=0x7f4530009500 comp rx=0 tx=0).stop 2024-09-13T08:45:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.590+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 msgr2=0x7f45400c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.590+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f45380098f0 tx=0x7f4538032c10 comp rx=0 tx=0).stop 2024-09-13T08:45:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 shutdown_connections 2024-09-13T08:45:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4520078f10 0x7f452007b3d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45400c7ea0 0x7f45400c8350 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f454015bfc0 0x7f4540151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 --2- 172.21.15.31:0/2453096161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454006fa60 0x7f45401f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.591+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 >> 172.21.15.31:0/2453096161 conn(0x7f45400cfbb0 msgr2=0x7f4540157450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.592+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 shutdown_connections 2024-09-13T08:45:31.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.593+0000 7f4545a00640 1 -- 172.21.15.31:0/2453096161 wait complete. 2024-09-13T08:45:31.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:45:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 -- 172.21.15.31:0/3400875549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 msgr2=0x7f03700d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 --2- 172.21.15.31:0/3400875549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700d28f0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0368005fd0 tx=0x7f036802a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 -- 172.21.15.31:0/3400875549 shutdown_connections 2024-09-13T08:45:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 --2- 172.21.15.31:0/3400875549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700d28f0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 -- 172.21.15.31:0/3400875549 >> 172.21.15.31:0/3400875549 conn(0x7f03700d1dd0 msgr2=0x7f03700d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 -- 172.21.15.31:0/3400875549 shutdown_connections 2024-09-13T08:45:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.788+0000 7f0376400640 1 -- 172.21.15.31:0/3400875549 wait complete. 2024-09-13T08:45:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.789+0000 7f0376400640 1 Processor -- start 2024-09-13T08:45:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.790+0000 7f0376400640 1 -- start start 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.790+0000 7f0376400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.790+0000 7f0376400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03700cf050 0x7f03700caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f0376400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 0x7f03700c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f0376400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03700b6070 con 0x7f03700ceb50 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f0376400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03700b5ef0 con 0x7f037015bec0 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f0376400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03700b61f0 con 0x7f03700cf050 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47692/0 (socket says 172.21.15.31:47692) 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 -- 172.21.15.31:0/3512632504 learned_addr learned my addr 172.21.15.31:0/3512632504 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036fe00640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 0x7f03700c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036ea00640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03700cf050 0x7f03700caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 -- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03700cf050 msgr2=0x7f03700caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03700cf050 0x7f03700caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 -- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 msgr2=0x7f03700c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 0x7f03700c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 -- 172.21.15.31:0/3512632504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037015dab0 con 0x7f03700ceb50 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036fe00640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 0x7f03700c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.791+0000 7f036f400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f036000c440 tx=0x7f036000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.792+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360017070 con 0x7f03700ceb50 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.792+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03600035f0 con 0x7f03700ceb50 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.792+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360003a40 con 0x7f03700ceb50 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.792+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037015dd90 con 0x7f03700ceb50 2024-09-13T08:45:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.792+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03700cb670 con 0x7f03700ceb50 2024-09-13T08:45:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.793+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03700b6710 con 0x7f03700ceb50 2024-09-13T08:45:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.794+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f036001a070 con 0x7f03700ceb50 2024-09-13T08:45:31.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.795+0000 7f0367400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 0x7f035007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:31.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.795+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5271+0+0 (secure 0 0 0) 0x7f036009a950 con 0x7f03700ceb50 2024-09-13T08:45:31.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.796+0000 7f036ea00640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 0x7f035007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.796+0000 7f036ea00640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 0x7f035007b3d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03701e96d0 tx=0x7f035c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.799+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0360021080 con 0x7f03700ceb50 2024-09-13T08:45:32.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:31.999+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f03700723c0 con 0x7f03700ceb50 2024-09-13T08:45:32.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.595+0000 7f0367400640 1 -- 172.21.15.31:0/3512632504 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v149) ==== 170+0+0 (secure 0 0 0) 0x7f0360062790 con 0x7f03700ceb50 2024-09-13T08:45:32.596 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:45:32.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.598+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 msgr2=0x7f035007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:32.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.598+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 0x7f035007b3d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03701e96d0 tx=0x7f035c009040 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.598+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 msgr2=0x7f03700ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.598+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f036000c440 tx=0x7f036000c900 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 shutdown_connections 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0350078f10 0x7f035007b3d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f037015bec0 0x7f03700c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03700cf050 0x7f03700caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 --2- 172.21.15.31:0/3512632504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03700ceb50 0x7f03700ca900 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 >> 172.21.15.31:0/3512632504 conn(0x7f03700d1dd0 msgr2=0x7f0370070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:32.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.599+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 shutdown_connections 2024-09-13T08:45:32.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.600+0000 7f0376400640 1 -- 172.21.15.31:0/3512632504 wait complete. 2024-09-13T08:45:32.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:45:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 -- 172.21.15.31:0/1737934341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f840406fec0 msgr2=0x7f84040b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 --2- 172.21.15.31:0/1737934341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f840406fec0 0x7f84040b29e0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f83fc005fd0 tx=0x7f83fc02aa90 comp rx=0 tx=0).stop 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 -- 172.21.15.31:0/1737934341 shutdown_connections 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 --2- 172.21.15.31:0/1737934341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f840406fec0 0x7f84040b29e0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 --2- 172.21.15.31:0/1737934341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f8404072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 -- 172.21.15.31:0/1737934341 >> 172.21.15.31:0/1737934341 conn(0x7f84040cfbb0 msgr2=0x7f84040cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 -- 172.21.15.31:0/1737934341 shutdown_connections 2024-09-13T08:45:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.796+0000 7f840b800640 1 -- 172.21.15.31:0/1737934341 wait complete. 2024-09-13T08:45:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.798+0000 7f840b800640 1 Processor -- start 2024-09-13T08:45:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.798+0000 7f840b800640 1 -- start start 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 0x7f840415f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f84041f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84040cbfd0 con 0x7f8404144680 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84040cbe50 con 0x7f840406fa60 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f840b800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f84040b2f20 con 0x7f840415bfc0 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f8409800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f8409800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47704/0 (socket says 172.21.15.31:47704) 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f8409800640 1 -- 172.21.15.31:0/2472796958 learned_addr learned my addr 172.21.15.31:0/2472796958 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f8408e00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 0x7f840415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.799+0000 7f83fbe00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f84041f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 -- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 msgr2=0x7f84041f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f84041f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 -- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 msgr2=0x7f840415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 0x7f840415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 -- 172.21.15.31:0/2472796958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84041f94e0 con 0x7f8404144680 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8408e00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 0x7f840415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f83fbe00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f84041f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8409800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009bf0 tx=0x7f83fc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc046070 con 0x7f8404144680 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83fc004040 con 0x7f8404144680 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc004450 con 0x7f8404144680 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8404151410 con 0x7f8404144680 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.800+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84041519d0 con 0x7f8404144680 2024-09-13T08:45:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.801+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84040b33d0 con 0x7f8404144680 2024-09-13T08:45:32.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.803+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f83fc049070 con 0x7f8404144680 2024-09-13T08:45:32.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.804+0000 7f8401600640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 0x7f83e007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:32.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.804+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 5653+0+0 (secure 0 0 0) 0x7f83fc0bc5f0 con 0x7f8404144680 2024-09-13T08:45:32.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.804+0000 7f8408e00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 0x7f83e007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:32.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.805+0000 7f8408e00640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 0x7f83e007b3d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f84040cf560 tx=0x7f83f4005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:32.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:32.808+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f83fc050080 con 0x7f8404144680 2024-09-13T08:45:33.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.040+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f84040cb310 con 0x7f8404144680 2024-09-13T08:45:33.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.607+0000 7f8401600640 1 -- 172.21.15.31:0/2472796958 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v96) ==== 174+0+0 (secure 0 0 0) 0x7f84041f94e0 con 0x7f8404144680 2024-09-13T08:45:33.612 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 22 and data pool 23 2024-09-13T08:45:33.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.616+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 msgr2=0x7f83e007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:33.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.616+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 0x7f83e007b3d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f84040cf560 tx=0x7f83f4005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:33.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.616+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 msgr2=0x7f84041524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:33.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.616+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009bf0 tx=0x7f83fc003ba0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 shutdown_connections 2024-09-13T08:45:33.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f83e0078f10 0x7f83e007b3d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8404144680 0x7f84041524b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f840415bfc0 0x7f84041f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 --2- 172.21.15.31:0/2472796958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f840406fa60 0x7f840415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 >> 172.21.15.31:0/2472796958 conn(0x7f84040cfbb0 msgr2=0x7f8404157950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:33.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 shutdown_connections 2024-09-13T08:45:33.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.617+0000 7f840b800640 1 -- 172.21.15.31:0/2472796958 wait complete. 2024-09-13T08:45:33.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 -- 172.21.15.31:0/1120529426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600ca5c0 msgr2=0x7f40600cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 --2- 172.21.15.31:0/1120529426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600ca5c0 0x7f40600cc300 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4048009830 tx=0x7f404802a5c0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 -- 172.21.15.31:0/1120529426 shutdown_connections 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 --2- 172.21.15.31:0/1120529426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600ca5c0 0x7f40600cc300 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 -- 172.21.15.31:0/1120529426 >> 172.21.15.31:0/1120529426 conn(0x7f40600cc910 msgr2=0x7f40600cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 -- 172.21.15.31:0/1120529426 shutdown_connections 2024-09-13T08:45:33.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.811+0000 7f4064e00640 1 -- 172.21.15.31:0/1120529426 wait complete. 2024-09-13T08:45:33.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.812+0000 7f4064e00640 1 Processor -- start 2024-09-13T08:45:33.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 -- start start 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 0x7f40600b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40600ca5c0 0x7f40600728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40600b1d60 con 0x7f406015dfc0 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40600740a0 con 0x7f40600b1910 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.813+0000 7f4064e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f40600b1ee0 con 0x7f40600ca5c0 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47718/0 (socket says 172.21.15.31:47718) 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 -- 172.21.15.31:0/1623849442 learned_addr learned my addr 172.21.15.31:0/1623849442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:33.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405de00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 0x7f40600b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 -- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40600ca5c0 msgr2=0x7f40600728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40600ca5c0 0x7f40600728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 -- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 msgr2=0x7f40600b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 0x7f40600b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 -- 172.21.15.31:0/1623849442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40600b59e0 con 0x7f406015dfc0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405de00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 0x7f40600b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.814+0000 7f405e800640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f405800cbc0 tx=0x7f405800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.815+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058016020 con 0x7f406015dfc0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.815+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f405800ae30 con 0x7f406015dfc0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.815+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058012680 con 0x7f406015dfc0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.815+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40600c70e0 con 0x7f406015dfc0 2024-09-13T08:45:33.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.815+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40600c76a0 con 0x7f406015dfc0 2024-09-13T08:45:33.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.816+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40600cc300 con 0x7f406015dfc0 2024-09-13T08:45:33.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.817+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f4058019020 con 0x7f406015dfc0 2024-09-13T08:45:33.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.818+0000 7f4056000640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 0x7f403c07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:33.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.818+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5753+0+0 (secure 0 0 0) 0x7f405809c260 con 0x7f406015dfc0 2024-09-13T08:45:33.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.818+0000 7f405de00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 0x7f403c07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:33.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.819+0000 7f405de00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 0x7f403c07b3d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4050000f60 tx=0x7f4050009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:33.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:33.822+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4058020020 con 0x7f406015dfc0 2024-09-13T08:45:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.016+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40600af5d0 con 0x7f406015dfc0 2024-09-13T08:45:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.017+0000 7f4056000640 1 -- 172.21.15.31:0/1623849442 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) ==== 74+0+16771 (secure 0 0 0) 0x7f40600cbe40 con 0x7f406015dfc0 2024-09-13T08:45:34.019 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:34.019 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":150,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:33.041162+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:45:30.846891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T08:45:32.010174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 msgr2=0x7f403c07b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 0x7f403c07b3d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4050000f60 tx=0x7f4050009040 comp rx=0 tx=0).stop 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 msgr2=0x7f406014de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f405800cbc0 tx=0x7f405800a430 comp rx=0 tx=0).stop 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 shutdown_connections 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f403c078f10 0x7f403c07b3d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f406015dfc0 0x7f406014de00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40600ca5c0 0x7f40600728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 --2- 172.21.15.31:0/1623849442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40600b1910 0x7f40600b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 >> 172.21.15.31:0/1623849442 conn(0x7f40600cc910 msgr2=0x7f40600cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 shutdown_connections 2024-09-13T08:45:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.020+0000 7f4064e00640 1 -- 172.21.15.31:0/1623849442 wait complete. 2024-09-13T08:45:34.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 -- 172.21.15.31:0/992677325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 msgr2=0x7f49040b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 --2- 172.21.15.31:0/992677325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040b1910 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f48fc005fd0 tx=0x7f48fc02aae0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 -- 172.21.15.31:0/992677325 shutdown_connections 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 --2- 172.21.15.31:0/992677325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040b1910 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 --2- 172.21.15.31:0/992677325 >> v2:172.21.15.45:3300/0 conn(0x7f4904152f10 0x7f49040cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 -- 172.21.15.31:0/992677325 >> 172.21.15.31:0/992677325 conn(0x7f49040cc910 msgr2=0x7f49040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:34.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 -- 172.21.15.31:0/992677325 shutdown_connections 2024-09-13T08:45:34.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.197+0000 7f4909a00640 1 -- 172.21.15.31:0/992677325 wait complete. 2024-09-13T08:45:34.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.198+0000 7f4909a00640 1 Processor -- start 2024-09-13T08:45:34.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.198+0000 7f4909a00640 1 -- start start 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4904072850 0x7f4904153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 0x7f490409b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49040ca660 con 0x7f49040746d0 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49040ca4e0 con 0x7f4904072850 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.199+0000 7f4909a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f49040b1e50 con 0x7f4904152f10 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47740/0 (socket says 172.21.15.31:47740) 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 -- 172.21.15.31:0/985043841 learned_addr learned my addr 172.21.15.31:0/985043841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f4902a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4904072850 0x7f4904153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f4903400640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 0x7f490409b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 -- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 msgr2=0x7f490409b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 0x7f490409b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 -- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4904072850 msgr2=0x7f4904153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4904072850 0x7f4904153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 -- 172.21.15.31:0/985043841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49040c4f30 con 0x7f49040746d0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f4903400640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 0x7f490409b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.200+0000 7f48fbe00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00c500 tx=0x7f48ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.201+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec0052d0 con 0x7f49040746d0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.201+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48ec005470 con 0x7f49040746d0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.201+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49040c5210 con 0x7f49040746d0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.201+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec012870 con 0x7f49040746d0 2024-09-13T08:45:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.201+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490409c640 con 0x7f49040746d0 2024-09-13T08:45:34.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.202+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f490409c840 con 0x7f49040746d0 2024-09-13T08:45:34.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.203+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f48ec0035f0 con 0x7f49040746d0 2024-09-13T08:45:34.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.204+0000 7f48faa00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 0x7f48e007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.204+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5753+0+0 (secure 0 0 0) 0x7f48ec09b130 con 0x7f49040746d0 2024-09-13T08:45:34.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.204+0000 7f4902a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 0x7f48e007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.205+0000 7f4902a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 0x7f48e007b3d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f48f4005fd0 tx=0x7f48f400b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:34.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.209+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f490409c840 con 0x7f49040746d0 2024-09-13T08:45:34.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.403+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49040c53e0 con 0x7f49040746d0 2024-09-13T08:45:34.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.404+0000 7f48faa00640 1 -- 172.21.15.31:0/985043841 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) ==== 74+0+16771 (secure 0 0 0) 0x7f48ec062da0 con 0x7f49040746d0 2024-09-13T08:45:34.406 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:34.406 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":150,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:33.041162+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:45:30.846891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T08:45:32.010174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:34.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 msgr2=0x7f48e007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 0x7f48e007b3d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f48f4005fd0 tx=0x7f48f400b2e0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 msgr2=0x7f49040c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00c500 tx=0x7f48ec00c9c0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 shutdown_connections 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f48e0078f10 0x7f48e007b3d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4904152f10 0x7f490409b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49040746d0 0x7f49040c4830 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 --2- 172.21.15.31:0/985043841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4904072850 0x7f4904153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 >> 172.21.15.31:0/985043841 conn(0x7f49040cc910 msgr2=0x7f490415e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 shutdown_connections 2024-09-13T08:45:34.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.407+0000 7f4909a00640 1 -- 172.21.15.31:0/985043841 wait complete. 2024-09-13T08:45:34.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:45:34.587 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:45:34.586+0000 7f6963600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:45:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 -- 172.21.15.31:0/909201474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209815bfc0 msgr2=0x7f2098072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 --2- 172.21.15.31:0/909201474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209815bfc0 0x7f2098072560 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f208c005fd0 tx=0x7f208c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 -- 172.21.15.31:0/909201474 shutdown_connections 2024-09-13T08:45:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 --2- 172.21.15.31:0/909201474 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f209806fa60 0x7f20980cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 --2- 172.21.15.31:0/909201474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209815bfc0 0x7f2098072560 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 -- 172.21.15.31:0/909201474 >> 172.21.15.31:0/909201474 conn(0x7f20980cfbb0 msgr2=0x7f20980cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:34.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 -- 172.21.15.31:0/909201474 shutdown_connections 2024-09-13T08:45:34.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.587+0000 7f209fe00640 1 -- 172.21.15.31:0/909201474 wait complete. 2024-09-13T08:45:34.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.589+0000 7f209fe00640 1 Processor -- start 2024-09-13T08:45:34.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.589+0000 7f209fe00640 1 -- start start 2024-09-13T08:45:34.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209815bfc0 0x7f20981e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 0x7f20981476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f20980b3270 con 0x7f209806fa60 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20980b30f0 con 0x7f209815bfc0 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f20980b33f0 con 0x7f20981e8d90 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47768/0 (socket says 172.21.15.31:47768) 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.590+0000 7f209d400640 1 -- 172.21.15.31:0/2412686323 learned_addr learned my addr 172.21.15.31:0/2412686323 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209ca00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209815bfc0 0x7f20981e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209de00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 0x7f20981476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 -- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 msgr2=0x7f20981476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 0x7f20981476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 -- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209815bfc0 msgr2=0x7f20981e8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209815bfc0 0x7f20981e8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 -- 172.21.15.31:0/2412686323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20981f9390 con 0x7f209806fa60 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209de00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 0x7f20981476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:34.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209d400640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f208c034040 tx=0x7f208c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c03e030 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f208c003ae0 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c035040 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.591+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20980cdab0 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.592+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20980ce070 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.593+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20980cbf20 con 0x7f209806fa60 2024-09-13T08:45:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.594+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f208c041030 con 0x7f209806fa60 2024-09-13T08:45:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.595+0000 7f2095600640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 0x7f207807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:34.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.595+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5753+0+0 (secure 0 0 0) 0x7f208c0bbef0 con 0x7f209806fa60 2024-09-13T08:45:34.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.595+0000 7f209ca00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 0x7f207807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:34.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.596+0000 7f209ca00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 0x7f207807b3d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f20980cf340 tx=0x7f2084005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:34.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.599+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f20981f9390 con 0x7f209806fa60 2024-09-13T08:45:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:34.836+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f20980cb310 con 0x7f209806fa60 2024-09-13T08:45:35.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.620+0000 7f2095600640 1 -- 172.21.15.31:0/2412686323 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v98) ==== 118+0+0 (secure 0 0 0) 0x7f208c0839c0 con 0x7f209806fa60 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.623+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 msgr2=0x7f207807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.623+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 0x7f207807b3d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f20980cf340 tx=0x7f2084005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.623+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 msgr2=0x7f20981f8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.623+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f208c034040 tx=0x7f208c0035f0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 shutdown_connections 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2078078f10 0x7f207807b3d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20981e8d90 0x7f20981476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209815bfc0 0x7f20981e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 --2- 172.21.15.31:0/2412686323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f209806fa60 0x7f20981f8e50 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 >> 172.21.15.31:0/2412686323 conn(0x7f20980cfbb0 msgr2=0x7f209815aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 shutdown_connections 2024-09-13T08:45:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.624+0000 7f209fe00640 1 -- 172.21.15.31:0/2412686323 wait complete. 2024-09-13T08:45:35.645 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:45:35.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 -- 172.21.15.31:0/2541843656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0360146f50 msgr2=0x7f0360161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 --2- 172.21.15.31:0/2541843656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0360146f50 0x7f0360161f00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f034c005fd0 tx=0x7f034c02a530 comp rx=0 tx=0).stop 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 -- 172.21.15.31:0/2541843656 shutdown_connections 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 --2- 172.21.15.31:0/2541843656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0360146f50 0x7f0360161f00 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 --2- 172.21.15.31:0/2541843656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03600cdf90 0x7f03600ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 -- 172.21.15.31:0/2541843656 >> 172.21.15.31:0/2541843656 conn(0x7f03600b1520 msgr2=0x7f03600b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 -- 172.21.15.31:0/2541843656 shutdown_connections 2024-09-13T08:45:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.814+0000 7f0367000640 1 -- 172.21.15.31:0/2541843656 wait complete. 2024-09-13T08:45:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.815+0000 7f0367000640 1 Processor -- start 2024-09-13T08:45:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.816+0000 7f0367000640 1 -- start start 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.816+0000 7f0367000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.816+0000 7f0367000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0360146f50 0x7f0360157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.816+0000 7f0367000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0360162440 0x7f03600b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0367000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03600cc560 con 0x7f03600cdf90 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0367000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03600cc3e0 con 0x7f0360146f50 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0367000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03600cc6e0 con 0x7f0360162440 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45390/0 (socket says 172.21.15.31:45390) 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 -- 172.21.15.31:0/274124733 learned_addr learned my addr 172.21.15.31:0/274124733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365200640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0360146f50 0x7f0360157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0366600640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0360162440 0x7f03600b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 -- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0360162440 msgr2=0x7f03600b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0360162440 0x7f03600b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 -- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0360146f50 msgr2=0x7f0360157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0360146f50 0x7f0360157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.817+0000 7f0365c00640 1 -- 172.21.15.31:0/274124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03600c7f90 con 0x7f03600cdf90 2024-09-13T08:45:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f0365c00640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f035400d7a0 tx=0x7f035400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354017070 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0354009cc0 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03600b2c10 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354005280 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.818+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03600b31d0 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.819+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03600ccca0 con 0x7f03600cdf90 2024-09-13T08:45:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.820+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f0354005420 con 0x7f03600cdf90 2024-09-13T08:45:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.821+0000 7f035d600640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 0x7f034007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.821+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f035409b860 con 0x7f03600cdf90 2024-09-13T08:45:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.822+0000 7f0365200640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 0x7f034007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.822+0000 7f0365200640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 0x7f034007b3d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f034c033000 tx=0x7f034c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:35.825+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0354062c50 con 0x7f03600cdf90 2024-09-13T08:45:36.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.056+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f03600af4c0 con 0x7f03600cdf90 2024-09-13T08:45:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.622+0000 7f035d600640 1 -- 172.21.15.31:0/274124733 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v99) ==== 122+0+0 (secure 0 0 0) 0x7f0354067680 con 0x7f03600cdf90 2024-09-13T08:45:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 msgr2=0x7f034007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 0x7f034007b3d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f034c033000 tx=0x7f034c004730 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 msgr2=0x7f036015e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f035400d7a0 tx=0x7f035400dc60 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 shutdown_connections 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0340078f10 0x7f034007b3d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0360162440 0x7f03600b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0360146f50 0x7f0360157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 --2- 172.21.15.31:0/274124733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03600cdf90 0x7f036015e8b0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.625+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 >> 172.21.15.31:0/274124733 conn(0x7f03600b1520 msgr2=0x7f03600723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.626+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 shutdown_connections 2024-09-13T08:45:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.626+0000 7f0367000640 1 -- 172.21.15.31:0/274124733 wait complete. 2024-09-13T08:45:36.646 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:45:36.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:36.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 -- 172.21.15.31:0/1008312671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7374004800 msgr2=0x7f7374004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:36.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 --2- 172.21.15.31:0/1008312671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7374004800 0x7f7374004c80 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f737c0016c0 tx=0x7f7374005830 comp rx=0 tx=0).stop 2024-09-13T08:45:36.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 -- 172.21.15.31:0/1008312671 shutdown_connections 2024-09-13T08:45:36.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 --2- 172.21.15.31:0/1008312671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7374004800 0x7f7374004c80 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 -- 172.21.15.31:0/1008312671 >> 172.21.15.31:0/1008312671 conn(0x7f738c074260 msgr2=0x7f738c074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:36.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 -- 172.21.15.31:0/1008312671 shutdown_connections 2024-09-13T08:45:36.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.813+0000 7f7394800640 1 -- 172.21.15.31:0/1008312671 wait complete. 2024-09-13T08:45:36.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.815+0000 7f7394800640 1 Processor -- start 2024-09-13T08:45:36.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.815+0000 7f7394800640 1 -- start start 2024-09-13T08:45:36.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 0x7f738c0c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:36.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:36.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738c0c52e0 0x7f738c152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:36.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f738c0b0920 con 0x7f738c0d0cd0 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f738c068ac0 con 0x7f738c0c52e0 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7394800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f738c0b0aa0 con 0x7f7374004800 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45402/0 (socket says 172.21.15.31:45402) 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 -- 172.21.15.31:0/2104001520 learned_addr learned my addr 172.21.15.31:0/2104001520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391e00640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 0x7f738c0c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7392800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738c0c52e0 0x7f738c152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 -- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 msgr2=0x7f738c0c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 0x7f738c0c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 -- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738c0c52e0 msgr2=0x7f738c152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738c0c52e0 0x7f738c152cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391400640 1 -- 172.21.15.31:0/2104001520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c0c7a60 con 0x7f738c0d0cd0 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.816+0000 7f7391e00640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 0x7f738c0c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7391400640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f737c0017e0 tx=0x7f737c0063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c042070 con 0x7f738c0d0cd0 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f737c039070 con 0x7f738c0d0cd0 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c003d70 con 0x7f738c0d0cd0 2024-09-13T08:45:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c0c7d40 con 0x7f738c0d0cd0 2024-09-13T08:45:36.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.817+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c0c8300 con 0x7f738c0d0cd0 2024-09-13T08:45:36.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.818+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f738c0b1230 con 0x7f738c0d0cd0 2024-09-13T08:45:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.819+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f737c03b070 con 0x7f738c0d0cd0 2024-09-13T08:45:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.820+0000 7f7386000640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 0x7f737007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.820+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f737c0baa40 con 0x7f738c0d0cd0 2024-09-13T08:45:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.820+0000 7f7391e00640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 0x7f737007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.821+0000 7f7391e00640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 0x7f737007b3d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f738c06b690 tx=0x7f73800067a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:36.825+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f737c0825a0 con 0x7f738c0d0cd0 2024-09-13T08:45:37.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.060+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f738c159880 con 0x7f738c0d0cd0 2024-09-13T08:45:37.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.060+0000 7f7386000640 1 -- 172.21.15.31:0/2104001520 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5227 (secure 0 0 0) 0x7f737c081ea0 con 0x7f738c0d0cd0 2024-09-13T08:45:37.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:37.062 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":99,"btime":"2024-09-13T08:45:36:620992+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:37.062 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 99 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 msgr2=0x7f737007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 0x7f737007b3d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f738c06b690 tx=0x7f73800067a0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 msgr2=0x7f738c0c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f737c0017e0 tx=0x7f737c0063a0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 shutdown_connections 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7370078f10 0x7f737007b3d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738c0c52e0 0x7f738c152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c0d0cd0 0x7f738c0c4da0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 --2- 172.21.15.31:0/2104001520 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7374004800 0x7f738c0c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 >> 172.21.15.31:0/2104001520 conn(0x7f738c074260 msgr2=0x7f738c157ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:37.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 shutdown_connections 2024-09-13T08:45:37.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.063+0000 7f7394800640 1 -- 172.21.15.31:0/2104001520 wait complete. 2024-09-13T08:45:37.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 -- 172.21.15.31:0/555285788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc15dfc0 msgr2=0x7fcffc0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 --2- 172.21.15.31:0/555285788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc15dfc0 0x7fcffc0cc300 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fcffc070b80 tx=0x7fcff002a550 comp rx=0 tx=0).stop 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 -- 172.21.15.31:0/555285788 shutdown_connections 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 --2- 172.21.15.31:0/555285788 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc0b1910 0x7fcffc0b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 --2- 172.21.15.31:0/555285788 >> v2:172.21.15.45:3300/0 conn(0x7fcffc0ca5c0 0x7fcffc0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 --2- 172.21.15.31:0/555285788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc15dfc0 0x7fcffc0cc300 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 -- 172.21.15.31:0/555285788 >> 172.21.15.31:0/555285788 conn(0x7fcffc0cc910 msgr2=0x7fcffc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 -- 172.21.15.31:0/555285788 shutdown_connections 2024-09-13T08:45:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.238+0000 7fd004000640 1 -- 172.21.15.31:0/555285788 wait complete. 2024-09-13T08:45:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.240+0000 7fd004000640 1 Processor -- start 2024-09-13T08:45:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.240+0000 7fd004000640 1 -- start start 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc0ca5c0 0x7fcffc1f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc15dfc0 0x7fcffc151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc073900 con 0x7fcffc0b1910 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc073780 con 0x7fcffc0ca5c0 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd004000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcffc073a80 con 0x7fcffc15dfc0 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd001600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd001600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45430/0 (socket says 172.21.15.31:45430) 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd001600640 1 -- 172.21.15.31:0/2568411269 learned_addr learned my addr 172.21.15.31:0/2568411269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.241+0000 7fd000c00640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc0ca5c0 0x7fcffc1f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd002000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc15dfc0 0x7fcffc151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 -- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc15dfc0 msgr2=0x7fcffc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc15dfc0 0x7fcffc151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 -- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc0ca5c0 msgr2=0x7fcffc1f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc0ca5c0 0x7fcffc1f8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 -- 172.21.15.31:0/2568411269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc0b5270 con 0x7fcffc0b1910 2024-09-13T08:45:37.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd001600640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034040 tx=0x7fcff0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff003e030 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcff0003e00 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.242+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcffc1f5160 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.243+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0035040 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.243+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcffc1f4dc0 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.244+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcffc074010 con 0x7fcffc0b1910 2024-09-13T08:45:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.245+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fcff0041030 con 0x7fcffc0b1910 2024-09-13T08:45:37.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.246+0000 7fcff9600640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 0x7fcfdc07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.246+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fcff00bbd20 con 0x7fcffc0b1910 2024-09-13T08:45:37.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.246+0000 7fd000c00640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 0x7fcfdc07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.247+0000 7fd000c00640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 0x7fcfdc07b3d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcffc09db30 tx=0x7fcfe800b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:37.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.250+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcff00831e0 con 0x7fcffc0b1910 2024-09-13T08:45:37.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:45:37.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:45:37.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.447+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcffc0af5d0 con 0x7fcffc0b1910 2024-09-13T08:45:37.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.448+0000 7fcff9600640 1 -- 172.21.15.31:0/2568411269 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16758 (secure 0 0 0) 0x7fcff0087c10 con 0x7fcffc0b1910 2024-09-13T08:45:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:37.449 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":151,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:34.608532+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:45:30.846891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T08:45:32.010174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 msgr2=0x7fcfdc07b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 0x7fcfdc07b3d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcffc09db30 tx=0x7fcfe800b390 comp rx=0 tx=0).stop 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 msgr2=0x7fcffc0b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034040 tx=0x7fcff0003910 comp rx=0 tx=0).stop 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 shutdown_connections 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcfdc078f10 0x7fcfdc07b3d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcffc15dfc0 0x7fcffc151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc0ca5c0 0x7fcffc1f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 --2- 172.21.15.31:0/2568411269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcffc0b1910 0x7fcffc0b4bd0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 >> 172.21.15.31:0/2568411269 conn(0x7fcffc0cc910 msgr2=0x7fcffc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:37.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 shutdown_connections 2024-09-13T08:45:37.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.451+0000 7fd004000640 1 -- 172.21.15.31:0/2568411269 wait complete. 2024-09-13T08:45:37.468 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:45:37.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 -- 172.21.15.31:0/4164497458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180b47f0 msgr2=0x7f8f180d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 --2- 172.21.15.31:0/4164497458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180b47f0 0x7f8f180d3bd0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f8f180d19a0 tx=0x7f8f0402a560 comp rx=0 tx=0).stop 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 -- 172.21.15.31:0/4164497458 shutdown_connections 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 --2- 172.21.15.31:0/4164497458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180d4110 0x7f8f181619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 --2- 172.21.15.31:0/4164497458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180b47f0 0x7f8f180d3bd0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 --2- 172.21.15.31:0/4164497458 >> v2:172.21.15.45:3300/0 conn(0x7f8f1815dfa0 0x7f8f180d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 -- 172.21.15.31:0/4164497458 >> 172.21.15.31:0/4164497458 conn(0x7f8f1808d8b0 msgr2=0x7f8f1809d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:37.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 -- 172.21.15.31:0/4164497458 shutdown_connections 2024-09-13T08:45:37.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.671+0000 7f8f1f000640 1 -- 172.21.15.31:0/4164497458 wait complete. 2024-09-13T08:45:37.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.672+0000 7f8f1f000640 1 Processor -- start 2024-09-13T08:45:37.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 -- start start 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180b47f0 0x7f8f18150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 0x7f8f181f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8f18162270 con 0x7f8f180d4110 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f181620f0 con 0x7f8f1815dfa0 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.673+0000 7f8f1f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8f181623f0 con 0x7f8f180b47f0 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45448/0 (socket says 172.21.15.31:45448) 2024-09-13T08:45:37.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 -- 172.21.15.31:0/3242841111 learned_addr learned my addr 172.21.15.31:0/3242841111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1dc00640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180b47f0 0x7f8f18150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1e600640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 0x7f8f181f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 -- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180b47f0 msgr2=0x7f8f18150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180b47f0 0x7f8f18150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 -- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 msgr2=0x7f8f181f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 0x7f8f181f4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1d200640 1 -- 172.21.15.31:0/3242841111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f180d0450 con 0x7f8f180d4110 2024-09-13T08:45:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.674+0000 7f8f1e600640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 0x7f8f181f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f1d200640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009850 tx=0x7f8f040039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0403f070 con 0x7f8f180d4110 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f04004890 con 0x7f8f180d4110 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f180d0730 con 0x7f8f180d4110 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04035030 con 0x7f8f180d4110 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.675+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f180d0cf0 con 0x7f8f180d4110 2024-09-13T08:45:37.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.676+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f181619a0 con 0x7f8f180d4110 2024-09-13T08:45:37.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.677+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f8f04042070 con 0x7f8f180d4110 2024-09-13T08:45:37.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.678+0000 7f8f15600640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 0x7f8ef807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:37.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.678+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f8f040bc440 con 0x7f8f180d4110 2024-09-13T08:45:37.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.678+0000 7f8f1dc00640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 0x7f8ef807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:37.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.679+0000 7f8f1dc00640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 0x7f8ef807b3d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005fd0 tx=0x7f8f0c00a250 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.682+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8f04083a50 con 0x7f8f180d4110 2024-09-13T08:45:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.874+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f180b9b10 con 0x7f8ef8078f10 2024-09-13T08:45:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.879+0000 7f8f15600640 1 -- 172.21.15.31:0/3242841111 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7f8f180b9b10 con 0x7f8ef8078f10 2024-09-13T08:45:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 msgr2=0x7f8ef807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 0x7f8ef807b3d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005fd0 tx=0x7f8f0c00a250 comp rx=0 tx=0).stop 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 msgr2=0x7f8f181f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009850 tx=0x7f8f040039e0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 shutdown_connections 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8ef8078f10 0x7f8ef807b3d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f1815dfa0 0x7f8f181f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f180d4110 0x7f8f181f4050 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 --2- 172.21.15.31:0/3242841111 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8f180b47f0 0x7f8f18150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 >> 172.21.15.31:0/3242841111 conn(0x7f8f1808d8b0 msgr2=0x7f8f180b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 shutdown_connections 2024-09-13T08:45:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:37.883+0000 7f8f1f000640 1 -- 172.21.15.31:0/3242841111 wait complete. 2024-09-13T08:45:37.900 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-13T08:45:37.005516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282116,"kb_used_data":15964,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692604,"statfs":{"total":773094113280,"available":772805226496,"internally_reserved":0,"allocated":16347136,"data_stored":7875972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.417095"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629893+0000","last_change":"2024-09-13T08:45:33.631458+0000","last_active":"2024-09-13T08:45:34.629893+0000","last_peered":"2024-09-13T08:45:34.629893+0000","last_clean":"2024-09-13T08:45:34.629893+0000","last_became_active":"2024-09-13T08:45:33.631198+0000","last_became_peered":"2024-09-13T08:45:33.631198+0000","last_unstale":"2024-09-13T08:45:34.629893+0000","last_undegraded":"2024-09-13T08:45:34.629893+0000","last_fullsized":"2024-09-13T08:45:34.629893+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:18:13.314925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630089+0000","last_change":"2024-09-13T08:45:33.620890+0000","last_active":"2024-09-13T08:45:34.630089+0000","last_peered":"2024-09-13T08:45:34.630089+0000","last_clean":"2024-09-13T08:45:34.630089+0000","last_became_active":"2024-09-13T08:45:33.620601+0000","last_became_peered":"2024-09-13T08:45:33.620601+0000","last_unstale":"2024-09-13T08:45:34.630089+0000","last_undegraded":"2024-09-13T08:45:34.630089+0000","last_fullsized":"2024-09-13T08:45:34.630089+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:09.666382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629921+0000","last_change":"2024-09-13T08:45:33.622332+0000","last_active":"2024-09-13T08:45:34.629921+0000","last_peered":"2024-09-13T08:45:34.629921+0000","last_clean":"2024-09-13T08:45:34.629921+0000","last_became_active":"2024-09-13T08:45:33.622054+0000","last_became_peered":"2024-09-13T08:45:33.622054+0000","last_unstale":"2024-09-13T08:45:34.629921+0000","last_undegraded":"2024-09-13T08:45:34.629921+0000","last_fullsized":"2024-09-13T08:45:34.629921+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:58.226891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618856+0000","last_change":"2024-09-13T08:45:33.619320+0000","last_active":"2024-09-13T08:45:34.618856+0000","last_peered":"2024-09-13T08:45:34.618856+0000","last_clean":"2024-09-13T08:45:34.618856+0000","last_became_active":"2024-09-13T08:45:33.618730+0000","last_became_peered":"2024-09-13T08:45:33.618730+0000","last_unstale":"2024-09-13T08:45:34.618856+0000","last_undegraded":"2024-09-13T08:45:34.618856+0000","last_fullsized":"2024-09-13T08:45:34.618856+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:16.453660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894470+0000","last_change":"2024-09-13T08:45:33.633860+0000","last_active":"2024-09-13T08:45:34.894470+0000","last_peered":"2024-09-13T08:45:34.894470+0000","last_clean":"2024-09-13T08:45:34.894470+0000","last_became_active":"2024-09-13T08:45:33.633564+0000","last_became_peered":"2024-09-13T08:45:33.633564+0000","last_unstale":"2024-09-13T08:45:34.894470+0000","last_undegraded":"2024-09-13T08:45:34.894470+0000","last_fullsized":"2024-09-13T08:45:34.894470+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:44.258133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619661+0000","last_change":"2024-09-13T08:45:33.620322+0000","last_active":"2024-09-13T08:45:34.619661+0000","last_peered":"2024-09-13T08:45:34.619661+0000","last_clean":"2024-09-13T08:45:34.619661+0000","last_became_active":"2024-09-13T08:45:33.620003+0000","last_became_peered":"2024-09-13T08:45:33.620003+0000","last_unstale":"2024-09-13T08:45:34.619661+0000","last_undegraded":"2024-09-13T08:45:34.619661+0000","last_fullsized":"2024-09-13T08:45:34.619661+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:16:28.449503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204382+0000","last_change":"2024-09-13T08:45:33.626916+0000","last_active":"2024-09-13T08:45:36.204382+0000","last_peered":"2024-09-13T08:45:36.204382+0000","last_clean":"2024-09-13T08:45:36.204382+0000","last_became_active":"2024-09-13T08:45:33.626414+0000","last_became_peered":"2024-09-13T08:45:33.626414+0000","last_unstale":"2024-09-13T08:45:36.204382+0000","last_undegraded":"2024-09-13T08:45:36.204382+0000","last_fullsized":"2024-09-13T08:45:36.204382+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:01.245998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203229+0000","last_change":"2024-09-13T08:45:33.618406+0000","last_active":"2024-09-13T08:45:36.203229+0000","last_peered":"2024-09-13T08:45:36.203229+0000","last_clean":"2024-09-13T08:45:36.203229+0000","last_became_active":"2024-09-13T08:45:33.617012+0000","last_became_peered":"2024-09-13T08:45:33.617012+0000","last_unstale":"2024-09-13T08:45:36.203229+0000","last_undegraded":"2024-09-13T08:45:36.203229+0000","last_fullsized":"2024-09-13T08:45:36.203229+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:38.020097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203377+0000","last_change":"2024-09-13T08:45:33.614597+0000","last_active":"2024-09-13T08:45:36.203377+0000","last_peered":"2024-09-13T08:45:36.203377+0000","last_clean":"2024-09-13T08:45:36.203377+0000","last_became_active":"2024-09-13T08:45:33.614275+0000","last_became_peered":"2024-09-13T08:45:33.614275+0000","last_unstale":"2024-09-13T08:45:36.203377+0000","last_undegraded":"2024-09-13T08:45:36.203377+0000","last_fullsized":"2024-09-13T08:45:36.203377+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:29.640042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203338+0000","last_change":"2024-09-13T08:45:33.624832+0000","last_active":"2024-09-13T08:45:36.203338+0000","last_peered":"2024-09-13T08:45:36.203338+0000","last_clean":"2024-09-13T08:45:36.203338+0000","last_became_active":"2024-09-13T08:45:33.624517+0000","last_became_peered":"2024-09-13T08:45:33.624517+0000","last_unstale":"2024-09-13T08:45:36.203338+0000","last_undegraded":"2024-09-13T08:45:36.203338+0000","last_fullsized":"2024-09-13T08:45:36.203338+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:46.512058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894696+0000","last_change":"2024-09-13T08:45:33.618707+0000","last_active":"2024-09-13T08:45:34.894696+0000","last_peered":"2024-09-13T08:45:34.894696+0000","last_clean":"2024-09-13T08:45:34.894696+0000","last_became_active":"2024-09-13T08:45:33.617802+0000","last_became_peered":"2024-09-13T08:45:33.617802+0000","last_unstale":"2024-09-13T08:45:34.894696+0000","last_undegraded":"2024-09-13T08:45:34.894696+0000","last_fullsized":"2024-09-13T08:45:34.894696+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:00.192533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617405+0000","last_change":"2024-09-13T08:45:33.627848+0000","last_active":"2024-09-13T08:45:34.617405+0000","last_peered":"2024-09-13T08:45:34.617405+0000","last_clean":"2024-09-13T08:45:34.617405+0000","last_became_active":"2024-09-13T08:45:33.626669+0000","last_became_peered":"2024-09-13T08:45:33.626669+0000","last_unstale":"2024-09-13T08:45:34.617405+0000","last_undegraded":"2024-09-13T08:45:34.617405+0000","last_fullsized":"2024-09-13T08:45:34.617405+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:25:23.372974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617381+0000","last_change":"2024-09-13T08:45:33.615945+0000","last_active":"2024-09-13T08:45:34.617381+0000","last_peered":"2024-09-13T08:45:34.617381+0000","last_clean":"2024-09-13T08:45:34.617381+0000","last_became_active":"2024-09-13T08:45:33.615609+0000","last_became_peered":"2024-09-13T08:45:33.615609+0000","last_unstale":"2024-09-13T08:45:34.617381+0000","last_undegraded":"2024-09-13T08:45:34.617381+0000","last_fullsized":"2024-09-13T08:45:34.617381+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:06.881884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203094+0000","last_change":"2024-09-13T08:45:33.631590+0000","last_active":"2024-09-13T08:45:36.203094+0000","last_peered":"2024-09-13T08:45:36.203094+0000","last_clean":"2024-09-13T08:45:36.203094+0000","last_became_active":"2024-09-13T08:45:33.631031+0000","last_became_peered":"2024-09-13T08:45:33.631031+0000","last_unstale":"2024-09-13T08:45:36.203094+0000","last_undegraded":"2024-09-13T08:45:36.203094+0000","last_fullsized":"2024-09-13T08:45:36.203094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:09.666382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203139+0000","last_change":"2024-09-13T08:45:33.615254+0000","last_active":"2024-09-13T08:45:36.203139+0000","last_peered":"2024-09-13T08:45:36.203139+0000","last_clean":"2024-09-13T08:45:36.203139+0000","last_became_active":"2024-09-13T08:45:33.614983+0000","last_became_peered":"2024-09-13T08:45:33.614983+0000","last_unstale":"2024-09-13T08:45:36.203139+0000","last_undegraded":"2024-09-13T08:45:36.203139+0000","last_fullsized":"2024-09-13T08:45:36.203139+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:22.375438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203999+0000","last_change":"2024-09-13T08:45:33.620887+0000","last_active":"2024-09-13T08:45:36.203999+0000","last_peered":"2024-09-13T08:45:36.203999+0000","last_clean":"2024-09-13T08:45:36.203999+0000","last_became_active":"2024-09-13T08:45:33.620607+0000","last_became_peered":"2024-09-13T08:45:33.620607+0000","last_unstale":"2024-09-13T08:45:36.203999+0000","last_undegraded":"2024-09-13T08:45:36.203999+0000","last_fullsized":"2024-09-13T08:45:36.203999+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.591042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618961+0000","last_change":"2024-09-13T08:45:33.621444+0000","last_active":"2024-09-13T08:45:34.618961+0000","last_peered":"2024-09-13T08:45:34.618961+0000","last_clean":"2024-09-13T08:45:34.618961+0000","last_became_active":"2024-09-13T08:45:33.621125+0000","last_became_peered":"2024-09-13T08:45:33.621125+0000","last_unstale":"2024-09-13T08:45:34.618961+0000","last_undegraded":"2024-09-13T08:45:34.618961+0000","last_fullsized":"2024-09-13T08:45:34.618961+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:41.533846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618328+0000","last_change":"2024-09-13T08:45:33.625473+0000","last_active":"2024-09-13T08:45:34.618328+0000","last_peered":"2024-09-13T08:45:34.618328+0000","last_clean":"2024-09-13T08:45:34.618328+0000","last_became_active":"2024-09-13T08:45:33.625194+0000","last_became_peered":"2024-09-13T08:45:33.625194+0000","last_unstale":"2024-09-13T08:45:34.618328+0000","last_undegraded":"2024-09-13T08:45:34.618328+0000","last_fullsized":"2024-09-13T08:45:34.618328+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:03.426038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205702+0000","last_change":"2024-09-13T08:45:33.625665+0000","last_active":"2024-09-13T08:45:36.205702+0000","last_peered":"2024-09-13T08:45:36.205702+0000","last_clean":"2024-09-13T08:45:36.205702+0000","last_became_active":"2024-09-13T08:45:33.625258+0000","last_became_peered":"2024-09-13T08:45:33.625258+0000","last_unstale":"2024-09-13T08:45:36.205702+0000","last_undegraded":"2024-09-13T08:45:36.205702+0000","last_fullsized":"2024-09-13T08:45:36.205702+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:27.096984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617457+0000","last_change":"2024-09-13T08:45:33.622254+0000","last_active":"2024-09-13T08:45:34.617457+0000","last_peered":"2024-09-13T08:45:34.617457+0000","last_clean":"2024-09-13T08:45:34.617457+0000","last_became_active":"2024-09-13T08:45:33.621966+0000","last_became_peered":"2024-09-13T08:45:33.621966+0000","last_unstale":"2024-09-13T08:45:34.617457+0000","last_undegraded":"2024-09-13T08:45:34.617457+0000","last_fullsized":"2024-09-13T08:45:34.617457+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:08.382851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619561+0000","last_change":"2024-09-13T08:45:33.616459+0000","last_active":"2024-09-13T08:45:34.619561+0000","last_peered":"2024-09-13T08:45:34.619561+0000","last_clean":"2024-09-13T08:45:34.619561+0000","last_became_active":"2024-09-13T08:45:33.616167+0000","last_became_peered":"2024-09-13T08:45:33.616167+0000","last_unstale":"2024-09-13T08:45:34.619561+0000","last_undegraded":"2024-09-13T08:45:34.619561+0000","last_fullsized":"2024-09-13T08:45:34.619561+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:02.806744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894423+0000","last_change":"2024-09-13T08:45:33.620595+0000","last_active":"2024-09-13T08:45:34.894423+0000","last_peered":"2024-09-13T08:45:34.894423+0000","last_clean":"2024-09-13T08:45:34.894423+0000","last_became_active":"2024-09-13T08:45:33.620328+0000","last_became_peered":"2024-09-13T08:45:33.620328+0000","last_unstale":"2024-09-13T08:45:34.894423+0000","last_undegraded":"2024-09-13T08:45:34.894423+0000","last_fullsized":"2024-09-13T08:45:34.894423+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:40.677013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204244+0000","last_change":"2024-09-13T08:45:33.621640+0000","last_active":"2024-09-13T08:45:36.204244+0000","last_peered":"2024-09-13T08:45:36.204244+0000","last_clean":"2024-09-13T08:45:36.204244+0000","last_became_active":"2024-09-13T08:45:33.620791+0000","last_became_peered":"2024-09-13T08:45:33.620791+0000","last_unstale":"2024-09-13T08:45:36.204244+0000","last_undegraded":"2024-09-13T08:45:36.204244+0000","last_fullsized":"2024-09-13T08:45:36.204244+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:20.182452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894335+0000","last_change":"2024-09-13T08:45:33.638184+0000","last_active":"2024-09-13T08:45:34.894335+0000","last_peered":"2024-09-13T08:45:34.894335+0000","last_clean":"2024-09-13T08:45:34.894335+0000","last_became_active":"2024-09-13T08:45:33.635223+0000","last_became_peered":"2024-09-13T08:45:33.635223+0000","last_unstale":"2024-09-13T08:45:34.894335+0000","last_undegraded":"2024-09-13T08:45:34.894335+0000","last_fullsized":"2024-09-13T08:45:34.894335+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:20.513044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894733+0000","last_change":"2024-09-13T08:45:33.622658+0000","last_active":"2024-09-13T08:45:34.894733+0000","last_peered":"2024-09-13T08:45:34.894733+0000","last_clean":"2024-09-13T08:45:34.894733+0000","last_became_active":"2024-09-13T08:45:33.622361+0000","last_became_peered":"2024-09-13T08:45:33.622361+0000","last_unstale":"2024-09-13T08:45:34.894733+0000","last_undegraded":"2024-09-13T08:45:34.894733+0000","last_fullsized":"2024-09-13T08:45:34.894733+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.567031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205342+0000","last_change":"2024-09-13T08:45:33.628316+0000","last_active":"2024-09-13T08:45:36.205342+0000","last_peered":"2024-09-13T08:45:36.205342+0000","last_clean":"2024-09-13T08:45:36.205342+0000","last_became_active":"2024-09-13T08:45:33.627994+0000","last_became_peered":"2024-09-13T08:45:33.627994+0000","last_unstale":"2024-09-13T08:45:36.205342+0000","last_undegraded":"2024-09-13T08:45:36.205342+0000","last_fullsized":"2024-09-13T08:45:36.205342+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:03.283784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894748+0000","last_change":"2024-09-13T08:45:33.636265+0000","last_active":"2024-09-13T08:45:34.894748+0000","last_peered":"2024-09-13T08:45:34.894748+0000","last_clean":"2024-09-13T08:45:34.894748+0000","last_became_active":"2024-09-13T08:45:33.635862+0000","last_became_peered":"2024-09-13T08:45:33.635862+0000","last_unstale":"2024-09-13T08:45:34.894748+0000","last_undegraded":"2024-09-13T08:45:34.894748+0000","last_fullsized":"2024-09-13T08:45:34.894748+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:21:24.800675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204665+0000","last_change":"2024-09-13T08:45:33.625565+0000","last_active":"2024-09-13T08:45:36.204665+0000","last_peered":"2024-09-13T08:45:36.204665+0000","last_clean":"2024-09-13T08:45:36.204665+0000","last_became_active":"2024-09-13T08:45:33.625186+0000","last_became_peered":"2024-09-13T08:45:33.625186+0000","last_unstale":"2024-09-13T08:45:36.204665+0000","last_undegraded":"2024-09-13T08:45:36.204665+0000","last_fullsized":"2024-09-13T08:45:36.204665+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:11.124390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629827+0000","last_change":"2024-09-13T08:45:33.621997+0000","last_active":"2024-09-13T08:45:34.629827+0000","last_peered":"2024-09-13T08:45:34.629827+0000","last_clean":"2024-09-13T08:45:34.629827+0000","last_became_active":"2024-09-13T08:45:33.621191+0000","last_became_peered":"2024-09-13T08:45:33.621191+0000","last_unstale":"2024-09-13T08:45:34.629827+0000","last_undegraded":"2024-09-13T08:45:34.629827+0000","last_fullsized":"2024-09-13T08:45:34.629827+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:50.966084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203363+0000","last_change":"2024-09-13T08:45:33.631285+0000","last_active":"2024-09-13T08:45:36.203363+0000","last_peered":"2024-09-13T08:45:36.203363+0000","last_clean":"2024-09-13T08:45:36.203363+0000","last_became_active":"2024-09-13T08:45:33.630642+0000","last_became_peered":"2024-09-13T08:45:33.630642+0000","last_unstale":"2024-09-13T08:45:36.203363+0000","last_undegraded":"2024-09-13T08:45:36.203363+0000","last_fullsized":"2024-09-13T08:45:36.203363+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:14.431761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629609+0000","last_change":"2024-09-13T08:45:33.610332+0000","last_active":"2024-09-13T08:45:34.629609+0000","last_peered":"2024-09-13T08:45:34.629609+0000","last_clean":"2024-09-13T08:45:34.629609+0000","last_became_active":"2024-09-13T08:45:33.610038+0000","last_became_peered":"2024-09-13T08:45:33.610038+0000","last_unstale":"2024-09-13T08:45:34.629609+0000","last_undegraded":"2024-09-13T08:45:34.629609+0000","last_fullsized":"2024-09-13T08:45:34.629609+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:11.124390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617966+0000","last_change":"2024-09-13T08:45:33.611932+0000","last_active":"2024-09-13T08:45:34.617966+0000","last_peered":"2024-09-13T08:45:34.617966+0000","last_clean":"2024-09-13T08:45:34.617966+0000","last_became_active":"2024-09-13T08:45:33.611635+0000","last_became_peered":"2024-09-13T08:45:33.611635+0000","last_unstale":"2024-09-13T08:45:34.617966+0000","last_undegraded":"2024-09-13T08:45:34.617966+0000","last_fullsized":"2024-09-13T08:45:34.617966+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:21:24.800675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619452+0000","last_change":"2024-09-13T08:45:33.619404+0000","last_active":"2024-09-13T08:45:34.619452+0000","last_peered":"2024-09-13T08:45:34.619452+0000","last_clean":"2024-09-13T08:45:34.619452+0000","last_became_active":"2024-09-13T08:45:33.619037+0000","last_became_peered":"2024-09-13T08:45:33.619037+0000","last_unstale":"2024-09-13T08:45:34.619452+0000","last_undegraded":"2024-09-13T08:45:34.619452+0000","last_fullsized":"2024-09-13T08:45:34.619452+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:46.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1f","version":"150'4","reported_seq":19,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629951+0000","last_change":"2024-09-13T08:45:32.621061+0000","last_active":"2024-09-13T08:45:34.629951+0000","last_peered":"2024-09-13T08:45:34.629951+0000","last_clean":"2024-09-13T08:45:34.629951+0000","last_became_active":"2024-09-13T08:45:32.620793+0000","last_became_peered":"2024-09-13T08:45:32.620793+0000","last_unstale":"2024-09-13T08:45:34.629951+0000","last_undegraded":"2024-09-13T08:45:34.629951+0000","last_fullsized":"2024-09-13T08:45:34.629951+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:51.771655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629962+0000","last_change":"2024-09-13T08:45:33.615156+0000","last_active":"2024-09-13T08:45:34.629962+0000","last_peered":"2024-09-13T08:45:34.629962+0000","last_clean":"2024-09-13T08:45:34.629962+0000","last_became_active":"2024-09-13T08:45:33.614860+0000","last_became_peered":"2024-09-13T08:45:33.614860+0000","last_unstale":"2024-09-13T08:45:34.629962+0000","last_undegraded":"2024-09-13T08:45:34.629962+0000","last_fullsized":"2024-09-13T08:45:34.629962+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:27.096984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203397+0000","last_change":"2024-09-13T08:45:32.620891+0000","last_active":"2024-09-13T08:45:36.203397+0000","last_peered":"2024-09-13T08:45:36.203397+0000","last_clean":"2024-09-13T08:45:36.203397+0000","last_became_active":"2024-09-13T08:45:32.620614+0000","last_became_peered":"2024-09-13T08:45:32.620614+0000","last_unstale":"2024-09-13T08:45:36.203397+0000","last_undegraded":"2024-09-13T08:45:36.203397+0000","last_fullsized":"2024-09-13T08:45:36.203397+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:29.582616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618808+0000","last_change":"2024-09-13T08:45:33.619592+0000","last_active":"2024-09-13T08:45:34.618808+0000","last_peered":"2024-09-13T08:45:34.618808+0000","last_clean":"2024-09-13T08:45:34.618808+0000","last_became_active":"2024-09-13T08:45:33.617503+0000","last_became_peered":"2024-09-13T08:45:33.617503+0000","last_unstale":"2024-09-13T08:45:34.618808+0000","last_undegraded":"2024-09-13T08:45:34.618808+0000","last_fullsized":"2024-09-13T08:45:34.618808+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:14.761774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618917+0000","last_change":"2024-09-13T08:45:32.613395+0000","last_active":"2024-09-13T08:45:34.618917+0000","last_peered":"2024-09-13T08:45:34.618917+0000","last_clean":"2024-09-13T08:45:34.618917+0000","last_became_active":"2024-09-13T08:45:32.613050+0000","last_became_peered":"2024-09-13T08:45:32.613050+0000","last_unstale":"2024-09-13T08:45:34.618917+0000","last_undegraded":"2024-09-13T08:45:34.618917+0000","last_fullsized":"2024-09-13T08:45:34.618917+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:33.672884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203335+0000","last_change":"2024-09-13T08:45:33.616470+0000","last_active":"2024-09-13T08:45:36.203335+0000","last_peered":"2024-09-13T08:45:36.203335+0000","last_clean":"2024-09-13T08:45:36.203335+0000","last_became_active":"2024-09-13T08:45:33.616053+0000","last_became_peered":"2024-09-13T08:45:33.616053+0000","last_unstale":"2024-09-13T08:45:36.203335+0000","last_undegraded":"2024-09-13T08:45:36.203335+0000","last_fullsized":"2024-09-13T08:45:36.203335+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:56:04.012311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1c","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205067+0000","last_change":"2024-09-13T08:45:32.618389+0000","last_active":"2024-09-13T08:45:36.205067+0000","last_peered":"2024-09-13T08:45:36.205067+0000","last_clean":"2024-09-13T08:45:36.205067+0000","last_became_active":"2024-09-13T08:45:32.617588+0000","last_became_peered":"2024-09-13T08:45:32.617588+0000","last_unstale":"2024-09-13T08:45:36.205067+0000","last_undegraded":"2024-09-13T08:45:36.205067+0000","last_fullsized":"2024-09-13T08:45:36.205067+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:21.548606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618132+0000","last_change":"2024-09-13T08:45:33.621587+0000","last_active":"2024-09-13T08:45:34.618132+0000","last_peered":"2024-09-13T08:45:34.618132+0000","last_clean":"2024-09-13T08:45:34.618132+0000","last_became_active":"2024-09-13T08:45:33.620130+0000","last_became_peered":"2024-09-13T08:45:33.620130+0000","last_unstale":"2024-09-13T08:45:34.618132+0000","last_undegraded":"2024-09-13T08:45:34.618132+0000","last_fullsized":"2024-09-13T08:45:34.618132+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:43.284644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629685+0000","last_change":"2024-09-13T08:45:32.610935+0000","last_active":"2024-09-13T08:45:34.629685+0000","last_peered":"2024-09-13T08:45:34.629685+0000","last_clean":"2024-09-13T08:45:34.629685+0000","last_became_active":"2024-09-13T08:45:32.610241+0000","last_became_peered":"2024-09-13T08:45:32.610241+0000","last_unstale":"2024-09-13T08:45:34.629685+0000","last_undegraded":"2024-09-13T08:45:34.629685+0000","last_fullsized":"2024-09-13T08:45:34.629685+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:38.326122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619571+0000","last_change":"2024-09-13T08:45:33.616609+0000","last_active":"2024-09-13T08:45:34.619571+0000","last_peered":"2024-09-13T08:45:34.619571+0000","last_clean":"2024-09-13T08:45:34.619571+0000","last_became_active":"2024-09-13T08:45:33.616081+0000","last_became_peered":"2024-09-13T08:45:33.616081+0000","last_unstale":"2024-09-13T08:45:34.619571+0000","last_undegraded":"2024-09-13T08:45:34.619571+0000","last_fullsized":"2024-09-13T08:45:34.619571+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:43.691761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1a","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618221+0000","last_change":"2024-09-13T08:45:32.613940+0000","last_active":"2024-09-13T08:45:34.618221+0000","last_peered":"2024-09-13T08:45:34.618221+0000","last_clean":"2024-09-13T08:45:34.618221+0000","last_became_active":"2024-09-13T08:45:32.613645+0000","last_became_peered":"2024-09-13T08:45:32.613645+0000","last_unstale":"2024-09-13T08:45:34.618221+0000","last_undegraded":"2024-09-13T08:45:34.618221+0000","last_fullsized":"2024-09-13T08:45:34.618221+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:14.080253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204833+0000","last_change":"2024-09-13T08:45:33.621259+0000","last_active":"2024-09-13T08:45:36.204833+0000","last_peered":"2024-09-13T08:45:36.204833+0000","last_clean":"2024-09-13T08:45:36.204833+0000","last_became_active":"2024-09-13T08:45:33.620982+0000","last_became_peered":"2024-09-13T08:45:33.620982+0000","last_unstale":"2024-09-13T08:45:36.204833+0000","last_undegraded":"2024-09-13T08:45:36.204833+0000","last_fullsized":"2024-09-13T08:45:36.204833+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:44:42.739868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629615+0000","last_change":"2024-09-13T08:45:32.620940+0000","last_active":"2024-09-13T08:45:34.629615+0000","last_peered":"2024-09-13T08:45:34.629615+0000","last_clean":"2024-09-13T08:45:34.629615+0000","last_became_active":"2024-09-13T08:45:32.620654+0000","last_became_peered":"2024-09-13T08:45:32.620654+0000","last_unstale":"2024-09-13T08:45:34.629615+0000","last_undegraded":"2024-09-13T08:45:34.629615+0000","last_fullsized":"2024-09-13T08:45:34.629615+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:06:15.310795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617875+0000","last_change":"2024-09-13T08:45:33.622411+0000","last_active":"2024-09-13T08:45:34.617875+0000","last_peered":"2024-09-13T08:45:34.617875+0000","last_clean":"2024-09-13T08:45:34.617875+0000","last_became_active":"2024-09-13T08:45:33.619412+0000","last_became_peered":"2024-09-13T08:45:33.619412+0000","last_unstale":"2024-09-13T08:45:34.617875+0000","last_undegraded":"2024-09-13T08:45:34.617875+0000","last_fullsized":"2024-09-13T08:45:34.617875+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:02.002010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203817+0000","last_change":"2024-09-13T08:45:32.619147+0000","last_active":"2024-09-13T08:45:36.203817+0000","last_peered":"2024-09-13T08:45:36.203817+0000","last_clean":"2024-09-13T08:45:36.203817+0000","last_became_active":"2024-09-13T08:45:32.618840+0000","last_became_peered":"2024-09-13T08:45:32.618840+0000","last_unstale":"2024-09-13T08:45:36.203817+0000","last_undegraded":"2024-09-13T08:45:36.203817+0000","last_fullsized":"2024-09-13T08:45:36.203817+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:43:33.945255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205589+0000","last_change":"2024-09-13T08:45:33.629576+0000","last_active":"2024-09-13T08:45:36.205589+0000","last_peered":"2024-09-13T08:45:36.205589+0000","last_clean":"2024-09-13T08:45:36.205589+0000","last_became_active":"2024-09-13T08:45:33.629297+0000","last_became_peered":"2024-09-13T08:45:33.629297+0000","last_unstale":"2024-09-13T08:45:36.205589+0000","last_undegraded":"2024-09-13T08:45:36.205589+0000","last_fullsized":"2024-09-13T08:45:36.205589+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:58.226891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619201+0000","last_change":"2024-09-13T08:45:32.621412+0000","last_active":"2024-09-13T08:45:34.619201+0000","last_peered":"2024-09-13T08:45:34.619201+0000","last_clean":"2024-09-13T08:45:34.619201+0000","last_became_active":"2024-09-13T08:45:32.621011+0000","last_became_peered":"2024-09-13T08:45:32.621011+0000","last_unstale":"2024-09-13T08:45:34.619201+0000","last_undegraded":"2024-09-13T08:45:34.619201+0000","last_fullsized":"2024-09-13T08:45:34.619201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:28.537636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204947+0000","last_change":"2024-09-13T08:45:33.628352+0000","last_active":"2024-09-13T08:45:36.204947+0000","last_peered":"2024-09-13T08:45:36.204947+0000","last_clean":"2024-09-13T08:45:36.204947+0000","last_became_active":"2024-09-13T08:45:33.627871+0000","last_became_peered":"2024-09-13T08:45:33.627871+0000","last_unstale":"2024-09-13T08:45:36.204947+0000","last_undegraded":"2024-09-13T08:45:36.204947+0000","last_fullsized":"2024-09-13T08:45:36.204947+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:40.374360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.15","version":"150'3","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205086+0000","last_change":"2024-09-13T08:45:32.622371+0000","last_active":"2024-09-13T08:45:36.205086+0000","last_peered":"2024-09-13T08:45:36.205086+0000","last_clean":"2024-09-13T08:45:36.205086+0000","last_became_active":"2024-09-13T08:45:32.622062+0000","last_became_peered":"2024-09-13T08:45:32.622062+0000","last_unstale":"2024-09-13T08:45:36.205086+0000","last_undegraded":"2024-09-13T08:45:36.205086+0000","last_fullsized":"2024-09-13T08:45:36.205086+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:00.615811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630121+0000","last_change":"2024-09-13T08:45:33.621415+0000","last_active":"2024-09-13T08:45:34.630121+0000","last_peered":"2024-09-13T08:45:34.630121+0000","last_clean":"2024-09-13T08:45:34.630121+0000","last_became_active":"2024-09-13T08:45:33.620745+0000","last_became_peered":"2024-09-13T08:45:33.620745+0000","last_unstale":"2024-09-13T08:45:34.630121+0000","last_undegraded":"2024-09-13T08:45:34.630121+0000","last_fullsized":"2024-09-13T08:45:34.630121+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:22.375438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.14","version":"150'6","reported_seq":21,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894469+0000","last_change":"2024-09-13T08:45:32.622424+0000","last_active":"2024-09-13T08:45:34.894469+0000","last_peered":"2024-09-13T08:45:34.894469+0000","last_clean":"2024-09-13T08:45:34.894469+0000","last_became_active":"2024-09-13T08:45:32.622090+0000","last_became_peered":"2024-09-13T08:45:32.622090+0000","last_unstale":"2024-09-13T08:45:34.894469+0000","last_undegraded":"2024-09-13T08:45:34.894469+0000","last_fullsized":"2024-09-13T08:45:34.894469+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:51.838355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204373+0000","last_change":"2024-09-13T08:45:33.619435+0000","last_active":"2024-09-13T08:45:36.204373+0000","last_peered":"2024-09-13T08:45:36.204373+0000","last_clean":"2024-09-13T08:45:36.204373+0000","last_became_active":"2024-09-13T08:45:33.619146+0000","last_became_peered":"2024-09-13T08:45:33.619146+0000","last_unstale":"2024-09-13T08:45:36.204373+0000","last_undegraded":"2024-09-13T08:45:36.204373+0000","last_fullsized":"2024-09-13T08:45:36.204373+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:44.238924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619144+0000","last_change":"2024-09-13T08:45:32.615626+0000","last_active":"2024-09-13T08:45:34.619144+0000","last_peered":"2024-09-13T08:45:34.619144+0000","last_clean":"2024-09-13T08:45:34.619144+0000","last_became_active":"2024-09-13T08:45:32.615358+0000","last_became_peered":"2024-09-13T08:45:32.615358+0000","last_unstale":"2024-09-13T08:45:34.619144+0000","last_undegraded":"2024-09-13T08:45:34.619144+0000","last_fullsized":"2024-09-13T08:45:34.619144+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:31.355664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204581+0000","last_change":"2024-09-13T08:45:33.624271+0000","last_active":"2024-09-13T08:45:36.204581+0000","last_peered":"2024-09-13T08:45:36.204581+0000","last_clean":"2024-09-13T08:45:36.204581+0000","last_became_active":"2024-09-13T08:45:33.623806+0000","last_became_peered":"2024-09-13T08:45:33.623806+0000","last_unstale":"2024-09-13T08:45:36.204581+0000","last_undegraded":"2024-09-13T08:45:36.204581+0000","last_fullsized":"2024-09-13T08:45:36.204581+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:56.955592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618781+0000","last_change":"2024-09-13T08:45:32.620219+0000","last_active":"2024-09-13T08:45:34.618781+0000","last_peered":"2024-09-13T08:45:34.618781+0000","last_clean":"2024-09-13T08:45:34.618781+0000","last_became_active":"2024-09-13T08:45:32.619750+0000","last_became_peered":"2024-09-13T08:45:32.619750+0000","last_unstale":"2024-09-13T08:45:34.618781+0000","last_undegraded":"2024-09-13T08:45:34.618781+0000","last_fullsized":"2024-09-13T08:45:34.618781+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:36.889900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618198+0000","last_change":"2024-09-13T08:45:33.619472+0000","last_active":"2024-09-13T08:45:34.618198+0000","last_peered":"2024-09-13T08:45:34.618198+0000","last_clean":"2024-09-13T08:45:34.618198+0000","last_became_active":"2024-09-13T08:45:33.617767+0000","last_became_peered":"2024-09-13T08:45:33.617767+0000","last_unstale":"2024-09-13T08:45:34.618198+0000","last_undegraded":"2024-09-13T08:45:34.618198+0000","last_fullsized":"2024-09-13T08:45:34.618198+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:29.210658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204253+0000","last_change":"2024-09-13T08:45:32.617190+0000","last_active":"2024-09-13T08:45:36.204253+0000","last_peered":"2024-09-13T08:45:36.204253+0000","last_clean":"2024-09-13T08:45:36.204253+0000","last_became_active":"2024-09-13T08:45:32.616867+0000","last_became_peered":"2024-09-13T08:45:32.616867+0000","last_unstale":"2024-09-13T08:45:36.204253+0000","last_undegraded":"2024-09-13T08:45:36.204253+0000","last_fullsized":"2024-09-13T08:45:36.204253+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:42.407504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203314+0000","last_change":"2024-09-13T08:45:33.625673+0000","last_active":"2024-09-13T08:45:36.203314+0000","last_peered":"2024-09-13T08:45:36.203314+0000","last_clean":"2024-09-13T08:45:36.203314+0000","last_became_active":"2024-09-13T08:45:33.625394+0000","last_became_peered":"2024-09-13T08:45:33.625394+0000","last_unstale":"2024-09-13T08:45:36.203314+0000","last_undegraded":"2024-09-13T08:45:36.203314+0000","last_fullsized":"2024-09-13T08:45:36.203314+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:48.509209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"150'8","reported_seq":23,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894233+0000","last_change":"2024-09-13T08:45:32.616810+0000","last_active":"2024-09-13T08:45:34.894233+0000","last_peered":"2024-09-13T08:45:34.894233+0000","last_clean":"2024-09-13T08:45:34.894233+0000","last_became_active":"2024-09-13T08:45:32.615280+0000","last_became_peered":"2024-09-13T08:45:32.615280+0000","last_unstale":"2024-09-13T08:45:34.894233+0000","last_undegraded":"2024-09-13T08:45:34.894233+0000","last_fullsized":"2024-09-13T08:45:34.894233+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:08.292180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205511+0000","last_change":"2024-09-13T08:45:33.633537+0000","last_active":"2024-09-13T08:45:36.205511+0000","last_peered":"2024-09-13T08:45:36.205511+0000","last_clean":"2024-09-13T08:45:36.205511+0000","last_became_active":"2024-09-13T08:45:33.632783+0000","last_became_peered":"2024-09-13T08:45:33.632783+0000","last_unstale":"2024-09-13T08:45:36.205511+0000","last_undegraded":"2024-09-13T08:45:36.205511+0000","last_fullsized":"2024-09-13T08:45:36.205511+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:50.966084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894694+0000","last_change":"2024-09-13T08:45:32.615167+0000","last_active":"2024-09-13T08:45:34.894694+0000","last_peered":"2024-09-13T08:45:34.894694+0000","last_clean":"2024-09-13T08:45:34.894694+0000","last_became_active":"2024-09-13T08:45:32.613724+0000","last_became_peered":"2024-09-13T08:45:32.613724+0000","last_unstale":"2024-09-13T08:45:34.894694+0000","last_undegraded":"2024-09-13T08:45:34.894694+0000","last_fullsized":"2024-09-13T08:45:34.894694+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:09.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619168+0000","last_change":"2024-09-13T08:45:33.621259+0000","last_active":"2024-09-13T08:45:34.619168+0000","last_peered":"2024-09-13T08:45:34.619168+0000","last_clean":"2024-09-13T08:45:34.619168+0000","last_became_active":"2024-09-13T08:45:33.620942+0000","last_became_peered":"2024-09-13T08:45:33.620942+0000","last_unstale":"2024-09-13T08:45:34.619168+0000","last_undegraded":"2024-09-13T08:45:34.619168+0000","last_fullsized":"2024-09-13T08:45:34.619168+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:19.918580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203448+0000","last_change":"2024-09-13T08:45:32.611972+0000","last_active":"2024-09-13T08:45:36.203448+0000","last_peered":"2024-09-13T08:45:36.203448+0000","last_clean":"2024-09-13T08:45:36.203448+0000","last_became_active":"2024-09-13T08:45:32.611131+0000","last_became_peered":"2024-09-13T08:45:32.611131+0000","last_unstale":"2024-09-13T08:45:36.203448+0000","last_undegraded":"2024-09-13T08:45:36.203448+0000","last_fullsized":"2024-09-13T08:45:36.203448+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:19.487842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619094+0000","last_change":"2024-09-13T08:45:33.615094+0000","last_active":"2024-09-13T08:45:34.619094+0000","last_peered":"2024-09-13T08:45:34.619094+0000","last_clean":"2024-09-13T08:45:34.619094+0000","last_became_active":"2024-09-13T08:45:33.614707+0000","last_became_peered":"2024-09-13T08:45:33.614707+0000","last_unstale":"2024-09-13T08:45:34.619094+0000","last_undegraded":"2024-09-13T08:45:34.619094+0000","last_fullsized":"2024-09-13T08:45:34.619094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:52.686751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.d","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894734+0000","last_change":"2024-09-13T08:45:32.619476+0000","last_active":"2024-09-13T08:45:34.894734+0000","last_peered":"2024-09-13T08:45:34.894734+0000","last_clean":"2024-09-13T08:45:34.894734+0000","last_became_active":"2024-09-13T08:45:32.619104+0000","last_became_peered":"2024-09-13T08:45:32.619104+0000","last_unstale":"2024-09-13T08:45:34.894734+0000","last_undegraded":"2024-09-13T08:45:34.894734+0000","last_fullsized":"2024-09-13T08:45:34.894734+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:14.006254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205223+0000","last_change":"2024-09-13T08:45:33.623632+0000","last_active":"2024-09-13T08:45:36.205223+0000","last_peered":"2024-09-13T08:45:36.205223+0000","last_clean":"2024-09-13T08:45:36.205223+0000","last_became_active":"2024-09-13T08:45:33.623316+0000","last_became_peered":"2024-09-13T08:45:33.623316+0000","last_unstale":"2024-09-13T08:45:36.205223+0000","last_undegraded":"2024-09-13T08:45:36.205223+0000","last_fullsized":"2024-09-13T08:45:36.205223+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:53.922270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617440+0000","last_change":"2024-09-13T08:45:32.617286+0000","last_active":"2024-09-13T08:45:34.617440+0000","last_peered":"2024-09-13T08:45:34.617440+0000","last_clean":"2024-09-13T08:45:34.617440+0000","last_became_active":"2024-09-13T08:45:32.617005+0000","last_became_peered":"2024-09-13T08:45:32.617005+0000","last_unstale":"2024-09-13T08:45:34.617440+0000","last_undegraded":"2024-09-13T08:45:34.617440+0000","last_fullsized":"2024-09-13T08:45:34.617440+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:16:42.655224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630392+0000","last_change":"2024-09-13T08:45:33.612130+0000","last_active":"2024-09-13T08:45:34.630392+0000","last_peered":"2024-09-13T08:45:34.630392+0000","last_clean":"2024-09-13T08:45:34.630392+0000","last_became_active":"2024-09-13T08:45:33.611708+0000","last_became_peered":"2024-09-13T08:45:33.611708+0000","last_unstale":"2024-09-13T08:45:34.630392+0000","last_undegraded":"2024-09-13T08:45:34.630392+0000","last_fullsized":"2024-09-13T08:45:34.630392+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:38.020097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.b","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204655+0000","last_change":"2024-09-13T08:45:32.621453+0000","last_active":"2024-09-13T08:45:36.204655+0000","last_peered":"2024-09-13T08:45:36.204655+0000","last_clean":"2024-09-13T08:45:36.204655+0000","last_became_active":"2024-09-13T08:45:32.619431+0000","last_became_peered":"2024-09-13T08:45:32.619431+0000","last_unstale":"2024-09-13T08:45:36.204655+0000","last_undegraded":"2024-09-13T08:45:36.204655+0000","last_fullsized":"2024-09-13T08:45:36.204655+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:34.535933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204636+0000","last_change":"2024-09-13T08:45:33.632239+0000","last_active":"2024-09-13T08:45:36.204636+0000","last_peered":"2024-09-13T08:45:36.204636+0000","last_clean":"2024-09-13T08:45:36.204636+0000","last_became_active":"2024-09-13T08:45:33.631948+0000","last_became_peered":"2024-09-13T08:45:33.631948+0000","last_unstale":"2024-09-13T08:45:36.204636+0000","last_undegraded":"2024-09-13T08:45:36.204636+0000","last_fullsized":"2024-09-13T08:45:36.204636+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:34.449653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.616960+0000","last_change":"2024-09-13T08:45:32.612100+0000","last_active":"2024-09-13T08:45:34.616960+0000","last_peered":"2024-09-13T08:45:34.616960+0000","last_clean":"2024-09-13T08:45:34.616960+0000","last_became_active":"2024-09-13T08:45:32.611834+0000","last_became_peered":"2024-09-13T08:45:32.611834+0000","last_unstale":"2024-09-13T08:45:34.616960+0000","last_undegraded":"2024-09-13T08:45:34.616960+0000","last_fullsized":"2024-09-13T08:45:34.616960+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:57.319688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629527+0000","last_change":"2024-09-13T08:45:33.619203+0000","last_active":"2024-09-13T08:45:34.629527+0000","last_peered":"2024-09-13T08:45:34.629527+0000","last_clean":"2024-09-13T08:45:34.629527+0000","last_became_active":"2024-09-13T08:45:33.618868+0000","last_became_peered":"2024-09-13T08:45:33.618868+0000","last_unstale":"2024-09-13T08:45:34.629527+0000","last_undegraded":"2024-09-13T08:45:34.629527+0000","last_fullsized":"2024-09-13T08:45:34.629527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:44.238924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617743+0000","last_change":"2024-09-13T08:45:32.614951+0000","last_active":"2024-09-13T08:45:34.617743+0000","last_peered":"2024-09-13T08:45:34.617743+0000","last_clean":"2024-09-13T08:45:34.617743+0000","last_became_active":"2024-09-13T08:45:32.614589+0000","last_became_peered":"2024-09-13T08:45:32.614589+0000","last_unstale":"2024-09-13T08:45:34.617743+0000","last_undegraded":"2024-09-13T08:45:34.617743+0000","last_fullsized":"2024-09-13T08:45:34.617743+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:37.315527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204129+0000","last_change":"2024-09-13T08:45:33.636399+0000","last_active":"2024-09-13T08:45:36.204129+0000","last_peered":"2024-09-13T08:45:36.204129+0000","last_clean":"2024-09-13T08:45:36.204129+0000","last_became_active":"2024-09-13T08:45:33.636041+0000","last_became_peered":"2024-09-13T08:45:33.636041+0000","last_unstale":"2024-09-13T08:45:36.204129+0000","last_undegraded":"2024-09-13T08:45:36.204129+0000","last_fullsized":"2024-09-13T08:45:36.204129+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:57.520787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617110+0000","last_change":"2024-09-13T08:45:32.621572+0000","last_active":"2024-09-13T08:45:34.617110+0000","last_peered":"2024-09-13T08:45:34.617110+0000","last_clean":"2024-09-13T08:45:34.617110+0000","last_became_active":"2024-09-13T08:45:32.621303+0000","last_became_peered":"2024-09-13T08:45:32.621303+0000","last_unstale":"2024-09-13T08:45:34.617110+0000","last_undegraded":"2024-09-13T08:45:34.617110+0000","last_fullsized":"2024-09-13T08:45:34.617110+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:16.977123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205496+0000","last_change":"2024-09-13T08:45:33.626962+0000","last_active":"2024-09-13T08:45:36.205496+0000","last_peered":"2024-09-13T08:45:36.205496+0000","last_clean":"2024-09-13T08:45:36.205496+0000","last_became_active":"2024-09-13T08:45:33.626357+0000","last_became_peered":"2024-09-13T08:45:33.626357+0000","last_unstale":"2024-09-13T08:45:36.205496+0000","last_undegraded":"2024-09-13T08:45:36.205496+0000","last_fullsized":"2024-09-13T08:45:36.205496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:21:09.054648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618931+0000","last_change":"2024-09-13T08:45:32.617486+0000","last_active":"2024-09-13T08:45:34.618931+0000","last_peered":"2024-09-13T08:45:34.618931+0000","last_clean":"2024-09-13T08:45:34.618931+0000","last_became_active":"2024-09-13T08:45:32.616893+0000","last_became_peered":"2024-09-13T08:45:32.616893+0000","last_unstale":"2024-09-13T08:45:34.618931+0000","last_undegraded":"2024-09-13T08:45:34.618931+0000","last_fullsized":"2024-09-13T08:45:34.618931+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:52.852645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619443+0000","last_change":"2024-09-13T08:45:33.618697+0000","last_active":"2024-09-13T08:45:34.619443+0000","last_peered":"2024-09-13T08:45:34.619443+0000","last_clean":"2024-09-13T08:45:34.619443+0000","last_became_active":"2024-09-13T08:45:33.618209+0000","last_became_peered":"2024-09-13T08:45:33.618209+0000","last_unstale":"2024-09-13T08:45:34.619443+0000","last_undegraded":"2024-09-13T08:45:34.619443+0000","last_fullsized":"2024-09-13T08:45:34.619443+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:57.571587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.6","version":"151'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617414+0000","last_change":"2024-09-13T08:45:32.621813+0000","last_active":"2024-09-13T08:45:34.617414+0000","last_peered":"2024-09-13T08:45:34.617414+0000","last_clean":"2024-09-13T08:45:34.617414+0000","last_became_active":"2024-09-13T08:45:32.621424+0000","last_became_peered":"2024-09-13T08:45:32.621424+0000","last_unstale":"2024-09-13T08:45:34.617414+0000","last_undegraded":"2024-09-13T08:45:34.617414+0000","last_fullsized":"2024-09-13T08:45:34.617414+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:51:20.741097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619333+0000","last_change":"2024-09-13T08:45:33.622422+0000","last_active":"2024-09-13T08:45:34.619333+0000","last_peered":"2024-09-13T08:45:34.619333+0000","last_clean":"2024-09-13T08:45:34.619333+0000","last_became_active":"2024-09-13T08:45:33.622131+0000","last_became_peered":"2024-09-13T08:45:33.622131+0000","last_unstale":"2024-09-13T08:45:34.619333+0000","last_undegraded":"2024-09-13T08:45:34.619333+0000","last_fullsized":"2024-09-13T08:45:34.619333+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:02:33.435544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630238+0000","last_change":"2024-09-13T08:45:32.614709+0000","last_active":"2024-09-13T08:45:34.630238+0000","last_peered":"2024-09-13T08:45:34.630238+0000","last_clean":"2024-09-13T08:45:34.630238+0000","last_became_active":"2024-09-13T08:45:32.614407+0000","last_became_peered":"2024-09-13T08:45:32.614407+0000","last_unstale":"2024-09-13T08:45:34.630238+0000","last_undegraded":"2024-09-13T08:45:34.630238+0000","last_fullsized":"2024-09-13T08:45:34.630238+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:41.458139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618146+0000","last_change":"2024-09-13T08:45:33.616093+0000","last_active":"2024-09-13T08:45:34.618146+0000","last_peered":"2024-09-13T08:45:34.618146+0000","last_clean":"2024-09-13T08:45:34.618146+0000","last_became_active":"2024-09-13T08:45:33.614646+0000","last_became_peered":"2024-09-13T08:45:33.614646+0000","last_unstale":"2024-09-13T08:45:34.618146+0000","last_undegraded":"2024-09-13T08:45:34.618146+0000","last_fullsized":"2024-09-13T08:45:34.618146+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:21.334184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204955+0000","last_change":"2024-09-13T08:45:32.621977+0000","last_active":"2024-09-13T08:45:36.204955+0000","last_peered":"2024-09-13T08:45:36.204955+0000","last_clean":"2024-09-13T08:45:36.204955+0000","last_became_active":"2024-09-13T08:45:32.621650+0000","last_became_peered":"2024-09-13T08:45:32.621650+0000","last_unstale":"2024-09-13T08:45:36.204955+0000","last_undegraded":"2024-09-13T08:45:36.204955+0000","last_fullsized":"2024-09-13T08:45:36.204955+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:42.407504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618711+0000","last_change":"2024-09-13T08:45:33.611854+0000","last_active":"2024-09-13T08:45:34.618711+0000","last_peered":"2024-09-13T08:45:34.618711+0000","last_clean":"2024-09-13T08:45:34.618711+0000","last_became_active":"2024-09-13T08:45:33.611530+0000","last_became_peered":"2024-09-13T08:45:33.611530+0000","last_unstale":"2024-09-13T08:45:34.618711+0000","last_undegraded":"2024-09-13T08:45:34.618711+0000","last_fullsized":"2024-09-13T08:45:34.618711+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:46.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894408+0000","last_change":"2024-09-13T08:45:32.616012+0000","last_active":"2024-09-13T08:45:34.894408+0000","last_peered":"2024-09-13T08:45:34.894408+0000","last_clean":"2024-09-13T08:45:34.894408+0000","last_became_active":"2024-09-13T08:45:32.615746+0000","last_became_peered":"2024-09-13T08:45:32.615746+0000","last_unstale":"2024-09-13T08:45:34.894408+0000","last_undegraded":"2024-09-13T08:45:34.894408+0000","last_fullsized":"2024-09-13T08:45:34.894408+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:29.505007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629615+0000","last_change":"2024-09-13T08:45:33.629666+0000","last_active":"2024-09-13T08:45:34.629615+0000","last_peered":"2024-09-13T08:45:34.629615+0000","last_clean":"2024-09-13T08:45:34.629615+0000","last_became_active":"2024-09-13T08:45:33.629353+0000","last_became_peered":"2024-09-13T08:45:33.629353+0000","last_unstale":"2024-09-13T08:45:34.629615+0000","last_undegraded":"2024-09-13T08:45:34.629615+0000","last_fullsized":"2024-09-13T08:45:34.629615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:56.955592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204420+0000","last_change":"2024-09-13T08:45:32.615086+0000","last_active":"2024-09-13T08:45:36.204420+0000","last_peered":"2024-09-13T08:45:36.204420+0000","last_clean":"2024-09-13T08:45:36.204420+0000","last_became_active":"2024-09-13T08:45:32.614772+0000","last_became_peered":"2024-09-13T08:45:32.614772+0000","last_unstale":"2024-09-13T08:45:36.204420+0000","last_undegraded":"2024-09-13T08:45:36.204420+0000","last_fullsized":"2024-09-13T08:45:36.204420+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:21.548606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204070+0000","last_change":"2024-09-13T08:45:33.626564+0000","last_active":"2024-09-13T08:45:36.204070+0000","last_peered":"2024-09-13T08:45:36.204070+0000","last_clean":"2024-09-13T08:45:36.204070+0000","last_became_active":"2024-09-13T08:45:33.626279+0000","last_became_peered":"2024-09-13T08:45:33.626279+0000","last_unstale":"2024-09-13T08:45:36.204070+0000","last_undegraded":"2024-09-13T08:45:36.204070+0000","last_fullsized":"2024-09-13T08:45:36.204070+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:20.496268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629689+0000","last_change":"2024-09-13T08:45:32.615367+0000","last_active":"2024-09-13T08:45:34.629689+0000","last_peered":"2024-09-13T08:45:34.629689+0000","last_clean":"2024-09-13T08:45:34.629689+0000","last_became_active":"2024-09-13T08:45:32.614975+0000","last_became_peered":"2024-09-13T08:45:32.614975+0000","last_unstale":"2024-09-13T08:45:34.629689+0000","last_undegraded":"2024-09-13T08:45:34.629689+0000","last_fullsized":"2024-09-13T08:45:34.629689+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:10:16.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629550+0000","last_change":"2024-09-13T08:45:33.619392+0000","last_active":"2024-09-13T08:45:34.629550+0000","last_peered":"2024-09-13T08:45:34.629550+0000","last_clean":"2024-09-13T08:45:34.629550+0000","last_became_active":"2024-09-13T08:45:33.619085+0000","last_became_peered":"2024-09-13T08:45:33.619085+0000","last_unstale":"2024-09-13T08:45:34.629550+0000","last_undegraded":"2024-09-13T08:45:34.629550+0000","last_fullsized":"2024-09-13T08:45:34.629550+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:01.245998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629330+0000","last_change":"2024-09-13T08:45:32.607846+0000","last_active":"2024-09-13T08:45:34.629330+0000","last_peered":"2024-09-13T08:45:34.629330+0000","last_clean":"2024-09-13T08:45:34.629330+0000","last_became_active":"2024-09-13T08:45:32.607548+0000","last_became_peered":"2024-09-13T08:45:32.607548+0000","last_unstale":"2024-09-13T08:45:34.629330+0000","last_undegraded":"2024-09-13T08:45:34.629330+0000","last_fullsized":"2024-09-13T08:45:34.629330+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:13.011757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205572+0000","last_change":"2024-09-13T08:45:33.634276+0000","last_active":"2024-09-13T08:45:36.205572+0000","last_peered":"2024-09-13T08:45:36.205572+0000","last_clean":"2024-09-13T08:45:36.205572+0000","last_became_active":"2024-09-13T08:45:33.633911+0000","last_became_peered":"2024-09-13T08:45:33.633911+0000","last_unstale":"2024-09-13T08:45:36.205572+0000","last_undegraded":"2024-09-13T08:45:36.205572+0000","last_fullsized":"2024-09-13T08:45:36.205572+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:18:13.314925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894576+0000","last_change":"2024-09-13T08:45:32.617353+0000","last_active":"2024-09-13T08:45:34.894576+0000","last_peered":"2024-09-13T08:45:34.894576+0000","last_clean":"2024-09-13T08:45:34.894576+0000","last_became_active":"2024-09-13T08:45:32.616890+0000","last_became_peered":"2024-09-13T08:45:32.616890+0000","last_unstale":"2024-09-13T08:45:34.894576+0000","last_undegraded":"2024-09-13T08:45:34.894576+0000","last_fullsized":"2024-09-13T08:45:34.894576+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:38.873438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":374,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894373+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:45:34.894373+0000","last_peered":"2024-09-13T08:45:34.894373+0000","last_clean":"2024-09-13T08:45:34.894373+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:45:34.894373+0000","last_undegraded":"2024-09-13T08:45:34.894373+0000","last_fullsized":"2024-09-13T08:45:34.894373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509547,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1888256,"data_stored":837095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509547,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":837129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509547,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37780,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334060,"statfs":{"total":96636764160,"available":96598077440,"internally_reserved":0,"allocated":2490368,"data_stored":1428135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542250,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":837129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509547,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1904640,"data_stored":838729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509547,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1892352,"data_stored":837677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509548,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33636,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338204,"statfs":{"total":96636764160,"available":96602320896,"internally_reserved":0,"allocated":2506752,"data_stored":1429257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509546,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1880064,"data_stored":830821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:45:37.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:45:38.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 -- 172.21.15.31:0/2659324714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 msgr2=0x7f0a7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2659324714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a7c0051c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00c640 tx=0x7f0a7c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:38.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a99800640 1 -- 172.21.15.31:0/2659324714 reap_dead start 2024-09-13T08:45:38.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 -- 172.21.15.31:0/2659324714 shutdown_connections 2024-09-13T08:45:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 -- 172.21.15.31:0/2659324714 >> 172.21.15.31:0/2659324714 conn(0x7f0a940cc910 msgr2=0x7f0a940cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 -- 172.21.15.31:0/2659324714 shutdown_connections 2024-09-13T08:45:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.101+0000 7f0a9b800640 1 -- 172.21.15.31:0/2659324714 wait complete. 2024-09-13T08:45:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.102+0000 7f0a9b800640 1 Processor -- start 2024-09-13T08:45:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.103+0000 7f0a9b800640 1 -- start start 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 0x7f0a9409b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 0x7f0a94156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940cf230 con 0x7f0a7c004d40 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940cf0b0 con 0x7f0a941563b0 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a9b800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0a940cf3b0 con 0x7f0a9409d600 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45468/0 (socket says 172.21.15.31:45468) 2024-09-13T08:45:38.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 -- 172.21.15.31:0/2740018894 learned_addr learned my addr 172.21.15.31:0/2740018894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a99800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 0x7f0a94156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a93e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 0x7f0a9409b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 -- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 msgr2=0x7f0a9409b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 0x7f0a9409b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 -- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 msgr2=0x7f0a94156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 0x7f0a94156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a98e00640 1 -- 172.21.15.31:0/2740018894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a94157d20 con 0x7f0a7c004d40 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a99800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 0x7f0a94156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.104+0000 7f0a93e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 0x7f0a9409b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a98e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0a88011ae0 tx=0x7f0a88009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800fcb0 con 0x7f0a7c004d40 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a8800fe50 con 0x7f0a7c004d40 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88004280 con 0x7f0a7c004d40 2024-09-13T08:45:38.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a940c5c50 con 0x7f0a7c004d40 2024-09-13T08:45:38.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.105+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a940c6210 con 0x7f0a7c004d40 2024-09-13T08:45:38.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.106+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9415f3d0 con 0x7f0a7c004d40 2024-09-13T08:45:38.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.107+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f0a880035f0 con 0x7f0a7c004d40 2024-09-13T08:45:38.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.108+0000 7f0a90c00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 0x7f0a7407b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.108+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f0a8809c0f0 con 0x7f0a7c004d40 2024-09-13T08:45:38.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.108+0000 7f0a93e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 0x7f0a7407b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.109+0000 7f0a93e00640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 0x7f0a7407b400 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a940ca470 tx=0x7f0a80005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:38.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.112+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0a88063c50 con 0x7f0a7c004d40 2024-09-13T08:45:38.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.307+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a941579a0 con 0x7f0a74078f40 2024-09-13T08:45:38.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.311+0000 7f0a90c00640 1 -- 172.21.15.31:0/2740018894 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7f0a8005ea60 con 0x7f0a74078f40 2024-09-13T08:45:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:38.316 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:45:38.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 msgr2=0x7f0a7407b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 0x7f0a7407b400 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a940ca470 tx=0x7f0a80005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 msgr2=0x7f0a940c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0a88011ae0 tx=0x7f0a88009f10 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 shutdown_connections 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a74078f40 0x7f0a7407b400 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a941563b0 0x7f0a94156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a9409d600 0x7f0a9409b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 --2- 172.21.15.31:0/2740018894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c004d40 0x7f0a940c5420 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 >> 172.21.15.31:0/2740018894 conn(0x7f0a940cc910 msgr2=0x7f0a940ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 shutdown_connections 2024-09-13T08:45:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.315+0000 7f0a9b800640 1 -- 172.21.15.31:0/2740018894 wait complete. 2024-09-13T08:45:38.332 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-13T08:45:37.005516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282116,"kb_used_data":15964,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692604,"statfs":{"total":773094113280,"available":772805226496,"internally_reserved":0,"allocated":16347136,"data_stored":7875972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.417095"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629893+0000","last_change":"2024-09-13T08:45:33.631458+0000","last_active":"2024-09-13T08:45:34.629893+0000","last_peered":"2024-09-13T08:45:34.629893+0000","last_clean":"2024-09-13T08:45:34.629893+0000","last_became_active":"2024-09-13T08:45:33.631198+0000","last_became_peered":"2024-09-13T08:45:33.631198+0000","last_unstale":"2024-09-13T08:45:34.629893+0000","last_undegraded":"2024-09-13T08:45:34.629893+0000","last_fullsized":"2024-09-13T08:45:34.629893+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:18:13.314925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630089+0000","last_change":"2024-09-13T08:45:33.620890+0000","last_active":"2024-09-13T08:45:34.630089+0000","last_peered":"2024-09-13T08:45:34.630089+0000","last_clean":"2024-09-13T08:45:34.630089+0000","last_became_active":"2024-09-13T08:45:33.620601+0000","last_became_peered":"2024-09-13T08:45:33.620601+0000","last_unstale":"2024-09-13T08:45:34.630089+0000","last_undegraded":"2024-09-13T08:45:34.630089+0000","last_fullsized":"2024-09-13T08:45:34.630089+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:09.666382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629921+0000","last_change":"2024-09-13T08:45:33.622332+0000","last_active":"2024-09-13T08:45:34.629921+0000","last_peered":"2024-09-13T08:45:34.629921+0000","last_clean":"2024-09-13T08:45:34.629921+0000","last_became_active":"2024-09-13T08:45:33.622054+0000","last_became_peered":"2024-09-13T08:45:33.622054+0000","last_unstale":"2024-09-13T08:45:34.629921+0000","last_undegraded":"2024-09-13T08:45:34.629921+0000","last_fullsized":"2024-09-13T08:45:34.629921+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:58.226891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618856+0000","last_change":"2024-09-13T08:45:33.619320+0000","last_active":"2024-09-13T08:45:34.618856+0000","last_peered":"2024-09-13T08:45:34.618856+0000","last_clean":"2024-09-13T08:45:34.618856+0000","last_became_active":"2024-09-13T08:45:33.618730+0000","last_became_peered":"2024-09-13T08:45:33.618730+0000","last_unstale":"2024-09-13T08:45:34.618856+0000","last_undegraded":"2024-09-13T08:45:34.618856+0000","last_fullsized":"2024-09-13T08:45:34.618856+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:58:16.453660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894470+0000","last_change":"2024-09-13T08:45:33.633860+0000","last_active":"2024-09-13T08:45:34.894470+0000","last_peered":"2024-09-13T08:45:34.894470+0000","last_clean":"2024-09-13T08:45:34.894470+0000","last_became_active":"2024-09-13T08:45:33.633564+0000","last_became_peered":"2024-09-13T08:45:33.633564+0000","last_unstale":"2024-09-13T08:45:34.894470+0000","last_undegraded":"2024-09-13T08:45:34.894470+0000","last_fullsized":"2024-09-13T08:45:34.894470+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:44.258133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619661+0000","last_change":"2024-09-13T08:45:33.620322+0000","last_active":"2024-09-13T08:45:34.619661+0000","last_peered":"2024-09-13T08:45:34.619661+0000","last_clean":"2024-09-13T08:45:34.619661+0000","last_became_active":"2024-09-13T08:45:33.620003+0000","last_became_peered":"2024-09-13T08:45:33.620003+0000","last_unstale":"2024-09-13T08:45:34.619661+0000","last_undegraded":"2024-09-13T08:45:34.619661+0000","last_fullsized":"2024-09-13T08:45:34.619661+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:16:28.449503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204382+0000","last_change":"2024-09-13T08:45:33.626916+0000","last_active":"2024-09-13T08:45:36.204382+0000","last_peered":"2024-09-13T08:45:36.204382+0000","last_clean":"2024-09-13T08:45:36.204382+0000","last_became_active":"2024-09-13T08:45:33.626414+0000","last_became_peered":"2024-09-13T08:45:33.626414+0000","last_unstale":"2024-09-13T08:45:36.204382+0000","last_undegraded":"2024-09-13T08:45:36.204382+0000","last_fullsized":"2024-09-13T08:45:36.204382+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:01.245998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203229+0000","last_change":"2024-09-13T08:45:33.618406+0000","last_active":"2024-09-13T08:45:36.203229+0000","last_peered":"2024-09-13T08:45:36.203229+0000","last_clean":"2024-09-13T08:45:36.203229+0000","last_became_active":"2024-09-13T08:45:33.617012+0000","last_became_peered":"2024-09-13T08:45:33.617012+0000","last_unstale":"2024-09-13T08:45:36.203229+0000","last_undegraded":"2024-09-13T08:45:36.203229+0000","last_fullsized":"2024-09-13T08:45:36.203229+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:38.020097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203377+0000","last_change":"2024-09-13T08:45:33.614597+0000","last_active":"2024-09-13T08:45:36.203377+0000","last_peered":"2024-09-13T08:45:36.203377+0000","last_clean":"2024-09-13T08:45:36.203377+0000","last_became_active":"2024-09-13T08:45:33.614275+0000","last_became_peered":"2024-09-13T08:45:33.614275+0000","last_unstale":"2024-09-13T08:45:36.203377+0000","last_undegraded":"2024-09-13T08:45:36.203377+0000","last_fullsized":"2024-09-13T08:45:36.203377+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:29.640042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203338+0000","last_change":"2024-09-13T08:45:33.624832+0000","last_active":"2024-09-13T08:45:36.203338+0000","last_peered":"2024-09-13T08:45:36.203338+0000","last_clean":"2024-09-13T08:45:36.203338+0000","last_became_active":"2024-09-13T08:45:33.624517+0000","last_became_peered":"2024-09-13T08:45:33.624517+0000","last_unstale":"2024-09-13T08:45:36.203338+0000","last_undegraded":"2024-09-13T08:45:36.203338+0000","last_fullsized":"2024-09-13T08:45:36.203338+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:27:46.512058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894696+0000","last_change":"2024-09-13T08:45:33.618707+0000","last_active":"2024-09-13T08:45:34.894696+0000","last_peered":"2024-09-13T08:45:34.894696+0000","last_clean":"2024-09-13T08:45:34.894696+0000","last_became_active":"2024-09-13T08:45:33.617802+0000","last_became_peered":"2024-09-13T08:45:33.617802+0000","last_unstale":"2024-09-13T08:45:34.894696+0000","last_undegraded":"2024-09-13T08:45:34.894696+0000","last_fullsized":"2024-09-13T08:45:34.894696+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:00.192533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617405+0000","last_change":"2024-09-13T08:45:33.627848+0000","last_active":"2024-09-13T08:45:34.617405+0000","last_peered":"2024-09-13T08:45:34.617405+0000","last_clean":"2024-09-13T08:45:34.617405+0000","last_became_active":"2024-09-13T08:45:33.626669+0000","last_became_peered":"2024-09-13T08:45:33.626669+0000","last_unstale":"2024-09-13T08:45:34.617405+0000","last_undegraded":"2024-09-13T08:45:34.617405+0000","last_fullsized":"2024-09-13T08:45:34.617405+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:25:23.372974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617381+0000","last_change":"2024-09-13T08:45:33.615945+0000","last_active":"2024-09-13T08:45:34.617381+0000","last_peered":"2024-09-13T08:45:34.617381+0000","last_clean":"2024-09-13T08:45:34.617381+0000","last_became_active":"2024-09-13T08:45:33.615609+0000","last_became_peered":"2024-09-13T08:45:33.615609+0000","last_unstale":"2024-09-13T08:45:34.617381+0000","last_undegraded":"2024-09-13T08:45:34.617381+0000","last_fullsized":"2024-09-13T08:45:34.617381+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:06.881884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203094+0000","last_change":"2024-09-13T08:45:33.631590+0000","last_active":"2024-09-13T08:45:36.203094+0000","last_peered":"2024-09-13T08:45:36.203094+0000","last_clean":"2024-09-13T08:45:36.203094+0000","last_became_active":"2024-09-13T08:45:33.631031+0000","last_became_peered":"2024-09-13T08:45:33.631031+0000","last_unstale":"2024-09-13T08:45:36.203094+0000","last_undegraded":"2024-09-13T08:45:36.203094+0000","last_fullsized":"2024-09-13T08:45:36.203094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:09.666382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203139+0000","last_change":"2024-09-13T08:45:33.615254+0000","last_active":"2024-09-13T08:45:36.203139+0000","last_peered":"2024-09-13T08:45:36.203139+0000","last_clean":"2024-09-13T08:45:36.203139+0000","last_became_active":"2024-09-13T08:45:33.614983+0000","last_became_peered":"2024-09-13T08:45:33.614983+0000","last_unstale":"2024-09-13T08:45:36.203139+0000","last_undegraded":"2024-09-13T08:45:36.203139+0000","last_fullsized":"2024-09-13T08:45:36.203139+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:22.375438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203999+0000","last_change":"2024-09-13T08:45:33.620887+0000","last_active":"2024-09-13T08:45:36.203999+0000","last_peered":"2024-09-13T08:45:36.203999+0000","last_clean":"2024-09-13T08:45:36.203999+0000","last_became_active":"2024-09-13T08:45:33.620607+0000","last_became_peered":"2024-09-13T08:45:33.620607+0000","last_unstale":"2024-09-13T08:45:36.203999+0000","last_undegraded":"2024-09-13T08:45:36.203999+0000","last_fullsized":"2024-09-13T08:45:36.203999+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.591042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618961+0000","last_change":"2024-09-13T08:45:33.621444+0000","last_active":"2024-09-13T08:45:34.618961+0000","last_peered":"2024-09-13T08:45:34.618961+0000","last_clean":"2024-09-13T08:45:34.618961+0000","last_became_active":"2024-09-13T08:45:33.621125+0000","last_became_peered":"2024-09-13T08:45:33.621125+0000","last_unstale":"2024-09-13T08:45:34.618961+0000","last_undegraded":"2024-09-13T08:45:34.618961+0000","last_fullsized":"2024-09-13T08:45:34.618961+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:59:41.533846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618328+0000","last_change":"2024-09-13T08:45:33.625473+0000","last_active":"2024-09-13T08:45:34.618328+0000","last_peered":"2024-09-13T08:45:34.618328+0000","last_clean":"2024-09-13T08:45:34.618328+0000","last_became_active":"2024-09-13T08:45:33.625194+0000","last_became_peered":"2024-09-13T08:45:33.625194+0000","last_unstale":"2024-09-13T08:45:34.618328+0000","last_undegraded":"2024-09-13T08:45:34.618328+0000","last_fullsized":"2024-09-13T08:45:34.618328+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:03.426038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205702+0000","last_change":"2024-09-13T08:45:33.625665+0000","last_active":"2024-09-13T08:45:36.205702+0000","last_peered":"2024-09-13T08:45:36.205702+0000","last_clean":"2024-09-13T08:45:36.205702+0000","last_became_active":"2024-09-13T08:45:33.625258+0000","last_became_peered":"2024-09-13T08:45:33.625258+0000","last_unstale":"2024-09-13T08:45:36.205702+0000","last_undegraded":"2024-09-13T08:45:36.205702+0000","last_fullsized":"2024-09-13T08:45:36.205702+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:27.096984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617457+0000","last_change":"2024-09-13T08:45:33.622254+0000","last_active":"2024-09-13T08:45:34.617457+0000","last_peered":"2024-09-13T08:45:34.617457+0000","last_clean":"2024-09-13T08:45:34.617457+0000","last_became_active":"2024-09-13T08:45:33.621966+0000","last_became_peered":"2024-09-13T08:45:33.621966+0000","last_unstale":"2024-09-13T08:45:34.617457+0000","last_undegraded":"2024-09-13T08:45:34.617457+0000","last_fullsized":"2024-09-13T08:45:34.617457+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:08.382851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619561+0000","last_change":"2024-09-13T08:45:33.616459+0000","last_active":"2024-09-13T08:45:34.619561+0000","last_peered":"2024-09-13T08:45:34.619561+0000","last_clean":"2024-09-13T08:45:34.619561+0000","last_became_active":"2024-09-13T08:45:33.616167+0000","last_became_peered":"2024-09-13T08:45:33.616167+0000","last_unstale":"2024-09-13T08:45:34.619561+0000","last_undegraded":"2024-09-13T08:45:34.619561+0000","last_fullsized":"2024-09-13T08:45:34.619561+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:02:02.806744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894423+0000","last_change":"2024-09-13T08:45:33.620595+0000","last_active":"2024-09-13T08:45:34.894423+0000","last_peered":"2024-09-13T08:45:34.894423+0000","last_clean":"2024-09-13T08:45:34.894423+0000","last_became_active":"2024-09-13T08:45:33.620328+0000","last_became_peered":"2024-09-13T08:45:33.620328+0000","last_unstale":"2024-09-13T08:45:34.894423+0000","last_undegraded":"2024-09-13T08:45:34.894423+0000","last_fullsized":"2024-09-13T08:45:34.894423+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:40.677013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204244+0000","last_change":"2024-09-13T08:45:33.621640+0000","last_active":"2024-09-13T08:45:36.204244+0000","last_peered":"2024-09-13T08:45:36.204244+0000","last_clean":"2024-09-13T08:45:36.204244+0000","last_became_active":"2024-09-13T08:45:33.620791+0000","last_became_peered":"2024-09-13T08:45:33.620791+0000","last_unstale":"2024-09-13T08:45:36.204244+0000","last_undegraded":"2024-09-13T08:45:36.204244+0000","last_fullsized":"2024-09-13T08:45:36.204244+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:20.182452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894335+0000","last_change":"2024-09-13T08:45:33.638184+0000","last_active":"2024-09-13T08:45:34.894335+0000","last_peered":"2024-09-13T08:45:34.894335+0000","last_clean":"2024-09-13T08:45:34.894335+0000","last_became_active":"2024-09-13T08:45:33.635223+0000","last_became_peered":"2024-09-13T08:45:33.635223+0000","last_unstale":"2024-09-13T08:45:34.894335+0000","last_undegraded":"2024-09-13T08:45:34.894335+0000","last_fullsized":"2024-09-13T08:45:34.894335+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:20.513044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894733+0000","last_change":"2024-09-13T08:45:33.622658+0000","last_active":"2024-09-13T08:45:34.894733+0000","last_peered":"2024-09-13T08:45:34.894733+0000","last_clean":"2024-09-13T08:45:34.894733+0000","last_became_active":"2024-09-13T08:45:33.622361+0000","last_became_peered":"2024-09-13T08:45:33.622361+0000","last_unstale":"2024-09-13T08:45:34.894733+0000","last_undegraded":"2024-09-13T08:45:34.894733+0000","last_fullsized":"2024-09-13T08:45:34.894733+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.567031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205342+0000","last_change":"2024-09-13T08:45:33.628316+0000","last_active":"2024-09-13T08:45:36.205342+0000","last_peered":"2024-09-13T08:45:36.205342+0000","last_clean":"2024-09-13T08:45:36.205342+0000","last_became_active":"2024-09-13T08:45:33.627994+0000","last_became_peered":"2024-09-13T08:45:33.627994+0000","last_unstale":"2024-09-13T08:45:36.205342+0000","last_undegraded":"2024-09-13T08:45:36.205342+0000","last_fullsized":"2024-09-13T08:45:36.205342+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:03.283784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894748+0000","last_change":"2024-09-13T08:45:33.636265+0000","last_active":"2024-09-13T08:45:34.894748+0000","last_peered":"2024-09-13T08:45:34.894748+0000","last_clean":"2024-09-13T08:45:34.894748+0000","last_became_active":"2024-09-13T08:45:33.635862+0000","last_became_peered":"2024-09-13T08:45:33.635862+0000","last_unstale":"2024-09-13T08:45:34.894748+0000","last_undegraded":"2024-09-13T08:45:34.894748+0000","last_fullsized":"2024-09-13T08:45:34.894748+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:21:24.800675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204665+0000","last_change":"2024-09-13T08:45:33.625565+0000","last_active":"2024-09-13T08:45:36.204665+0000","last_peered":"2024-09-13T08:45:36.204665+0000","last_clean":"2024-09-13T08:45:36.204665+0000","last_became_active":"2024-09-13T08:45:33.625186+0000","last_became_peered":"2024-09-13T08:45:33.625186+0000","last_unstale":"2024-09-13T08:45:36.204665+0000","last_undegraded":"2024-09-13T08:45:36.204665+0000","last_fullsized":"2024-09-13T08:45:36.204665+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:11.124390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629827+0000","last_change":"2024-09-13T08:45:33.621997+0000","last_active":"2024-09-13T08:45:34.629827+0000","last_peered":"2024-09-13T08:45:34.629827+0000","last_clean":"2024-09-13T08:45:34.629827+0000","last_became_active":"2024-09-13T08:45:33.621191+0000","last_became_peered":"2024-09-13T08:45:33.621191+0000","last_unstale":"2024-09-13T08:45:34.629827+0000","last_undegraded":"2024-09-13T08:45:34.629827+0000","last_fullsized":"2024-09-13T08:45:34.629827+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:50.966084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203363+0000","last_change":"2024-09-13T08:45:33.631285+0000","last_active":"2024-09-13T08:45:36.203363+0000","last_peered":"2024-09-13T08:45:36.203363+0000","last_clean":"2024-09-13T08:45:36.203363+0000","last_became_active":"2024-09-13T08:45:33.630642+0000","last_became_peered":"2024-09-13T08:45:33.630642+0000","last_unstale":"2024-09-13T08:45:36.203363+0000","last_undegraded":"2024-09-13T08:45:36.203363+0000","last_fullsized":"2024-09-13T08:45:36.203363+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:14.431761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629609+0000","last_change":"2024-09-13T08:45:33.610332+0000","last_active":"2024-09-13T08:45:34.629609+0000","last_peered":"2024-09-13T08:45:34.629609+0000","last_clean":"2024-09-13T08:45:34.629609+0000","last_became_active":"2024-09-13T08:45:33.610038+0000","last_became_peered":"2024-09-13T08:45:33.610038+0000","last_unstale":"2024-09-13T08:45:34.629609+0000","last_undegraded":"2024-09-13T08:45:34.629609+0000","last_fullsized":"2024-09-13T08:45:34.629609+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:11.124390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617966+0000","last_change":"2024-09-13T08:45:33.611932+0000","last_active":"2024-09-13T08:45:34.617966+0000","last_peered":"2024-09-13T08:45:34.617966+0000","last_clean":"2024-09-13T08:45:34.617966+0000","last_became_active":"2024-09-13T08:45:33.611635+0000","last_became_peered":"2024-09-13T08:45:33.611635+0000","last_unstale":"2024-09-13T08:45:34.617966+0000","last_undegraded":"2024-09-13T08:45:34.617966+0000","last_fullsized":"2024-09-13T08:45:34.617966+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:21:24.800675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619452+0000","last_change":"2024-09-13T08:45:33.619404+0000","last_active":"2024-09-13T08:45:34.619452+0000","last_peered":"2024-09-13T08:45:34.619452+0000","last_clean":"2024-09-13T08:45:34.619452+0000","last_became_active":"2024-09-13T08:45:33.619037+0000","last_became_peered":"2024-09-13T08:45:33.619037+0000","last_unstale":"2024-09-13T08:45:34.619452+0000","last_undegraded":"2024-09-13T08:45:34.619452+0000","last_fullsized":"2024-09-13T08:45:34.619452+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:19:46.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1f","version":"150'4","reported_seq":19,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629951+0000","last_change":"2024-09-13T08:45:32.621061+0000","last_active":"2024-09-13T08:45:34.629951+0000","last_peered":"2024-09-13T08:45:34.629951+0000","last_clean":"2024-09-13T08:45:34.629951+0000","last_became_active":"2024-09-13T08:45:32.620793+0000","last_became_peered":"2024-09-13T08:45:32.620793+0000","last_unstale":"2024-09-13T08:45:34.629951+0000","last_undegraded":"2024-09-13T08:45:34.629951+0000","last_fullsized":"2024-09-13T08:45:34.629951+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:51.771655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629962+0000","last_change":"2024-09-13T08:45:33.615156+0000","last_active":"2024-09-13T08:45:34.629962+0000","last_peered":"2024-09-13T08:45:34.629962+0000","last_clean":"2024-09-13T08:45:34.629962+0000","last_became_active":"2024-09-13T08:45:33.614860+0000","last_became_peered":"2024-09-13T08:45:33.614860+0000","last_unstale":"2024-09-13T08:45:34.629962+0000","last_undegraded":"2024-09-13T08:45:34.629962+0000","last_fullsized":"2024-09-13T08:45:34.629962+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:46:27.096984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203397+0000","last_change":"2024-09-13T08:45:32.620891+0000","last_active":"2024-09-13T08:45:36.203397+0000","last_peered":"2024-09-13T08:45:36.203397+0000","last_clean":"2024-09-13T08:45:36.203397+0000","last_became_active":"2024-09-13T08:45:32.620614+0000","last_became_peered":"2024-09-13T08:45:32.620614+0000","last_unstale":"2024-09-13T08:45:36.203397+0000","last_undegraded":"2024-09-13T08:45:36.203397+0000","last_fullsized":"2024-09-13T08:45:36.203397+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:29.582616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618808+0000","last_change":"2024-09-13T08:45:33.619592+0000","last_active":"2024-09-13T08:45:34.618808+0000","last_peered":"2024-09-13T08:45:34.618808+0000","last_clean":"2024-09-13T08:45:34.618808+0000","last_became_active":"2024-09-13T08:45:33.617503+0000","last_became_peered":"2024-09-13T08:45:33.617503+0000","last_unstale":"2024-09-13T08:45:34.618808+0000","last_undegraded":"2024-09-13T08:45:34.618808+0000","last_fullsized":"2024-09-13T08:45:34.618808+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:14.761774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618917+0000","last_change":"2024-09-13T08:45:32.613395+0000","last_active":"2024-09-13T08:45:34.618917+0000","last_peered":"2024-09-13T08:45:34.618917+0000","last_clean":"2024-09-13T08:45:34.618917+0000","last_became_active":"2024-09-13T08:45:32.613050+0000","last_became_peered":"2024-09-13T08:45:32.613050+0000","last_unstale":"2024-09-13T08:45:34.618917+0000","last_undegraded":"2024-09-13T08:45:34.618917+0000","last_fullsized":"2024-09-13T08:45:34.618917+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:33.672884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203335+0000","last_change":"2024-09-13T08:45:33.616470+0000","last_active":"2024-09-13T08:45:36.203335+0000","last_peered":"2024-09-13T08:45:36.203335+0000","last_clean":"2024-09-13T08:45:36.203335+0000","last_became_active":"2024-09-13T08:45:33.616053+0000","last_became_peered":"2024-09-13T08:45:33.616053+0000","last_unstale":"2024-09-13T08:45:36.203335+0000","last_undegraded":"2024-09-13T08:45:36.203335+0000","last_fullsized":"2024-09-13T08:45:36.203335+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:56:04.012311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1c","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205067+0000","last_change":"2024-09-13T08:45:32.618389+0000","last_active":"2024-09-13T08:45:36.205067+0000","last_peered":"2024-09-13T08:45:36.205067+0000","last_clean":"2024-09-13T08:45:36.205067+0000","last_became_active":"2024-09-13T08:45:32.617588+0000","last_became_peered":"2024-09-13T08:45:32.617588+0000","last_unstale":"2024-09-13T08:45:36.205067+0000","last_undegraded":"2024-09-13T08:45:36.205067+0000","last_fullsized":"2024-09-13T08:45:36.205067+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:21.548606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618132+0000","last_change":"2024-09-13T08:45:33.621587+0000","last_active":"2024-09-13T08:45:34.618132+0000","last_peered":"2024-09-13T08:45:34.618132+0000","last_clean":"2024-09-13T08:45:34.618132+0000","last_became_active":"2024-09-13T08:45:33.620130+0000","last_became_peered":"2024-09-13T08:45:33.620130+0000","last_unstale":"2024-09-13T08:45:34.618132+0000","last_undegraded":"2024-09-13T08:45:34.618132+0000","last_fullsized":"2024-09-13T08:45:34.618132+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:54:43.284644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629685+0000","last_change":"2024-09-13T08:45:32.610935+0000","last_active":"2024-09-13T08:45:34.629685+0000","last_peered":"2024-09-13T08:45:34.629685+0000","last_clean":"2024-09-13T08:45:34.629685+0000","last_became_active":"2024-09-13T08:45:32.610241+0000","last_became_peered":"2024-09-13T08:45:32.610241+0000","last_unstale":"2024-09-13T08:45:34.629685+0000","last_undegraded":"2024-09-13T08:45:34.629685+0000","last_fullsized":"2024-09-13T08:45:34.629685+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:38.326122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619571+0000","last_change":"2024-09-13T08:45:33.616609+0000","last_active":"2024-09-13T08:45:34.619571+0000","last_peered":"2024-09-13T08:45:34.619571+0000","last_clean":"2024-09-13T08:45:34.619571+0000","last_became_active":"2024-09-13T08:45:33.616081+0000","last_became_peered":"2024-09-13T08:45:33.616081+0000","last_unstale":"2024-09-13T08:45:34.619571+0000","last_undegraded":"2024-09-13T08:45:34.619571+0000","last_fullsized":"2024-09-13T08:45:34.619571+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:30:43.691761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1a","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618221+0000","last_change":"2024-09-13T08:45:32.613940+0000","last_active":"2024-09-13T08:45:34.618221+0000","last_peered":"2024-09-13T08:45:34.618221+0000","last_clean":"2024-09-13T08:45:34.618221+0000","last_became_active":"2024-09-13T08:45:32.613645+0000","last_became_peered":"2024-09-13T08:45:32.613645+0000","last_unstale":"2024-09-13T08:45:34.618221+0000","last_undegraded":"2024-09-13T08:45:34.618221+0000","last_fullsized":"2024-09-13T08:45:34.618221+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:14.080253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204833+0000","last_change":"2024-09-13T08:45:33.621259+0000","last_active":"2024-09-13T08:45:36.204833+0000","last_peered":"2024-09-13T08:45:36.204833+0000","last_clean":"2024-09-13T08:45:36.204833+0000","last_became_active":"2024-09-13T08:45:33.620982+0000","last_became_peered":"2024-09-13T08:45:33.620982+0000","last_unstale":"2024-09-13T08:45:36.204833+0000","last_undegraded":"2024-09-13T08:45:36.204833+0000","last_fullsized":"2024-09-13T08:45:36.204833+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:44:42.739868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629615+0000","last_change":"2024-09-13T08:45:32.620940+0000","last_active":"2024-09-13T08:45:34.629615+0000","last_peered":"2024-09-13T08:45:34.629615+0000","last_clean":"2024-09-13T08:45:34.629615+0000","last_became_active":"2024-09-13T08:45:32.620654+0000","last_became_peered":"2024-09-13T08:45:32.620654+0000","last_unstale":"2024-09-13T08:45:34.629615+0000","last_undegraded":"2024-09-13T08:45:34.629615+0000","last_fullsized":"2024-09-13T08:45:34.629615+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:06:15.310795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617875+0000","last_change":"2024-09-13T08:45:33.622411+0000","last_active":"2024-09-13T08:45:34.617875+0000","last_peered":"2024-09-13T08:45:34.617875+0000","last_clean":"2024-09-13T08:45:34.617875+0000","last_became_active":"2024-09-13T08:45:33.619412+0000","last_became_peered":"2024-09-13T08:45:33.619412+0000","last_unstale":"2024-09-13T08:45:34.617875+0000","last_undegraded":"2024-09-13T08:45:34.617875+0000","last_fullsized":"2024-09-13T08:45:34.617875+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:02.002010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203817+0000","last_change":"2024-09-13T08:45:32.619147+0000","last_active":"2024-09-13T08:45:36.203817+0000","last_peered":"2024-09-13T08:45:36.203817+0000","last_clean":"2024-09-13T08:45:36.203817+0000","last_became_active":"2024-09-13T08:45:32.618840+0000","last_became_peered":"2024-09-13T08:45:32.618840+0000","last_unstale":"2024-09-13T08:45:36.203817+0000","last_undegraded":"2024-09-13T08:45:36.203817+0000","last_fullsized":"2024-09-13T08:45:36.203817+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:43:33.945255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205589+0000","last_change":"2024-09-13T08:45:33.629576+0000","last_active":"2024-09-13T08:45:36.205589+0000","last_peered":"2024-09-13T08:45:36.205589+0000","last_clean":"2024-09-13T08:45:36.205589+0000","last_became_active":"2024-09-13T08:45:33.629297+0000","last_became_peered":"2024-09-13T08:45:33.629297+0000","last_unstale":"2024-09-13T08:45:36.205589+0000","last_undegraded":"2024-09-13T08:45:36.205589+0000","last_fullsized":"2024-09-13T08:45:36.205589+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:24:58.226891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619201+0000","last_change":"2024-09-13T08:45:32.621412+0000","last_active":"2024-09-13T08:45:34.619201+0000","last_peered":"2024-09-13T08:45:34.619201+0000","last_clean":"2024-09-13T08:45:34.619201+0000","last_became_active":"2024-09-13T08:45:32.621011+0000","last_became_peered":"2024-09-13T08:45:32.621011+0000","last_unstale":"2024-09-13T08:45:34.619201+0000","last_undegraded":"2024-09-13T08:45:34.619201+0000","last_fullsized":"2024-09-13T08:45:34.619201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:28.537636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204947+0000","last_change":"2024-09-13T08:45:33.628352+0000","last_active":"2024-09-13T08:45:36.204947+0000","last_peered":"2024-09-13T08:45:36.204947+0000","last_clean":"2024-09-13T08:45:36.204947+0000","last_became_active":"2024-09-13T08:45:33.627871+0000","last_became_peered":"2024-09-13T08:45:33.627871+0000","last_unstale":"2024-09-13T08:45:36.204947+0000","last_undegraded":"2024-09-13T08:45:36.204947+0000","last_fullsized":"2024-09-13T08:45:36.204947+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:40.374360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.15","version":"150'3","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205086+0000","last_change":"2024-09-13T08:45:32.622371+0000","last_active":"2024-09-13T08:45:36.205086+0000","last_peered":"2024-09-13T08:45:36.205086+0000","last_clean":"2024-09-13T08:45:36.205086+0000","last_became_active":"2024-09-13T08:45:32.622062+0000","last_became_peered":"2024-09-13T08:45:32.622062+0000","last_unstale":"2024-09-13T08:45:36.205086+0000","last_undegraded":"2024-09-13T08:45:36.205086+0000","last_fullsized":"2024-09-13T08:45:36.205086+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:00.615811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630121+0000","last_change":"2024-09-13T08:45:33.621415+0000","last_active":"2024-09-13T08:45:34.630121+0000","last_peered":"2024-09-13T08:45:34.630121+0000","last_clean":"2024-09-13T08:45:34.630121+0000","last_became_active":"2024-09-13T08:45:33.620745+0000","last_became_peered":"2024-09-13T08:45:33.620745+0000","last_unstale":"2024-09-13T08:45:34.630121+0000","last_undegraded":"2024-09-13T08:45:34.630121+0000","last_fullsized":"2024-09-13T08:45:34.630121+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:22.375438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.14","version":"150'6","reported_seq":21,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894469+0000","last_change":"2024-09-13T08:45:32.622424+0000","last_active":"2024-09-13T08:45:34.894469+0000","last_peered":"2024-09-13T08:45:34.894469+0000","last_clean":"2024-09-13T08:45:34.894469+0000","last_became_active":"2024-09-13T08:45:32.622090+0000","last_became_peered":"2024-09-13T08:45:32.622090+0000","last_unstale":"2024-09-13T08:45:34.894469+0000","last_undegraded":"2024-09-13T08:45:34.894469+0000","last_fullsized":"2024-09-13T08:45:34.894469+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:51.838355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204373+0000","last_change":"2024-09-13T08:45:33.619435+0000","last_active":"2024-09-13T08:45:36.204373+0000","last_peered":"2024-09-13T08:45:36.204373+0000","last_clean":"2024-09-13T08:45:36.204373+0000","last_became_active":"2024-09-13T08:45:33.619146+0000","last_became_peered":"2024-09-13T08:45:33.619146+0000","last_unstale":"2024-09-13T08:45:36.204373+0000","last_undegraded":"2024-09-13T08:45:36.204373+0000","last_fullsized":"2024-09-13T08:45:36.204373+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:44.238924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619144+0000","last_change":"2024-09-13T08:45:32.615626+0000","last_active":"2024-09-13T08:45:34.619144+0000","last_peered":"2024-09-13T08:45:34.619144+0000","last_clean":"2024-09-13T08:45:34.619144+0000","last_became_active":"2024-09-13T08:45:32.615358+0000","last_became_peered":"2024-09-13T08:45:32.615358+0000","last_unstale":"2024-09-13T08:45:34.619144+0000","last_undegraded":"2024-09-13T08:45:34.619144+0000","last_fullsized":"2024-09-13T08:45:34.619144+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:31.355664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204581+0000","last_change":"2024-09-13T08:45:33.624271+0000","last_active":"2024-09-13T08:45:36.204581+0000","last_peered":"2024-09-13T08:45:36.204581+0000","last_clean":"2024-09-13T08:45:36.204581+0000","last_became_active":"2024-09-13T08:45:33.623806+0000","last_became_peered":"2024-09-13T08:45:33.623806+0000","last_unstale":"2024-09-13T08:45:36.204581+0000","last_undegraded":"2024-09-13T08:45:36.204581+0000","last_fullsized":"2024-09-13T08:45:36.204581+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:56.955592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618781+0000","last_change":"2024-09-13T08:45:32.620219+0000","last_active":"2024-09-13T08:45:34.618781+0000","last_peered":"2024-09-13T08:45:34.618781+0000","last_clean":"2024-09-13T08:45:34.618781+0000","last_became_active":"2024-09-13T08:45:32.619750+0000","last_became_peered":"2024-09-13T08:45:32.619750+0000","last_unstale":"2024-09-13T08:45:34.618781+0000","last_undegraded":"2024-09-13T08:45:34.618781+0000","last_fullsized":"2024-09-13T08:45:34.618781+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:36.889900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618198+0000","last_change":"2024-09-13T08:45:33.619472+0000","last_active":"2024-09-13T08:45:34.618198+0000","last_peered":"2024-09-13T08:45:34.618198+0000","last_clean":"2024-09-13T08:45:34.618198+0000","last_became_active":"2024-09-13T08:45:33.617767+0000","last_became_peered":"2024-09-13T08:45:33.617767+0000","last_unstale":"2024-09-13T08:45:34.618198+0000","last_undegraded":"2024-09-13T08:45:34.618198+0000","last_fullsized":"2024-09-13T08:45:34.618198+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:29.210658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204253+0000","last_change":"2024-09-13T08:45:32.617190+0000","last_active":"2024-09-13T08:45:36.204253+0000","last_peered":"2024-09-13T08:45:36.204253+0000","last_clean":"2024-09-13T08:45:36.204253+0000","last_became_active":"2024-09-13T08:45:32.616867+0000","last_became_peered":"2024-09-13T08:45:32.616867+0000","last_unstale":"2024-09-13T08:45:36.204253+0000","last_undegraded":"2024-09-13T08:45:36.204253+0000","last_fullsized":"2024-09-13T08:45:36.204253+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:42.407504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203314+0000","last_change":"2024-09-13T08:45:33.625673+0000","last_active":"2024-09-13T08:45:36.203314+0000","last_peered":"2024-09-13T08:45:36.203314+0000","last_clean":"2024-09-13T08:45:36.203314+0000","last_became_active":"2024-09-13T08:45:33.625394+0000","last_became_peered":"2024-09-13T08:45:33.625394+0000","last_unstale":"2024-09-13T08:45:36.203314+0000","last_undegraded":"2024-09-13T08:45:36.203314+0000","last_fullsized":"2024-09-13T08:45:36.203314+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:48.509209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"150'8","reported_seq":23,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894233+0000","last_change":"2024-09-13T08:45:32.616810+0000","last_active":"2024-09-13T08:45:34.894233+0000","last_peered":"2024-09-13T08:45:34.894233+0000","last_clean":"2024-09-13T08:45:34.894233+0000","last_became_active":"2024-09-13T08:45:32.615280+0000","last_became_peered":"2024-09-13T08:45:32.615280+0000","last_unstale":"2024-09-13T08:45:34.894233+0000","last_undegraded":"2024-09-13T08:45:34.894233+0000","last_fullsized":"2024-09-13T08:45:34.894233+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:08.292180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205511+0000","last_change":"2024-09-13T08:45:33.633537+0000","last_active":"2024-09-13T08:45:36.205511+0000","last_peered":"2024-09-13T08:45:36.205511+0000","last_clean":"2024-09-13T08:45:36.205511+0000","last_became_active":"2024-09-13T08:45:33.632783+0000","last_became_peered":"2024-09-13T08:45:33.632783+0000","last_unstale":"2024-09-13T08:45:36.205511+0000","last_undegraded":"2024-09-13T08:45:36.205511+0000","last_fullsized":"2024-09-13T08:45:36.205511+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:50.966084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894694+0000","last_change":"2024-09-13T08:45:32.615167+0000","last_active":"2024-09-13T08:45:34.894694+0000","last_peered":"2024-09-13T08:45:34.894694+0000","last_clean":"2024-09-13T08:45:34.894694+0000","last_became_active":"2024-09-13T08:45:32.613724+0000","last_became_peered":"2024-09-13T08:45:32.613724+0000","last_unstale":"2024-09-13T08:45:34.894694+0000","last_undegraded":"2024-09-13T08:45:34.894694+0000","last_fullsized":"2024-09-13T08:45:34.894694+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:09.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619168+0000","last_change":"2024-09-13T08:45:33.621259+0000","last_active":"2024-09-13T08:45:34.619168+0000","last_peered":"2024-09-13T08:45:34.619168+0000","last_clean":"2024-09-13T08:45:34.619168+0000","last_became_active":"2024-09-13T08:45:33.620942+0000","last_became_peered":"2024-09-13T08:45:33.620942+0000","last_unstale":"2024-09-13T08:45:34.619168+0000","last_undegraded":"2024-09-13T08:45:34.619168+0000","last_fullsized":"2024-09-13T08:45:34.619168+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:19.918580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.203448+0000","last_change":"2024-09-13T08:45:32.611972+0000","last_active":"2024-09-13T08:45:36.203448+0000","last_peered":"2024-09-13T08:45:36.203448+0000","last_clean":"2024-09-13T08:45:36.203448+0000","last_became_active":"2024-09-13T08:45:32.611131+0000","last_became_peered":"2024-09-13T08:45:32.611131+0000","last_unstale":"2024-09-13T08:45:36.203448+0000","last_undegraded":"2024-09-13T08:45:36.203448+0000","last_fullsized":"2024-09-13T08:45:36.203448+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:19.487842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619094+0000","last_change":"2024-09-13T08:45:33.615094+0000","last_active":"2024-09-13T08:45:34.619094+0000","last_peered":"2024-09-13T08:45:34.619094+0000","last_clean":"2024-09-13T08:45:34.619094+0000","last_became_active":"2024-09-13T08:45:33.614707+0000","last_became_peered":"2024-09-13T08:45:33.614707+0000","last_unstale":"2024-09-13T08:45:34.619094+0000","last_undegraded":"2024-09-13T08:45:34.619094+0000","last_fullsized":"2024-09-13T08:45:34.619094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:52.686751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.d","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894734+0000","last_change":"2024-09-13T08:45:32.619476+0000","last_active":"2024-09-13T08:45:34.894734+0000","last_peered":"2024-09-13T08:45:34.894734+0000","last_clean":"2024-09-13T08:45:34.894734+0000","last_became_active":"2024-09-13T08:45:32.619104+0000","last_became_peered":"2024-09-13T08:45:32.619104+0000","last_unstale":"2024-09-13T08:45:34.894734+0000","last_undegraded":"2024-09-13T08:45:34.894734+0000","last_fullsized":"2024-09-13T08:45:34.894734+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:14.006254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205223+0000","last_change":"2024-09-13T08:45:33.623632+0000","last_active":"2024-09-13T08:45:36.205223+0000","last_peered":"2024-09-13T08:45:36.205223+0000","last_clean":"2024-09-13T08:45:36.205223+0000","last_became_active":"2024-09-13T08:45:33.623316+0000","last_became_peered":"2024-09-13T08:45:33.623316+0000","last_unstale":"2024-09-13T08:45:36.205223+0000","last_undegraded":"2024-09-13T08:45:36.205223+0000","last_fullsized":"2024-09-13T08:45:36.205223+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:53.922270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617440+0000","last_change":"2024-09-13T08:45:32.617286+0000","last_active":"2024-09-13T08:45:34.617440+0000","last_peered":"2024-09-13T08:45:34.617440+0000","last_clean":"2024-09-13T08:45:34.617440+0000","last_became_active":"2024-09-13T08:45:32.617005+0000","last_became_peered":"2024-09-13T08:45:32.617005+0000","last_unstale":"2024-09-13T08:45:34.617440+0000","last_undegraded":"2024-09-13T08:45:34.617440+0000","last_fullsized":"2024-09-13T08:45:34.617440+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:16:42.655224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630392+0000","last_change":"2024-09-13T08:45:33.612130+0000","last_active":"2024-09-13T08:45:34.630392+0000","last_peered":"2024-09-13T08:45:34.630392+0000","last_clean":"2024-09-13T08:45:34.630392+0000","last_became_active":"2024-09-13T08:45:33.611708+0000","last_became_peered":"2024-09-13T08:45:33.611708+0000","last_unstale":"2024-09-13T08:45:34.630392+0000","last_undegraded":"2024-09-13T08:45:34.630392+0000","last_fullsized":"2024-09-13T08:45:34.630392+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:38.020097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.b","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204655+0000","last_change":"2024-09-13T08:45:32.621453+0000","last_active":"2024-09-13T08:45:36.204655+0000","last_peered":"2024-09-13T08:45:36.204655+0000","last_clean":"2024-09-13T08:45:36.204655+0000","last_became_active":"2024-09-13T08:45:32.619431+0000","last_became_peered":"2024-09-13T08:45:32.619431+0000","last_unstale":"2024-09-13T08:45:36.204655+0000","last_undegraded":"2024-09-13T08:45:36.204655+0000","last_fullsized":"2024-09-13T08:45:36.204655+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:34.535933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204636+0000","last_change":"2024-09-13T08:45:33.632239+0000","last_active":"2024-09-13T08:45:36.204636+0000","last_peered":"2024-09-13T08:45:36.204636+0000","last_clean":"2024-09-13T08:45:36.204636+0000","last_became_active":"2024-09-13T08:45:33.631948+0000","last_became_peered":"2024-09-13T08:45:33.631948+0000","last_unstale":"2024-09-13T08:45:36.204636+0000","last_undegraded":"2024-09-13T08:45:36.204636+0000","last_fullsized":"2024-09-13T08:45:36.204636+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:21:34.449653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.616960+0000","last_change":"2024-09-13T08:45:32.612100+0000","last_active":"2024-09-13T08:45:34.616960+0000","last_peered":"2024-09-13T08:45:34.616960+0000","last_clean":"2024-09-13T08:45:34.616960+0000","last_became_active":"2024-09-13T08:45:32.611834+0000","last_became_peered":"2024-09-13T08:45:32.611834+0000","last_unstale":"2024-09-13T08:45:34.616960+0000","last_undegraded":"2024-09-13T08:45:34.616960+0000","last_fullsized":"2024-09-13T08:45:34.616960+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:57.319688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629527+0000","last_change":"2024-09-13T08:45:33.619203+0000","last_active":"2024-09-13T08:45:34.629527+0000","last_peered":"2024-09-13T08:45:34.629527+0000","last_clean":"2024-09-13T08:45:34.629527+0000","last_became_active":"2024-09-13T08:45:33.618868+0000","last_became_peered":"2024-09-13T08:45:33.618868+0000","last_unstale":"2024-09-13T08:45:34.629527+0000","last_undegraded":"2024-09-13T08:45:34.629527+0000","last_fullsized":"2024-09-13T08:45:34.629527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:44.238924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617743+0000","last_change":"2024-09-13T08:45:32.614951+0000","last_active":"2024-09-13T08:45:34.617743+0000","last_peered":"2024-09-13T08:45:34.617743+0000","last_clean":"2024-09-13T08:45:34.617743+0000","last_became_active":"2024-09-13T08:45:32.614589+0000","last_became_peered":"2024-09-13T08:45:32.614589+0000","last_unstale":"2024-09-13T08:45:34.617743+0000","last_undegraded":"2024-09-13T08:45:34.617743+0000","last_fullsized":"2024-09-13T08:45:34.617743+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:37.315527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204129+0000","last_change":"2024-09-13T08:45:33.636399+0000","last_active":"2024-09-13T08:45:36.204129+0000","last_peered":"2024-09-13T08:45:36.204129+0000","last_clean":"2024-09-13T08:45:36.204129+0000","last_became_active":"2024-09-13T08:45:33.636041+0000","last_became_peered":"2024-09-13T08:45:33.636041+0000","last_unstale":"2024-09-13T08:45:36.204129+0000","last_undegraded":"2024-09-13T08:45:36.204129+0000","last_fullsized":"2024-09-13T08:45:36.204129+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:57.520787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617110+0000","last_change":"2024-09-13T08:45:32.621572+0000","last_active":"2024-09-13T08:45:34.617110+0000","last_peered":"2024-09-13T08:45:34.617110+0000","last_clean":"2024-09-13T08:45:34.617110+0000","last_became_active":"2024-09-13T08:45:32.621303+0000","last_became_peered":"2024-09-13T08:45:32.621303+0000","last_unstale":"2024-09-13T08:45:34.617110+0000","last_undegraded":"2024-09-13T08:45:34.617110+0000","last_fullsized":"2024-09-13T08:45:34.617110+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:16.977123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205496+0000","last_change":"2024-09-13T08:45:33.626962+0000","last_active":"2024-09-13T08:45:36.205496+0000","last_peered":"2024-09-13T08:45:36.205496+0000","last_clean":"2024-09-13T08:45:36.205496+0000","last_became_active":"2024-09-13T08:45:33.626357+0000","last_became_peered":"2024-09-13T08:45:33.626357+0000","last_unstale":"2024-09-13T08:45:36.205496+0000","last_undegraded":"2024-09-13T08:45:36.205496+0000","last_fullsized":"2024-09-13T08:45:36.205496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:21:09.054648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618931+0000","last_change":"2024-09-13T08:45:32.617486+0000","last_active":"2024-09-13T08:45:34.618931+0000","last_peered":"2024-09-13T08:45:34.618931+0000","last_clean":"2024-09-13T08:45:34.618931+0000","last_became_active":"2024-09-13T08:45:32.616893+0000","last_became_peered":"2024-09-13T08:45:32.616893+0000","last_unstale":"2024-09-13T08:45:34.618931+0000","last_undegraded":"2024-09-13T08:45:34.618931+0000","last_fullsized":"2024-09-13T08:45:34.618931+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:47:52.852645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619443+0000","last_change":"2024-09-13T08:45:33.618697+0000","last_active":"2024-09-13T08:45:34.619443+0000","last_peered":"2024-09-13T08:45:34.619443+0000","last_clean":"2024-09-13T08:45:34.619443+0000","last_became_active":"2024-09-13T08:45:33.618209+0000","last_became_peered":"2024-09-13T08:45:33.618209+0000","last_unstale":"2024-09-13T08:45:34.619443+0000","last_undegraded":"2024-09-13T08:45:34.619443+0000","last_fullsized":"2024-09-13T08:45:34.619443+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:57.571587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.6","version":"151'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.617414+0000","last_change":"2024-09-13T08:45:32.621813+0000","last_active":"2024-09-13T08:45:34.617414+0000","last_peered":"2024-09-13T08:45:34.617414+0000","last_clean":"2024-09-13T08:45:34.617414+0000","last_became_active":"2024-09-13T08:45:32.621424+0000","last_became_peered":"2024-09-13T08:45:32.621424+0000","last_unstale":"2024-09-13T08:45:34.617414+0000","last_undegraded":"2024-09-13T08:45:34.617414+0000","last_fullsized":"2024-09-13T08:45:34.617414+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:51:20.741097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.619333+0000","last_change":"2024-09-13T08:45:33.622422+0000","last_active":"2024-09-13T08:45:34.619333+0000","last_peered":"2024-09-13T08:45:34.619333+0000","last_clean":"2024-09-13T08:45:34.619333+0000","last_became_active":"2024-09-13T08:45:33.622131+0000","last_became_peered":"2024-09-13T08:45:33.622131+0000","last_unstale":"2024-09-13T08:45:34.619333+0000","last_undegraded":"2024-09-13T08:45:34.619333+0000","last_fullsized":"2024-09-13T08:45:34.619333+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:02:33.435544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.630238+0000","last_change":"2024-09-13T08:45:32.614709+0000","last_active":"2024-09-13T08:45:34.630238+0000","last_peered":"2024-09-13T08:45:34.630238+0000","last_clean":"2024-09-13T08:45:34.630238+0000","last_became_active":"2024-09-13T08:45:32.614407+0000","last_became_peered":"2024-09-13T08:45:32.614407+0000","last_unstale":"2024-09-13T08:45:34.630238+0000","last_undegraded":"2024-09-13T08:45:34.630238+0000","last_fullsized":"2024-09-13T08:45:34.630238+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:59:41.458139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618146+0000","last_change":"2024-09-13T08:45:33.616093+0000","last_active":"2024-09-13T08:45:34.618146+0000","last_peered":"2024-09-13T08:45:34.618146+0000","last_clean":"2024-09-13T08:45:34.618146+0000","last_became_active":"2024-09-13T08:45:33.614646+0000","last_became_peered":"2024-09-13T08:45:33.614646+0000","last_unstale":"2024-09-13T08:45:34.618146+0000","last_undegraded":"2024-09-13T08:45:34.618146+0000","last_fullsized":"2024-09-13T08:45:34.618146+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:50:21.334184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204955+0000","last_change":"2024-09-13T08:45:32.621977+0000","last_active":"2024-09-13T08:45:36.204955+0000","last_peered":"2024-09-13T08:45:36.204955+0000","last_clean":"2024-09-13T08:45:36.204955+0000","last_became_active":"2024-09-13T08:45:32.621650+0000","last_became_peered":"2024-09-13T08:45:32.621650+0000","last_unstale":"2024-09-13T08:45:36.204955+0000","last_undegraded":"2024-09-13T08:45:36.204955+0000","last_fullsized":"2024-09-13T08:45:36.204955+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:42.407504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.618711+0000","last_change":"2024-09-13T08:45:33.611854+0000","last_active":"2024-09-13T08:45:34.618711+0000","last_peered":"2024-09-13T08:45:34.618711+0000","last_clean":"2024-09-13T08:45:34.618711+0000","last_became_active":"2024-09-13T08:45:33.611530+0000","last_became_peered":"2024-09-13T08:45:33.611530+0000","last_unstale":"2024-09-13T08:45:34.618711+0000","last_undegraded":"2024-09-13T08:45:34.618711+0000","last_fullsized":"2024-09-13T08:45:34.618711+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:46.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894408+0000","last_change":"2024-09-13T08:45:32.616012+0000","last_active":"2024-09-13T08:45:34.894408+0000","last_peered":"2024-09-13T08:45:34.894408+0000","last_clean":"2024-09-13T08:45:34.894408+0000","last_became_active":"2024-09-13T08:45:32.615746+0000","last_became_peered":"2024-09-13T08:45:32.615746+0000","last_unstale":"2024-09-13T08:45:34.894408+0000","last_undegraded":"2024-09-13T08:45:34.894408+0000","last_fullsized":"2024-09-13T08:45:34.894408+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:29.505007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629615+0000","last_change":"2024-09-13T08:45:33.629666+0000","last_active":"2024-09-13T08:45:34.629615+0000","last_peered":"2024-09-13T08:45:34.629615+0000","last_clean":"2024-09-13T08:45:34.629615+0000","last_became_active":"2024-09-13T08:45:33.629353+0000","last_became_peered":"2024-09-13T08:45:33.629353+0000","last_unstale":"2024-09-13T08:45:34.629615+0000","last_undegraded":"2024-09-13T08:45:34.629615+0000","last_fullsized":"2024-09-13T08:45:34.629615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:56.955592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204420+0000","last_change":"2024-09-13T08:45:32.615086+0000","last_active":"2024-09-13T08:45:36.204420+0000","last_peered":"2024-09-13T08:45:36.204420+0000","last_clean":"2024-09-13T08:45:36.204420+0000","last_became_active":"2024-09-13T08:45:32.614772+0000","last_became_peered":"2024-09-13T08:45:32.614772+0000","last_unstale":"2024-09-13T08:45:36.204420+0000","last_undegraded":"2024-09-13T08:45:36.204420+0000","last_fullsized":"2024-09-13T08:45:36.204420+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:27:21.548606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.204070+0000","last_change":"2024-09-13T08:45:33.626564+0000","last_active":"2024-09-13T08:45:36.204070+0000","last_peered":"2024-09-13T08:45:36.204070+0000","last_clean":"2024-09-13T08:45:36.204070+0000","last_became_active":"2024-09-13T08:45:33.626279+0000","last_became_peered":"2024-09-13T08:45:33.626279+0000","last_unstale":"2024-09-13T08:45:36.204070+0000","last_undegraded":"2024-09-13T08:45:36.204070+0000","last_fullsized":"2024-09-13T08:45:36.204070+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:20.496268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629689+0000","last_change":"2024-09-13T08:45:32.615367+0000","last_active":"2024-09-13T08:45:34.629689+0000","last_peered":"2024-09-13T08:45:34.629689+0000","last_clean":"2024-09-13T08:45:34.629689+0000","last_became_active":"2024-09-13T08:45:32.614975+0000","last_became_peered":"2024-09-13T08:45:32.614975+0000","last_unstale":"2024-09-13T08:45:34.629689+0000","last_undegraded":"2024-09-13T08:45:34.629689+0000","last_fullsized":"2024-09-13T08:45:34.629689+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:10:16.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629550+0000","last_change":"2024-09-13T08:45:33.619392+0000","last_active":"2024-09-13T08:45:34.629550+0000","last_peered":"2024-09-13T08:45:34.629550+0000","last_clean":"2024-09-13T08:45:34.629550+0000","last_became_active":"2024-09-13T08:45:33.619085+0000","last_became_peered":"2024-09-13T08:45:33.619085+0000","last_unstale":"2024-09-13T08:45:34.629550+0000","last_undegraded":"2024-09-13T08:45:34.629550+0000","last_fullsized":"2024-09-13T08:45:34.629550+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:01.245998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.629330+0000","last_change":"2024-09-13T08:45:32.607846+0000","last_active":"2024-09-13T08:45:34.629330+0000","last_peered":"2024-09-13T08:45:34.629330+0000","last_clean":"2024-09-13T08:45:34.629330+0000","last_became_active":"2024-09-13T08:45:32.607548+0000","last_became_peered":"2024-09-13T08:45:32.607548+0000","last_unstale":"2024-09-13T08:45:34.629330+0000","last_undegraded":"2024-09-13T08:45:34.629330+0000","last_fullsized":"2024-09-13T08:45:34.629330+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:13.011757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:36.205572+0000","last_change":"2024-09-13T08:45:33.634276+0000","last_active":"2024-09-13T08:45:36.205572+0000","last_peered":"2024-09-13T08:45:36.205572+0000","last_clean":"2024-09-13T08:45:36.205572+0000","last_became_active":"2024-09-13T08:45:33.633911+0000","last_became_peered":"2024-09-13T08:45:33.633911+0000","last_unstale":"2024-09-13T08:45:36.205572+0000","last_undegraded":"2024-09-13T08:45:36.205572+0000","last_fullsized":"2024-09-13T08:45:36.205572+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:32.588174+0000","last_clean_scrub_stamp":"2024-09-13T08:45:32.588174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:18:13.314925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894576+0000","last_change":"2024-09-13T08:45:32.617353+0000","last_active":"2024-09-13T08:45:34.894576+0000","last_peered":"2024-09-13T08:45:34.894576+0000","last_clean":"2024-09-13T08:45:34.894576+0000","last_became_active":"2024-09-13T08:45:32.616890+0000","last_became_peered":"2024-09-13T08:45:32.616890+0000","last_unstale":"2024-09-13T08:45:34.894576+0000","last_undegraded":"2024-09-13T08:45:34.894576+0000","last_fullsized":"2024-09-13T08:45:34.894576+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:45:31.579747+0000","last_clean_scrub_stamp":"2024-09-13T08:45:31.579747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:38.873438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":374,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T08:45:34.894373+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:45:34.894373+0000","last_peered":"2024-09-13T08:45:34.894373+0000","last_clean":"2024-09-13T08:45:34.894373+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:45:34.894373+0000","last_undegraded":"2024-09-13T08:45:34.894373+0000","last_fullsized":"2024-09-13T08:45:34.894373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509547,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1888256,"data_stored":837095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509547,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":837129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509547,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37780,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334060,"statfs":{"total":96636764160,"available":96598077440,"internally_reserved":0,"allocated":2490368,"data_stored":1428135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542250,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":837129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509547,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1904640,"data_stored":838729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509547,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1892352,"data_stored":837677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509548,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33636,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338204,"statfs":{"total":96636764160,"available":96602320896,"internally_reserved":0,"allocated":2506752,"data_stored":1429257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509546,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1880064,"data_stored":830821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:45:38.336 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:45:38.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.539+0000 7fcb6f000640 1 -- 172.21.15.31:0/2290291702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb680cbe50 msgr2=0x7fcb680b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.539+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2290291702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb680cbe50 0x7fcb680b29e0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fcb60005fd0 tx=0x7fcb6002ab30 comp rx=0 tx=0).stop 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.539+0000 7fcb6f000640 1 -- 172.21.15.31:0/2290291702 shutdown_connections 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.539+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2290291702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb680cbe50 0x7fcb680b29e0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.539+0000 7fcb6f000640 1 -- 172.21.15.31:0/2290291702 >> 172.21.15.31:0/2290291702 conn(0x7fcb680cfbb0 msgr2=0x7fcb680cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.540+0000 7fcb6f000640 1 -- 172.21.15.31:0/2290291702 shutdown_connections 2024-09-13T08:45:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.540+0000 7fcb6f000640 1 -- 172.21.15.31:0/2290291702 wait complete. 2024-09-13T08:45:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.541+0000 7fcb6f000640 1 Processor -- start 2024-09-13T08:45:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.541+0000 7fcb6f000640 1 -- start start 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb680cbe50 0x7fcb6815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb6815bfc0 0x7fcb681f8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb680b30a0 con 0x7fcb68072560 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb680b2f20 con 0x7fcb6815bfc0 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb6f000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fcb680b3220 con 0x7fcb680cbe50 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.542+0000 7fcb67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45480/0 (socket says 172.21.15.31:45480) 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 -- 172.21.15.31:0/2455658425 learned_addr learned my addr 172.21.15.31:0/2455658425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb6d000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb6815bfc0 0x7fcb681f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67400640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb680cbe50 0x7fcb6815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 -- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb680cbe50 msgr2=0x7fcb6815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb680cbe50 0x7fcb6815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 -- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb6815bfc0 msgr2=0x7fcb681f8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb6815bfc0 0x7fcb681f8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 -- 172.21.15.31:0/2455658425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6815d480 con 0x7fcb68072560 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.543+0000 7fcb67e00640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c3a0 tx=0x7fcb5800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.544+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58017070 con 0x7fcb68072560 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.544+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb580053c0 con 0x7fcb68072560 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.544+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58003850 con 0x7fcb68072560 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.544+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6815d760 con 0x7fcb68072560 2024-09-13T08:45:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.544+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb680c50a0 con 0x7fcb68072560 2024-09-13T08:45:38.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.545+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb680b29e0 con 0x7fcb68072560 2024-09-13T08:45:38.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.546+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fcb5801a070 con 0x7fcb68072560 2024-09-13T08:45:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.547+0000 7fcb5fe00640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 0x7fcb4807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.547+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fcb5809a960 con 0x7fcb68072560 2024-09-13T08:45:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.547+0000 7fcb67400640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 0x7fcb4807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.548+0000 7fcb67400640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 0x7fcb4807b3d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcb680b1ad0 tx=0x7fcb54009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:38.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.551+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fcb58061e70 con 0x7fcb68072560 2024-09-13T08:45:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.822+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fcb680b5690 con 0x7fcb68072560 2024-09-13T08:45:38.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.825+0000 7fcb5fe00640 1 -- 172.21.15.31:0/2455658425 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7fcb580668a0 con 0x7fcb68072560 2024-09-13T08:45:38.825 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:45:38.826 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:45:38.826 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:45:38.826 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:45:38.826 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:45:38.826 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 msgr2=0x7fcb4807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 0x7fcb4807b3d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcb680b1ad0 tx=0x7fcb54009040 comp rx=0 tx=0).stop 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 msgr2=0x7fcb680c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c3a0 tx=0x7fcb5800c860 comp rx=0 tx=0).stop 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 shutdown_connections 2024-09-13T08:45:38.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fcb48078f10 0x7fcb4807b3d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb6815bfc0 0x7fcb681f8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fcb680cbe50 0x7fcb6815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 --2- 172.21.15.31:0/2455658425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb68072560 0x7fcb680c93f0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 >> 172.21.15.31:0/2455658425 conn(0x7fcb680cfbb0 msgr2=0x7fcb680cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 shutdown_connections 2024-09-13T08:45:38.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:38.827+0000 7fcb6f000640 1 -- 172.21.15.31:0/2455658425 wait complete. 2024-09-13T08:45:38.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 -- 172.21.15.31:0/1863755107 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 msgr2=0x7f7c5c0cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 --2- 172.21.15.31:0/1863755107 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c0cbc50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7c54005fd0 tx=0x7f7c5402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 -- 172.21.15.31:0/1863755107 shutdown_connections 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 --2- 172.21.15.31:0/1863755107 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c0cbc50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 --2- 172.21.15.31:0/1863755107 >> v2:172.21.15.45:3300/0 conn(0x7f7c5c0cf850 0x7f7c5c0ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 -- 172.21.15.31:0/1863755107 >> 172.21.15.31:0/1863755107 conn(0x7f7c5c0cc910 msgr2=0x7f7c5c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 -- 172.21.15.31:0/1863755107 shutdown_connections 2024-09-13T08:45:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.004+0000 7f7c64600640 1 -- 172.21.15.31:0/1863755107 wait complete. 2024-09-13T08:45:39.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.006+0000 7f7c64600640 1 Processor -- start 2024-09-13T08:45:39.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.006+0000 7f7c64600640 1 -- start start 2024-09-13T08:45:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c0cf850 0x7f7c5c09af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1600f0 con 0x7f7c5c15bd70 2024-09-13T08:45:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c0cb850 con 0x7f7c5c0cf850 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c64600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7c5c09dab0 con 0x7f7c5c0ce900 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c62600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c62600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45486/0 (socket says 172.21.15.31:45486) 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c62600640 1 -- 172.21.15.31:0/2085013656 learned_addr learned my addr 172.21.15.31:0/2085013656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c61c00640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.007+0000 7f7c61200640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c0cf850 0x7f7c5c09af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 -- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 msgr2=0x7f7c5c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 -- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c0cf850 msgr2=0x7f7c5c09af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c0cf850 0x7f7c5c09af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 -- 172.21.15.31:0/2085013656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5c09cf30 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c61c00640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c62600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009cd0 tx=0x7f7c54031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.008+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54040070 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.009+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c54004040 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.009+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54004450 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.009+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5c09d210 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.009+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c5c15f0d0 con 0x7f7c5c15bd70 2024-09-13T08:45:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.010+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5c0ce3c0 con 0x7f7c5c15bd70 2024-09-13T08:45:39.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.011+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f7c54039040 con 0x7f7c5c15bd70 2024-09-13T08:45:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.012+0000 7f7c59600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 0x7f7c3c07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.012+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f7c540bcbb0 con 0x7f7c5c15bd70 2024-09-13T08:45:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.012+0000 7f7c61c00640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 0x7f7c3c07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.013+0000 7f7c61c00640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 0x7f7c3c07b3d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7c50001b10 tx=0x7f7c50009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.016+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7c54084030 con 0x7f7c5c15bd70 2024-09-13T08:45:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.251+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c5c0c35b0 con 0x7f7c5c15bd70 2024-09-13T08:45:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.251+0000 7f7c59600640 1 -- 172.21.15.31:0/2085013656 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5227 (secure 0 0 0) 0x7f7c54088a60 con 0x7f7c5c15bd70 2024-09-13T08:45:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:39.253 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":99,"btime":"2024-09-13T08:45:36:620992+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:39.253 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 99 2024-09-13T08:45:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 msgr2=0x7f7c3c07b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 0x7f7c3c07b3d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7c50001b10 tx=0x7f7c50009040 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 msgr2=0x7f7c5c158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009cd0 tx=0x7f7c54031760 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 shutdown_connections 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c3c078f10 0x7f7c3c07b3d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c5c15bd70 0x7f7c5c158ad0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c0cf850 0x7f7c5c09af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 --2- 172.21.15.31:0/2085013656 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7c5c0ce900 0x7f7c5c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 >> 172.21.15.31:0/2085013656 conn(0x7f7c5c0cc910 msgr2=0x7f7c5c0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.254+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 shutdown_connections 2024-09-13T08:45:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.255+0000 7f7c64600640 1 -- 172.21.15.31:0/2085013656 wait complete. 2024-09-13T08:45:39.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:45:33.041130+0000', 'modified': '2024-09-13T08:45:36.057928+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8454}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8454': {'gid': 8454, 'name': 'c', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6834/3730730208', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 3730730208}, {'type': 'v1', 'addr': '172.21.15.31:6834', 'nonce': 3730730208}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8454, 'qdb_cluster': [8454]} 2024-09-13T08:45:39.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:45:39.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:45:39.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 -- 172.21.15.31:0/702189565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0004800 msgr2=0x7f0cf0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 --2- 172.21.15.31:0/702189565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0004800 0x7f0cf0004c80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400c2e0 tx=0x7f0cf0005830 comp rx=0 tx=0).stop 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d0e400640 1 -- 172.21.15.31:0/702189565 reap_dead start 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 -- 172.21.15.31:0/702189565 shutdown_connections 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 -- 172.21.15.31:0/702189565 >> 172.21.15.31:0/702189565 conn(0x7f0d080b18f0 msgr2=0x7f0d080b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 -- 172.21.15.31:0/702189565 shutdown_connections 2024-09-13T08:45:39.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.840+0000 7f0d10400640 1 -- 172.21.15.31:0/702189565 wait complete. 2024-09-13T08:45:39.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.842+0000 7f0d10400640 1 Processor -- start 2024-09-13T08:45:39.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.842+0000 7f0d10400640 1 -- start start 2024-09-13T08:45:39.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cf0004800 0x7f0d0815b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d08158810 0x7f0d081f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d080ce950 con 0x7f0d08148d70 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d080ce7d0 con 0x7f0cf0004800 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d10400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0d08073cf0 con 0x7f0d08158810 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cf0004800 0x7f0d0815b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45508/0 (socket says 172.21.15.31:45508) 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d0d000640 1 -- 172.21.15.31:0/2035303410 learned_addr learned my addr 172.21.15.31:0/2035303410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.843+0000 7f0d0e400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d08158810 0x7f0d081f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 -- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d08158810 msgr2=0x7f0d081f7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d08158810 0x7f0d081f7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 -- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cf0004800 msgr2=0x7f0d0815b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cf0004800 0x7f0d0815b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 -- 172.21.15.31:0/2035303410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0815a0c0 con 0x7f0d08148d70 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d0d000640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402b140 tx=0x7f0cf4006c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d05600640 1 -- 172.21.15.31:0/2035303410 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4031040 con 0x7f0d08148d70 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d05600640 1 -- 172.21.15.31:0/2035303410 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf40035f0 con 0x7f0d08148d70 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d05600640 1 -- 172.21.15.31:0/2035303410 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4003a40 con 0x7f0d08148d70 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.844+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0815a3a0 con 0x7f0d08148d70 2024-09-13T08:45:39.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.845+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0815a960 con 0x7f0d08148d70 2024-09-13T08:45:39.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.847+0000 7f0d05600640 1 -- 172.21.15.31:0/2035303410 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f0cf4034070 con 0x7f0d08148d70 2024-09-13T08:45:39.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.848+0000 7f0d05600640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 0x7f0ce807b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.849+0000 7f0d05600640 1 -- 172.21.15.31:0/2035303410 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f0cf40bb420 con 0x7f0d08148d70 2024-09-13T08:45:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.849+0000 7f0d132ca740 1 Processor -- start 2024-09-13T08:45:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.849+0000 7f0d0da00640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 0x7f0ce807b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.849+0000 7f0d132ca740 1 -- start start 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.849+0000 7f0d0da00640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 0x7f0ce807b400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009bd0 tx=0x7f0cfc005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff5a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff562fa0 0x5652ff5a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5652ff4e5430 con 0x5652ff562a20 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5652ff4d55d0 con 0x5652ff5a78b0 2024-09-13T08:45:39.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x5652ff4c4a00 con 0x5652ff562fa0 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45520/0 (socket says 172.21.15.31:45520) 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 -- 172.21.15.31:0/820051723 learned_addr learned my addr 172.21.15.31:0/820051723 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0da00640 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff5a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0e400640 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff562fa0 0x5652ff5a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 -- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff562fa0 msgr2=0x5652ff5a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff562fa0 0x5652ff5a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 -- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 msgr2=0x5652ff5a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.850+0000 7f0d0d000640 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff5a7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.851+0000 7f0d0d000640 1 -- 172.21.15.31:0/820051723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652ff48d520 con 0x5652ff562a20 2024-09-13T08:45:39.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.851+0000 7f0d0d000640 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40828e0 tx=0x7f0cf4082720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a677eaa57f6909 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.851+0000 7f0cfa000640 1 -- 172.21.15.31:0/820051723 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4082460 con 0x5652ff562a20 2024-09-13T08:45:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.851+0000 7f0cfa000640 1 -- 172.21.15.31:0/820051723 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf4046060 con 0x5652ff562a20 2024-09-13T08:45:39.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.851+0000 7f0cfa000640 1 -- 172.21.15.31:0/820051723 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4082950 con 0x5652ff562a20 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 -- 172.21.15.31:0/820051723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 msgr2=0x5652ff5a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40828e0 tx=0x7f0cf4082720 comp rx=0 tx=0).stop 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 -- 172.21.15.31:0/820051723 shutdown_connections 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff562fa0 0x5652ff5a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562a20 0x5652ff5a6390 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 --2- 172.21.15.31:0/820051723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff5a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 -- 172.21.15.31:0/820051723 >> 172.21.15.31:0/820051723 conn(0x5652ff55dff0 msgr2=0x5652ff561a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 -- 172.21.15.31:0/820051723 shutdown_connections 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.852+0000 7f0d132ca740 1 -- 172.21.15.31:0/820051723 wait complete. 2024-09-13T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.853+0000 7f0d132ca740 1 -- start start 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.853+0000 7f0d132ca740 1 Processor -- start 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5a78b0 0x5652ff5b9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5863c0 0x5652ff5b9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5652ff46b330 con 0x5652ff562fa0 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5652ff4c8810 con 0x5652ff5863c0 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x5652ff235620 con 0x5652ff5a78b0 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d132ca740 10 client.0 _ll_register_callbacks cb 0x5652ff470420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45528/0 (socket says 172.21.15.31:45528) 2024-09-13T08:45:39.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d0da00640 1 -- 172.21.15.31:0/3952175669 learned_addr learned my addr 172.21.15.31:0/3952175669 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d0e400640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5863c0 0x5652ff5b9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.854+0000 7f0d0d000640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5a78b0 0x5652ff5b9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 -- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5a78b0 msgr2=0x5652ff5b9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5a78b0 0x5652ff5b9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 -- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5863c0 msgr2=0x5652ff5b9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5863c0 0x5652ff5b9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 -- 172.21.15.31:0/3952175669 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652ff226850 con 0x5652ff562fa0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d0da00640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc01fa60 tx=0x7f0cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767540543dd197ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc004050 con 0x5652ff562fa0 2024-09-13T08:45:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf4046060 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f0cfc0041f0 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0cfc032050 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d132ca740 10 client.8577 fetch_fsmap learned FSMap version 99 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.855+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5652ff29d6b0 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf4046060 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 99) ==== 3502+0+0 (secure 0 0 0) 0x7f0cfc011940 con 0x5652ff562fa0 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 10 client.8577 fetch_fsmap finished waiting for FSMap version 99 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 10 client.8577 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 20 client.8577 populate_metadata read hostname 'smithi031' 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff589f10 0x5652ff5c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 4 client.8577 mds_command: new command op to 8454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:45:39.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 --> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652ff4d7520 con 0x5652ff589f10 2024-09-13T08:45:39.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.856+0000 7f0d0d000640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff589f10 0x5652ff5c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.857+0000 7f0d0d000640 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff589f10 0x5652ff5c3fb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e88e5782100c85f server_cookie=be2075679178f847 in_seq=0 out_seq=0 2024-09-13T08:45:39.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.857+0000 7f0ceea00640 10 client.8577 ms_handle_connect on v2:172.21.15.31:6832/3730730208 2024-09-13T08:45:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0ceea00640 1 -- 172.21.15.31:0/3952175669 <== mds.0 v2:172.21.15.31:6832/3730730208 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5652ff4d7520 con 0x5652ff589f10 2024-09-13T08:45:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0ceea00640 10 client.8577 handle_command_reply: tid=0 2024-09-13T08:45:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 client.8577 shutdown 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 msgr2=0x5652ff57aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc01fa60 tx=0x7f0cfc0046c0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 shutdown_connections 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff589f10 0x5652ff5c3fb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5863c0 0x5652ff5b9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5a78b0 0x5652ff5b9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 --2- 172.21.15.31:0/3952175669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff57aef0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 >> 172.21.15.31:0/3952175669 conn(0x5652ff562a20 msgr2=0x5652ff55b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0ceea00640 0 client.8577 ms_handle_reset on v2:172.21.15.31:6832/3730730208 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 shutdown_connections 2024-09-13T08:45:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 1 -- 172.21.15.31:0/3952175669 wait complete. 2024-09-13T08:45:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.858+0000 7f0d132ca740 20 client.8577 trim_cache size 0 max 16384 2024-09-13T08:45:39.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 Processor -- start 2024-09-13T08:45:39.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 -- start start 2024-09-13T08:45:39.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff55f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff55f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d08200b20 con 0x5652ff562fa0 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d081f5580 con 0x5652ff5a78b0 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.878+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0d082004d0 con 0x5652ff5863c0 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45542/0 (socket says 172.21.15.31:45542) 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 -- 172.21.15.31:0/1505973195 learned_addr learned my addr 172.21.15.31:0/1505973195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 -- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 msgr2=0x5652ff55f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0d000640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff55f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0e400640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff55f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff55f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 -- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 msgr2=0x5652ff55f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff55f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 -- 172.21.15.31:0/1505973195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652ff5a8000 con 0x5652ff562fa0 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0d000640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff55f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0e400640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff55f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.879+0000 7f0d0da00640 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc004650 tx=0x7f0cfc012930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845e57fe8a4efe63 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0cf9600640 1 -- 172.21.15.31:0/1505973195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc03b070 con 0x5652ff562fa0 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0cf9600640 1 -- 172.21.15.31:0/1505973195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cfc011a10 con 0x5652ff562fa0 2024-09-13T08:45:39.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0cf9600640 1 -- 172.21.15.31:0/1505973195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc036630 con 0x5652ff562fa0 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 -- 172.21.15.31:0/1505973195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 msgr2=0x5652ff5b9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc004650 tx=0x7f0cfc012930 comp rx=0 tx=0).stop 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 -- 172.21.15.31:0/1505973195 shutdown_connections 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff5a78b0 0x5652ff55f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff55f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 --2- 172.21.15.31:0/1505973195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff562fa0 0x5652ff5b9730 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 -- 172.21.15.31:0/1505973195 >> 172.21.15.31:0/1505973195 conn(0x5652ff562a20 msgr2=0x5652ff55cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 -- 172.21.15.31:0/1505973195 shutdown_connections 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.880+0000 7f0d132ca740 1 -- 172.21.15.31:0/1505973195 wait complete. 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.881+0000 7f0d132ca740 1 -- start start 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.881+0000 7f0d132ca740 1 Processor -- start 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff620950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 0x5652ff55ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5652ff589d90 con 0x5652ff5a78b0 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5652ff5893a0 con 0x5652ff562fa0 2024-09-13T08:45:39.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x5652ff58a3b0 con 0x5652ff5863c0 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.882+0000 7f0d132ca740 10 client.0 _ll_register_callbacks cb 0x7f0cfc01f890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 0x5652ff55ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:36288/0 (socket says 172.21.15.31:36288) 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0da00640 1 -- 172.21.15.31:0/4205294044 learned_addr learned my addr 172.21.15.31:0/4205294044 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0d000640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff620950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 -- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 msgr2=0x5652ff620950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff620950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 -- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 msgr2=0x5652ff62aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:45:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 -- 172.21.15.31:0/4205294044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652ff5bd210 con 0x5652ff5a78b0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0da00640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0d0e400640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 0x5652ff55ae40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f0d080cf770 tx=0x7f0d00011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c80c80a83b125e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.883+0000 7f0ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00003b70 con 0x5652ff5a78b0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f0d00003d10 con 0x5652ff5a78b0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cfc011a10 con 0x5652ff5a78b0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0d00027050 con 0x5652ff5a78b0 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 10 client.8583 fetch_fsmap learned FSMap version 99 2024-09-13T08:45:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5652ff620e90 con 0x5652ff5a78b0 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cfc011a10 con 0x5652ff5a78b0 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 99) ==== 3502+0+0 (secure 0 0 0) 0x7f0d0002a020 con 0x5652ff5a78b0 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 10 client.8583 fetch_fsmap finished waiting for FSMap version 99 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 10 client.8583 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 20 client.8583 populate_metadata read hostname 'smithi031' 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff627290 0x5652ff5a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 4 client.8583 mds_command: new command op to 8454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 --> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] -- command(tid 0: {"prefix": "status"}) -- 0x5652ff4d2fd0 con 0x5652ff627290 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.884+0000 7f0d0da00640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff627290 0x5652ff5a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0d0da00640 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff627290 0x5652ff5a60f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6659a23973ee025a server_cookie=784543c035b2dac2 in_seq=0 out_seq=0 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0ceea00640 10 client.8583 ms_handle_connect on v2:172.21.15.31:6832/3730730208 2024-09-13T08:45:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0ceea00640 1 -- 172.21.15.31:0/4205294044 <== mds.0 v2:172.21.15.31:6832/3730730208 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5652ff4d2fd0 con 0x5652ff627290 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0ceea00640 10 client.8583 handle_command_reply: tid=0 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0d132ca740 1 client.8583 shutdown 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 msgr2=0x5652ff55ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.885+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 0x5652ff55ae40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f0d080cf770 tx=0x7f0d00011ae0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 shutdown_connections 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:6832/3730730208,v1:172.21.15.31:6834/3730730208] conn(0x5652ff627290 0x5652ff5a60f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652ff5a78b0 0x5652ff55ae40 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x5652ff5863c0 0x5652ff620950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 --2- 172.21.15.31:0/4205294044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652ff562fa0 0x5652ff62aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 >> 172.21.15.31:0/4205294044 conn(0x5652ff562a20 msgr2=0x5652ff5b9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0ceea00640 0 client.8583 ms_handle_reset on v2:172.21.15.31:6832/3730730208 2024-09-13T08:45:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 shutdown_connections 2024-09-13T08:45:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 1 -- 172.21.15.31:0/4205294044 wait complete. 2024-09-13T08:45:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d132ca740 20 client.8583 trim_cache size 0 max 16384 2024-09-13T08:45:39.893 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:45:39.893 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 8454, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2754400229999998, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 99, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 151, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 151, 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.548535912 2024-09-13T08:45:39.894 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:45:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 msgr2=0x7f0ce807b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 0x7f0ce807b400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009bd0 tx=0x7f0cfc005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 msgr2=0x7f0d081582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.886+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402b140 tx=0x7f0cf4006c50 comp rx=0 tx=0).stop 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 shutdown_connections 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0ce8078f40 0x7f0ce807b400 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0d08158810 0x7f0d081f7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d08148d70 0x7f0d081582d0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 --2- 172.21.15.31:0/2035303410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cf0004800 0x7f0d0815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 >> 172.21.15.31:0/2035303410 conn(0x7f0d080b18f0 msgr2=0x7f0d0806e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 shutdown_connections 2024-09-13T08:45:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:39.887+0000 7f0d10400640 1 -- 172.21.15.31:0/2035303410 wait complete. 2024-09-13T08:45:39.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:45:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 -- 172.21.15.31:0/1739103771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c0746d0 msgr2=0x7fec5c152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 --2- 172.21.15.31:0/1739103771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c0746d0 0x7fec5c152e50 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fec54005fd0 tx=0x7fec5402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 -- 172.21.15.31:0/1739103771 shutdown_connections 2024-09-13T08:45:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 --2- 172.21.15.31:0/1739103771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c0746d0 0x7fec5c152e50 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 -- 172.21.15.31:0/1739103771 >> 172.21.15.31:0/1739103771 conn(0x7fec5c0cc910 msgr2=0x7fec5c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:40.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 -- 172.21.15.31:0/1739103771 shutdown_connections 2024-09-13T08:45:40.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.062+0000 7fec63000640 1 -- 172.21.15.31:0/1739103771 wait complete. 2024-09-13T08:45:40.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.064+0000 7fec63000640 1 Processor -- start 2024-09-13T08:45:40.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.064+0000 7fec63000640 1 -- start start 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec5c0746d0 0x7fec5c09ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec5c0cf980 0x7fec5c15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c0b1ca0 con 0x7fec5c152a70 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c0b1b20 con 0x7fec5c0746d0 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec63000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fec5c0b1e20 con 0x7fec5c0cf980 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec61000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec61000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45564/0 (socket says 172.21.15.31:45564) 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec61000640 1 -- 172.21.15.31:0/689521362 learned_addr learned my addr 172.21.15.31:0/689521362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec5be00640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec5c0746d0 0x7fec5c09ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.065+0000 7fec53400640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec5c0cf980 0x7fec5c15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 -- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec5c0cf980 msgr2=0x7fec5c15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec5c0cf980 0x7fec5c15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 -- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec5c0746d0 msgr2=0x7fec5c09ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec5c0746d0 0x7fec5c09ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 -- 172.21.15.31:0/689521362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5c09cfb0 con 0x7fec5c152a70 2024-09-13T08:45:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec61000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fec54006010 tx=0x7fec54003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54031b90 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec54031e20 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5c15a2e0 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec540332f0 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.066+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec5c15a8a0 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.068+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec5c0b2280 con 0x7fec5c152a70 2024-09-13T08:45:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.069+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fec54031660 con 0x7fec5c152a70 2024-09-13T08:45:40.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.069+0000 7fec58c00640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 0x7fec3807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:40.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.070+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fec540bcb90 con 0x7fec5c152a70 2024-09-13T08:45:40.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.070+0000 7fec5be00640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 0x7fec3807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:40.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.070+0000 7fec5be00640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 0x7fec3807b3d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004f60 tx=0x7fec4c007e30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:40.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.074+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fec5403f030 con 0x7fec5c152a70 2024-09-13T08:45:40.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.306+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fec5c0cd100 con 0x7fec5c152a70 2024-09-13T08:45:40.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.307+0000 7fec58c00640 1 -- 172.21.15.31:0/689521362 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7fec54084010 con 0x7fec5c152a70 2024-09-13T08:45:40.308 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:45:40.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 msgr2=0x7fec3807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 0x7fec3807b3d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004f60 tx=0x7fec4c007e30 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 msgr2=0x7fec5c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fec54006010 tx=0x7fec54003ba0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 shutdown_connections 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec38078f10 0x7fec3807b3d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c152a70 0x7fec5c06f540 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec5c0cf980 0x7fec5c15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 --2- 172.21.15.31:0/689521362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec5c0746d0 0x7fec5c09ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.309+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 >> 172.21.15.31:0/689521362 conn(0x7fec5c0cc910 msgr2=0x7fec5c0ace40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.310+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 shutdown_connections 2024-09-13T08:45:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40.310+0000 7fec63000640 1 -- 172.21.15.31:0/689521362 wait complete. 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:45:40.326 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:40.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.327 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:40.363 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42191sec preferred_lft 42191sec 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:40.364 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:45:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:45:40.366 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:40.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.366 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:45:40.456 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:45:40.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.460 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:45:40.497 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:45:40.497 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:45:40.497 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:45:40.497 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:45:40.497 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:45:40.497 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:45:40.498 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:45:40.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:45:40.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:45:40.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:45:40.498 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:45:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:45:40.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:40.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.696 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:45:40.732 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:45:40.732 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:45:40.732 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:40.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:45:40.829 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:40.830 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:40.865 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:45:40.865 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:40.866 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:40.941 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:45:40.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:40.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:41.069 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:41.070 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:41.097 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:41.097 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:41.097 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:45:41.097 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:45:41.097 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:45:41.098 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:45:41.099 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:45:41.099 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:45:41.099 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:45:41.099 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:45:41.101 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:45:41.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:41.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:41.159 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:45:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:45:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:41.256 INFO:teuthology.orchestra.run.smithi031.stdout:client7217 2024-09-13T08:45:41.257 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:41.257 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:41.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 -- 172.21.15.31:0/205240915 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7900ca5c0 msgr2=0x7fb7900cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 --2- 172.21.15.31:0/205240915 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7900ca5c0 0x7fb7900cc300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb778009860 tx=0x7fb77802a530 comp rx=0 tx=0).stop 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 -- 172.21.15.31:0/205240915 shutdown_connections 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 --2- 172.21.15.31:0/205240915 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb7900ca5c0 0x7fb7900cc300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 -- 172.21.15.31:0/205240915 >> 172.21.15.31:0/205240915 conn(0x7fb7900cc910 msgr2=0x7fb7900cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:41.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 -- 172.21.15.31:0/205240915 shutdown_connections 2024-09-13T08:45:41.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.493+0000 7fb795a00640 1 -- 172.21.15.31:0/205240915 wait complete. 2024-09-13T08:45:41.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.495+0000 7fb795a00640 1 Processor -- start 2024-09-13T08:45:41.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.495+0000 7fb795a00640 1 -- start start 2024-09-13T08:45:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7900b1910 0x7fb790151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb79015dfc0 0x7fb7900afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7900b1da0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7900740a0 con 0x7fb7900b1910 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb795a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb7900b1f20 con 0x7fb79015dfc0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45566/0 (socket says 172.21.15.31:45566) 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 -- 172.21.15.31:0/2946559307 learned_addr learned my addr 172.21.15.31:0/2946559307 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb78f400640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb79015dfc0 0x7fb7900afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb78ea00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7900b1910 0x7fb790151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 -- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb79015dfc0 msgr2=0x7fb7900afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb79015dfc0 0x7fb7900afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 -- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7900b1910 msgr2=0x7fb790151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7900b1910 0x7fb790151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.496+0000 7fb786000640 1 -- 172.21.15.31:0/2946559307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7900b2cc0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb786000640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb778009980 tx=0x7fb778003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778005c90 con 0x7fb7900ca5c0 2024-09-13T08:45:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7901f4dc0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb778005020 con 0x7fb7900ca5c0 2024-09-13T08:45:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778005410 con 0x7fb7900ca5c0 2024-09-13T08:45:41.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.497+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7901f5b40 con 0x7fb7900ca5c0 2024-09-13T08:45:41.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.498+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7900af5d0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.499+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fb778039040 con 0x7fb7900ca5c0 2024-09-13T08:45:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.500+0000 7fb787400640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 0x7fb76c07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.500+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fb7780bc6c0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.501+0000 7fb78ea00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 0x7fb76c07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.501+0000 7fb78ea00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 0x7fb76c07b3d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb780001120 tx=0x7fb780009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.504+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb778040030 con 0x7fb7900ca5c0 2024-09-13T08:45:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.745+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb790073b00 con 0x7fb7900ca5c0 2024-09-13T08:45:41.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.745+0000 7fb787400640 1 -- 172.21.15.31:0/2946559307 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7fb7780841d0 con 0x7fb7900ca5c0 2024-09-13T08:45:41.746 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:45:41.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 msgr2=0x7fb76c07b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 0x7fb76c07b3d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb780001120 tx=0x7fb780009040 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 msgr2=0x7fb79009c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb778009980 tx=0x7fb778003c10 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 shutdown_connections 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb76c078f10 0x7fb76c07b3d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb79015dfc0 0x7fb7900afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7900ca5c0 0x7fb79009c6f0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 --2- 172.21.15.31:0/2946559307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7900b1910 0x7fb790151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 >> 172.21.15.31:0/2946559307 conn(0x7fb7900cc910 msgr2=0x7fb7900ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 shutdown_connections 2024-09-13T08:45:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:41.747+0000 7fb795a00640 1 -- 172.21.15.31:0/2946559307 wait complete. 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:45:41.764 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:41.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:41.765 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:41.805 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42195sec preferred_lft 42195sec 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.806 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.807 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:45:41.808 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:45:41.808 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:45:41.808 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:45:41.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:41.809 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:45:41.907 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:45:41.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:41.910 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:45:41.947 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:45:41.947 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:45:41.947 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:45:41.947 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:45:41.947 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:45:41.947 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:45:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:45:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:42.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:42.145 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:45:42.179 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:45:42.180 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:45:42.180 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:42.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:42.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:45:42.278 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:42.278 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:42.314 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:45:42.314 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:45:42.314 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:42.388 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:45:42.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:42.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:42.517 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:42.517 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:45:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:45:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:45:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:45:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:45:42.546 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:45:42.546 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:45:42.546 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:45:42.546 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:45:42.546 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:45:42.548 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:45:42.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:42.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:42.604 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:45:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:45:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:45:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:42.694 INFO:teuthology.orchestra.run.smithi045.stdout:client7226 2024-09-13T08:45:42.694 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:42.694 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:42.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 -- 172.21.15.31:0/1738348485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 msgr2=0x7f40b40ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 --2- 172.21.15.31:0/1738348485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40ca9e0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f40b40694f0 tx=0x7f40ac02a560 comp rx=0 tx=0).stop 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 -- 172.21.15.31:0/1738348485 shutdown_connections 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 --2- 172.21.15.31:0/1738348485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40ca9e0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 --2- 172.21.15.31:0/1738348485 >> v2:172.21.15.45:3300/0 conn(0x7f40b415b620 0x7f40b40ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 --2- 172.21.15.31:0/1738348485 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:42.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 -- 172.21.15.31:0/1738348485 >> 172.21.15.31:0/1738348485 conn(0x7f40b40cfbb0 msgr2=0x7f40b40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:42.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 -- 172.21.15.31:0/1738348485 shutdown_connections 2024-09-13T08:45:42.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.907+0000 7f40ba600640 1 -- 172.21.15.31:0/1738348485 wait complete. 2024-09-13T08:45:42.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.908+0000 7f40ba600640 1 Processor -- start 2024-09-13T08:45:42.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.909+0000 7f40ba600640 1 -- start start 2024-09-13T08:45:42.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.909+0000 7f40ba600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.909+0000 7f40ba600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b40b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40ba600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 0x7f40b40c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40ba600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40b415df90 con 0x7f40b406ee50 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40ba600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40b406ebc0 con 0x7f40b415b620 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40ba600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f40b415e110 con 0x7f40b415ad00 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45578/0 (socket says 172.21.15.31:45578) 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 -- 172.21.15.31:0/2034837855 learned_addr learned my addr 172.21.15.31:0/2034837855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b2a00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b40b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3e00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 0x7f40b40c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 -- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 msgr2=0x7f40b40b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b40b1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 -- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 msgr2=0x7f40b40c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 0x7f40b40c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 -- 172.21.15.31:0/2034837855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b40c41e0 con 0x7f40b406ee50 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b2a00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b40b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3e00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 0x7f40b40c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.910+0000 7f40b3400640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f40a400ece0 tx=0x7f40a400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.911+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4004d20 con 0x7f40b406ee50 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.911+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40a400b570 con 0x7f40b406ee50 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.911+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a400b9c0 con 0x7f40b406ee50 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.911+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b40c44c0 con 0x7f40b406ee50 2024-09-13T08:45:42.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.911+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b40b5330 con 0x7f40b406ee50 2024-09-13T08:45:42.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.912+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b40c4700 con 0x7f40b406ee50 2024-09-13T08:45:42.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.913+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f40a400bb60 con 0x7f40b406ee50 2024-09-13T08:45:42.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.914+0000 7f40ab400640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 0x7f409407b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:42.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.914+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f40a409c060 con 0x7f40b406ee50 2024-09-13T08:45:42.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.914+0000 7f40b2a00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 0x7f409407b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:42.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.915+0000 7f40b2a00640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 0x7f409407b400 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40b40b4e60 tx=0x7f40a0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:42.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:42.918+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f40a4063740 con 0x7f40b406ee50 2024-09-13T08:45:43.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.152+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f40b415b0e0 con 0x7f40b406ee50 2024-09-13T08:45:43.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.843+0000 7f40ab400640 1 -- 172.21.15.31:0/2034837855 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v100) ==== 136+0+0 (secure 0 0 0) 0x7f40a4068170 con 0x7f40b406ee50 2024-09-13T08:45:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 msgr2=0x7f409407b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 0x7f409407b400 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40b40b4e60 tx=0x7f40a0005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 msgr2=0x7f40b40b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f40a400ece0 tx=0x7f40a400c7a0 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 shutdown_connections 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4094078f40 0x7f409407b400 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40b415b620 0x7f40b40c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f40b415ad00 0x7f40b40b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 --2- 172.21.15.31:0/2034837855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40b406ee50 0x7f40b40b19d0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.846+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 >> 172.21.15.31:0/2034837855 conn(0x7f40b40cfbb0 msgr2=0x7f40b415d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.847+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 shutdown_connections 2024-09-13T08:45:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:43.847+0000 7f40ba600640 1 -- 172.21.15.31:0/2034837855 wait complete. 2024-09-13T08:45:43.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:45:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 -- 172.21.15.31:0/2352413575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 msgr2=0x7f5a9c0ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 --2- 172.21.15.31:0/2352413575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0ce0c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c06e610 tx=0x7f5a8c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 -- 172.21.15.31:0/2352413575 shutdown_connections 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 --2- 172.21.15.31:0/2352413575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0ce0c0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 --2- 172.21.15.31:0/2352413575 >> v2:172.21.15.45:3300/0 conn(0x7f5a9c15be30 0x7f5a9c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 -- 172.21.15.31:0/2352413575 >> 172.21.15.31:0/2352413575 conn(0x7f5a9c0b18f0 msgr2=0x7f5a9c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 -- 172.21.15.31:0/2352413575 shutdown_connections 2024-09-13T08:45:44.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.031+0000 7f5aa4800640 1 -- 172.21.15.31:0/2352413575 wait complete. 2024-09-13T08:45:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.033+0000 7f5aa4800640 1 Processor -- start 2024-09-13T08:45:44.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.033+0000 7f5aa4800640 1 -- start start 2024-09-13T08:45:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 0x7f5a9c0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 0x7f5a9c0b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c0749f0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c074870 con 0x7f5a9c15be30 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa4800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9c160160 con 0x7f5a9c0ce600 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45604/0 (socket says 172.21.15.31:45604) 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.034+0000 7f5aa1400640 1 -- 172.21.15.31:0/3743483058 learned_addr learned my addr 172.21.15.31:0/3743483058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1e00640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 0x7f5a9c0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa2800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 0x7f5a9c0b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 -- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 msgr2=0x7f5a9c0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 0x7f5a9c0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 -- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 msgr2=0x7f5a9c0b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 0x7f5a9c0b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 -- 172.21.15.31:0/3743483058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c1f80e0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1e00640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 0x7f5a9c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa2800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 0x7f5a9c0b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa1400640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0099c0 tx=0x7f5a8c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c03f070 con 0x7f5a9c0ced20 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a8c0047a0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9c1f83c0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.035+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c035030 con 0x7f5a9c0ced20 2024-09-13T08:45:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.036+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9c1f8980 con 0x7f5a9c0ced20 2024-09-13T08:45:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.036+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9c1605c0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.038+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f5a8c042070 con 0x7f5a9c0ced20 2024-09-13T08:45:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.039+0000 7f5a96000640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 0x7f5a8007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.039+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f5a8c0bc840 con 0x7f5a9c0ced20 2024-09-13T08:45:44.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.039+0000 7f5aa1e00640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 0x7f5a8007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.040+0000 7f5aa1e00640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 0x7f5a8007b3d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0b3410 tx=0x7f5a90005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.043+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5a8c0843a0 con 0x7f5a9c0ced20 2024-09-13T08:45:44.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.263+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f5a9c0b25f0 con 0x7f5a80078f10 2024-09-13T08:45:44.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.265+0000 7f5a96000640 1 -- 172.21.15.31:0/3743483058 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f5a9c0b25f0 con 0x7f5a80078f10 2024-09-13T08:45:44.266 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:45:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:45:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:45:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:45:44.266 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:45:44.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 msgr2=0x7f5a8007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 0x7f5a8007b3d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0b3410 tx=0x7f5a90005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 msgr2=0x7f5a9c0afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0099c0 tx=0x7f5a8c0038a0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 shutdown_connections 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5a80078f10 0x7f5a8007b3d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c15be30 0x7f5a9c0b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c0ced20 0x7f5a9c0afbe0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 --2- 172.21.15.31:0/3743483058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5a9c0ce600 0x7f5a9c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 >> 172.21.15.31:0/3743483058 conn(0x7f5a9c0b18f0 msgr2=0x7f5a9c0af2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.267+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 shutdown_connections 2024-09-13T08:45:44.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.268+0000 7f5aa4800640 1 -- 172.21.15.31:0/3743483058 wait complete. 2024-09-13T08:45:44.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 -- 172.21.15.31:0/1656038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 msgr2=0x7fdf140cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 --2- 172.21.15.31:0/1656038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140cd780 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdf08005fd0 tx=0x7fdf0802a4b0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 -- 172.21.15.31:0/1656038387 shutdown_connections 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 --2- 172.21.15.31:0/1656038387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 0x7fdf140cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 --2- 172.21.15.31:0/1656038387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140cd780 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 -- 172.21.15.31:0/1656038387 >> 172.21.15.31:0/1656038387 conn(0x7fdf140cfbb0 msgr2=0x7fdf140cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 -- 172.21.15.31:0/1656038387 shutdown_connections 2024-09-13T08:45:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.448+0000 7fdf1be00640 1 -- 172.21.15.31:0/1656038387 wait complete. 2024-09-13T08:45:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.450+0000 7fdf1be00640 1 Processor -- start 2024-09-13T08:45:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.450+0000 7fdf1be00640 1 -- start start 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 0x7fdf140c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf140cc190 0x7fdf14153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf140cbe50 con 0x7fdf14161480 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf14070040 con 0x7fdf140cc190 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf1be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdf140cbfd0 con 0x7fdf140cb380 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45612/0 (socket says 172.21.15.31:45612) 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf19e00640 1 -- 172.21.15.31:0/349658439 learned_addr learned my addr 172.21.15.31:0/349658439 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:44.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf19400640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 0x7fdf140c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.451+0000 7fdf18a00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf140cc190 0x7fdf14153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 -- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 msgr2=0x7fdf140c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 0x7fdf140c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 -- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf140cc190 msgr2=0x7fdf14153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf140cc190 0x7fdf14153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 -- 172.21.15.31:0/349658439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf141544a0 con 0x7fdf14161480 2024-09-13T08:45:44.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf19e00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00eaa0 tx=0x7fdf0c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c00cdb0 con 0x7fdf14161480 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf0c004c10 con 0x7fdf14161480 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1415da70 con 0x7fdf14161480 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.452+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c005020 con 0x7fdf14161480 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.453+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf140c46f0 con 0x7fdf14161480 2024-09-13T08:45:44.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.454+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf141625f0 con 0x7fdf14161480 2024-09-13T08:45:44.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.455+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7fdf0c0051c0 con 0x7fdf14161480 2024-09-13T08:45:44.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.456+0000 7fdf11600640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 0x7fdef407b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.456+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fdf0c09bf50 con 0x7fdf14161480 2024-09-13T08:45:44.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.456+0000 7fdf19400640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 0x7fdef407b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.457+0000 7fdf19400640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 0x7fdef407b3d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdf08033000 tx=0x7fdf08004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.461+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdf0c016030 con 0x7fdf14161480 2024-09-13T08:45:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.656+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fdf140728e0 con 0x7fdf14161480 2024-09-13T08:45:44.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.656+0000 7fdf11600640 1 -- 172.21.15.31:0/349658439 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7fdf0c063460 con 0x7fdf14161480 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 msgr2=0x7fdef407b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 0x7fdef407b3d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdf08033000 tx=0x7fdf08004730 comp rx=0 tx=0).stop 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 msgr2=0x7fdf140c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00eaa0 tx=0x7fdf0c00ef60 comp rx=0 tx=0).stop 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 shutdown_connections 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdef4078f10 0x7fdef407b3d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf14161480 0x7fdf140c51d0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf140cc190 0x7fdf14153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 --2- 172.21.15.31:0/349658439 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdf140cb380 0x7fdf140c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 >> 172.21.15.31:0/349658439 conn(0x7fdf140cfbb0 msgr2=0x7fdf140cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 shutdown_connections 2024-09-13T08:45:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.659+0000 7fdf1be00640 1 -- 172.21.15.31:0/349658439 wait complete. 2024-09-13T08:45:44.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:44.676 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol3760) 2024-09-13T08:45:44.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:44.740 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp79030/subvol83941) 2024-09-13T08:45:44.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol3760 2024-09-13T08:45:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 -- 172.21.15.31:0/3033342633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26ec004800 msgr2=0x7f26ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 --2- 172.21.15.31:0/3033342633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26ec004800 0x7f26ec004c80 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009b80 tx=0x7f26ec005830 comp rx=0 tx=0).stop 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f2707e00640 1 -- 172.21.15.31:0/3033342633 reap_dead start 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 -- 172.21.15.31:0/3033342633 shutdown_connections 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 -- 172.21.15.31:0/3033342633 >> 172.21.15.31:0/3033342633 conn(0x7f27080cfbb0 msgr2=0x7f27080cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 -- 172.21.15.31:0/3033342633 shutdown_connections 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.964+0000 7f270e600640 1 -- 172.21.15.31:0/3033342633 wait complete. 2024-09-13T08:45:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.966+0000 7f270e600640 1 Processor -- start 2024-09-13T08:45:44.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.966+0000 7f270e600640 1 -- start start 2024-09-13T08:45:44.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f26ec004800 0x7f27080c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27081416f0 0x7f2708141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f270806e740 con 0x7f27081420b0 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f270806e5c0 con 0x7f27081416f0 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f270e600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f27080cb740 con 0x7f26ec004800 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f2707e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f2707e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56574/0 (socket says 172.21.15.31:56574) 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f2707e00640 1 -- 172.21.15.31:0/1102677329 learned_addr learned my addr 172.21.15.31:0/1102677329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f2707400640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f26ec004800 0x7f27080c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.967+0000 7f2706a00640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27081416f0 0x7f2708141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 -- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f26ec004800 msgr2=0x7f27080c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f26ec004800 0x7f27080c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 -- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27081416f0 msgr2=0x7f2708141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27081416f0 0x7f2708141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 -- 172.21.15.31:0/1102677329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27080c4bb0 con 0x7f27081420b0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f2707e00640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2700005fd0 tx=0x7f270000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700003d80 con 0x7f27081420b0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2700004b20 con 0x7f27081420b0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27080c4e90 con 0x7f27081420b0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700004030 con 0x7f27081420b0 2024-09-13T08:45:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.968+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27080c5450 con 0x7f27081420b0 2024-09-13T08:45:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.969+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270806fa60 con 0x7f27081420b0 2024-09-13T08:45:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.971+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f27000041d0 con 0x7f27081420b0 2024-09-13T08:45:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.972+0000 7f26ff400640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 0x7f26e807b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.972+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f270009c560 con 0x7f27081420b0 2024-09-13T08:45:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.972+0000 7f2707400640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 0x7f26e807b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.972+0000 7f2707400640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 0x7f26e807b3d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26f8001550 tx=0x7f26f8006c10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:44.976+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2700064100 con 0x7f27081420b0 2024-09-13T08:45:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.199+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol3760", "target": ["mon-mgr", ""]}) -- 0x7f27080cb310 con 0x7f26e8078f10 2024-09-13T08:45:45.206 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:45.205+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:45.206 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:45.205+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:45.206 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:45.205+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:45.206 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:45.205+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:45.206 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:45.205+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.234+0000 7f26ff400640 1 -- 172.21.15.31:0/1102677329 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+29 (secure 0 0 0) 0x7f27080cb310 con 0x7f26e8078f10 2024-09-13T08:45:45.235 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol3760 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 msgr2=0x7f26e807b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 0x7f26e807b3d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26f8001550 tx=0x7f26f8006c10 comp rx=0 tx=0).stop 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 msgr2=0x7f27081507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2700005fd0 tx=0x7f270000f790 comp rx=0 tx=0).stop 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 shutdown_connections 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f26e8078f10 0x7f26e807b3d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27081420b0 0x7f27081507d0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27081416f0 0x7f2708141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 --2- 172.21.15.31:0/1102677329 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f26ec004800 0x7f27080c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 >> 172.21.15.31:0/1102677329 conn(0x7f27080cfbb0 msgr2=0x7f270815ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 shutdown_connections 2024-09-13T08:45:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.237+0000 7f270e600640 1 -- 172.21.15.31:0/1102677329 wait complete. 2024-09-13T08:45:45.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol83941 subvol_grp79030 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.419+0000 7f033b600640 1 -- 172.21.15.31:0/1288901696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cc190 msgr2=0x7f03340caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.419+0000 7f033b600640 1 --2- 172.21.15.31:0/1288901696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cc190 0x7f03340caa80 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f032c005fd0 tx=0x7f032c02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.419+0000 7f033b600640 1 -- 172.21.15.31:0/1288901696 shutdown_connections 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.419+0000 7f033b600640 1 --2- 172.21.15.31:0/1288901696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03340cc190 0x7f03340caa80 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.419+0000 7f033b600640 1 -- 172.21.15.31:0/1288901696 >> 172.21.15.31:0/1288901696 conn(0x7f03340cfbb0 msgr2=0x7f03340cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.420+0000 7f033b600640 1 -- 172.21.15.31:0/1288901696 shutdown_connections 2024-09-13T08:45:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.420+0000 7f033b600640 1 -- 172.21.15.31:0/1288901696 wait complete. 2024-09-13T08:45:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.421+0000 7f033b600640 1 Processor -- start 2024-09-13T08:45:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.421+0000 7f033b600640 1 -- start start 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03340cc190 0x7f03341f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 0x7f03340b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03340cbfd0 con 0x7f0334161480 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03340cbe50 con 0x7f03340cde90 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.422+0000 7f033b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03340cafc0 con 0x7f03340cc190 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56612/0 (socket says 172.21.15.31:56612) 2024-09-13T08:45:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 -- 172.21.15.31:0/2183966837 learned_addr learned my addr 172.21.15.31:0/2183966837 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f032be00640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 0x7f03340b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 -- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03340cc190 msgr2=0x7f03341f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03340cc190 0x7f03341f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 -- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 msgr2=0x7f03340b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 0x7f03340b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 -- 172.21.15.31:0/2183966837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03341f50f0 con 0x7f0334161480 2024-09-13T08:45:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f032be00640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 0x7f03340b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.423+0000 7f0339600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f032c009cd0 tx=0x7f032c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.424+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c040070 con 0x7f0334161480 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.424+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f032c004150 con 0x7f0334161480 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.424+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c0045a0 con 0x7f0334161480 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.424+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03340b3c00 con 0x7f0334161480 2024-09-13T08:45:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.424+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03341f5670 con 0x7f0334161480 2024-09-13T08:45:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.425+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03340cb470 con 0x7f0334161480 2024-09-13T08:45:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.426+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 101899+0+0 (secure 0 0 0) 0x7f032c039040 con 0x7f0334161480 2024-09-13T08:45:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.427+0000 7f0331600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 0x7f031007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.427+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f032c0bcd40 con 0x7f0334161480 2024-09-13T08:45:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.427+0000 7f0338c00640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 0x7f031007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.428+0000 7f0338c00640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 0x7f031007b3d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f03240078c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.431+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f032c084810 con 0x7f0334161480 2024-09-13T08:45:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.653+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol83941", "group_name": "subvol_grp79030", "target": ["mon-mgr", ""]}) -- 0x7f03340b5b30 con 0x7f0310078f10 2024-09-13T08:45:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.670+0000 7f0331600640 1 -- 172.21.15.31:0/2183966837 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f03340b5b30 con 0x7f0310078f10 2024-09-13T08:45:45.671 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp79030/subvol83941 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 msgr2=0x7f031007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 0x7f031007b3d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0324007fd0 tx=0x7f03240078c0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 msgr2=0x7f0334157c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f032c009cd0 tx=0x7f032c0316f0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 shutdown_connections 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0310078f10 0x7f031007b3d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0334161480 0x7f0334157c60 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03340cde90 0x7f03340b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 --2- 172.21.15.31:0/2183966837 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f03340cc190 0x7f03341f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 >> 172.21.15.31:0/2183966837 conn(0x7f03340cfbb0 msgr2=0x7f03340ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 shutdown_connections 2024-09-13T08:45:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.673+0000 7f033b600640 1 -- 172.21.15.31:0/2183966837 wait complete. 2024-09-13T08:45:45.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol3760 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.861+0000 7ff8e7c00640 1 -- 172.21.15.31:0/2951393599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e00cce50 msgr2=0x7ff8e00cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.861+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/2951393599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e00cce50 0x7ff8e00cd230 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009830 tx=0x7ff8d402a5b0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.862+0000 7ff8e7c00640 1 -- 172.21.15.31:0/2951393599 shutdown_connections 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.862+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/2951393599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e00cce50 0x7ff8e00cd230 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.862+0000 7ff8e7c00640 1 -- 172.21.15.31:0/2951393599 >> 172.21.15.31:0/2951393599 conn(0x7ff8e00d0440 msgr2=0x7ff8e00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.862+0000 7ff8e7c00640 1 -- 172.21.15.31:0/2951393599 shutdown_connections 2024-09-13T08:45:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.862+0000 7ff8e7c00640 1 -- 172.21.15.31:0/2951393599 wait complete. 2024-09-13T08:45:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.863+0000 7ff8e7c00640 1 Processor -- start 2024-09-13T08:45:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.863+0000 7ff8e7c00640 1 -- start start 2024-09-13T08:45:45.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8e00688b0 0x7ff8e01f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0068cf0 0x7ff8e01f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e0159e00 con 0x7ff8e00cce50 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e0159c80 con 0x7ff8e0068cf0 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.864+0000 7ff8e7c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e00b1000 con 0x7ff8e00688b0 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56620/0 (socket says 172.21.15.31:56620) 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 -- 172.21.15.31:0/4280045869 learned_addr learned my addr 172.21.15.31:0/4280045869 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8dfe00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0068cf0 0x7ff8e01f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5200640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8e00688b0 0x7ff8e01f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 -- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8e00688b0 msgr2=0x7ff8e01f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8e00688b0 0x7ff8e01f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 -- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0068cf0 msgr2=0x7ff8e01f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0068cf0 0x7ff8e01f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 -- 172.21.15.31:0/4280045869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e00c80a0 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.865+0000 7ff8e5c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800cb10 tx=0x7ff8d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.866+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8016020 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.866+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8d800ad60 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.866+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8012680 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.866+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e00c8380 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.866+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e015a250 con 0x7ff8e00cce50 2024-09-13T08:45:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.867+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e00cd230 con 0x7ff8e00cce50 2024-09-13T08:45:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.868+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 101950+0+0 (secure 0 0 0) 0x7ff8d8019020 con 0x7ff8e00cce50 2024-09-13T08:45:45.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.869+0000 7ff8dcc00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 0x7ff8c007b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:45.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.869+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7ff8d809bf10 con 0x7ff8e00cce50 2024-09-13T08:45:45.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.870+0000 7ff8e5200640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 0x7ff8c007b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:45.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.870+0000 7ff8e5200640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 0x7ff8c007b460 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4034000 tx=0x7ff8d4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:45.874+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff8d8020030 con 0x7ff8e00cce50 2024-09-13T08:45:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.097+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol3760", "target": ["mon-mgr", ""]}) -- 0x7ff8e00d0820 con 0x7ff8c0078fa0 2024-09-13T08:45:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.102+0000 7ff8dcc00640 1 -- 172.21.15.31:0/4280045869 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+29 (secure 0 0 0) 0x7ff8e00d0820 con 0x7ff8c0078fa0 2024-09-13T08:45:46.102 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol3760 2024-09-13T08:45:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 msgr2=0x7ff8c007b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 0x7ff8c007b460 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4034000 tx=0x7ff8d4004730 comp rx=0 tx=0).stop 2024-09-13T08:45:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 msgr2=0x7ff8e01f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800cb10 tx=0x7ff8d800a060 comp rx=0 tx=0).stop 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 shutdown_connections 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8c0078fa0 0x7ff8c007b460 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e00cce50 0x7ff8e01f66b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0068cf0 0x7ff8e01f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 --2- 172.21.15.31:0/4280045869 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8e00688b0 0x7ff8e01f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 >> 172.21.15.31:0/4280045869 conn(0x7ff8e00d0440 msgr2=0x7ff8e00afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.104+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 shutdown_connections 2024-09-13T08:45:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.105+0000 7ff8e7c00640 1 -- 172.21.15.31:0/4280045869 wait complete. 2024-09-13T08:45:46.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:46.122 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/24eebb960efd56fdf47f8c028fd17af9.meta) 2024-09-13T08:45:46.183 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:45:46.183 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:45:46.183 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:45:46.184 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol3760 2024-09-13T08:45:46.184 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:45:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:46.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol83941 subvol_grp79030 2024-09-13T08:45:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 -- 172.21.15.31:0/345743642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c000b2ce0 msgr2=0x7f8c000d0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 --2- 172.21.15.31:0/345743642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c000b2ce0 0x7f8c000d0280 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8c0006fe60 tx=0x7f8be802a560 comp rx=0 tx=0).stop 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 -- 172.21.15.31:0/345743642 shutdown_connections 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 --2- 172.21.15.31:0/345743642 >> v2:172.21.15.45:3300/0 conn(0x7f8c000d07c0 0x7f8c000af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 --2- 172.21.15.31:0/345743642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c000b2ce0 0x7f8c000d0280 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 -- 172.21.15.31:0/345743642 >> 172.21.15.31:0/345743642 conn(0x7f8c000d1c00 msgr2=0x7f8c000d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 -- 172.21.15.31:0/345743642 shutdown_connections 2024-09-13T08:45:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.351+0000 7f8bff200640 1 -- 172.21.15.31:0/345743642 wait complete. 2024-09-13T08:45:46.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.353+0000 7f8bff200640 1 Processor -- start 2024-09-13T08:45:46.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.353+0000 7f8bff200640 1 -- start start 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8c000b2ce0 0x7f8c000b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c000d07c0 0x7f8c001530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8c000afed0 con 0x7f8c0015f250 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c000afd50 con 0x7f8c000d07c0 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bff200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8c000b0050 con 0x7f8c000b2ce0 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bfe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bfe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56644/0 (socket says 172.21.15.31:56644) 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bfe800640 1 -- 172.21.15.31:0/1071773835 learned_addr learned my addr 172.21.15.31:0/1071773835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bf5400640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c000d07c0 0x7f8c001530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.354+0000 7f8bfde00640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8c000b2ce0 0x7f8c000b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 -- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8c000b2ce0 msgr2=0x7f8c000b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8c000b2ce0 0x7f8c000b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 -- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c000d07c0 msgr2=0x7f8c001530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c000d07c0 0x7f8c001530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 -- 172.21.15.31:0/1071773835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c000b5c60 con 0x7f8c0015f250 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bfe800640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800eaa0 tx=0x7f8bf800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf800cdb0 con 0x7f8c0015f250 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bf8004c10 con 0x7f8c0015f250 2024-09-13T08:45:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8005020 con 0x7f8c0015f250 2024-09-13T08:45:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c000b5f40 con 0x7f8c0015f250 2024-09-13T08:45:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.355+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c000b5120 con 0x7f8c0015f250 2024-09-13T08:45:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.356+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c000b05e0 con 0x7f8c0015f250 2024-09-13T08:45:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.358+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 101950+0+0 (secure 0 0 0) 0x7f8bf80051c0 con 0x7f8c0015f250 2024-09-13T08:45:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.358+0000 7f8bf6a00640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 0x7f8bdc07b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.359+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f8bf809c5a0 con 0x7f8c0015f250 2024-09-13T08:45:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.359+0000 7f8bfde00640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 0x7f8bdc07b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.359+0000 7f8bfde00640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 0x7f8bdc07b460 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0001720 tx=0x7f8bf0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.363+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8bf8016030 con 0x7f8c0015f250 2024-09-13T08:45:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.585+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol83941", "group_name": "subvol_grp79030", "target": ["mon-mgr", ""]}) -- 0x7f8c000c8a70 con 0x7f8bdc078fa0 2024-09-13T08:45:46.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.590+0000 7f8bf6a00640 1 -- 172.21.15.31:0/1071773835 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8c000c8a70 con 0x7f8bdc078fa0 2024-09-13T08:45:46.591 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp79030/subvol83941 2024-09-13T08:45:46.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 msgr2=0x7f8bdc07b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 0x7f8bdc07b460 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0001720 tx=0x7f8bf0009040 comp rx=0 tx=0).stop 2024-09-13T08:45:46.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 msgr2=0x7f8c001549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800eaa0 tx=0x7f8bf800ef60 comp rx=0 tx=0).stop 2024-09-13T08:45:46.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 shutdown_connections 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8bdc078fa0 0x7f8bdc07b460 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c0015f250 0x7f8c001549b0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c000d07c0 0x7f8c001530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 --2- 172.21.15.31:0/1071773835 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8c000b2ce0 0x7f8c000b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 >> 172.21.15.31:0/1071773835 conn(0x7f8c000d1c00 msgr2=0x7f8c00071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 shutdown_connections 2024-09-13T08:45:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.592+0000 7f8bff200640 1 -- 172.21.15.31:0/1071773835 wait complete. 2024-09-13T08:45:46.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:46.609 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/d3e387ff64ea8ad361652341b2843e11.meta) 2024-09-13T08:45:46.668 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:45:46.668 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:45:46.668 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:45:46.669 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/subvol_grp79030/subvol83941 2024-09-13T08:45:46.669 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:45:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:46.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol3760 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 -- 172.21.15.31:0/2895931466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c415bfc0 msgr2=0x7f77c4072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 --2- 172.21.15.31:0/2895931466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c415bfc0 0x7f77c4072560 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f77b4005fd0 tx=0x7f77b402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 -- 172.21.15.31:0/2895931466 shutdown_connections 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 --2- 172.21.15.31:0/2895931466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c415bfc0 0x7f77c4072560 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 -- 172.21.15.31:0/2895931466 >> 172.21.15.31:0/2895931466 conn(0x7f77c40cfbb0 msgr2=0x7f77c40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 -- 172.21.15.31:0/2895931466 shutdown_connections 2024-09-13T08:45:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.835+0000 7f77cb000640 1 -- 172.21.15.31:0/2895931466 wait complete. 2024-09-13T08:45:46.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.837+0000 7f77cb000640 1 Processor -- start 2024-09-13T08:45:46.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.837+0000 7f77cb000640 1 -- start start 2024-09-13T08:45:46.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77c415bfc0 0x7f77c41f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f77c415d710 0x7f77c40c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77c40cbfd0 con 0x7f77c406fa60 2024-09-13T08:45:46.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f77c40cbe50 con 0x7f77c415bfc0 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77cb000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f77c40b30f0 con 0x7f77c415d710 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77c3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77c3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56660/0 (socket says 172.21.15.31:56660) 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.838+0000 7f77c3e00640 1 -- 172.21.15.31:0/2956703451 learned_addr learned my addr 172.21.15.31:0/2956703451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3400640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77c415bfc0 0x7f77c41f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c9000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f77c415d710 0x7f77c40c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 -- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f77c415d710 msgr2=0x7f77c40c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f77c415d710 0x7f77c40c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 -- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77c415bfc0 msgr2=0x7f77c41f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77c415bfc0 0x7f77c41f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 -- 172.21.15.31:0/2956703451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c415c990 con 0x7f77c406fa60 2024-09-13T08:45:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77c3e00640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f77b4034040 tx=0x7f77b40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b403e030 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77b4003ca0 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4035040 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c415cc30 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.839+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c415d1f0 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.841+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c4072560 con 0x7f77c406fa60 2024-09-13T08:45:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.842+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 101950+0+0 (secure 0 0 0) 0x7f77b4041030 con 0x7f77c406fa60 2024-09-13T08:45:46.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.843+0000 7f77bbe00640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 0x7f77a407b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.843+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f77b40bc110 con 0x7f77c406fa60 2024-09-13T08:45:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.843+0000 7f77c3400640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 0x7f77a407b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.843+0000 7f77c3400640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 0x7f77a407b460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f77c41f08f0 tx=0x7f77b0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:46.847+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f77b4083700 con 0x7f77c406fa60 2024-09-13T08:45:47.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.074+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol3760", "target": ["mon-mgr", ""]}) -- 0x7f77c4159200 con 0x7f77a4078fa0 2024-09-13T08:45:47.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.102+0000 7f77bbe00640 1 -- 172.21.15.31:0/2956703451 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f77c4159200 con 0x7f77a4078fa0 2024-09-13T08:45:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 msgr2=0x7f77a407b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 0x7f77a407b460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f77c41f08f0 tx=0x7f77b0009040 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 msgr2=0x7f77c4159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f77b4034040 tx=0x7f77b40038a0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 shutdown_connections 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f77a4078fa0 0x7f77a407b460 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f77c415d710 0x7f77c40c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77c415bfc0 0x7f77c41f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 --2- 172.21.15.31:0/2956703451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77c406fa60 0x7f77c4159a70 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 >> 172.21.15.31:0/2956703451 conn(0x7f77c40cfbb0 msgr2=0x7f77c40b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 shutdown_connections 2024-09-13T08:45:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.105+0000 7f77cb000640 1 -- 172.21.15.31:0/2956703451 wait complete. 2024-09-13T08:45:47.110 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.109+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.110 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.109+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.110 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.109+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.110 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.109+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.110 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.109+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol83941 subvol_grp79030 2024-09-13T08:45:47.129 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.127+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.130 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.127+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.130 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.127+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.130 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.127+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.130 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:45:47.127+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:45:47.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 -- 172.21.15.31:0/430202709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4650072560 msgr2=0x7f46500cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 --2- 172.21.15.31:0/430202709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4650072560 0x7f46500cb740 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4638005fd0 tx=0x7f463802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 -- 172.21.15.31:0/430202709 shutdown_connections 2024-09-13T08:45:47.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 --2- 172.21.15.31:0/430202709 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f46500cbc80 0x7f46500b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 --2- 172.21.15.31:0/430202709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4650072560 0x7f46500cb740 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 -- 172.21.15.31:0/430202709 >> 172.21.15.31:0/430202709 conn(0x7f46500cfbb0 msgr2=0x7f46500cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 -- 172.21.15.31:0/430202709 shutdown_connections 2024-09-13T08:45:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.292+0000 7f4656400640 1 -- 172.21.15.31:0/430202709 wait complete. 2024-09-13T08:45:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.293+0000 7f4656400640 1 Processor -- start 2024-09-13T08:45:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.294+0000 7f4656400640 1 -- start start 2024-09-13T08:45:47.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46500cbc80 0x7f46500c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 0x7f46501f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f465006e740 con 0x7f4650072560 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f465006e5c0 con 0x7f46500cbc80 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4656400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f46500b2f20 con 0x7f465015bfc0 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56704/0 (socket says 172.21.15.31:56704) 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 -- 172.21.15.31:0/68527236 learned_addr learned my addr 172.21.15.31:0/68527236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f4647e00640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46500cbc80 0x7f46500c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464fe00640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 0x7f46501f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 -- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 msgr2=0x7f46501f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 0x7f46501f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 -- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46500cbc80 msgr2=0x7f46500c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46500cbc80 0x7f46500c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464f400640 1 -- 172.21.15.31:0/68527236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46500c4d80 con 0x7f4650072560 2024-09-13T08:45:47.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.295+0000 7f464fe00640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 0x7f46501f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:47.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f464f400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f464000c3a0 tx=0x7f464000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:47.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640017070 con 0x7f4650072560 2024-09-13T08:45:47.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46400053c0 con 0x7f4650072560 2024-09-13T08:45:47.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46500c5060 con 0x7f4650072560 2024-09-13T08:45:47.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640003850 con 0x7f4650072560 2024-09-13T08:45:47.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.296+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46500c5620 con 0x7f4650072560 2024-09-13T08:45:47.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.297+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46500b3350 con 0x7f4650072560 2024-09-13T08:45:47.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.298+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 101950+0+0 (secure 0 0 0) 0x7f464001a070 con 0x7f4650072560 2024-09-13T08:45:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.300+0000 7f4647400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 0x7f462c07b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.300+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f464009ae30 con 0x7f4650072560 2024-09-13T08:45:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.300+0000 7f4647e00640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 0x7f462c07b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.301+0000 7f4647e00640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 0x7f462c07b460 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4638033000 tx=0x7f4638004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:47.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.304+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f46400a0020 con 0x7f4650072560 2024-09-13T08:45:47.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.526+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol83941", "group_name": "subvol_grp79030", "target": ["mon-mgr", ""]}) -- 0x7f46500b3a20 con 0x7f462c078fa0 2024-09-13T08:45:47.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.552+0000 7f4647400640 1 -- 172.21.15.31:0/68527236 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f46500b3a20 con 0x7f462c078fa0 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 msgr2=0x7f462c07b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 0x7f462c07b460 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4638033000 tx=0x7f4638004730 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 msgr2=0x7f46500c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f464000c3a0 tx=0x7f464000c860 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 shutdown_connections 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f462c078fa0 0x7f462c07b460 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f465015bfc0 0x7f46501f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46500cbc80 0x7f46500c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 --2- 172.21.15.31:0/68527236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4650072560 0x7f46500c9950 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.554+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 >> 172.21.15.31:0/68527236 conn(0x7f46500cfbb0 msgr2=0x7f46501579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:47.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.555+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 shutdown_connections 2024-09-13T08:45:47.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.555+0000 7f4656400640 1 -- 172.21.15.31:0/68527236 wait complete. 2024-09-13T08:45:47.574 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:45:47.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:45:47.660 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:45:47.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:45:47.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp79030 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.831+0000 7f6e2ea00640 1 -- 172.21.15.31:0/1535005708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2815bfc0 msgr2=0x7f6e28072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.831+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/1535005708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2815bfc0 0x7f6e28072560 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005fd0 tx=0x7f6e1802a670 comp rx=0 tx=0).stop 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.832+0000 7f6e2ea00640 1 -- 172.21.15.31:0/1535005708 shutdown_connections 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.832+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/1535005708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2815bfc0 0x7f6e28072560 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.832+0000 7f6e2ea00640 1 -- 172.21.15.31:0/1535005708 >> 172.21.15.31:0/1535005708 conn(0x7f6e280cfbb0 msgr2=0x7f6e280cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.832+0000 7f6e2ea00640 1 -- 172.21.15.31:0/1535005708 shutdown_connections 2024-09-13T08:45:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.832+0000 7f6e2ea00640 1 -- 172.21.15.31:0/1535005708 wait complete. 2024-09-13T08:45:47.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.833+0000 7f6e2ea00640 1 Processor -- start 2024-09-13T08:45:47.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.834+0000 7f6e2ea00640 1 -- start start 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.834+0000 7f6e2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.834+0000 7f6e2ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e2815bfc0 0x7f6e28151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e2ea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e281f9830 0x7f6e281f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e2ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6e280cbfd0 con 0x7f6e2806fa60 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e2ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e280cbe50 con 0x7f6e2815bfc0 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e2ea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6e280b30f0 con 0x7f6e281f9830 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56722/0 (socket says 172.21.15.31:56722) 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 -- 172.21.15.31:0/18257631 learned_addr learned my addr 172.21.15.31:0/18257631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27400640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e2815bfc0 0x7f6e28151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e2ca00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e281f9830 0x7f6e281f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 -- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e281f9830 msgr2=0x7f6e281f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e281f9830 0x7f6e281f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 -- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e2815bfc0 msgr2=0x7f6e28151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e2815bfc0 0x7f6e28151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 -- 172.21.15.31:0/18257631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2815c700 con 0x7f6e2806fa60 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.835+0000 7f6e27e00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f6e18034040 tx=0x7f6e18003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.836+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1803e030 con 0x7f6e2806fa60 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.836+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e180047a0 con 0x7f6e2806fa60 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.836+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18035040 con 0x7f6e2806fa60 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.836+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2815c9e0 con 0x7f6e2806fa60 2024-09-13T08:45:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.836+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2815cfa0 con 0x7f6e2806fa60 2024-09-13T08:45:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.837+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e28072ee0 con 0x7f6e2806fa60 2024-09-13T08:45:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.838+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 101950+0+0 (secure 0 0 0) 0x7f6e18041030 con 0x7f6e2806fa60 2024-09-13T08:45:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.839+0000 7f6e1fe00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 0x7f6e0807b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.839+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f6e180bc2b0 con 0x7f6e2806fa60 2024-09-13T08:45:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.839+0000 7f6e27400640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 0x7f6e0807b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.840+0000 7f6e27400640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 0x7f6e0807b460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6e14001680 tx=0x7f6e14009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:47.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.844+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6e28072ee0 con 0x7f6e2806fa60 2024-09-13T08:45:47.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:47.867+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f6e1807f4a0 con 0x7f6e2806fa60 2024-09-13T08:45:48.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.065+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp79030", "target": ["mon-mgr", ""]}) -- 0x7f6e280b5850 con 0x7f6e08078fa0 2024-09-13T08:45:48.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.082+0000 7f6e1fe00640 1 -- 172.21.15.31:0/18257631 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6e280b5850 con 0x7f6e08078fa0 2024-09-13T08:45:48.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 msgr2=0x7f6e0807b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 0x7f6e0807b460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6e14001680 tx=0x7f6e14009040 comp rx=0 tx=0).stop 2024-09-13T08:45:48.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 msgr2=0x7f6e280b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f6e18034040 tx=0x7f6e18003970 comp rx=0 tx=0).stop 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 shutdown_connections 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6e08078fa0 0x7f6e0807b460 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6e281f9830 0x7f6e281f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e2815bfc0 0x7f6e28151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 --2- 172.21.15.31:0/18257631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2806fa60 0x7f6e280b3920 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 >> 172.21.15.31:0/18257631 conn(0x7f6e280cfbb0 msgr2=0x7f6e28153000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 shutdown_connections 2024-09-13T08:45:48.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.084+0000 7f6e2ea00640 1 -- 172.21.15.31:0/18257631 wait complete. 2024-09-13T08:45:48.101 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:45:48.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:48.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:48.294 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:45:48.294 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:48.294 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:48.294 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:48.330 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:48.330 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:48.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.606+0000 7f9d2c600640 1 -- 172.21.15.31:0/3770054603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d24073860 msgr2=0x7f9d24073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.606+0000 7f9d2c600640 1 --2- 172.21.15.31:0/3770054603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d24073860 0x7f9d24073cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c005fd0 tx=0x7f9d1c02aa90 comp rx=0 tx=0).stop 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.606+0000 7f9d2c600640 1 -- 172.21.15.31:0/3770054603 shutdown_connections 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.606+0000 7f9d2c600640 1 --2- 172.21.15.31:0/3770054603 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d24073860 0x7f9d24073cf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.606+0000 7f9d2c600640 1 --2- 172.21.15.31:0/3770054603 >> v2:172.21.15.45:3300/0 conn(0x7f9d2415be30 0x7f9d240cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.607+0000 7f9d2c600640 1 -- 172.21.15.31:0/3770054603 >> 172.21.15.31:0/3770054603 conn(0x7f9d240b18f0 msgr2=0x7f9d240b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.607+0000 7f9d2c600640 1 -- 172.21.15.31:0/3770054603 shutdown_connections 2024-09-13T08:45:48.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.607+0000 7f9d2c600640 1 -- 172.21.15.31:0/3770054603 wait complete. 2024-09-13T08:45:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.608+0000 7f9d2c600640 1 Processor -- start 2024-09-13T08:45:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.608+0000 7f9d2c600640 1 -- start start 2024-09-13T08:45:48.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:48.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d240ced20 0x7f9d24154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:48.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d2415be30 0x7f9d240cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:48.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9d240ce950 con 0x7f9d24073860 2024-09-13T08:45:48.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d240ce7d0 con 0x7f9d240ced20 2024-09-13T08:45:48.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.609+0000 7f9d2c600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9d24074230 con 0x7f9d2415be30 2024-09-13T08:45:48.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:48.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56734/0 (socket says 172.21.15.31:56734) 2024-09-13T08:45:48.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 -- 172.21.15.31:0/1414339710 learned_addr learned my addr 172.21.15.31:0/1414339710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d2a600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d2415be30 0x7f9d240cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29200640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d240ced20 0x7f9d24154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 -- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d2415be30 msgr2=0x7f9d240cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d2415be30 0x7f9d240cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 -- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d240ced20 msgr2=0x7f9d24154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d240ced20 0x7f9d24154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 -- 172.21.15.31:0/1414339710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d240cb3d0 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.610+0000 7f9d29c00640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800e790 tx=0x7f9d1800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.611+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800eea0 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.611+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d1800cd50 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.611+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18010690 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.611+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d240cb6b0 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.611+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d24154460 con 0x7f9d24073860 2024-09-13T08:45:48.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.612+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d24074770 con 0x7f9d24073860 2024-09-13T08:45:48.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.613+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f9d18010830 con 0x7f9d24073860 2024-09-13T08:45:48.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.615+0000 7f9d21600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 0x7f9d0407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:48.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.615+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f9d18099ae0 con 0x7f9d24073860 2024-09-13T08:45:48.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.615+0000 7f9d29200640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 0x7f9d0407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:48.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.616+0000 7f9d29200640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 0x7f9d0407b580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9d240cc0c0 tx=0x7f9d10009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:48.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.620+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9d18063c00 con 0x7f9d24073860 2024-09-13T08:45:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.831+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9d240ad430 con 0x7f9d24073860 2024-09-13T08:45:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.832+0000 7f9d21600640 1 -- 172.21.15.31:0/1414339710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v151) ==== 80+0+0 (secure 0 0 0) 0x7f9d18063500 con 0x7f9d24073860 2024-09-13T08:45:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:45:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 msgr2=0x7f9d0407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 0x7f9d0407b580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9d240cc0c0 tx=0x7f9d10009040 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 msgr2=0x7f9d2414c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800e790 tx=0x7f9d1800ec50 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 shutdown_connections 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9d040790c0 0x7f9d0407b580 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9d2415be30 0x7f9d240cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d240ced20 0x7f9d24154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 --2- 172.21.15.31:0/1414339710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24073860 0x7f9d2414c980 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 >> 172.21.15.31:0/1414339710 conn(0x7f9d240b18f0 msgr2=0x7f9d240cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 shutdown_connections 2024-09-13T08:45:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:48.834+0000 7f9d2c600640 1 -- 172.21.15.31:0/1414339710 wait complete. 2024-09-13T08:45:48.850 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:45:48.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:48.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:45:48.915 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:45:48.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:48.915 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:45:48.952 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:48.952 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:48.952 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:48.993 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:48.993 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:49.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:45:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 -- 172.21.15.31:0/3370295315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 msgr2=0x7f051c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 --2- 172.21.15.31:0/3370295315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f051c003e20 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f052800c290 tx=0x7f051c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0536800640 1 -- 172.21.15.31:0/3370295315 reap_dead start 2024-09-13T08:45:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 -- 172.21.15.31:0/3370295315 shutdown_connections 2024-09-13T08:45:49.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 -- 172.21.15.31:0/3370295315 >> 172.21.15.31:0/3370295315 conn(0x7f05380b2000 msgr2=0x7f05380b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:49.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 -- 172.21.15.31:0/3370295315 shutdown_connections 2024-09-13T08:45:49.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.235+0000 7f0537200640 1 -- 172.21.15.31:0/3370295315 wait complete. 2024-09-13T08:45:49.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.236+0000 7f0537200640 1 Processor -- start 2024-09-13T08:45:49.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.237+0000 7f0537200640 1 -- start start 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.237+0000 7f0537200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.237+0000 7f0537200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05381f7db0 0x7f053814f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0537200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f053814fb10 0x7f05380b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0537200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f053806d970 con 0x7f051c003a00 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0537200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f053806d7f0 con 0x7f05381f7db0 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0537200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f053815d480 con 0x7f053814fb10 2024-09-13T08:45:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56752/0 (socket says 172.21.15.31:56752) 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 -- 172.21.15.31:0/3516343276 learned_addr learned my addr 172.21.15.31:0/3516343276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535400640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05381f7db0 0x7f053814f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0536800640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f053814fb10 0x7f05380b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 -- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f053814fb10 msgr2=0x7f05380b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f053814fb10 0x7f05380b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 -- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05381f7db0 msgr2=0x7f053814f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05381f7db0 0x7f053814f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.238+0000 7f0535e00640 1 -- 172.21.15.31:0/3516343276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053814e530 con 0x7f051c003a00 2024-09-13T08:45:49.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f0535e00640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f052800c380 tx=0x7f0528001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:49.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528031040 con 0x7f051c003a00 2024-09-13T08:45:49.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05280035f0 con 0x7f051c003a00 2024-09-13T08:45:49.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528003a00 con 0x7f051c003a00 2024-09-13T08:45:49.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053814e810 con 0x7f051c003a00 2024-09-13T08:45:49.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.239+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053814edd0 con 0x7f051c003a00 2024-09-13T08:45:49.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.240+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05380cc140 con 0x7f051c003a00 2024-09-13T08:45:49.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.242+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f0528034070 con 0x7f051c003a00 2024-09-13T08:45:49.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.243+0000 7f052e000640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 0x7f051807b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.243+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f05280b9270 con 0x7f051c003a00 2024-09-13T08:45:49.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.243+0000 7f0535400640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 0x7f051807b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.244+0000 7f0535400640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 0x7f051807b580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05380cdf30 tx=0x7f052400d360 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:49.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.248+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0528083370 con 0x7f051c003a00 2024-09-13T08:45:49.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.454+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f053815b8e0 con 0x7f051c003a00 2024-09-13T08:45:49.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.454+0000 7f052e000640 1 -- 172.21.15.31:0/3516343276 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v151) ==== 80+0+0 (secure 0 0 0) 0x7f0528082c70 con 0x7f051c003a00 2024-09-13T08:45:49.456 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:45:49.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 msgr2=0x7f051807b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 0x7f051807b580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05380cdf30 tx=0x7f052400d360 comp rx=0 tx=0).stop 2024-09-13T08:45:49.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 msgr2=0x7f0538155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f052800c380 tx=0x7f0528001cb0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 shutdown_connections 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05180790c0 0x7f051807b580 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f053814fb10 0x7f05380b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05381f7db0 0x7f053814f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 --2- 172.21.15.31:0/3516343276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f051c003a00 0x7f0538155850 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 >> 172.21.15.31:0/3516343276 conn(0x7f05380b2000 msgr2=0x7f05380cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 shutdown_connections 2024-09-13T08:45:49.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.457+0000 7f0537200640 1 -- 172.21.15.31:0/3516343276 wait complete. 2024-09-13T08:45:49.473 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:45:49.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:49.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:45:49.541 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:45:49.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:45:49.542 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:45:49.579 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:49.580 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:49.580 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:49.616 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:45:49.616 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:45:49.616 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:49.616 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:49.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:45:49.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 -- 172.21.15.31:0/2864455093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 msgr2=0x7f7d780cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 --2- 172.21.15.31:0/2864455093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780cf850 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7d780c6f10 tx=0x7f7d6802a560 comp rx=0 tx=0).stop 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 -- 172.21.15.31:0/2864455093 shutdown_connections 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 --2- 172.21.15.31:0/2864455093 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 --2- 172.21.15.31:0/2864455093 >> v2:172.21.15.45:3300/0 conn(0x7f7d7815f9e0 0x7f7d7815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 --2- 172.21.15.31:0/2864455093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780cf850 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 -- 172.21.15.31:0/2864455093 >> 172.21.15.31:0/2864455093 conn(0x7f7d780cc910 msgr2=0x7f7d780cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 -- 172.21.15.31:0/2864455093 shutdown_connections 2024-09-13T08:45:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.818+0000 7f7d7d600640 1 -- 172.21.15.31:0/2864455093 wait complete. 2024-09-13T08:45:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.820+0000 7f7d7d600640 1 Processor -- start 2024-09-13T08:45:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.820+0000 7f7d7d600640 1 -- start start 2024-09-13T08:45:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 0x7f7d781f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d780cb9d0 con 0x7f7d7815bd70 2024-09-13T08:45:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d780cb850 con 0x7f7d7815f9e0 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d7d600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7d780cbc50 con 0x7f7d780ce3c0 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d76000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d76a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d76000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56766/0 (socket says 172.21.15.31:56766) 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d76000640 1 -- 172.21.15.31:0/3271248518 learned_addr learned my addr 172.21.15.31:0/3271248518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d76a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780c9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:36464/0 (socket says 172.21.15.31:36464) 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.821+0000 7f7d77400640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 0x7f7d781f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 -- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 msgr2=0x7f7d780c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 -- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 msgr2=0x7f7d781f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 0x7f7d781f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 -- 172.21.15.31:0/3271248518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d780c4a60 con 0x7f7d7815bd70 2024-09-13T08:45:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d77400640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 0x7f7d781f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d76000640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c970 tx=0x7f7d6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d64007bb0 con 0x7f7d7815bd70 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d64007e40 con 0x7f7d7815bd70 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400f690 con 0x7f7d7815bd70 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d780c4d40 con 0x7f7d7815bd70 2024-09-13T08:45:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.822+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d780c5300 con 0x7f7d7815bd70 2024-09-13T08:45:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.823+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d781f5bc0 con 0x7f7d7815bd70 2024-09-13T08:45:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.825+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f7d6400f830 con 0x7f7d7815bd70 2024-09-13T08:45:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.825+0000 7f7d6ea00640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 0x7f7d5807b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.826+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f7d6409c350 con 0x7f7d7815bd70 2024-09-13T08:45:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.826+0000 7f7d76a00640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 0x7f7d5807b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.826+0000 7f7d76a00640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 0x7f7d5807b580 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7d68033000 tx=0x7f7d68004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:49.830+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7d64014030 con 0x7f7d7815bd70 2024-09-13T08:45:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.062+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d780ce7f0 con 0x7f7d7815bd70 2024-09-13T08:45:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.063+0000 7f7d6ea00640 1 -- 172.21.15.31:0/3271248518 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5228 (secure 0 0 0) 0x7f7d780ce7f0 con 0x7f7d7815bd70 2024-09-13T08:45:50.064 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:50.064 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":100,"btime":"2024-09-13T08:45:43:836819+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 100 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 msgr2=0x7f7d5807b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 0x7f7d5807b580 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7d68033000 tx=0x7f7d68004730 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 msgr2=0x7f7d780c4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c970 tx=0x7f7d6400ce30 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 shutdown_connections 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7d580790c0 0x7f7d5807b580 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d7815f9e0 0x7f7d781f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d7815bd70 0x7f7d780c4330 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 --2- 172.21.15.31:0/3271248518 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7d780ce3c0 0x7f7d780c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 >> 172.21.15.31:0/3271248518 conn(0x7f7d780cc910 msgr2=0x7f7d780ad120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 shutdown_connections 2024-09-13T08:45:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.066+0000 7f7d7d600640 1 -- 172.21.15.31:0/3271248518 wait complete. 2024-09-13T08:45:50.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:50.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 -- 172.21.15.31:0/3351699231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f211015dfc0 msgr2=0x7f21100cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 --2- 172.21.15.31:0/3351699231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f211015dfc0 0x7f21100cc300 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f2110070b80 tx=0x7f210002a540 comp rx=0 tx=0).stop 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 -- 172.21.15.31:0/3351699231 shutdown_connections 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 --2- 172.21.15.31:0/3351699231 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 0x7f21100b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 --2- 172.21.15.31:0/3351699231 >> v2:172.21.15.45:3300/0 conn(0x7f21100ca5c0 0x7f21100ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 --2- 172.21.15.31:0/3351699231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f211015dfc0 0x7f21100cc300 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 -- 172.21.15.31:0/3351699231 >> 172.21.15.31:0/3351699231 conn(0x7f21100cc910 msgr2=0x7f21100cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 -- 172.21.15.31:0/3351699231 shutdown_connections 2024-09-13T08:45:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.244+0000 7f2116600640 1 -- 172.21.15.31:0/3351699231 wait complete. 2024-09-13T08:45:50.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.246+0000 7f2116600640 1 Processor -- start 2024-09-13T08:45:50.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.246+0000 7f2116600640 1 -- start start 2024-09-13T08:45:50.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 0x7f21101f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 0x7f21100b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2110073900 con 0x7f21100ca5c0 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2110073780 con 0x7f211015dfc0 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2116600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2110073a80 con 0x7f21100b1910 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2106a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2106a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56784/0 (socket says 172.21.15.31:56784) 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f2106a00640 1 -- 172.21.15.31:0/1732950944 learned_addr learned my addr 172.21.15.31:0/1732950944 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f210fe00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 0x7f21100b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.247+0000 7f210f400640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 0x7f21101f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 -- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 msgr2=0x7f21101f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 0x7f21101f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 -- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 msgr2=0x7f21100b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 0x7f21100b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 -- 172.21.15.31:0/1732950944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21101f8b20 con 0x7f21100ca5c0 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f210fe00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 0x7f21100b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:50.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2106a00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f20f800d980 tx=0x7f20f800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8017070 con 0x7f21100ca5c0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f8009d20 con 0x7f21100ca5c0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80106c0 con 0x7f21100ca5c0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21101f8e00 con 0x7f21100ca5c0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.248+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21100b2d60 con 0x7f21100ca5c0 2024-09-13T08:45:50.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.249+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2110073f80 con 0x7f21100ca5c0 2024-09-13T08:45:50.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.251+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f20f8010860 con 0x7f21100ca5c0 2024-09-13T08:45:50.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.252+0000 7f2107e00640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 0x7f20ec07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.252+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f20f80994a0 con 0x7f21100ca5c0 2024-09-13T08:45:50.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.252+0000 7f210f400640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 0x7f20ec07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.252+0000 7f210f400640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 0x7f20ec07b580 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2100033000 tx=0x7f2100004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:50.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.256+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f20f8062f70 con 0x7f21100ca5c0 2024-09-13T08:45:50.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.488+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21100aee40 con 0x7f21100ca5c0 2024-09-13T08:45:50.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.489+0000 7f2107e00640 1 -- 172.21.15.31:0/1732950944 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5228 (secure 0 0 0) 0x7f20f80679a0 con 0x7f21100ca5c0 2024-09-13T08:45:50.490 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:50.490 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":100,"btime":"2024-09-13T08:45:43:836819+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:50.490 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 100 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.491+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 msgr2=0x7f20ec07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.491+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 0x7f20ec07b580 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2100033000 tx=0x7f2100004730 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.491+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 msgr2=0x7f2110151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.491+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f20f800d980 tx=0x7f20f800de40 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 shutdown_connections 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f20ec0790c0 0x7f20ec07b580 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f211015dfc0 0x7f21100b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21100ca5c0 0x7f2110151c90 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 --2- 172.21.15.31:0/1732950944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f21100b1910 0x7f21101f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 >> 172.21.15.31:0/1732950944 conn(0x7f21100cc910 msgr2=0x7f21100d0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 shutdown_connections 2024-09-13T08:45:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.492+0000 7f2116600640 1 -- 172.21.15.31:0/1732950944 wait complete. 2024-09-13T08:45:50.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 -- 172.21.15.31:0/702870568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 msgr2=0x7fa0bc0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 --2- 172.21.15.31:0/702870568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc0cf850 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009830 tx=0x7fa0b002a510 comp rx=0 tx=0).stop 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 -- 172.21.15.31:0/702870568 shutdown_connections 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 --2- 172.21.15.31:0/702870568 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc15f9e0 0x7fa0bc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 --2- 172.21.15.31:0/702870568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc0cf850 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 -- 172.21.15.31:0/702870568 >> 172.21.15.31:0/702870568 conn(0x7fa0bc0cc910 msgr2=0x7fa0bc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 -- 172.21.15.31:0/702870568 shutdown_connections 2024-09-13T08:45:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.669+0000 7fa0c4600640 1 -- 172.21.15.31:0/702870568 wait complete. 2024-09-13T08:45:50.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.671+0000 7fa0c4600640 1 Processor -- start 2024-09-13T08:45:50.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.671+0000 7fa0c4600640 1 -- start start 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc0ce3c0 0x7fa0bc1f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f9e0 0x7fa0bc1f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc0cb9d0 con 0x7fa0bc15bd70 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc0cb850 con 0x7fa0bc15f9e0 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c4600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa0bc0cc360 con 0x7fa0bc0ce3c0 2024-09-13T08:45:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56794/0 (socket says 172.21.15.31:56794) 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c1200640 1 -- 172.21.15.31:0/2517230565 learned_addr learned my addr 172.21.15.31:0/2517230565 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c1c00640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc0ce3c0 0x7fa0bc1f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.672+0000 7fa0c2600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f9e0 0x7fa0bc1f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 -- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc0ce3c0 msgr2=0x7fa0bc1f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc0ce3c0 0x7fa0bc1f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 -- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f9e0 msgr2=0x7fa0bc1f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f9e0 0x7fa0bc1f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 -- 172.21.15.31:0/2517230565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc1efe20 con 0x7fa0bc15bd70 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c1200640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800d700 tx=0x7fa0a800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8017070 con 0x7fa0bc15bd70 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0a8004310 con 0x7fa0bc15bd70 2024-09-13T08:45:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc0c6740 con 0x7fa0bc15bd70 2024-09-13T08:45:50.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8009a80 con 0x7fa0bc15bd70 2024-09-13T08:45:50.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.673+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc1f2720 con 0x7fa0bc15bd70 2024-09-13T08:45:50.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.674+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0bc0cf850 con 0x7fa0bc15bd70 2024-09-13T08:45:50.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.676+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fa0a8009c20 con 0x7fa0bc15bd70 2024-09-13T08:45:50.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.676+0000 7fa0b9600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 0x7fa09c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:50.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.677+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fa0a8099170 con 0x7fa0bc15bd70 2024-09-13T08:45:50.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.677+0000 7fa0c1c00640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 0x7fa09c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:50.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.677+0000 7fa0c1c00640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 0x7fa09c07b580 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0034000 tx=0x7fa0b0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:50.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.681+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa0a8062c40 con 0x7fa0bc15bd70 2024-09-13T08:45:50.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.876+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0bc0ce8f0 con 0x7fa0bc15bd70 2024-09-13T08:45:50.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.877+0000 7fa0b9600640 1 -- 172.21.15.31:0/2517230565 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16758 (secure 0 0 0) 0x7fa0a8067670 con 0x7fa0bc15bd70 2024-09-13T08:45:50.878 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:50.878 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":151,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:34.608532+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:45:30.846891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T08:45:32.010174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:50.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 msgr2=0x7fa09c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 0x7fa09c07b580 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0034000 tx=0x7fa0b0004730 comp rx=0 tx=0).stop 2024-09-13T08:45:50.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 msgr2=0x7fa0bc1ef8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800d700 tx=0x7fa0a800dbc0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 shutdown_connections 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa09c0790c0 0x7fa09c07b580 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f9e0 0x7fa0bc1f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0bc15bd70 0x7fa0bc1ef8e0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 --2- 172.21.15.31:0/2517230565 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa0bc0ce3c0 0x7fa0bc1f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 >> 172.21.15.31:0/2517230565 conn(0x7fa0bc0cc910 msgr2=0x7fa0bc0ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 shutdown_connections 2024-09-13T08:45:50.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:50.879+0000 7fa0c4600640 1 -- 172.21.15.31:0/2517230565 wait complete. 2024-09-13T08:45:50.897 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:45:50.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 -- 172.21.15.31:0/2407987872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 msgr2=0x7f1c9c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 --2- 172.21.15.31:0/2407987872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1c9c003e20 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c260 tx=0x7f1c9c005830 comp rx=0 tx=0).stop 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cb9a00640 1 -- 172.21.15.31:0/2407987872 reap_dead start 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 -- 172.21.15.31:0/2407987872 shutdown_connections 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 -- 172.21.15.31:0/2407987872 >> 172.21.15.31:0/2407987872 conn(0x7f1cb408d8b0 msgr2=0x7f1cb409d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 -- 172.21.15.31:0/2407987872 shutdown_connections 2024-09-13T08:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.058+0000 7f1cba400640 1 -- 172.21.15.31:0/2407987872 wait complete. 2024-09-13T08:45:51.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.060+0000 7f1cba400640 1 Processor -- start 2024-09-13T08:45:51.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.060+0000 7f1cba400640 1 -- start start 2024-09-13T08:45:51.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cb40cf7b0 0x7f1cb406dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 0x7f1cb40cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb40b1310 con 0x7f1c9c003a00 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb40b1190 con 0x7f1cb40cf7b0 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cba400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb40b1490 con 0x7f1cb41f52d0 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56814/0 (socket says 172.21.15.31:56814) 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 -- 172.21.15.31:0/710088022 learned_addr learned my addr 172.21.15.31:0/710088022 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb3e00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cb40cf7b0 0x7f1cb406dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9a00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 0x7f1cb40cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 -- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 msgr2=0x7f1cb40cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 0x7f1cb40cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 -- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cb40cf7b0 msgr2=0x7f1cb406dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cb40cf7b0 0x7f1cb406dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9000640 1 -- 172.21.15.31:0/710088022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb40ce9f0 con 0x7f1c9c003a00 2024-09-13T08:45:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.061+0000 7f1cb9a00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 0x7f1cb40cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cb9000640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000c00 tx=0x7f1ca8001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8031040 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ca8003610 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8003a40 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb40cecd0 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.062+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb40d0810 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.063+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb40d3bd0 con 0x7f1c9c003a00 2024-09-13T08:45:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.064+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f1ca8034070 con 0x7f1c9c003a00 2024-09-13T08:45:51.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.065+0000 7f1cb0c00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 0x7f1c9407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.065+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7f1ca80b90e0 con 0x7f1c9c003a00 2024-09-13T08:45:51.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.065+0000 7f1cb3e00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 0x7f1c9407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.066+0000 7f1cb3e00640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 0x7f1c9407b580 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40cfbb0 tx=0x7f1ca0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.069+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1ca80831e0 con 0x7f1c9c003a00 2024-09-13T08:45:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.299+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cb40d4cf0 con 0x7f1c9c003a00 2024-09-13T08:45:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.299+0000 7f1cb0c00640 1 -- 172.21.15.31:0/710088022 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5228 (secure 0 0 0) 0x7f1ca8082ae0 con 0x7f1c9c003a00 2024-09-13T08:45:51.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:51.301 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":100,"btime":"2024-09-13T08:45:43:836819+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:51.301 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 100 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 msgr2=0x7f1c9407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 0x7f1c9407b580 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40cfbb0 tx=0x7f1ca0005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 msgr2=0x7f1cb40d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000c00 tx=0x7f1ca8001d60 comp rx=0 tx=0).stop 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 shutdown_connections 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1c940790c0 0x7f1c9407b580 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1cb41f52d0 0x7f1cb40cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cb40cf7b0 0x7f1cb406dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 --2- 172.21.15.31:0/710088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c9c003a00 0x7f1cb40d02d0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 >> 172.21.15.31:0/710088022 conn(0x7f1cb408d8b0 msgr2=0x7f1cb40b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 shutdown_connections 2024-09-13T08:45:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.302+0000 7f1cba400640 1 -- 172.21.15.31:0/710088022 wait complete. 2024-09-13T08:45:51.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:45:51.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 -- 172.21.15.31:0/4232277534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 msgr2=0x7fe7f0152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 --2- 172.21.15.31:0/4232277534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f0152e50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00b2a00 tx=0x7fe7e002a560 comp rx=0 tx=0).stop 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 -- 172.21.15.31:0/4232277534 shutdown_connections 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 --2- 172.21.15.31:0/4232277534 >> v2:172.21.15.45:3300/0 conn(0x7fe7f00cf980 0x7fe7f00cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 --2- 172.21.15.31:0/4232277534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f0152e50 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 -- 172.21.15.31:0/4232277534 >> 172.21.15.31:0/4232277534 conn(0x7fe7f00cc910 msgr2=0x7fe7f00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 -- 172.21.15.31:0/4232277534 shutdown_connections 2024-09-13T08:45:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.478+0000 7fe7f7800640 1 -- 172.21.15.31:0/4232277534 wait complete. 2024-09-13T08:45:51.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.480+0000 7fe7f7800640 1 Processor -- start 2024-09-13T08:45:51.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.480+0000 7fe7f7800640 1 -- start start 2024-09-13T08:45:51.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f0073fc0 0x7fe7f009d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f00cf980 0x7fe7f0159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f00b1ca0 con 0x7fe7f0152a70 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f00b1b20 con 0x7fe7f0073fc0 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f7800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f00b1e20 con 0x7fe7f00cf980 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f4e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f0073fc0 0x7fe7f009d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56830/0 (socket says 172.21.15.31:56830) 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7f5800640 1 -- 172.21.15.31:0/1630497425 learned_addr learned my addr 172.21.15.31:0/1630497425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.481+0000 7fe7e7e00640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f00cf980 0x7fe7f0159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 -- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f00cf980 msgr2=0x7fe7f0159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f00cf980 0x7fe7f0159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 -- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f0073fc0 msgr2=0x7fe7f009d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f0073fc0 0x7fe7f009d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 -- 172.21.15.31:0/1630497425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f009cb40 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f5800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800b990 tx=0x7fe7e800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8019070 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7e800cb90 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8014440 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f0156bc0 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.482+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f0157180 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.483+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f00b22d0 con 0x7fe7f0152a70 2024-09-13T08:45:51.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.485+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fe7e801c070 con 0x7fe7f0152a70 2024-09-13T08:45:51.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.486+0000 7fe7ed600640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 0x7fe7cc07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.486+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5753+0+0 (secure 0 0 0) 0x7fe7e809b8a0 con 0x7fe7f0152a70 2024-09-13T08:45:51.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.486+0000 7fe7f4e00640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 0x7fe7cc07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.487+0000 7fe7f4e00640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 0x7fe7cc07b580 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0033000 tx=0x7fe7e0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.490+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe7e8062870 con 0x7fe7f0152a70 2024-09-13T08:45:51.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.721+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7f009dc10 con 0x7fe7f0152a70 2024-09-13T08:45:51.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.721+0000 7fe7ed600640 1 -- 172.21.15.31:0/1630497425 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5228 (secure 0 0 0) 0x7fe7e80672a0 con 0x7fe7f0152a70 2024-09-13T08:45:51.723 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:51.723 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":100,"btime":"2024-09-13T08:45:43:836819+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:45:33.041130+0000","modified":"2024-09-13T08:45:36.057928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8454":{"gid":8454,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6834/3730730208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3730730208},{"type":"v1","addr":"172.21.15.31:6834","nonce":3730730208}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8454,"qdb_cluster":[8454]},"id":11}]} 2024-09-13T08:45:51.723 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 100 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 msgr2=0x7fe7cc07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 0x7fe7cc07b580 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0033000 tx=0x7fe7e0004730 comp rx=0 tx=0).stop 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 msgr2=0x7fe7f015d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800b990 tx=0x7fe7e800bd90 comp rx=0 tx=0).stop 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 shutdown_connections 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe7cc0790c0 0x7fe7cc07b580 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7f0152a70 0x7fe7f015d440 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe7f00cf980 0x7fe7f0159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 --2- 172.21.15.31:0/1630497425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7f0073fc0 0x7fe7f009d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 >> 172.21.15.31:0/1630497425 conn(0x7fe7f00cc910 msgr2=0x7fe7f00ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:51.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 shutdown_connections 2024-09-13T08:45:51.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.724+0000 7fe7f7800640 1 -- 172.21.15.31:0/1630497425 wait complete. 2024-09-13T08:45:51.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:51.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 -- 172.21.15.31:0/1718374569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500d4110 msgr2=0x7ff1501619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 --2- 172.21.15.31:0/1718374569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500d4110 0x7ff1501619a0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff15006a510 tx=0x7ff14802a560 comp rx=0 tx=0).stop 2024-09-13T08:45:51.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 -- 172.21.15.31:0/1718374569 shutdown_connections 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 --2- 172.21.15.31:0/1718374569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500d4110 0x7ff1501619a0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 --2- 172.21.15.31:0/1718374569 >> v2:172.21.15.45:3300/0 conn(0x7ff1500b47f0 0x7ff1500d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 --2- 172.21.15.31:0/1718374569 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff15015dfa0 0x7ff1500d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 -- 172.21.15.31:0/1718374569 >> 172.21.15.31:0/1718374569 conn(0x7ff15008d8b0 msgr2=0x7ff15009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 -- 172.21.15.31:0/1718374569 shutdown_connections 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.904+0000 7ff155a00640 1 -- 172.21.15.31:0/1718374569 wait complete. 2024-09-13T08:45:51.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.906+0000 7ff155a00640 1 Processor -- start 2024-09-13T08:45:51.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.906+0000 7ff155a00640 1 -- start start 2024-09-13T08:45:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 0x7ff150160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 0x7ff150161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff150162270 con 0x7ff1500b47f0 2024-09-13T08:45:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501620f0 con 0x7ff15015dfa0 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff1501623f0 con 0x7ff1500d4110 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56854/0 (socket says 172.21.15.31:56854) 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14fe00640 1 -- 172.21.15.31:0/2898642753 learned_addr learned my addr 172.21.15.31:0/2898642753 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff155000640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 0x7ff150161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14f400640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 0x7ff150160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14fe00640 1 -- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 msgr2=0x7ff150160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.907+0000 7ff14fe00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 0x7ff150160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff14fe00640 1 -- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 msgr2=0x7ff150161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff14fe00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 0x7ff150161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff14fe00640 1 -- 172.21.15.31:0/2898642753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15014b610 con 0x7ff1500b47f0 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff155000640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 0x7ff150161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:51.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff14f400640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 0x7ff150160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff14fe00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7ff14000eb00 tx=0x7ff14000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140004b20 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff140004ce0 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15014e250 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140005280 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.908+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15014bbb0 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.909+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1501619a0 con 0x7ff1500b47f0 2024-09-13T08:45:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.910+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7ff140005420 con 0x7ff1500b47f0 2024-09-13T08:45:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.911+0000 7ff147e00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 0x7ff12c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.911+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5782+0+0 (secure 0 0 0) 0x7ff140099e70 con 0x7ff1500b47f0 2024-09-13T08:45:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.911+0000 7ff14f400640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 0x7ff12c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:51.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.912+0000 7ff14f400640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 0x7ff12c07b580 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff1500cad50 tx=0x7ff138005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:51.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:51.916+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff140063f90 con 0x7ff1500b47f0 2024-09-13T08:45:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.113+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1500d4cf0 con 0x7ff1500b47f0 2024-09-13T08:45:52.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.114+0000 7ff147e00640 1 -- 172.21.15.31:0/2898642753 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16807 (secure 0 0 0) 0x7ff140063890 con 0x7ff1500b47f0 2024-09-13T08:45:52.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:52.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:51.871211+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:45:30.846891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T08:45:32.010174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.d","mappings":[{"from":5,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.116+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 msgr2=0x7ff12c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.116+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 0x7ff12c07b580 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff1500cad50 tx=0x7ff138005f90 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.116+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 msgr2=0x7ff1500b9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.116+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7ff14000eb00 tx=0x7ff14000c2d0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 shutdown_connections 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff12c0790c0 0x7ff12c07b580 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15015dfa0 0x7ff150161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff1500d4110 0x7ff150160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 --2- 172.21.15.31:0/2898642753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1500b47f0 0x7ff1500b9540 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 >> 172.21.15.31:0/2898642753 conn(0x7ff15008d8b0 msgr2=0x7ff15009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 shutdown_connections 2024-09-13T08:45:52.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.117+0000 7ff155a00640 1 -- 172.21.15.31:0/2898642753 wait complete. 2024-09-13T08:45:52.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:45:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 -- 172.21.15.31:0/2826319527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae90003a00 msgr2=0x7fae90003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 --2- 172.21.15.31:0/2826319527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae90003a00 0x7fae90003e20 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009f00 tx=0x7fae90005830 comp rx=0 tx=0).stop 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeae200640 1 -- 172.21.15.31:0/2826319527 reap_dead start 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 -- 172.21.15.31:0/2826319527 shutdown_connections 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 -- 172.21.15.31:0/2826319527 >> 172.21.15.31:0/2826319527 conn(0x7faea80b18f0 msgr2=0x7faea80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 -- 172.21.15.31:0/2826319527 shutdown_connections 2024-09-13T08:45:52.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.296+0000 7faeb0200640 1 -- 172.21.15.31:0/2826319527 wait complete. 2024-09-13T08:45:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.298+0000 7faeb0200640 1 Processor -- start 2024-09-13T08:45:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.298+0000 7faeb0200640 1 -- start start 2024-09-13T08:45:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae90003a00 0x7faea815f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faea80b3230 0x7faea80ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faea8073380 con 0x7faea81f5c50 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faea80ce9e0 con 0x7fae90003a00 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeb0200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7faea80cdb80 con 0x7faea80b3230 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeace00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeace00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56870/0 (socket says 172.21.15.31:56870) 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.299+0000 7faeace00640 1 -- 172.21.15.31:0/577449099 learned_addr learned my addr 172.21.15.31:0/577449099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faead800640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae90003a00 0x7faea815f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeae200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faea80b3230 0x7faea80ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 -- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faea80b3230 msgr2=0x7faea80ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faea80b3230 0x7faea80ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 -- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae90003a00 msgr2=0x7faea815f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae90003a00 0x7faea815f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 -- 172.21.15.31:0/577449099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea8154fd0 con 0x7faea81f5c50 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeace00640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fae94000fb0 tx=0x7fae94001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae94015070 con 0x7faea81f5c50 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae94010980 con 0x7faea81f5c50 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.300+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea80c93a0 con 0x7faea81f5c50 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.301+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae94010dd0 con 0x7faea81f5c50 2024-09-13T08:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.301+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea80c9960 con 0x7faea81f5c50 2024-09-13T08:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.302+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea80ce090 con 0x7faea81f5c50 2024-09-13T08:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.303+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fae94018070 con 0x7faea81f5c50 2024-09-13T08:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.304+0000 7faea5600640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 0x7fae8807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.304+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5782+0+0 (secure 0 0 0) 0x7fae940993e0 con 0x7faea81f5c50 2024-09-13T08:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.304+0000 7faead800640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 0x7fae8807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.305+0000 7faead800640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 0x7fae8807b5b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fae9c035000 tx=0x7fae9c02d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:52.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.308+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fae94062eb0 con 0x7faea81f5c50 2024-09-13T08:45:52.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.543+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faea80b25f0 con 0x7faea81f5c50 2024-09-13T08:45:52.544 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:45:52.543+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:45:52.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.545+0000 7faea5600640 1 -- 172.21.15.31:0/577449099 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v101) ==== 161+0+0 (secure 0 0 0) 0x7fae940678e0 con 0x7faea81f5c50 2024-09-13T08:45:52.546 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 msgr2=0x7fae8807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 0x7fae8807b5b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fae9c035000 tx=0x7fae9c02d040 comp rx=0 tx=0).stop 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 msgr2=0x7faea80b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fae94000fb0 tx=0x7fae94001820 comp rx=0 tx=0).stop 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 shutdown_connections 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae880790f0 0x7fae8807b5b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faea80b3230 0x7faea80ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea81f5c50 0x7faea80b2cf0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 --2- 172.21.15.31:0/577449099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae90003a00 0x7faea815f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 >> 172.21.15.31:0/577449099 conn(0x7faea80b18f0 msgr2=0x7faea80cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 shutdown_connections 2024-09-13T08:45:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.549+0000 7faeb0200640 1 -- 172.21.15.31:0/577449099 wait complete. 2024-09-13T08:45:52.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:45:52.616 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 -- 172.21.15.31:0/133522198 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00cafd0 msgr2=0x7f62d006e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 --2- 172.21.15.31:0/133522198 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00cafd0 0x7f62d006e8a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009860 tx=0x7f62bc02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 -- 172.21.15.31:0/133522198 shutdown_connections 2024-09-13T08:45:52.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 --2- 172.21.15.31:0/133522198 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00cafd0 0x7f62d006e8a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 -- 172.21.15.31:0/133522198 >> 172.21.15.31:0/133522198 conn(0x7f62d00cfb70 msgr2=0x7f62d00cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:52.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 -- 172.21.15.31:0/133522198 shutdown_connections 2024-09-13T08:45:52.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.734+0000 7f62d7800640 1 -- 172.21.15.31:0/133522198 wait complete. 2024-09-13T08:45:52.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.736+0000 7f62d7800640 1 Processor -- start 2024-09-13T08:45:52.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.736+0000 7f62d7800640 1 -- start start 2024-09-13T08:45:52.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 0x7f62d00c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62d015e310 0x7f62d015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00cdfd0 con 0x7f62d00cafd0 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00cde50 con 0x7f62d015e310 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d7800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00ce150 con 0x7f62d00ae660 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56892/0 (socket says 172.21.15.31:56892) 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62cfe00640 1 -- 172.21.15.31:0/1946365552 learned_addr learned my addr 172.21.15.31:0/1946365552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d5800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62d015e310 0x7f62d015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62d4e00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 0x7f62d00c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62cfe00640 1 -- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 msgr2=0x7f62d00c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.737+0000 7f62cfe00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 0x7f62d00c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62cfe00640 1 -- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62d015e310 msgr2=0x7f62d015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62cfe00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62d015e310 0x7f62d015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62cfe00640 1 -- 172.21.15.31:0/1946365552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00ad240 con 0x7f62d00cafd0 2024-09-13T08:45:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62d4e00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 0x7f62d00c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62cfe00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009980 tx=0x7f62bc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc040070 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62bc005310 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc005c90 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d01f29d0 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.738+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01f2f90 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.739+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d006e8a0 con 0x7f62d00cafd0 2024-09-13T08:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.741+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f62bc039040 con 0x7f62d00cafd0 2024-09-13T08:45:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.742+0000 7f62ccc00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 0x7f62b007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.742+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5868+0+0 (secure 0 0 0) 0x7f62bc0ba0a0 con 0x7f62d00cafd0 2024-09-13T08:45:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.742+0000 7f62d4e00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 0x7f62b007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.743+0000 7f62d4e00640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 0x7f62b007b580 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f62c40016d0 tx=0x7f62c4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.746+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f62bc0841c0 con 0x7f62d00cafd0 2024-09-13T08:45:52.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:52.983+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f62d015d3a0 con 0x7f62d00cafd0 2024-09-13T08:45:53.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.547+0000 7f62ccc00640 1 -- 172.21.15.31:0/1946365552 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v102) ==== 104+0+0 (secure 0 0 0) 0x7f62bc083ac0 con 0x7f62d00cafd0 2024-09-13T08:45:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.550+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 msgr2=0x7f62b007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.550+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 0x7f62b007b580 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f62c40016d0 tx=0x7f62c4009040 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.550+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 msgr2=0x7f62d01f2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009980 tx=0x7f62bc032c10 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 shutdown_connections 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62b00790c0 0x7f62b007b580 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62d015e310 0x7f62d015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62d00cafd0 0x7f62d01f2230 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 --2- 172.21.15.31:0/1946365552 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62d00ae660 0x7f62d00c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 >> 172.21.15.31:0/1946365552 conn(0x7f62d00cfb70 msgr2=0x7f62d00cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:53.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 shutdown_connections 2024-09-13T08:45:53.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.551+0000 7f62d7800640 1 -- 172.21.15.31:0/1946365552 wait complete. 2024-09-13T08:45:53.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:45:53.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 -- 172.21.15.31:0/3000418271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 msgr2=0x7f8dd0073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/3000418271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd0073ac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005fd0 tx=0x7f8dc002a540 comp rx=0 tx=0).stop 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 -- 172.21.15.31:0/3000418271 shutdown_connections 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/3000418271 >> v2:172.21.15.45:3300/0 conn(0x7f8dd00b2820 0x7f8dd006fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/3000418271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd0073ac0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 -- 172.21.15.31:0/3000418271 >> 172.21.15.31:0/3000418271 conn(0x7f8dd00b1520 msgr2=0x7f8dd00b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 -- 172.21.15.31:0/3000418271 shutdown_connections 2024-09-13T08:45:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.739+0000 7f8dcfe00640 1 -- 172.21.15.31:0/3000418271 wait complete. 2024-09-13T08:45:53.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.741+0000 7f8dcfe00640 1 Processor -- start 2024-09-13T08:45:53.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.741+0000 7f8dcfe00640 1 -- start start 2024-09-13T08:45:53.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd00b2820 0x7f8dd0159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:53.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd01f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd00ccbe0 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd00cca60 con 0x7f8dd00b2820 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcfe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd00ccd60 con 0x7f8dd015bc00 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dc6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dc6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56904/0 (socket says 172.21.15.31:56904) 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dc6000640 1 -- 172.21.15.31:0/240525802 learned_addr learned my addr 172.21.15.31:0/240525802 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd00b2820 0x7f8dd0159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.742+0000 7f8dcf400640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd01f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 -- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 msgr2=0x7f8dd01f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd01f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 -- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd00b2820 msgr2=0x7f8dd0159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd00b2820 0x7f8dd0159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 -- 172.21.15.31:0/240525802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd015e7f0 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc6000640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8db800e9c0 tx=0x7f8db800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db800cde0 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db8010520 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd01f8a30 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8010970 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.743+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd015a200 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.744+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd00cb370 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.746+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f8db8010b10 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.747+0000 7f8dc7400640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 0x7f8dac07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.747+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5868+0+0 (secure 0 0 0) 0x7f8db809c740 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.747+0000 7f8dcea00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 0x7f8dac07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.747+0000 7f8dcea00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 0x7f8dac07b580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005fd0 tx=0x7f8dc0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.751+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8db80636d0 con 0x7f8dd00cc3e0 2024-09-13T08:45:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:53.945+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8dd00c9c60 con 0x7f8dd00cc3e0 2024-09-13T08:45:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.890+0000 7f8dc7400640 1 -- 172.21.15.31:0/240525802 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v154) ==== 181+0+0 (secure 0 0 0) 0x7f8db8068100 con 0x7f8dd00cc3e0 2024-09-13T08:45:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 msgr2=0x7f8dac07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 0x7f8dac07b580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005fd0 tx=0x7f8dc0004730 comp rx=0 tx=0).stop 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 msgr2=0x7f8dd01f7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8db800e9c0 tx=0x7f8db800ee80 comp rx=0 tx=0).stop 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 shutdown_connections 2024-09-13T08:45:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8dac0790c0 0x7f8dac07b580 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8dd015bc00 0x7f8dd01f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dd00cc3e0 0x7f8dd01f7e60 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 --2- 172.21.15.31:0/240525802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd00b2820 0x7f8dd0159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 >> 172.21.15.31:0/240525802 conn(0x7f8dd00b1520 msgr2=0x7f8dd00cf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 shutdown_connections 2024-09-13T08:45:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:54.894+0000 7f8dcfe00640 1 -- 172.21.15.31:0/240525802 wait complete. 2024-09-13T08:45:54.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 -- 172.21.15.31:0/2739006819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d9415acb0 msgr2=0x7f1d940cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 --2- 172.21.15.31:0/2739006819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d9415acb0 0x7f1d940cf4d0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f1d84005fd0 tx=0x7f1d8402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 -- 172.21.15.31:0/2739006819 shutdown_connections 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 --2- 172.21.15.31:0/2739006819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d9415acb0 0x7f1d940cf4d0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 -- 172.21.15.31:0/2739006819 >> 172.21.15.31:0/2739006819 conn(0x7f1d940cfbb0 msgr2=0x7f1d940cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 -- 172.21.15.31:0/2739006819 shutdown_connections 2024-09-13T08:45:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.086+0000 7f1d99000640 1 -- 172.21.15.31:0/2739006819 wait complete. 2024-09-13T08:45:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.088+0000 7f1d99000640 1 Processor -- start 2024-09-13T08:45:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.088+0000 7f1d99000640 1 -- start start 2024-09-13T08:45:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d940741b0 0x7f1d94159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1d9415acb0 0x7f1d941f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9415f0b0 con 0x7f1d940745f0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9415ef30 con 0x7f1d940741b0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d99000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9415f230 con 0x7f1d9415acb0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d89400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d89400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51606/0 (socket says 172.21.15.31:51606) 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.089+0000 7f1d89400640 1 -- 172.21.15.31:0/3550283347 learned_addr learned my addr 172.21.15.31:0/3550283347 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d91e00640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d940741b0 0x7f1d94159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d92800640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1d9415acb0 0x7f1d941f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 -- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1d9415acb0 msgr2=0x7f1d941f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1d9415acb0 0x7f1d941f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 -- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d940741b0 msgr2=0x7f1d94159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d940741b0 0x7f1d94159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 -- 172.21.15.31:0/3550283347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d941f3d30 con 0x7f1d940745f0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d89400640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c450 tx=0x7f1d7c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0052d0 con 0x7f1d940745f0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7c005470 con 0x7f1d940745f0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.090+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c012870 con 0x7f1d940745f0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.091+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d941f36a0 con 0x7f1d940745f0 2024-09-13T08:45:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.091+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d940b0f20 con 0x7f1d940745f0 2024-09-13T08:45:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.092+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d9415f6e0 con 0x7f1d940745f0 2024-09-13T08:45:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.093+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f1d7c0035f0 con 0x7f1d940745f0 2024-09-13T08:45:55.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.094+0000 7f1d8aa00640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 0x7f1d7007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:55.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.094+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5430+0+0 (secure 0 0 0) 0x7f1d7c09b4a0 con 0x7f1d940745f0 2024-09-13T08:45:55.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.094+0000 7f1d91e00640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 0x7f1d7007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:55.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.095+0000 7f1d91e00640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 0x7f1d7007b580 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1d84033000 tx=0x7f1d84004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.098+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1d941f3d30 con 0x7f1d940745f0 2024-09-13T08:45:55.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.297+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1d940b10f0 con 0x7f1d940745f0 2024-09-13T08:45:55.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.896+0000 7f1d8aa00640 1 -- 172.21.15.31:0/3550283347 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v155) ==== 169+0+0 (secure 0 0 0) 0x7f1d7c062d30 con 0x7f1d940745f0 2024-09-13T08:45:55.898 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 msgr2=0x7f1d7007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 0x7f1d7007b580 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1d84033000 tx=0x7f1d84004730 comp rx=0 tx=0).stop 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 msgr2=0x7f1d941f2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c450 tx=0x7f1d7c00c910 comp rx=0 tx=0).stop 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 shutdown_connections 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1d700790c0 0x7f1d7007b580 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1d9415acb0 0x7f1d941f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d940745f0 0x7f1d941f2b20 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 --2- 172.21.15.31:0/3550283347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d940741b0 0x7f1d94159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:55.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 >> 172.21.15.31:0/3550283347 conn(0x7f1d940cfbb0 msgr2=0x7f1d9415de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:55.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.901+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 shutdown_connections 2024-09-13T08:45:55.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:55.902+0000 7f1d99000640 1 -- 172.21.15.31:0/3550283347 wait complete. 2024-09-13T08:45:55.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 -- 172.21.15.31:0/2633978449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 msgr2=0x7f85ec0cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2633978449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec0cdb80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f85dc005fd0 tx=0x7f85dc02ab30 comp rx=0 tx=0).stop 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 -- 172.21.15.31:0/2633978449 shutdown_connections 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2633978449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec0cdb80 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 -- 172.21.15.31:0/2633978449 >> 172.21.15.31:0/2633978449 conn(0x7f85ec0b18f0 msgr2=0x7f85ec0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 -- 172.21.15.31:0/2633978449 shutdown_connections 2024-09-13T08:45:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.093+0000 7f85f2e00640 1 -- 172.21.15.31:0/2633978449 wait complete. 2024-09-13T08:45:56.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.095+0000 7f85f2e00640 1 Processor -- start 2024-09-13T08:45:56.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.095+0000 7f85f2e00640 1 -- start start 2024-09-13T08:45:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85ec073860 0x7f85ec0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85ec0ced20 0x7f85ec0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0743c0 con 0x7f85ec15be30 2024-09-13T08:45:56.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0ce9e0 con 0x7f85ec0ced20 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f2e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec074540 con 0x7f85ec073860 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85ebe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85ec073860 0x7f85ec0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51634/0 (socket says 172.21.15.31:51634) 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85f0e00640 1 -- 172.21.15.31:0/2839363458 learned_addr learned my addr 172.21.15.31:0/2839363458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:56.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.096+0000 7f85eb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85ec0ced20 0x7f85ec0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 -- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85ec073860 msgr2=0x7f85ec0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85ec073860 0x7f85ec0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 -- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85ec0ced20 msgr2=0x7f85ec0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85ec0ced20 0x7f85ec0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 -- 172.21.15.31:0/2839363458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec1f4ce0 con 0x7f85ec15be30 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f0e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c430 tx=0x7f85e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4017070 con 0x7f85ec15be30 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85e40053c0 con 0x7f85ec15be30 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4003850 con 0x7f85ec15be30 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.097+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec0c9c00 con 0x7f85ec15be30 2024-09-13T08:45:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.098+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec0ca1c0 con 0x7f85ec15be30 2024-09-13T08:45:56.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.099+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec0749c0 con 0x7f85ec15be30 2024-09-13T08:45:56.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.100+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f85e401a070 con 0x7f85ec15be30 2024-09-13T08:45:56.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.101+0000 7f85e3e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 0x7f85cc07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.101+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7f85e409ac70 con 0x7f85ec15be30 2024-09-13T08:45:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.101+0000 7f85ebe00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 0x7f85cc07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.102+0000 7f85ebe00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 0x7f85cc07b580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f85dc035000 tx=0x7f85dc038040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.105+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f85e4061fc0 con 0x7f85ec15be30 2024-09-13T08:45:56.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.302+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f85ec0b25f0 con 0x7f85ec15be30 2024-09-13T08:45:56.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.306+0000 7f85e3e00640 1 -- 172.21.15.31:0/2839363458 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v45) ==== 71+0+0 (secure 0 0 0) 0x7f85e40669f0 con 0x7f85ec15be30 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 msgr2=0x7f85cc07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 0x7f85cc07b580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f85dc035000 tx=0x7f85dc038040 comp rx=0 tx=0).stop 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 msgr2=0x7f85ec09d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c430 tx=0x7f85e400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 shutdown_connections 2024-09-13T08:45:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.308+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85cc0790c0 0x7f85cc07b580 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85ec15be30 0x7f85ec09d9e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85ec0ced20 0x7f85ec0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 --2- 172.21.15.31:0/2839363458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85ec073860 0x7f85ec0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 >> 172.21.15.31:0/2839363458 conn(0x7f85ec0b18f0 msgr2=0x7f85ec0b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 shutdown_connections 2024-09-13T08:45:56.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.309+0000 7f85f2e00640 1 -- 172.21.15.31:0/2839363458 wait complete. 2024-09-13T08:45:56.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-13T08:45:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 -- 172.21.15.31:0/61720930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd80cf980 msgr2=0x7effd8152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 --2- 172.21.15.31:0/61720930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd80cf980 0x7effd8152e50 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7effd8073570 tx=0x7effc402a560 comp rx=0 tx=0).stop 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 -- 172.21.15.31:0/61720930 shutdown_connections 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 --2- 172.21.15.31:0/61720930 >> v2:172.21.15.45:3300/0 conn(0x7effd8073fc0 0x7effd80743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 --2- 172.21.15.31:0/61720930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd80cf980 0x7effd8152e50 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 -- 172.21.15.31:0/61720930 >> 172.21.15.31:0/61720930 conn(0x7effd80cc910 msgr2=0x7effd80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 -- 172.21.15.31:0/61720930 shutdown_connections 2024-09-13T08:45:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.484+0000 7effdfe00640 1 -- 172.21.15.31:0/61720930 wait complete. 2024-09-13T08:45:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.486+0000 7effdfe00640 1 Processor -- start 2024-09-13T08:45:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.486+0000 7effdfe00640 1 -- start start 2024-09-13T08:45:56.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7effd8073fc0 0x7effd81f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80cf980 0x7effd8153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7effd80b1ca0 con 0x7effd8152a70 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7effd80b1b20 con 0x7effd80cf980 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdfe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7effd80b1e20 con 0x7effd8073fc0 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51656/0 (socket says 172.21.15.31:51656) 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdde00640 1 -- 172.21.15.31:0/3624128735 learned_addr learned my addr 172.21.15.31:0/3624128735 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdca00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80cf980 0x7effd8153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.487+0000 7effdd400640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7effd8073fc0 0x7effd81f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 -- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7effd8073fc0 msgr2=0x7effd81f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7effd8073fc0 0x7effd81f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 -- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80cf980 msgr2=0x7effd8153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80cf980 0x7effd8153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 -- 172.21.15.31:0/3624128735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd8070210 con 0x7effd8152a70 2024-09-13T08:45:56.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdde00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7effd000b970 tx=0x7effd000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0019070 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effd000cc50 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd00146a0 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd81f5610 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.488+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd81f58f0 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.489+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd80b22f0 con 0x7effd8152a70 2024-09-13T08:45:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.491+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7effd001c070 con 0x7effd8152a70 2024-09-13T08:45:56.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.491+0000 7effd5600640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 0x7effb807b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:56.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.492+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7effd0098e60 con 0x7effd8152a70 2024-09-13T08:45:56.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.492+0000 7effdd400640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 0x7effb807b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:56.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.492+0000 7effdd400640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 0x7effb807b580 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7effcc0017f0 tx=0x7effcc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:56.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.496+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7effd0062930 con 0x7effd8152a70 2024-09-13T08:45:56.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.763+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7effd806e790 con 0x7effd8152a70 2024-09-13T08:45:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.898+0000 7effd5600640 1 -- 172.21.15.31:0/3624128735 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v485) ==== 149+0+0 (secure 0 0 0) 0x7effd0067360 con 0x7effd8152a70 2024-09-13T08:45:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 msgr2=0x7effb807b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 0x7effb807b580 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7effcc0017f0 tx=0x7effcc009040 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 msgr2=0x7effd8154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7effd000b970 tx=0x7effd000be30 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 shutdown_connections 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7effb80790c0 0x7effb807b580 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd8152a70 0x7effd8154350 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80cf980 0x7effd8153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 --2- 172.21.15.31:0/3624128735 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7effd8073fc0 0x7effd81f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 >> 172.21.15.31:0/3624128735 conn(0x7effd80cc910 msgr2=0x7effd80ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.900+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 shutdown_connections 2024-09-13T08:45:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:56.901+0000 7effdfe00640 1 -- 172.21.15.31:0/3624128735 wait complete. 2024-09-13T08:45:56.919 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- rather than going through a full upgrade cycle, ... ok 2024-09-13T08:45:56.919 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-13T08:45:56.920 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:45:56.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:45:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 -- 172.21.15.31:0/2721890671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 msgr2=0x7f9b30003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 --2- 172.21.15.31:0/2721890671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b30003e70 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009b50 tx=0x7f9b30005830 comp rx=0 tx=0).stop 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b50c00640 1 -- 172.21.15.31:0/2721890671 reap_dead start 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 -- 172.21.15.31:0/2721890671 shutdown_connections 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 -- 172.21.15.31:0/2721890671 >> 172.21.15.31:0/2721890671 conn(0x7f9b4c0cc910 msgr2=0x7f9b4c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 -- 172.21.15.31:0/2721890671 shutdown_connections 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.082+0000 7f9b52c00640 1 -- 172.21.15.31:0/2721890671 wait complete. 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.084+0000 7f9b52c00640 1 Processor -- start 2024-09-13T08:45:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.084+0000 7f9b52c00640 1 -- start start 2024-09-13T08:45:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 0x7f9b4c0b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 0x7f9b4c15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c0cec50 con 0x7f9b30003a50 2024-09-13T08:45:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c0cead0 con 0x7f9b4c1f4600 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b52c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c0cb850 con 0x7f9b4c0c51e0 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51682/0 (socket says 172.21.15.31:51682) 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 -- 172.21.15.31:0/3045537470 learned_addr learned my addr 172.21.15.31:0/3045537470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b50c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 0x7f9b4c15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 -- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 msgr2=0x7f9b4c0b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b43400640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 0x7f9b4c0b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 0x7f9b4c0b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 -- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 msgr2=0x7f9b4c15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 0x7f9b4c15d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b4be00640 1 -- 172.21.15.31:0/3045537470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c15b900 con 0x7f9b30003a50 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b50c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 0x7f9b4c15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.085+0000 7f9b43400640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 0x7f9b4c0b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b4be00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009a70 tx=0x7f9b3c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c038040 con 0x7f9b30003a50 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b3c004020 con 0x7f9b30003a50 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c1f4dc0 con 0x7f9b30003a50 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c004470 con 0x7f9b30003a50 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.086+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c1f5380 con 0x7f9b30003a50 2024-09-13T08:45:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.087+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b4c0cc270 con 0x7f9b30003a50 2024-09-13T08:45:57.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.088+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f9b3c045020 con 0x7f9b30003a50 2024-09-13T08:45:57.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.089+0000 7f9b48c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 0x7f9b2807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.089+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7f9b3c0bc100 con 0x7f9b30003a50 2024-09-13T08:45:57.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.090+0000 7f9b43400640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 0x7f9b2807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.090+0000 7f9b43400640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 0x7f9b2807b5b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b34001eb0 tx=0x7f9b34001600 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.094+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9b3c083aa0 con 0x7f9b30003a50 2024-09-13T08:45:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.290+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b4c0c39a0 con 0x7f9b30003a50 2024-09-13T08:45:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.291+0000 7f9b48c00640 1 -- 172.21.15.31:0/3045537470 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+13061 (secure 0 0 0) 0x7f9b3c0833a0 con 0x7f9b30003a50 2024-09-13T08:45:57.293 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:45:57.293 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":155,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:45:55.892320+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6834/3730730208":"2024-09-14T08:45:52.544233+0000","172.21.15.31:6832/3730730208":"2024-09-14T08:45:52.544233+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.293+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 msgr2=0x7f9b2807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.293+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 0x7f9b2807b5b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b34001eb0 tx=0x7f9b34001600 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.293+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 msgr2=0x7f9b4c154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.293+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009a70 tx=0x7f9b3c033670 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 shutdown_connections 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9b280790f0 0x7f9b2807b5b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b4c1f4600 0x7f9b4c15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b4c0c51e0 0x7f9b4c0b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 --2- 172.21.15.31:0/3045537470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b30003a50 0x7f9b4c154500 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 >> 172.21.15.31:0/3045537470 conn(0x7f9b4c0cc910 msgr2=0x7f9b4c0c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 shutdown_connections 2024-09-13T08:45:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.294+0000 7f9b52c00640 1 -- 172.21.15.31:0/3045537470 wait complete. 2024-09-13T08:45:57.312 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:45:57.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:45:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.473+0000 7f8a6a600640 1 -- 172.21.15.31:0/1061710672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 msgr2=0x7f8a640b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.473+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1061710672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a640b4840 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c005fd0 tx=0x7f8a5c02a940 comp rx=0 tx=0).stop 2024-09-13T08:45:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.474+0000 7f8a6a600640 1 -- 172.21.15.31:0/1061710672 shutdown_connections 2024-09-13T08:45:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.474+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1061710672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a640b4840 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.474+0000 7f8a6a600640 1 -- 172.21.15.31:0/1061710672 >> 172.21.15.31:0/1061710672 conn(0x7f8a640cf9b0 msgr2=0x7f8a640cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.474+0000 7f8a6a600640 1 -- 172.21.15.31:0/1061710672 shutdown_connections 2024-09-13T08:45:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.474+0000 7f8a6a600640 1 -- 172.21.15.31:0/1061710672 wait complete. 2024-09-13T08:45:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.475+0000 7f8a6a600640 1 Processor -- start 2024-09-13T08:45:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.475+0000 7f8a6a600640 1 -- start start 2024-09-13T08:45:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a64072450 0x7f8a641f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a64152a30 0x7f8a641f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a640b4d80 con 0x7f8a6406f550 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a64074500 con 0x7f8a64072450 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a6a600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8a640b4f00 con 0x7f8a64152a30 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.476+0000 7f8a69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51704/0 (socket says 172.21.15.31:51704) 2024-09-13T08:45:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 -- 172.21.15.31:0/1752326763 learned_addr learned my addr 172.21.15.31:0/1752326763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69c00640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a64152a30 0x7f8a641f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a5be00640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a64072450 0x7f8a641f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 -- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a64152a30 msgr2=0x7f8a641f84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a64152a30 0x7f8a641f84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 -- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a64072450 msgr2=0x7f8a641f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a64072450 0x7f8a641f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 -- 172.21.15.31:0/1752326763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a640caa10 con 0x7f8a6406f550 2024-09-13T08:45:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.477+0000 7f8a69200640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400c400 tx=0x7f8a5400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.478+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54011790 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.478+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a54017070 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.478+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54011ce0 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.478+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a640c70c0 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.478+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a640caf90 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.479+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a640b53b0 con 0x7f8a6406f550 2024-09-13T08:45:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.480+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f8a5401c460 con 0x7f8a6406f550 2024-09-13T08:45:57.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.481+0000 7f8a61600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 0x7f8a4007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.482+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7f8a5409b580 con 0x7f8a6406f550 2024-09-13T08:45:57.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.482+0000 7f8a5be00640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 0x7f8a4007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.482+0000 7f8a5be00640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 0x7f8a4007b580 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8a6414ec20 tx=0x7f8a4c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.486+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8a54011e80 con 0x7f8a6406f550 2024-09-13T08:45:57.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.681+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8a640ce110 con 0x7f8a6406f550 2024-09-13T08:45:57.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.681+0000 7f8a61600640 1 -- 172.21.15.31:0/1752326763 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) ==== 93+0+10 (secure 0 0 0) 0x7f8a540628d0 con 0x7f8a6406f550 2024-09-13T08:45:57.683 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:45:57.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 msgr2=0x7f8a4007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 0x7f8a4007b580 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8a6414ec20 tx=0x7f8a4c009040 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 msgr2=0x7f8a641e84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400c400 tx=0x7f8a5400c8c0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 shutdown_connections 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8a400790c0 0x7f8a4007b580 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a64152a30 0x7f8a641f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a64072450 0x7f8a641f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 --2- 172.21.15.31:0/1752326763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a6406f550 0x7f8a641e84b0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 >> 172.21.15.31:0/1752326763 conn(0x7f8a640cf9b0 msgr2=0x7f8a640af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 shutdown_connections 2024-09-13T08:45:57.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.684+0000 7f8a6a600640 1 -- 172.21.15.31:0/1752326763 wait complete. 2024-09-13T08:45:57.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 -- 172.21.15.31:0/2529452443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca80cf850 msgr2=0x7f3ca80ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 --2- 172.21.15.31:0/2529452443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca80cf850 0x7f3ca80ce3c0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80cfe60 tx=0x7f3c9002a670 comp rx=0 tx=0).stop 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 -- 172.21.15.31:0/2529452443 shutdown_connections 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 --2- 172.21.15.31:0/2529452443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca80cf850 0x7f3ca80ce3c0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 -- 172.21.15.31:0/2529452443 >> 172.21.15.31:0/2529452443 conn(0x7f3ca80cc910 msgr2=0x7f3ca80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 -- 172.21.15.31:0/2529452443 shutdown_connections 2024-09-13T08:45:57.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.862+0000 7f3caf600640 1 -- 172.21.15.31:0/2529452443 wait complete. 2024-09-13T08:45:57.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.863+0000 7f3caf600640 1 Processor -- start 2024-09-13T08:45:57.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.863+0000 7f3caf600640 1 -- start start 2024-09-13T08:45:57.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 0x7f3ca80b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca80cf850 0x7f3ca815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca81600f0 con 0x7f3ca815bd70 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca80cb850 con 0x7f3ca80cf850 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.864+0000 7f3caf600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca81f81c0 con 0x7f3ca80ce900 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51720/0 (socket says 172.21.15.31:51720) 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 -- 172.21.15.31:0/3562285122 learned_addr learned my addr 172.21.15.31:0/3562285122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3c9fe00640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca80cf850 0x7f3ca815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cacc00640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 0x7f3ca80b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 -- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 msgr2=0x7f3ca80b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 0x7f3ca80b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 -- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca80cf850 msgr2=0x7f3ca815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:57.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca80cf850 0x7f3ca815ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 -- 172.21.15.31:0/3562285122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca80c5ba0 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cacc00640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 0x7f3ca80b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.865+0000 7f3cad600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000c7a0 tx=0x7f3ca000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.866+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0003b70 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.866+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ca0003d10 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.866+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca000aab0 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.866+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca80c5e80 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.866+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca80b5b30 con 0x7f3ca815bd70 2024-09-13T08:45:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.867+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca806a190 con 0x7f3ca815bd70 2024-09-13T08:45:57.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.868+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f3ca000ac50 con 0x7f3ca815bd70 2024-09-13T08:45:57.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.869+0000 7f3ca5600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 0x7f3c8407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:57.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.869+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7f3ca009b6c0 con 0x7f3ca815bd70 2024-09-13T08:45:57.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.869+0000 7f3cacc00640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 0x7f3c8407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:57.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.870+0000 7f3cacc00640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 0x7f3c8407b580 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000f60 tx=0x7f3c98009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:57.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:57.873+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3ca806a190 con 0x7f3ca815bd70 2024-09-13T08:45:58.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.148+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7f3ca80acd40 con 0x7f3ca815bd70 2024-09-13T08:45:58.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.905+0000 7f3ca5600640 1 -- 172.21.15.31:0/3562285122 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v487) ==== 148+0+0 (secure 0 0 0) 0x7f3ca80acd40 con 0x7f3ca815bd70 2024-09-13T08:45:58.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 msgr2=0x7f3c8407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:58.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 0x7f3c8407b580 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000f60 tx=0x7f3c98009040 comp rx=0 tx=0).stop 2024-09-13T08:45:58.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 msgr2=0x7f3ca81f8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:58.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000c7a0 tx=0x7f3ca000cc60 comp rx=0 tx=0).stop 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 shutdown_connections 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3c840790c0 0x7f3c8407b580 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ca815bd70 0x7f3ca81f8950 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca80cf850 0x7f3ca815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 --2- 172.21.15.31:0/3562285122 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ca80ce900 0x7f3ca80b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 >> 172.21.15.31:0/3562285122 conn(0x7f3ca80cc910 msgr2=0x7f3ca815e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 shutdown_connections 2024-09-13T08:45:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:58.908+0000 7f3caf600640 1 -- 172.21.15.31:0/3562285122 wait complete. 2024-09-13T08:45:58.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 -- 172.21.15.31:0/1802291811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15ad60 msgr2=0x7f741c15f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 --2- 172.21.15.31:0/1802291811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15ad60 0x7f741c15f3d0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f741c070c60 tx=0x7f741002a4d0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 -- 172.21.15.31:0/1802291811 shutdown_connections 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 --2- 172.21.15.31:0/1802291811 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f741c15b140 0x7f741c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 --2- 172.21.15.31:0/1802291811 >> v2:172.21.15.45:3300/0 conn(0x7f741c15b5b0 0x7f741c0cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 --2- 172.21.15.31:0/1802291811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15ad60 0x7f741c15f3d0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 -- 172.21.15.31:0/1802291811 >> 172.21.15.31:0/1802291811 conn(0x7f741c0cc910 msgr2=0x7f741c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 -- 172.21.15.31:0/1802291811 shutdown_connections 2024-09-13T08:45:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.093+0000 7f7424200640 1 -- 172.21.15.31:0/1802291811 wait complete. 2024-09-13T08:45:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.094+0000 7f7424200640 1 Processor -- start 2024-09-13T08:45:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 -- start start 2024-09-13T08:45:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c15ad60 0x7f741c1f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f741c15b5b0 0x7f741c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f741c06fc30 con 0x7f741c15b140 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f741c06fab0 con 0x7f741c15ad60 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.095+0000 7f7424200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f741c06fdb0 con 0x7f741c15b5b0 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51734/0 (socket says 172.21.15.31:51734) 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 -- 172.21.15.31:0/989811580 learned_addr learned my addr 172.21.15.31:0/989811580 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 -- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f741c15b5b0 msgr2=0x7f741c15c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f741c15b5b0 0x7f741c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 -- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c15ad60 msgr2=0x7f741c1f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c15ad60 0x7f741c1f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 -- 172.21.15.31:0/989811580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c1f5fa0 con 0x7f741c15b140 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.096+0000 7f7420e00640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f740800d7a0 tx=0x7f740800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.097+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408017070 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.097+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7408009d20 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.097+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741c0b50c0 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.097+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080106c0 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.097+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741c0b5680 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.098+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741c0ae380 con 0x7f741c15b140 2024-09-13T08:45:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.099+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f740801a070 con 0x7f741c15b140 2024-09-13T08:45:59.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.100+0000 7f7419600640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 0x7f73fc07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.100+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7f74080992e0 con 0x7f741c15b140 2024-09-13T08:45:59.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.100+0000 7f7421800640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 0x7f73fc07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.101+0000 7f7421800640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 0x7f73fc07b580 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7410033000 tx=0x7f74100046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.104+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f741c0ae380 con 0x7f741c15b140 2024-09-13T08:45:59.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.302+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f741c072670 con 0x7f741c15b140 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.303+0000 7f7419600640 1 -- 172.21.15.31:0/989811580 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v45) ==== 59+0+1998 (secure 0 0 0) 0x7f741c072670 con 0x7f741c15b140 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:45:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:45:59.306 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:45:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 msgr2=0x7f73fc07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 0x7f73fc07b580 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7410033000 tx=0x7f74100046c0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 msgr2=0x7f741c0c2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f740800d7a0 tx=0x7f740800dc60 comp rx=0 tx=0).stop 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 shutdown_connections 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f73fc0790c0 0x7f73fc07b580 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f741c15b5b0 0x7f741c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f741c15b140 0x7f741c0c2c90 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 --2- 172.21.15.31:0/989811580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f741c15ad60 0x7f741c1f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.305+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 >> 172.21.15.31:0/989811580 conn(0x7f741c0cc910 msgr2=0x7f741c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:59.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.306+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 shutdown_connections 2024-09-13T08:45:59.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.306+0000 7f7424200640 1 -- 172.21.15.31:0/989811580 wait complete. 2024-09-13T08:45:59.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:45:59.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 -- 172.21.15.31:0/200118314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 msgr2=0x7fd864004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 --2- 172.21.15.31:0/200118314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd864004800 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd868009b80 tx=0x7fd864005830 comp rx=0 tx=0).stop 2024-09-13T08:45:59.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd87fe00640 1 -- 172.21.15.31:0/200118314 reap_dead start 2024-09-13T08:45:59.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 -- 172.21.15.31:0/200118314 shutdown_connections 2024-09-13T08:45:59.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 -- 172.21.15.31:0/200118314 >> 172.21.15.31:0/200118314 conn(0x7fd8800cfbb0 msgr2=0x7fd8800cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:59.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 -- 172.21.15.31:0/200118314 shutdown_connections 2024-09-13T08:45:59.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.488+0000 7fd886400640 1 -- 172.21.15.31:0/200118314 wait complete. 2024-09-13T08:45:59.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.490+0000 7fd886400640 1 Processor -- start 2024-09-13T08:45:59.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.490+0000 7fd886400640 1 -- start start 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8801f5ac0 0x7fd8800cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd8800cf170 0x7fd8801f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd88006e740 con 0x7fd864003fd0 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd88006e5c0 con 0x7fd8801f5ac0 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd886400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd8800b29e0 con 0x7fd8800cf170 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd87f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd87f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51748/0 (socket says 172.21.15.31:51748) 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd87f400640 1 -- 172.21.15.31:0/3994922737 learned_addr learned my addr 172.21.15.31:0/3994922737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd87fe00640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd8800cf170 0x7fd8801f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd876a00640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8801f5ac0 0x7fd8800cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.491+0000 7fd87f400640 1 -- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd8800cf170 msgr2=0x7fd8801f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd87f400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd8800cf170 0x7fd8801f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd87f400640 1 -- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8801f5ac0 msgr2=0x7fd8800cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd87f400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8801f5ac0 0x7fd8800cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd87f400640 1 -- 172.21.15.31:0/3994922737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8801484f0 con 0x7fd864003fd0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd87f400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fd870006630 tx=0x7fd8700039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870015070 con 0x7fd864003fd0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8700119c0 con 0x7fd864003fd0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870011e10 con 0x7fd864003fd0 2024-09-13T08:45:59.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801f9010 con 0x7fd864003fd0 2024-09-13T08:45:59.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.492+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801f52c0 con 0x7fd864003fd0 2024-09-13T08:45:59.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.493+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8800c34d0 con 0x7fd864003fd0 2024-09-13T08:45:59.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.495+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fd870018070 con 0x7fd864003fd0 2024-09-13T08:45:59.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.495+0000 7fd877e00640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 0x7fd85c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:45:59.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.496+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7fd87009c6a0 con 0x7fd864003fd0 2024-09-13T08:45:59.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.496+0000 7fd876a00640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 0x7fd85c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:45:59.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.496+0000 7fd876a00640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 0x7fd85c07b580 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd868035000 tx=0x7fd8680046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:45:59.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.500+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd8800c34d0 con 0x7fd864003fd0 2024-09-13T08:45:59.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.695+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd8800cb740 con 0x7fd864003fd0 2024-09-13T08:45:59.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.699+0000 7fd877e00640 1 -- 172.21.15.31:0/3994922737 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7fd870064040 con 0x7fd864003fd0 2024-09-13T08:45:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 msgr2=0x7fd85c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 0x7fd85c07b580 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd868035000 tx=0x7fd8680046c0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 msgr2=0x7fd880147fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fd870006630 tx=0x7fd8700039c0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 shutdown_connections 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd85c0790c0 0x7fd85c07b580 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.701+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd8800cf170 0x7fd8801f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.702+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8801f5ac0 0x7fd8800cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.702+0000 7fd886400640 1 --2- 172.21.15.31:0/3994922737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd864003fd0 0x7fd880147fb0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.702+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 >> 172.21.15.31:0/3994922737 conn(0x7fd8800cfbb0 msgr2=0x7fd8800b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.702+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 shutdown_connections 2024-09-13T08:45:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:45:59.702+0000 7fd886400640 1 -- 172.21.15.31:0/3994922737 wait complete. 2024-09-13T08:45:59.719 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:45:59.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:59.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:45:59.900 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:45:59.901 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:59.901 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:59.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:45:59.938 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:45:59.938 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:45:59.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:45:59.975 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:45:59.975 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:00.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:46:00.013 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:00.013 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:00.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:46:00.051 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:46:00.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:00.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 -- 172.21.15.31:0/2744381962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8015dfc0 msgr2=0x7fde800cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 --2- 172.21.15.31:0/2744381962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8015dfc0 0x7fde800cc300 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fde70009830 tx=0x7fde7002a510 comp rx=0 tx=0).stop 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 -- 172.21.15.31:0/2744381962 shutdown_connections 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 --2- 172.21.15.31:0/2744381962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde8015dfc0 0x7fde800cc300 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 -- 172.21.15.31:0/2744381962 >> 172.21.15.31:0/2744381962 conn(0x7fde800cc910 msgr2=0x7fde800cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 -- 172.21.15.31:0/2744381962 shutdown_connections 2024-09-13T08:46:00.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.212+0000 7fde86600640 1 -- 172.21.15.31:0/2744381962 wait complete. 2024-09-13T08:46:00.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.214+0000 7fde86600640 1 Processor -- start 2024-09-13T08:46:00.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.214+0000 7fde86600640 1 -- start start 2024-09-13T08:46:00.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde800ca5c0 0x7fde8015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8015dfc0 0x7fde8015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde800b1d60 con 0x7fde800b1910 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde800740a0 con 0x7fde8015dfc0 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde86600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fde800b1ee0 con 0x7fde800ca5c0 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde7f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde7f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51774/0 (socket says 172.21.15.31:51774) 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde7f400640 1 -- 172.21.15.31:0/1253514246 learned_addr learned my addr 172.21.15.31:0/1253514246 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.215+0000 7fde7fe00640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8015dfc0 0x7fde8015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde76a00640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde800ca5c0 0x7fde8015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 -- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde800ca5c0 msgr2=0x7fde8015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde800ca5c0 0x7fde8015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 -- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8015dfc0 msgr2=0x7fde8015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8015dfc0 0x7fde8015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 -- 172.21.15.31:0/1253514246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde801f82f0 con 0x7fde800b1910 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde7f400640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fde70009950 tx=0x7fde70033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70031d60 con 0x7fde800b1910 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde70005020 con 0x7fde800b1910 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70005470 con 0x7fde800b1910 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.216+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde801f85d0 con 0x7fde800b1910 2024-09-13T08:46:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.217+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde801f8b90 con 0x7fde800b1910 2024-09-13T08:46:00.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.218+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde800cc300 con 0x7fde800b1910 2024-09-13T08:46:00.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.219+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fde700047a0 con 0x7fde800b1910 2024-09-13T08:46:00.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.220+0000 7fde77e00640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 0x7fde5c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.220+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7fde700ba2d0 con 0x7fde800b1910 2024-09-13T08:46:00.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.220+0000 7fde76a00640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 0x7fde5c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.221+0000 7fde76a00640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 0x7fde5c07b580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fde801f8250 tx=0x7fde68009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:00.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.224+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fde7003f030 con 0x7fde800b1910 2024-09-13T08:46:00.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.457+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde800ce940 con 0x7fde800b1910 2024-09-13T08:46:00.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.458+0000 7fde77e00640 1 -- 172.21.15.31:0/1253514246 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3913 (secure 0 0 0) 0x7fde700843f0 con 0x7fde800b1910 2024-09-13T08:46:00.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:00.461 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-13T08:45:53:546255+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7133,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/4055454817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4055454817},{"type":"v1","addr":"172.21.15.45:6801","nonce":4055454817}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":7136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1486010974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1486010974},{"type":"v1","addr":"172.21.15.45:6835","nonce":1486010974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":8451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1529254080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1529254080},{"type":"v1","addr":"172.21.15.31:6835","nonce":1529254080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8748,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1496861638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1496861638},{"type":"v1","addr":"172.21.15.31:6834","nonce":1496861638}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[]} 2024-09-13T08:46:00.461 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-13T08:46:00.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 msgr2=0x7fde5c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 0x7fde5c07b580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fde801f8250 tx=0x7fde68009040 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 msgr2=0x7fde8014f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fde70009950 tx=0x7fde70033ce0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 shutdown_connections 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fde5c0790c0 0x7fde5c07b580 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8015dfc0 0x7fde8015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fde800ca5c0 0x7fde8015ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 --2- 172.21.15.31:0/1253514246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde800b1910 0x7fde8014f2b0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.460+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 >> 172.21.15.31:0/1253514246 conn(0x7fde800cc910 msgr2=0x7fde80072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.461+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 shutdown_connections 2024-09-13T08:46:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.461+0000 7fde86600640 1 -- 172.21.15.31:0/1253514246 wait complete. 2024-09-13T08:46:00.479 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:46:00.479 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:46:00.479 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:46:00.479 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:00.480 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:46:00.480 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:46:00.480 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:46:00.480 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:00.481 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:46:00.481 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:46:00.481 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:46:00.481 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:00.481 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:46:00.481 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:46:00.481 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:46:00.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:00.497 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:46:00.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:46:00.498 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:46:00.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:46:00.499 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:46:00.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:46:00.500 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:46:00.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:46:00.541 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:46:00.542 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:00.543 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:46:00.543 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:00.545 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:46:00.546 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:00.548 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:46:00.548 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:00.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:46:00.673 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:46:00.674 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:46:00.685 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:46:00.686 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.715+0000 7ff3bea00640 1 -- 172.21.15.31:0/315947673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 msgr2=0x7ff3b81619e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 --2- 172.21.15.31:0/315947673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b81619e0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4005fd0 tx=0x7ff3a402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 -- 172.21.15.31:0/315947673 shutdown_connections 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 --2- 172.21.15.31:0/315947673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b81619e0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 --2- 172.21.15.31:0/315947673 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b80ce560 0x7ff3b815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 -- 172.21.15.31:0/315947673 >> 172.21.15.31:0/315947673 conn(0x7ff3b80d1d30 msgr2=0x7ff3b80d2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 -- 172.21.15.31:0/315947673 shutdown_connections 2024-09-13T08:46:00.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.716+0000 7ff3bea00640 1 -- 172.21.15.31:0/315947673 wait complete. 2024-09-13T08:46:00.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.717+0000 7ff3bea00640 1 Processor -- start 2024-09-13T08:46:00.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.717+0000 7ff3bea00640 1 -- start start 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80ce560 0x7ff3b814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8161f20 0x7ff3b80afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80ccc00 con 0x7ff3b80ce940 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80cca80 con 0x7ff3b80ce560 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.718+0000 7ff3bea00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b80cf4f0 con 0x7ff3b8161f20 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51830/0 (socket says 172.21.15.31:51830) 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 -- 172.21.15.31:0/1252715339 learned_addr learned my addr 172.21.15.31:0/1252715339 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bd600640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80ce560 0x7ff3b814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3be000640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8161f20 0x7ff3b80afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 -- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8161f20 msgr2=0x7ff3b80afb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8161f20 0x7ff3b80afb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 -- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80ce560 msgr2=0x7ff3b814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80ce560 0x7ff3b814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 -- 172.21.15.31:0/1252715339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b8152210 con 0x7ff3b80ce940 2024-09-13T08:46:00.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.719+0000 7ff3bcc00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009850 tx=0x7ff3a40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:00.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.720+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a403f070 con 0x7ff3b80ce940 2024-09-13T08:46:00.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.720+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3a4003ce0 con 0x7ff3b80ce940 2024-09-13T08:46:00.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.720+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4035030 con 0x7ff3b80ce940 2024-09-13T08:46:00.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.720+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b815e020 con 0x7ff3b80ce940 2024-09-13T08:46:00.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.720+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b814c640 con 0x7ff3b80ce940 2024-09-13T08:46:00.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.721+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b81624e0 con 0x7ff3b80ce940 2024-09-13T08:46:00.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.722+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7ff3a4042070 con 0x7ff3b80ce940 2024-09-13T08:46:00.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.723+0000 7ff3b5600640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 0x7ff39807b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:00.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.723+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4987+0+0 (secure 0 0 0) 0x7ff3a40bc3d0 con 0x7ff3b80ce940 2024-09-13T08:46:00.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.723+0000 7ff3bd600640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 0x7ff39807b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:00.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.724+0000 7ff3bd600640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 0x7ff39807b580 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009d20 tx=0x7ff3ac005f30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:00.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.727+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff3a40837e0 con 0x7ff3b80ce940 2024-09-13T08:46:00.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:00.942+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff3b815eb70 con 0x7ff3b80ce940 2024-09-13T08:46:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.934+0000 7ff3b5600640 1 -- 172.21.15.31:0/1252715339 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v160) ==== 97+0+0 (secure 0 0 0) 0x7ff3a4088210 con 0x7ff3b80ce940 2024-09-13T08:46:01.936 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.937+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 msgr2=0x7ff39807b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.937+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 0x7ff39807b580 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009d20 tx=0x7ff3ac005f30 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.937+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 msgr2=0x7ff3b815ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009850 tx=0x7ff3a40035f0 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 shutdown_connections 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff3980790c0 0x7ff39807b580 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8161f20 0x7ff3b80afb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ce940 0x7ff3b815ca20 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 --2- 172.21.15.31:0/1252715339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80ce560 0x7ff3b814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 >> 172.21.15.31:0/1252715339 conn(0x7ff3b80d1d30 msgr2=0x7ff3b80b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:01.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 shutdown_connections 2024-09-13T08:46:01.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:01.938+0000 7ff3bea00640 1 -- 172.21.15.31:0/1252715339 wait complete. 2024-09-13T08:46:01.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:46:02.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2193448407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 msgr2=0x7f7ab40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2193448407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40d2b90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4005fd0 tx=0x7f7aa402a540 comp rx=0 tx=0).stop 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2193448407 shutdown_connections 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2193448407 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40b4b60 0x7f7ab40d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2193448407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40d2b90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2193448407 >> 172.21.15.31:0/2193448407 conn(0x7f7ab409d190 msgr2=0x7f7ab40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2193448407 shutdown_connections 2024-09-13T08:46:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.124+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2193448407 wait complete. 2024-09-13T08:46:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.126+0000 7f7ab9a00640 1 Processor -- start 2024-09-13T08:46:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.126+0000 7f7ab9a00640 1 -- start start 2024-09-13T08:46:02.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 0x7f7ab40cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab40b3440 con 0x7f7ab40b4b60 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab40d13a0 con 0x7f7ab40d27b0 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab9a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab40b35c0 con 0x7f7ab40d05f0 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51846/0 (socket says 172.21.15.31:51846) 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.127+0000 7f7ab3e00640 1 -- 172.21.15.31:0/2196019809 learned_addr learned my addr 172.21.15.31:0/2196019809 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab9000640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3400640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 0x7f7ab40cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 -- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 msgr2=0x7f7ab40cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 0x7f7ab40cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 -- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 msgr2=0x7f7ab40b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 -- 172.21.15.31:0/2196019809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab40c8eb0 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3400640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 0x7f7ab40cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab3e00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4034040 tx=0x7f7aa40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa403e030 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7aa4003d90 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4035040 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.128+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab40cdd50 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.129+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab40c9430 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.130+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab40d1b10 con 0x7f7ab40b4b60 2024-09-13T08:46:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.131+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f7aa4041030 con 0x7f7ab40b4b60 2024-09-13T08:46:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.132+0000 7f7aabe00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 0x7f7a9407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.132+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 4901+0+0 (secure 0 0 0) 0x7f7aa40bc360 con 0x7f7ab40b4b60 2024-09-13T08:46:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.132+0000 7f7ab3400640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 0x7f7a9407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.133+0000 7f7ab3400640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 0x7f7a9407b580 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ab41566e0 tx=0x7f7aa0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.136+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7aa4083d00 con 0x7f7ab40b4b60 2024-09-13T08:46:02.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.400+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7ab406e5f0 con 0x7f7ab40b4b60 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.401+0000 7f7aabe00640 1 -- 172.21.15.31:0/2196019809 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+5304 (secure 0 0 0) 0x7f7aa4083600 con 0x7f7ab40b4b60 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:46:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:46:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:46:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:46:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:46:02.419 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:46:02.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 msgr2=0x7f7a9407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 0x7f7a9407b580 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ab41566e0 tx=0x7f7aa0005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 msgr2=0x7f7ab40b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4034040 tx=0x7f7aa40038a0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 shutdown_connections 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7a940790c0 0x7f7a9407b580 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ab40d27b0 0x7f7ab40b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7ab40d05f0 0x7f7ab40cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 --2- 172.21.15.31:0/2196019809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ab40b4b60 0x7f7ab40b8a10 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.403+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 >> 172.21.15.31:0/2196019809 conn(0x7f7ab409d190 msgr2=0x7f7ab40d0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.404+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 shutdown_connections 2024-09-13T08:46:02.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.404+0000 7f7ab9a00640 1 -- 172.21.15.31:0/2196019809 wait complete. 2024-09-13T08:46:02.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:46:02.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.582+0000 7f1618e00640 1 -- 172.21.15.31:0/4030586120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140b52d0 msgr2=0x7f1614152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.582+0000 7f1618e00640 1 --2- 172.21.15.31:0/4030586120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140b52d0 0x7f1614152680 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f160c005fd0 tx=0x7f160c02a8a0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 -- 172.21.15.31:0/4030586120 shutdown_connections 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 --2- 172.21.15.31:0/4030586120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140b52d0 0x7f1614152680 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 --2- 172.21.15.31:0/4030586120 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140cc720 0x7f161406f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 -- 172.21.15.31:0/4030586120 >> 172.21.15.31:0/4030586120 conn(0x7f161406cb00 msgr2=0x7f161406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 -- 172.21.15.31:0/4030586120 shutdown_connections 2024-09-13T08:46:02.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.583+0000 7f1618e00640 1 -- 172.21.15.31:0/4030586120 wait complete. 2024-09-13T08:46:02.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.584+0000 7f1618e00640 1 Processor -- start 2024-09-13T08:46:02.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.585+0000 7f1618e00640 1 -- start start 2024-09-13T08:46:02.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.585+0000 7f1618e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140b52d0 0x7f16140c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.585+0000 7f1618e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.585+0000 7f1618e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16140ccb00 0x7f1614150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f1618e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1614152bc0 con 0x7f16140cc720 2024-09-13T08:46:02.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f1618e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1614070470 con 0x7f16140ccb00 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f1618e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1614152d40 con 0x7f16140b52d0 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51864/0 (socket says 172.21.15.31:51864) 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 -- 172.21.15.31:0/1177232513 learned_addr learned my addr 172.21.15.31:0/1177232513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f1613e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16140ccb00 0x7f1614150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f1613400640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140b52d0 0x7f16140c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 -- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140b52d0 msgr2=0x7f16140c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140b52d0 0x7f16140c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 -- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16140ccb00 msgr2=0x7f1614150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16140ccb00 0x7f1614150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.586+0000 7f160be00640 1 -- 172.21.15.31:0/1177232513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161415e760 con 0x7f16140cc720 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f160be00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c400 tx=0x7f15fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc011790 con 0x7f16140cc720 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15fc017070 con 0x7f16140cc720 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161415b620 con 0x7f16140cc720 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc011cc0 con 0x7f16140cc720 2024-09-13T08:46:02.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.587+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161415ece0 con 0x7f16140cc720 2024-09-13T08:46:02.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.588+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1614153260 con 0x7f16140cc720 2024-09-13T08:46:02.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.589+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f15fc01c460 con 0x7f16140cc720 2024-09-13T08:46:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.591+0000 7f160b400640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 0x7f15f007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.591+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 4901+0+0 (secure 0 0 0) 0x7f15fc09b680 con 0x7f16140cc720 2024-09-13T08:46:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.591+0000 7f1613400640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 0x7f15f007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.592+0000 7f1613400640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 0x7f15f007b580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1604009b60 tx=0x7f1604005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.595+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f15fc0630a0 con 0x7f16140cc720 2024-09-13T08:46:02.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.795+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f16140c6ff0 con 0x7f16140cc720 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.796+0000 7f160b400640 1 -- 172.21.15.31:0/1177232513 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v160) ==== 84+0+65 (secure 0 0 0) 0x7f15fc0629a0 con 0x7f16140cc720 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:02.798 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:46:02.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 msgr2=0x7f15f007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 0x7f15f007b580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1604009b60 tx=0x7f1604005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 msgr2=0x7f1614156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c400 tx=0x7f15fc00c8c0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 shutdown_connections 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15f00790c0 0x7f15f007b580 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16140ccb00 0x7f1614150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16140cc720 0x7f1614156a30 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 --2- 172.21.15.31:0/1177232513 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f16140b52d0 0x7f16140c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 >> 172.21.15.31:0/1177232513 conn(0x7f161406cb00 msgr2=0x7f16140b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 shutdown_connections 2024-09-13T08:46:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.798+0000 7f1618e00640 1 -- 172.21.15.31:0/1177232513 wait complete. 2024-09-13T08:46:02.816 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:46:02.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:46:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 -- 172.21.15.31:0/3102037616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b100cf4d0 msgr2=0x7f9b100cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 --2- 172.21.15.31:0/3102037616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b100cf4d0 0x7f9b100cd780 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009830 tx=0x7f9af802a510 comp rx=0 tx=0).stop 2024-09-13T08:46:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 -- 172.21.15.31:0/3102037616 shutdown_connections 2024-09-13T08:46:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 --2- 172.21.15.31:0/3102037616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b100cf4d0 0x7f9b100cd780 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 -- 172.21.15.31:0/3102037616 >> 172.21.15.31:0/3102037616 conn(0x7f9b100cfbb0 msgr2=0x7f9b100cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 -- 172.21.15.31:0/3102037616 shutdown_connections 2024-09-13T08:46:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.980+0000 7f9b17400640 1 -- 172.21.15.31:0/3102037616 wait complete. 2024-09-13T08:46:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.982+0000 7f9b17400640 1 Processor -- start 2024-09-13T08:46:02.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.982+0000 7f9b17400640 1 -- start start 2024-09-13T08:46:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b100cdcc0 0x7f9b101f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b100cf4d0 0x7f9b1014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1015f0b0 con 0x7f9b1015acb0 2024-09-13T08:46:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1015ef30 con 0x7f9b100cf4d0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b17400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1015f230 con 0x7f9b100cdcc0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51878/0 (socket says 172.21.15.31:51878) 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 -- 172.21.15.31:0/497204525 learned_addr learned my addr 172.21.15.31:0/497204525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b07e00640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b100cf4d0 0x7f9b1014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b14a00640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b100cdcc0 0x7f9b101f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 -- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b100cdcc0 msgr2=0x7f9b101f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b100cdcc0 0x7f9b101f8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 -- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b100cf4d0 msgr2=0x7f9b1014bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b100cf4d0 0x7f9b1014bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.983+0000 7f9b15400640 1 -- 172.21.15.31:0/497204525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b100b2ba0 con 0x7f9b1015acb0 2024-09-13T08:46:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b15400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c7b0 tx=0x7f9b0800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08017070 con 0x7f9b1015acb0 2024-09-13T08:46:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b08005070 con 0x7f9b1015acb0 2024-09-13T08:46:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b100b2e80 con 0x7f9b1015acb0 2024-09-13T08:46:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b080054c0 con 0x7f9b1015acb0 2024-09-13T08:46:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.984+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b100b3440 con 0x7f9b1015acb0 2024-09-13T08:46:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.985+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1015f700 con 0x7f9b1015acb0 2024-09-13T08:46:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.986+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f9b08005660 con 0x7f9b1015acb0 2024-09-13T08:46:02.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.987+0000 7f9b0d600640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 0x7f9aec07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:02.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.987+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 4901+0+0 (secure 0 0 0) 0x7f9b0809aeb0 con 0x7f9b1015acb0 2024-09-13T08:46:02.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.987+0000 7f9b14a00640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 0x7f9aec07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:02.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.988+0000 7f9b14a00640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 0x7f9aec07b580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9b000015e0 tx=0x7f9b00009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:02.991+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9b08062790 con 0x7f9b1015acb0 2024-09-13T08:46:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.189+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9b10050ee0 con 0x7f9b1015acb0 2024-09-13T08:46:03.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.931+0000 7f9b0d600640 1 -- 172.21.15.31:0/497204525 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v161) ==== 138+0+0 (secure 0 0 0) 0x7f9b08062090 con 0x7f9b1015acb0 2024-09-13T08:46:03.933 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:46:03.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 msgr2=0x7f9aec07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:03.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 0x7f9aec07b580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9b000015e0 tx=0x7f9b00009040 comp rx=0 tx=0).stop 2024-09-13T08:46:03.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 msgr2=0x7f9b1014c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:03.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c7b0 tx=0x7f9b0800cc70 comp rx=0 tx=0).stop 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 shutdown_connections 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9aec0790c0 0x7f9aec07b580 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b1015acb0 0x7f9b1014c270 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b100cf4d0 0x7f9b1014bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 --2- 172.21.15.31:0/497204525 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9b100cdcc0 0x7f9b101f8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 >> 172.21.15.31:0/497204525 conn(0x7f9b100cfbb0 msgr2=0x7f9b10070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 shutdown_connections 2024-09-13T08:46:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:03.935+0000 7f9b17400640 1 -- 172.21.15.31:0/497204525 wait complete. 2024-09-13T08:46:03.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:46:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 -- 172.21.15.31:0/637443377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd000cbe50 msgr2=0x7fdd000b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 --2- 172.21.15.31:0/637443377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd000cbe50 0x7fdd000b29e0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8005fd0 tx=0x7fdcf802a940 comp rx=0 tx=0).stop 2024-09-13T08:46:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 -- 172.21.15.31:0/637443377 shutdown_connections 2024-09-13T08:46:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 --2- 172.21.15.31:0/637443377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd000cbe50 0x7fdd000b29e0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 -- 172.21.15.31:0/637443377 >> 172.21.15.31:0/637443377 conn(0x7fdd000cfbb0 msgr2=0x7fdd000cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 -- 172.21.15.31:0/637443377 shutdown_connections 2024-09-13T08:46:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.132+0000 7fdd06e00640 1 -- 172.21.15.31:0/637443377 wait complete. 2024-09-13T08:46:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.133+0000 7fdd06e00640 1 Processor -- start 2024-09-13T08:46:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.134+0000 7fdd06e00640 1 -- start start 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.134+0000 7fdd06e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd00072560 0x7fdd001f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd06e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd000cbe50 0x7fdd001f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd06e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd06e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000b2f20 con 0x7fdd0015bfc0 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd06e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000d0a00 con 0x7fdd000cbe50 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd06e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fdd000b30a0 con 0x7fdd00072560 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51892/0 (socket says 172.21.15.31:51892) 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 -- 172.21.15.31:0/1558091944 learned_addr learned my addr 172.21.15.31:0/1558091944 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdcffe00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd00072560 0x7fdd001f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdcff400640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd000cbe50 0x7fdd001f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 -- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd00072560 msgr2=0x7fdd001f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd00072560 0x7fdd001f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 -- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd000cbe50 msgr2=0x7fdd001f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd000cbe50 0x7fdd001f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.135+0000 7fdd04e00640 1 -- 172.21.15.31:0/1558091944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd001f6d50 con 0x7fdd0015bfc0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdd04e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006010 tx=0x7fdcf8031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8004240 con 0x7fdd0015bfc0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcf80043e0 con 0x7fdd0015bfc0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8004830 con 0x7fdd0015bfc0 2024-09-13T08:46:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd00153630 con 0x7fdd0015bfc0 2024-09-13T08:46:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.136+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd000c3930 con 0x7fdd0015bfc0 2024-09-13T08:46:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.137+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd000c4100 con 0x7fdd0015bfc0 2024-09-13T08:46:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.139+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fdcf8039440 con 0x7fdd0015bfc0 2024-09-13T08:46:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.140+0000 7fdcf7e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 0x7fdce007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:04.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.140+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 5271+0+0 (secure 0 0 0) 0x7fdcf80badd0 con 0x7fdd0015bfc0 2024-09-13T08:46:04.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.140+0000 7fdcffe00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 0x7fdce007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:04.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.140+0000 7fdcffe00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 0x7fdce007b580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00046b0 tx=0x7fdcf0007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:04.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.143+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fdd00153630 con 0x7fdd0015bfc0 2024-09-13T08:46:04.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.345+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdd000c3740 con 0x7fdd0015bfc0 2024-09-13T08:46:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.940+0000 7fdcf7e00640 1 -- 172.21.15.31:0/1558091944 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v162) ==== 170+0+0 (secure 0 0 0) 0x7fdd001f6d50 con 0x7fdd0015bfc0 2024-09-13T08:46:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:46:04.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.945+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 msgr2=0x7fdce007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:04.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.945+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 0x7fdce007b580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00046b0 tx=0x7fdcf0007e90 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.945+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 msgr2=0x7fdd001530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.945+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006010 tx=0x7fdcf8031e20 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 shutdown_connections 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fdce00790c0 0x7fdce007b580 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd0015bfc0 0x7fdd001530f0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd000cbe50 0x7fdd001f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 --2- 172.21.15.31:0/1558091944 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fdd00072560 0x7fdd001f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.946+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 >> 172.21.15.31:0/1558091944 conn(0x7fdd000cfbb0 msgr2=0x7fdd0015c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.947+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 shutdown_connections 2024-09-13T08:46:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:04.949+0000 7fdd06e00640 1 -- 172.21.15.31:0/1558091944 wait complete. 2024-09-13T08:46:04.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.148+0000 7f2494600640 1 -- 172.21.15.31:0/4013217857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 msgr2=0x7f24740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.148+0000 7f2494600640 1 --2- 172.21.15.31:0/4013217857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f24740051c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f2484001660 tx=0x7f2474005830 comp rx=0 tx=0).stop 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.148+0000 7f2492600640 1 -- 172.21.15.31:0/4013217857 reap_dead start 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.149+0000 7f2494600640 1 -- 172.21.15.31:0/4013217857 shutdown_connections 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.149+0000 7f2494600640 1 -- 172.21.15.31:0/4013217857 >> 172.21.15.31:0/4013217857 conn(0x7f248c0cfbb0 msgr2=0x7f248c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.149+0000 7f2494600640 1 -- 172.21.15.31:0/4013217857 shutdown_connections 2024-09-13T08:46:05.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.149+0000 7f2494600640 1 -- 172.21.15.31:0/4013217857 wait complete. 2024-09-13T08:46:05.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.150+0000 7f2494600640 1 Processor -- start 2024-09-13T08:46:05.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.151+0000 7f2494600640 1 -- start start 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.151+0000 7f2494600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.151+0000 7f2494600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f248c152c80 0x7f248c152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2494600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f248c1531b0 0x7f248c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2494600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f248c06e740 con 0x7f2474004d40 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2494600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f248c06e5c0 con 0x7f248c152c80 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2494600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f248c0cb740 con 0x7f248c1531b0 2024-09-13T08:46:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36964/0 (socket says 172.21.15.31:36964) 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 -- 172.21.15.31:0/1262953458 learned_addr learned my addr 172.21.15.31:0/1262953458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491200640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f248c152c80 0x7f248c152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2492600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f248c1531b0 0x7f248c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 -- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f248c1531b0 msgr2=0x7f248c152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f248c1531b0 0x7f248c152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 -- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f248c152c80 msgr2=0x7f248c152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f248c152c80 0x7f248c152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 -- 172.21.15.31:0/1262953458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c1e4440 con 0x7f2474004d40 2024-09-13T08:46:05.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.152+0000 7f2491c00640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f248000e990 tx=0x7f248000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.153+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480015070 con 0x7f2474004d40 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.153+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f248000cde0 con 0x7f2474004d40 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.153+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480003bd0 con 0x7f2474004d40 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.153+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248c1e4720 con 0x7f2474004d40 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.153+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248c1e4ce0 con 0x7f2474004d40 2024-09-13T08:46:05.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.154+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f248c06fa60 con 0x7f2474004d40 2024-09-13T08:46:05.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.156+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f2480023480 con 0x7f2474004d40 2024-09-13T08:46:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.157+0000 7f2489600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 0x7f246c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.157+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5653+0+0 (secure 0 0 0) 0x7f248009c3e0 con 0x7f2474004d40 2024-09-13T08:46:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.158+0000 7f2491200640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 0x7f246c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:05.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.158+0000 7f2491200640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 0x7f246c07b580 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f248c0ca040 tx=0x7f2478005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:05.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.161+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2480063ee0 con 0x7f2474004d40 2024-09-13T08:46:05.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.394+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f248c0cb310 con 0x7f2474004d40 2024-09-13T08:46:05.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.942+0000 7f2489600640 1 -- 172.21.15.31:0/1262953458 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v105) ==== 174+0+0 (secure 0 0 0) 0x7f24800637e0 con 0x7f2474004d40 2024-09-13T08:46:05.944 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 24 and data pool 25 2024-09-13T08:46:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.946+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 msgr2=0x7f246c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.946+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 0x7f246c07b580 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f248c0ca040 tx=0x7f2478005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.946+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 msgr2=0x7f248c0c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.946+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f248000e990 tx=0x7f248000ee50 comp rx=0 tx=0).stop 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 shutdown_connections 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f246c0790c0 0x7f246c07b580 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f248c1531b0 0x7f248c152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f248c152c80 0x7f248c152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 --2- 172.21.15.31:0/1262953458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474004d40 0x7f248c0c9220 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.947+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 >> 172.21.15.31:0/1262953458 conn(0x7f248c0cfbb0 msgr2=0x7f248c15a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.948+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 shutdown_connections 2024-09-13T08:46:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:05.948+0000 7f2494600640 1 -- 172.21.15.31:0/1262953458 wait complete. 2024-09-13T08:46:05.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 -- 172.21.15.31:0/3031013821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015bd70 msgr2=0x7f47d00cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 --2- 172.21.15.31:0/3031013821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015bd70 0x7f47d00cf850 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009830 tx=0x7f47c002a460 comp rx=0 tx=0).stop 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 -- 172.21.15.31:0/3031013821 shutdown_connections 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 --2- 172.21.15.31:0/3031013821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015bd70 0x7f47d00cf850 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 -- 172.21.15.31:0/3031013821 >> 172.21.15.31:0/3031013821 conn(0x7f47d00cc910 msgr2=0x7f47d00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 -- 172.21.15.31:0/3031013821 shutdown_connections 2024-09-13T08:46:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.148+0000 7f47d6400640 1 -- 172.21.15.31:0/3031013821 wait complete. 2024-09-13T08:46:06.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.149+0000 7f47d6400640 1 Processor -- start 2024-09-13T08:46:06.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 -- start start 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d015bd70 0x7f47d009aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015fe20 0x7f47d01f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47d00cec50 con 0x7f47d015f9e0 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47d00cead0 con 0x7f47d015fe20 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.150+0000 7f47d6400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f47d00cb850 con 0x7f47d015bd70 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36970/0 (socket says 172.21.15.31:36970) 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 -- 172.21.15.31:0/521629201 learned_addr learned my addr 172.21.15.31:0/521629201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47cfe00640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015fe20 0x7f47d01f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47cf400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d015bd70 0x7f47d009aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 -- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d015bd70 msgr2=0x7f47d009aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d015bd70 0x7f47d009aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 -- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015fe20 msgr2=0x7f47d01f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015fe20 0x7f47d01f7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.151+0000 7f47c6a00640 1 -- 172.21.15.31:0/521629201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d01f88d0 con 0x7f47d015f9e0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47c6a00640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f47b8001220 tx=0x7f47b800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8017070 con 0x7f47d015f9e0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47b8005380 con 0x7f47d015f9e0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b800a8a0 con 0x7f47d015f9e0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d01f8bb0 con 0x7f47d015f9e0 2024-09-13T08:46:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.152+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d0159f90 con 0x7f47d015f9e0 2024-09-13T08:46:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.153+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d015b400 con 0x7f47d015f9e0 2024-09-13T08:46:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.154+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f47b801a070 con 0x7f47d015f9e0 2024-09-13T08:46:06.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.155+0000 7f47c7e00640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 0x7f47ac07b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.155+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5753+0+0 (secure 0 0 0) 0x7f47b809b080 con 0x7f47d015f9e0 2024-09-13T08:46:06.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.156+0000 7f47cf400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 0x7f47ac07b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.156+0000 7f47cf400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 0x7f47ac07b5b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034000 tx=0x7f47c002c040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.159+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f47d015b400 con 0x7f47d015f9e0 2024-09-13T08:46:06.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.355+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47d015f1c0 con 0x7f47d015f9e0 2024-09-13T08:46:06.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.356+0000 7f47c7e00640 1 -- 172.21.15.31:0/521629201 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+16715 (secure 0 0 0) 0x7f47d015f1c0 con 0x7f47d015f9e0 2024-09-13T08:46:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:06.359 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":163,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:05.396671+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:03.200753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:04.357008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.358+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 msgr2=0x7f47ac07b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.358+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 0x7f47ac07b5b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034000 tx=0x7f47c002c040 comp rx=0 tx=0).stop 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.358+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 msgr2=0x7f47d0159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.358+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f47b8001220 tx=0x7f47b800cb90 comp rx=0 tx=0).stop 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.358+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 shutdown_connections 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f47ac0790f0 0x7f47ac07b5b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015fe20 0x7f47d01f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d015f9e0 0x7f47d0159a50 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 --2- 172.21.15.31:0/521629201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f47d015bd70 0x7f47d009aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 >> 172.21.15.31:0/521629201 conn(0x7f47d00cc910 msgr2=0x7f47d00ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 shutdown_connections 2024-09-13T08:46:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.359+0000 7f47d6400640 1 -- 172.21.15.31:0/521629201 wait complete. 2024-09-13T08:46:06.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:06.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 -- 172.21.15.31:0/2638274179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815ad60 msgr2=0x7f39a815f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 --2- 172.21.15.31:0/2638274179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815ad60 0x7f39a815f3d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f39a815c470 tx=0x7f399802a510 comp rx=0 tx=0).stop 2024-09-13T08:46:06.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 -- 172.21.15.31:0/2638274179 shutdown_connections 2024-09-13T08:46:06.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 --2- 172.21.15.31:0/2638274179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815ad60 0x7f39a815f3d0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 -- 172.21.15.31:0/2638274179 >> 172.21.15.31:0/2638274179 conn(0x7f39a80cc910 msgr2=0x7f39a80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:06.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 -- 172.21.15.31:0/2638274179 shutdown_connections 2024-09-13T08:46:06.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.536+0000 7f39af200640 1 -- 172.21.15.31:0/2638274179 wait complete. 2024-09-13T08:46:06.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.538+0000 7f39af200640 1 Processor -- start 2024-09-13T08:46:06.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.538+0000 7f39af200640 1 -- start start 2024-09-13T08:46:06.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 0x7f39a80c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 0x7f39a81f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39a80cb4f0 con 0x7f39a815b5b0 2024-09-13T08:46:06.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39a806ff10 con 0x7f39a815ad60 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39af200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39a80cb670 con 0x7f39a815b140 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39ad200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39ad200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36992/0 (socket says 172.21.15.31:36992) 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39ad200640 1 -- 172.21.15.31:0/3330577759 learned_addr learned my addr 172.21.15.31:0/3330577759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39a7e00640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 0x7f39a80c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f39ad200640 1 -- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 msgr2=0x7f39a81f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.539+0000 7f399f400640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 0x7f39a81f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39ad200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 0x7f39a81f8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39ad200640 1 -- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 msgr2=0x7f39a80c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39ad200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 0x7f39a80c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39ad200640 1 -- 172.21.15.31:0/3330577759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a80c5670 con 0x7f39a815b5b0 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39a7e00640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 0x7f39a80c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:06.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f399f400640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 0x7f39a81f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39ad200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c700 tx=0x7f39a000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0003b70 con 0x7f39a815b5b0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39a0003d10 con 0x7f39a815b5b0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a80c5950 con 0x7f39a815b5b0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a000aab0 con 0x7f39a815b5b0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.540+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a80c5f10 con 0x7f39a815b5b0 2024-09-13T08:46:06.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.541+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a815f3d0 con 0x7f39a815b5b0 2024-09-13T08:46:06.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.542+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f39a000ac50 con 0x7f39a815b5b0 2024-09-13T08:46:06.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.543+0000 7f39a4c00640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 0x7f398407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.544+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5753+0+0 (secure 0 0 0) 0x7f39a009b9f0 con 0x7f39a815b5b0 2024-09-13T08:46:06.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.544+0000 7f39a7e00640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 0x7f398407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.544+0000 7f39a7e00640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 0x7f398407b580 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3998034000 tx=0x7f3998004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.548+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f39a0063090 con 0x7f39a815b5b0 2024-09-13T08:46:06.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.743+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39a80c60e0 con 0x7f39a815b5b0 2024-09-13T08:46:06.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.744+0000 7f39a4c00640 1 -- 172.21.15.31:0/3330577759 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+16715 (secure 0 0 0) 0x7f39a0062990 con 0x7f39a815b5b0 2024-09-13T08:46:06.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:06.746 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":163,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:05.396671+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:03.200753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:04.357008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.746+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 msgr2=0x7f398407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.746+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 0x7f398407b580 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3998034000 tx=0x7f3998004730 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.746+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 msgr2=0x7f39a81f8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.746+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c700 tx=0x7f39a000cbc0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 shutdown_connections 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f39840790c0 0x7f398407b580 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a815b5b0 0x7f39a81f8d60 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f39a815b140 0x7f39a81f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 --2- 172.21.15.31:0/3330577759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a815ad60 0x7f39a80c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 >> 172.21.15.31:0/3330577759 conn(0x7f39a80cc910 msgr2=0x7f39a80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:06.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 shutdown_connections 2024-09-13T08:46:06.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.747+0000 7f39af200640 1 -- 172.21.15.31:0/3330577759 wait complete. 2024-09-13T08:46:06.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:46:06.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 -- 172.21.15.31:0/862564997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 msgr2=0x7ff0280caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 --2- 172.21.15.31:0/862564997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff0280caa80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff020005fd0 tx=0x7ff02002a840 comp rx=0 tx=0).stop 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 -- 172.21.15.31:0/862564997 shutdown_connections 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 --2- 172.21.15.31:0/862564997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff0280caa80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 -- 172.21.15.31:0/862564997 >> 172.21.15.31:0/862564997 conn(0x7ff0280cfbb0 msgr2=0x7ff0280cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 -- 172.21.15.31:0/862564997 shutdown_connections 2024-09-13T08:46:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.929+0000 7ff02d800640 1 -- 172.21.15.31:0/862564997 wait complete. 2024-09-13T08:46:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.931+0000 7ff02d800640 1 Processor -- start 2024-09-13T08:46:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.931+0000 7ff02d800640 1 -- start start 2024-09-13T08:46:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff02814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0280cde90 0x7ff0280b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0280cbfd0 con 0x7ff028161480 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0280cbe50 con 0x7ff0280cc190 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff02d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff0280cafc0 con 0x7ff0280cde90 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff027400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff027400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36998/0 (socket says 172.21.15.31:36998) 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff027400640 1 -- 172.21.15.31:0/3638516460 learned_addr learned my addr 172.21.15.31:0/3638516460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff026a00640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff02814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.932+0000 7ff01e000640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0280cde90 0x7ff0280b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 -- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0280cde90 msgr2=0x7ff0280b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0280cde90 0x7ff0280b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 -- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 msgr2=0x7ff02814fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff02814fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 -- 172.21.15.31:0/3638516460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0280af4a0 con 0x7ff028161480 2024-09-13T08:46:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff027400640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7ff020034040 tx=0x7ff020031610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020042030 con 0x7ff028161480 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff020031a10 con 0x7ff028161480 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020031e60 con 0x7ff028161480 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0281f8c70 con 0x7ff028161480 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.933+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0280afa40 con 0x7ff028161480 2024-09-13T08:46:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.934+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0280cb490 con 0x7ff028161480 2024-09-13T08:46:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.936+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7ff020045070 con 0x7ff028161480 2024-09-13T08:46:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.937+0000 7ff01f400640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 0x7ff00407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.937+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5753+0+0 (secure 0 0 0) 0x7ff0200bc880 con 0x7ff028161480 2024-09-13T08:46:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.937+0000 7ff026a00640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 0x7ff00407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.937+0000 7ff026a00640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 0x7ff00407b580 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff018001220 tx=0x7ff018001c50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:06.941+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff02004c070 con 0x7ff028161480 2024-09-13T08:46:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.178+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff0280b5b30 con 0x7ff028161480 2024-09-13T08:46:07.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:46:07.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:46:07.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.966+0000 7ff01f400640 1 -- 172.21.15.31:0/3638516460 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v108) ==== 118+0+0 (secure 0 0 0) 0x7ff020083d10 con 0x7ff028161480 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.968+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 msgr2=0x7ff00407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.968+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 0x7ff00407b580 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff018001220 tx=0x7ff018001c50 comp rx=0 tx=0).stop 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.968+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 msgr2=0x7ff0280c36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.968+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7ff020034040 tx=0x7ff020031610 comp rx=0 tx=0).stop 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 shutdown_connections 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff0040790c0 0x7ff00407b580 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff028161480 0x7ff0280c36a0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0280cde90 0x7ff0280b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 --2- 172.21.15.31:0/3638516460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0280cc190 0x7ff02814fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 >> 172.21.15.31:0/3638516460 conn(0x7ff0280cfbb0 msgr2=0x7ff0281605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 shutdown_connections 2024-09-13T08:46:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:07.969+0000 7ff02d800640 1 -- 172.21.15.31:0/3638516460 wait complete. 2024-09-13T08:46:07.987 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:46:07.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:46:08.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 -- 172.21.15.31:0/2210067184 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc161480 msgr2=0x7f55bc0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 --2- 172.21.15.31:0/2210067184 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc161480 0x7f55bc0cd780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009830 tx=0x7f55ac02a410 comp rx=0 tx=0).stop 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 -- 172.21.15.31:0/2210067184 shutdown_connections 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 --2- 172.21.15.31:0/2210067184 >> v2:172.21.15.45:3300/0 conn(0x7f55bc0cc190 0x7f55bc0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 --2- 172.21.15.31:0/2210067184 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc161480 0x7f55bc0cd780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 -- 172.21.15.31:0/2210067184 >> 172.21.15.31:0/2210067184 conn(0x7f55bc0cfbb0 msgr2=0x7f55bc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 -- 172.21.15.31:0/2210067184 shutdown_connections 2024-09-13T08:46:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.151+0000 7f55c3400640 1 -- 172.21.15.31:0/2210067184 wait complete. 2024-09-13T08:46:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.153+0000 7f55c3400640 1 Processor -- start 2024-09-13T08:46:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.153+0000 7f55c3400640 1 -- start start 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cbc40 0x7f55bc0c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc161480 0x7f55bc0b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc070020 con 0x7f55bc0cc190 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc0cb590 con 0x7f55bc161480 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c3400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc1622c0 con 0x7f55bc0cbc40 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37006/0 (socket says 172.21.15.31:37006) 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 -- 172.21.15.31:0/274403613 learned_addr learned my addr 172.21.15.31:0/274403613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c1400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc161480 0x7f55bc0b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55c0a00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cbc40 0x7f55bc0c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 -- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cbc40 msgr2=0x7f55bc0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cbc40 0x7f55bc0c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 -- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc161480 msgr2=0x7f55bc0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc161480 0x7f55bc0b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.154+0000 7f55b3e00640 1 -- 172.21.15.31:0/274403613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1f5ad0 con 0x7f55bc0cc190 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55b3e00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e9c0 tx=0x7f55a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a400cde0 con 0x7f55bc0cc190 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55a4010520 con 0x7f55bc0cc190 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4010970 con 0x7f55bc0cc190 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc0c39d0 con 0x7f55bc0cc190 2024-09-13T08:46:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.155+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc0c3f90 con 0x7f55bc0cc190 2024-09-13T08:46:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.156+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc0d06a0 con 0x7f55bc0cc190 2024-09-13T08:46:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.157+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f55a4010b10 con 0x7f55bc0cc190 2024-09-13T08:46:08.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.158+0000 7f55b9600640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 0x7f559807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:08.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.158+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f55a409c6c0 con 0x7f55bc0cc190 2024-09-13T08:46:08.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.159+0000 7f55c0a00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 0x7f559807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:08.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.159+0000 7f55c0a00640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 0x7f559807b5b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55ac034000 tx=0x7f55ac02c040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:08.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.163+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f55bc0d06a0 con 0x7f55bc0cc190 2024-09-13T08:46:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.398+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f55bc162770 con 0x7f55bc0cc190 2024-09-13T08:46:08.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.969+0000 7f55b9600640 1 -- 172.21.15.31:0/274403613 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v109) ==== 122+0+0 (secure 0 0 0) 0x7f55bc1f5ad0 con 0x7f55bc0cc190 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 msgr2=0x7f559807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 0x7f559807b5b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55ac034000 tx=0x7f55ac02c040 comp rx=0 tx=0).stop 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 msgr2=0x7f55bc1f5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e9c0 tx=0x7f55a400ee80 comp rx=0 tx=0).stop 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 shutdown_connections 2024-09-13T08:46:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f55980790f0 0x7f559807b5b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55bc161480 0x7f55bc0b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55bc0cc190 0x7f55bc1f5590 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 --2- 172.21.15.31:0/274403613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f55bc0cbc40 0x7f55bc0c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 >> 172.21.15.31:0/274403613 conn(0x7f55bc0cfbb0 msgr2=0x7f55bc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 shutdown_connections 2024-09-13T08:46:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:08.972+0000 7f55c3400640 1 -- 172.21.15.31:0/274403613 wait complete. 2024-09-13T08:46:08.993 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:46:08.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 -- 172.21.15.31:0/958957482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 msgr2=0x7fb11000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 --2- 172.21.15.31:0/958957482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb11000b9c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0cece0 tx=0x7fb110005830 comp rx=0 tx=0).stop 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb130a00640 1 -- 172.21.15.31:0/958957482 reap_dead start 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 -- 172.21.15.31:0/958957482 shutdown_connections 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 -- 172.21.15.31:0/958957482 >> 172.21.15.31:0/958957482 conn(0x7fb12c0cc910 msgr2=0x7fb12c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 -- 172.21.15.31:0/958957482 shutdown_connections 2024-09-13T08:46:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.154+0000 7fb132a00640 1 -- 172.21.15.31:0/958957482 wait complete. 2024-09-13T08:46:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.156+0000 7fb132a00640 1 Processor -- start 2024-09-13T08:46:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.156+0000 7fb132a00640 1 -- start start 2024-09-13T08:46:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb12c15b1c0 0x7fb12c154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12c0c39a0 0x7fb12c0cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c1600f0 con 0x7fb1100051c0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c0cb850 con 0x7fb12c0c39a0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb132a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fb12c0c8260 con 0x7fb12c15b1c0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb12be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb12be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37028/0 (socket says 172.21.15.31:37028) 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb12be00640 1 -- 172.21.15.31:0/3536341077 learned_addr learned my addr 172.21.15.31:0/3536341077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb123e00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb12c15b1c0 0x7fb12c154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.157+0000 7fb130a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12c0c39a0 0x7fb12c0cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 -- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb12c15b1c0 msgr2=0x7fb12c154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb12c15b1c0 0x7fb12c154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 -- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12c0c39a0 msgr2=0x7fb12c0cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12c0c39a0 0x7fb12c0cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 -- 172.21.15.31:0/3536341077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c0c8540 con 0x7fb1100051c0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb12be00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fb11c006610 tx=0x7fb11c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c015070 con 0x7fb1100051c0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb11c011900 con 0x7fb1100051c0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c0c8820 con 0x7fb1100051c0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.158+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c011d30 con 0x7fb1100051c0 2024-09-13T08:46:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.159+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c0c8de0 con 0x7fb1100051c0 2024-09-13T08:46:09.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.159+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c0c8fe0 con 0x7fb1100051c0 2024-09-13T08:46:09.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.161+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fb11c018070 con 0x7fb1100051c0 2024-09-13T08:46:09.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.162+0000 7fb128c00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 0x7fb10807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.162+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fb11c09c970 con 0x7fb1100051c0 2024-09-13T08:46:09.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.162+0000 7fb123e00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 0x7fb10807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.163+0000 7fb123e00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 0x7fb10807b5b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0b0e50 tx=0x7fb114005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.166+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fb12c0c8fe0 con 0x7fb1100051c0 2024-09-13T08:46:09.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.401+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb12c0ce3c0 con 0x7fb1100051c0 2024-09-13T08:46:09.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.401+0000 7fb128c00640 1 -- 172.21.15.31:0/3536341077 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5224 (secure 0 0 0) 0x7fb11c063940 con 0x7fb1100051c0 2024-09-13T08:46:09.404 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:09.404 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":109,"btime":"2024-09-13T08:46:08:967593+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:09.404 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 109 2024-09-13T08:46:09.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 msgr2=0x7fb10807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 0x7fb10807b5b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0b0e50 tx=0x7fb114005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:09.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 msgr2=0x7fb12c09d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fb11c006610 tx=0x7fb11c0039c0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 shutdown_connections 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fb1080790f0 0x7fb10807b5b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb12c0c39a0 0x7fb12c0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fb12c15b1c0 0x7fb12c154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 --2- 172.21.15.31:0/3536341077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1100051c0 0x7fb12c09d530 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 >> 172.21.15.31:0/3536341077 conn(0x7fb12c0cc910 msgr2=0x7fb12c0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 shutdown_connections 2024-09-13T08:46:09.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.404+0000 7fb132a00640 1 -- 172.21.15.31:0/3536341077 wait complete. 2024-09-13T08:46:09.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 -- 172.21.15.31:0/3606009017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a100051c0 msgr2=0x7f0a1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3606009017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a100051c0 0x7f0a1000b9c0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400c610 tx=0x7f0a10005830 comp rx=0 tx=0).stop 2024-09-13T08:46:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 -- 172.21.15.31:0/3606009017 shutdown_connections 2024-09-13T08:46:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3606009017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a100051c0 0x7f0a1000b9c0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 -- 172.21.15.31:0/3606009017 >> 172.21.15.31:0/3606009017 conn(0x7f0a2c0cc910 msgr2=0x7f0a2c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:09.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 -- 172.21.15.31:0/3606009017 shutdown_connections 2024-09-13T08:46:09.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.582+0000 7f0a32e00640 1 -- 172.21.15.31:0/3606009017 wait complete. 2024-09-13T08:46:09.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.583+0000 7f0a32e00640 1 Processor -- start 2024-09-13T08:46:09.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.584+0000 7f0a32e00640 1 -- start start 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.584+0000 7f0a32e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a100051c0 0x7f0a2c1f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.584+0000 7f0a32e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a32e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a2c1f3e30 0x7f0a2c1f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a32e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c0cbf60 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a32e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c153340 con 0x7f0a2c1f3e30 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a32e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2c0cc0e0 con 0x7f0a100051c0 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37052/0 (socket says 172.21.15.31:37052) 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 -- 172.21.15.31:0/3853445130 learned_addr learned my addr 172.21.15.31:0/3853445130 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2be00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a100051c0 0x7f0a2c1f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a30e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a2c1f3e30 0x7f0a2c1f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 -- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a100051c0 msgr2=0x7f0a2c1f50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a100051c0 0x7f0a2c1f50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 -- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a2c1f3e30 msgr2=0x7f0a2c1f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a2c1f3e30 0x7f0a2c1f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.585+0000 7f0a2b400640 1 -- 172.21.15.31:0/3853445130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c1f5630 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a2b400640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0a180117d0 tx=0x7f0a18011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18004460 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a18016070 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800fb60 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2c0b4930 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.586+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2c0b4ef0 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.587+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2c0743f0 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.588+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f0a18018030 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.589+0000 7f0a23e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 0x7f0a0c07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.589+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f0a1809bf40 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.590+0000 7f0a2be00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 0x7f0a0c07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.590+0000 7f0a2be00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 0x7f0a0c07b580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c006630 tx=0x7f0a1c006310 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.594+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0a2c1f5630 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.790+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a2c0cd030 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.791+0000 7f0a23e00640 1 -- 172.21.15.31:0/3853445130 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+16702 (secure 0 0 0) 0x7f0a180633a0 con 0x7f0a2c0b2cc0 2024-09-13T08:46:09.794 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:09.794 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":164,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:06.943667+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:03.200753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:04.357008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 msgr2=0x7f0a0c07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 0x7f0a0c07b580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c006630 tx=0x7f0a1c006310 comp rx=0 tx=0).stop 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 msgr2=0x7f0a2c0b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0a180117d0 tx=0x7f0a18011c90 comp rx=0 tx=0).stop 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 shutdown_connections 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a0c0790c0 0x7f0a0c07b580 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a2c1f3e30 0x7f0a2c1f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a2c0b2cc0 0x7f0a2c0b56f0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 --2- 172.21.15.31:0/3853445130 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a100051c0 0x7f0a2c1f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 >> 172.21.15.31:0/3853445130 conn(0x7f0a2c0cc910 msgr2=0x7f0a2c159150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 shutdown_connections 2024-09-13T08:46:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.794+0000 7f0a32e00640 1 -- 172.21.15.31:0/3853445130 wait complete. 2024-09-13T08:46:09.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:46:09.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:46:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 -- 172.21.15.31:0/323445655 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415bd70 msgr2=0x7fa7f40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/323445655 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415bd70 0x7fa7f40cf850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009860 tx=0x7fa7e402a5a0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 -- 172.21.15.31:0/323445655 shutdown_connections 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/323445655 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415bd70 0x7fa7f40cf850 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 -- 172.21.15.31:0/323445655 >> 172.21.15.31:0/323445655 conn(0x7fa7f40cc910 msgr2=0x7fa7f40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 -- 172.21.15.31:0/323445655 shutdown_connections 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.971+0000 7fa7f8c00640 1 -- 172.21.15.31:0/323445655 wait complete. 2024-09-13T08:46:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.973+0000 7fa7f8c00640 1 Processor -- start 2024-09-13T08:46:09.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.973+0000 7fa7f8c00640 1 -- start start 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7f415bd70 0x7fa7f415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415f9e0 0x7fa7f40b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f40cec50 con 0x7fa7f415fe20 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f40cead0 con 0x7fa7f415bd70 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f8c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa7f40cb850 con 0x7fa7f415f9e0 2024-09-13T08:46:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37060/0 (socket says 172.21.15.31:37060) 2024-09-13T08:46:09.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f2800640 1 -- 172.21.15.31:0/3919644341 learned_addr learned my addr 172.21.15.31:0/3919644341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f1e00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7f415bd70 0x7fa7f415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.974+0000 7fa7f1400640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415f9e0 0x7fa7f40b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 -- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415f9e0 msgr2=0x7fa7f40b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415f9e0 0x7fa7f40b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 -- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7f415bd70 msgr2=0x7fa7f415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7f415bd70 0x7fa7f415acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 -- 172.21.15.31:0/3919644341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f41f8c70 con 0x7fa7f415fe20 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f2800640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00cb40 tx=0x7fa7ec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec016020 con 0x7fa7f415fe20 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7ec00adb0 con 0x7fa7f415fe20 2024-09-13T08:46:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec012680 con 0x7fa7f415fe20 2024-09-13T08:46:09.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.975+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f40c4cd0 con 0x7fa7f415fe20 2024-09-13T08:46:09.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.976+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f40c5290 con 0x7fa7f415fe20 2024-09-13T08:46:09.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.977+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f40cc6e0 con 0x7fa7f415fe20 2024-09-13T08:46:09.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.978+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fa7ec019020 con 0x7fa7f415fe20 2024-09-13T08:46:09.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.979+0000 7fa7ea000640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 0x7fa7d407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:09.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.979+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa7ec09c1a0 con 0x7fa7f415fe20 2024-09-13T08:46:09.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.979+0000 7fa7f1e00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 0x7fa7d407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:09.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.980+0000 7fa7f1e00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 0x7fa7d407b580 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4034000 tx=0x7fa7e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:09.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:09.984+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa7f40cc6e0 con 0x7fa7f415fe20 2024-09-13T08:46:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.175+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7f415f5b0 con 0x7fa7d40790c0 2024-09-13T08:46:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.179+0000 7fa7ea000640 1 -- 172.21.15.31:0/3919644341 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7fa7f415f5b0 con 0x7fa7d40790c0 2024-09-13T08:46:10.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:10.186 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 msgr2=0x7fa7d407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 0x7fa7d407b580 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4034000 tx=0x7fa7e4004730 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 msgr2=0x7fa7f406fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00cb40 tx=0x7fa7ec00a060 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 shutdown_connections 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa7d40790c0 0x7fa7d407b580 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7f415fe20 0x7fa7f406fab0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7f415f9e0 0x7fa7f40b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 --2- 172.21.15.31:0/3919644341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7f415bd70 0x7fa7f415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.183+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 >> 172.21.15.31:0/3919644341 conn(0x7fa7f40cc910 msgr2=0x7fa7f40ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.184+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 shutdown_connections 2024-09-13T08:46:10.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.184+0000 7fa7f8c00640 1 -- 172.21.15.31:0/3919644341 wait complete. 2024-09-13T08:46:10.201 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-13T08:46:09.011685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287528,"kb_used_data":17192,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687192,"statfs":{"total":773094113280,"available":772799684608,"internally_reserved":0,"allocated":17604608,"data_stored":8432660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.077756"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977918+0000","last_change":"2024-09-13T08:46:05.975752+0000","last_active":"2024-09-13T08:46:05.977918+0000","last_peered":"2024-09-13T08:46:05.977918+0000","last_clean":"2024-09-13T08:46:05.977918+0000","last_became_active":"2024-09-13T08:46:05.975493+0000","last_became_peered":"2024-09-13T08:46:05.975493+0000","last_unstale":"2024-09-13T08:46:05.977918+0000","last_undegraded":"2024-09-13T08:46:05.977918+0000","last_fullsized":"2024-09-13T08:46:05.977918+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:06.844604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948447+0000","last_change":"2024-09-13T08:46:05.979888+0000","last_active":"2024-09-13T08:46:06.948447+0000","last_peered":"2024-09-13T08:46:06.948447+0000","last_clean":"2024-09-13T08:46:06.948447+0000","last_became_active":"2024-09-13T08:46:05.979607+0000","last_became_peered":"2024-09-13T08:46:05.979607+0000","last_unstale":"2024-09-13T08:46:06.948447+0000","last_undegraded":"2024-09-13T08:46:06.948447+0000","last_fullsized":"2024-09-13T08:46:06.948447+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:47.278648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987594+0000","last_change":"2024-09-13T08:46:05.985405+0000","last_active":"2024-09-13T08:46:05.987594+0000","last_peered":"2024-09-13T08:46:05.987594+0000","last_clean":"2024-09-13T08:46:05.987594+0000","last_became_active":"2024-09-13T08:46:05.984800+0000","last_became_peered":"2024-09-13T08:46:05.984800+0000","last_unstale":"2024-09-13T08:46:05.987594+0000","last_undegraded":"2024-09-13T08:46:05.987594+0000","last_fullsized":"2024-09-13T08:46:05.987594+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.895527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947885+0000","last_change":"2024-09-13T08:46:05.967792+0000","last_active":"2024-09-13T08:46:06.947885+0000","last_peered":"2024-09-13T08:46:06.947885+0000","last_clean":"2024-09-13T08:46:06.947885+0000","last_became_active":"2024-09-13T08:46:05.967483+0000","last_became_peered":"2024-09-13T08:46:05.967483+0000","last_unstale":"2024-09-13T08:46:06.947885+0000","last_undegraded":"2024-09-13T08:46:06.947885+0000","last_fullsized":"2024-09-13T08:46:06.947885+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:11.410024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949902+0000","last_change":"2024-09-13T08:46:05.970808+0000","last_active":"2024-09-13T08:46:06.949902+0000","last_peered":"2024-09-13T08:46:06.949902+0000","last_clean":"2024-09-13T08:46:06.949902+0000","last_became_active":"2024-09-13T08:46:05.970504+0000","last_became_peered":"2024-09-13T08:46:05.970504+0000","last_unstale":"2024-09-13T08:46:06.949902+0000","last_undegraded":"2024-09-13T08:46:06.949902+0000","last_fullsized":"2024-09-13T08:46:06.949902+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:03.960394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949527+0000","last_change":"2024-09-13T08:46:05.962854+0000","last_active":"2024-09-13T08:46:06.949527+0000","last_peered":"2024-09-13T08:46:06.949527+0000","last_clean":"2024-09-13T08:46:06.949527+0000","last_became_active":"2024-09-13T08:46:05.962575+0000","last_became_peered":"2024-09-13T08:46:05.962575+0000","last_unstale":"2024-09-13T08:46:06.949527+0000","last_undegraded":"2024-09-13T08:46:06.949527+0000","last_fullsized":"2024-09-13T08:46:06.949527+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:50.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977828+0000","last_change":"2024-09-13T08:46:05.970940+0000","last_active":"2024-09-13T08:46:05.977828+0000","last_peered":"2024-09-13T08:46:05.977828+0000","last_clean":"2024-09-13T08:46:05.977828+0000","last_became_active":"2024-09-13T08:46:05.966346+0000","last_became_peered":"2024-09-13T08:46:05.966346+0000","last_unstale":"2024-09-13T08:46:05.977828+0000","last_undegraded":"2024-09-13T08:46:05.977828+0000","last_fullsized":"2024-09-13T08:46:05.977828+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949004+0000","last_change":"2024-09-13T08:46:05.970763+0000","last_active":"2024-09-13T08:46:06.949004+0000","last_peered":"2024-09-13T08:46:06.949004+0000","last_clean":"2024-09-13T08:46:06.949004+0000","last_became_active":"2024-09-13T08:46:05.970021+0000","last_became_peered":"2024-09-13T08:46:05.970021+0000","last_unstale":"2024-09-13T08:46:06.949004+0000","last_undegraded":"2024-09-13T08:46:06.949004+0000","last_fullsized":"2024-09-13T08:46:06.949004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:01.985808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948400+0000","last_change":"2024-09-13T08:46:05.971432+0000","last_active":"2024-09-13T08:46:06.948400+0000","last_peered":"2024-09-13T08:46:06.948400+0000","last_clean":"2024-09-13T08:46:06.948400+0000","last_became_active":"2024-09-13T08:46:05.970556+0000","last_became_peered":"2024-09-13T08:46:05.970556+0000","last_unstale":"2024-09-13T08:46:06.948400+0000","last_undegraded":"2024-09-13T08:46:06.948400+0000","last_fullsized":"2024-09-13T08:46:06.948400+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.584690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979501+0000","last_change":"2024-09-13T08:46:05.977431+0000","last_active":"2024-09-13T08:46:05.979501+0000","last_peered":"2024-09-13T08:46:05.979501+0000","last_clean":"2024-09-13T08:46:05.979501+0000","last_became_active":"2024-09-13T08:46:05.976716+0000","last_became_peered":"2024-09-13T08:46:05.976716+0000","last_unstale":"2024-09-13T08:46:05.979501+0000","last_undegraded":"2024-09-13T08:46:05.979501+0000","last_fullsized":"2024-09-13T08:46:05.979501+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:53.808701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.975840+0000","last_change":"2024-09-13T08:46:05.974464+0000","last_active":"2024-09-13T08:46:05.975840+0000","last_peered":"2024-09-13T08:46:05.975840+0000","last_clean":"2024-09-13T08:46:05.975840+0000","last_became_active":"2024-09-13T08:46:05.974163+0000","last_became_peered":"2024-09-13T08:46:05.974163+0000","last_unstale":"2024-09-13T08:46:05.975840+0000","last_undegraded":"2024-09-13T08:46:05.975840+0000","last_fullsized":"2024-09-13T08:46:05.975840+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:59.885201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969437+0000","last_change":"2024-09-13T08:46:05.960936+0000","last_active":"2024-09-13T08:46:05.969437+0000","last_peered":"2024-09-13T08:46:05.969437+0000","last_clean":"2024-09-13T08:46:05.969437+0000","last_became_active":"2024-09-13T08:46:05.960634+0000","last_became_peered":"2024-09-13T08:46:05.960634+0000","last_unstale":"2024-09-13T08:46:05.969437+0000","last_undegraded":"2024-09-13T08:46:05.969437+0000","last_fullsized":"2024-09-13T08:46:05.969437+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:30:39.469478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979214+0000","last_change":"2024-09-13T08:46:05.977401+0000","last_active":"2024-09-13T08:46:05.979214+0000","last_peered":"2024-09-13T08:46:05.979214+0000","last_clean":"2024-09-13T08:46:05.979214+0000","last_became_active":"2024-09-13T08:46:05.977135+0000","last_became_peered":"2024-09-13T08:46:05.977135+0000","last_unstale":"2024-09-13T08:46:05.979214+0000","last_undegraded":"2024-09-13T08:46:05.979214+0000","last_fullsized":"2024-09-13T08:46:05.979214+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:07.103853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947544+0000","last_change":"2024-09-13T08:46:05.973530+0000","last_active":"2024-09-13T08:46:06.947544+0000","last_peered":"2024-09-13T08:46:06.947544+0000","last_clean":"2024-09-13T08:46:06.947544+0000","last_became_active":"2024-09-13T08:46:05.973253+0000","last_became_peered":"2024-09-13T08:46:05.973253+0000","last_unstale":"2024-09-13T08:46:06.947544+0000","last_undegraded":"2024-09-13T08:46:06.947544+0000","last_fullsized":"2024-09-13T08:46:06.947544+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:18.254730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977212+0000","last_change":"2024-09-13T08:46:05.972439+0000","last_active":"2024-09-13T08:46:05.977212+0000","last_peered":"2024-09-13T08:46:05.977212+0000","last_clean":"2024-09-13T08:46:05.977212+0000","last_became_active":"2024-09-13T08:46:05.972034+0000","last_became_peered":"2024-09-13T08:46:05.972034+0000","last_unstale":"2024-09-13T08:46:05.977212+0000","last_undegraded":"2024-09-13T08:46:05.977212+0000","last_fullsized":"2024-09-13T08:46:05.977212+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:19.611048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987547+0000","last_change":"2024-09-13T08:46:05.982404+0000","last_active":"2024-09-13T08:46:05.987547+0000","last_peered":"2024-09-13T08:46:05.987547+0000","last_clean":"2024-09-13T08:46:05.987547+0000","last_became_active":"2024-09-13T08:46:05.982084+0000","last_became_peered":"2024-09-13T08:46:05.982084+0000","last_unstale":"2024-09-13T08:46:05.987547+0000","last_undegraded":"2024-09-13T08:46:05.987547+0000","last_fullsized":"2024-09-13T08:46:05.987547+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950351+0000","last_change":"2024-09-13T08:46:05.978395+0000","last_active":"2024-09-13T08:46:06.950351+0000","last_peered":"2024-09-13T08:46:06.950351+0000","last_clean":"2024-09-13T08:46:06.950351+0000","last_became_active":"2024-09-13T08:46:05.978038+0000","last_became_peered":"2024-09-13T08:46:05.978038+0000","last_unstale":"2024-09-13T08:46:06.950351+0000","last_undegraded":"2024-09-13T08:46:06.950351+0000","last_fullsized":"2024-09-13T08:46:06.950351+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:28.765521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.978189+0000","last_change":"2024-09-13T08:46:05.972336+0000","last_active":"2024-09-13T08:46:05.978189+0000","last_peered":"2024-09-13T08:46:05.978189+0000","last_clean":"2024-09-13T08:46:05.978189+0000","last_became_active":"2024-09-13T08:46:05.972090+0000","last_became_peered":"2024-09-13T08:46:05.972090+0000","last_unstale":"2024-09-13T08:46:05.978189+0000","last_undegraded":"2024-09-13T08:46:05.978189+0000","last_fullsized":"2024-09-13T08:46:05.978189+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987210+0000","last_change":"2024-09-13T08:46:05.975851+0000","last_active":"2024-09-13T08:46:05.987210+0000","last_peered":"2024-09-13T08:46:05.987210+0000","last_clean":"2024-09-13T08:46:05.987210+0000","last_became_active":"2024-09-13T08:46:05.975542+0000","last_became_peered":"2024-09-13T08:46:05.975542+0000","last_unstale":"2024-09-13T08:46:05.987210+0000","last_undegraded":"2024-09-13T08:46:05.987210+0000","last_fullsized":"2024-09-13T08:46:05.987210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949333+0000","last_change":"2024-09-13T08:46:05.968613+0000","last_active":"2024-09-13T08:46:06.949333+0000","last_peered":"2024-09-13T08:46:06.949333+0000","last_clean":"2024-09-13T08:46:06.949333+0000","last_became_active":"2024-09-13T08:46:05.968257+0000","last_became_peered":"2024-09-13T08:46:05.968257+0000","last_unstale":"2024-09-13T08:46:06.949333+0000","last_undegraded":"2024-09-13T08:46:06.949333+0000","last_fullsized":"2024-09-13T08:46:06.949333+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:46.022535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.986720+0000","last_change":"2024-09-13T08:46:05.975824+0000","last_active":"2024-09-13T08:46:05.986720+0000","last_peered":"2024-09-13T08:46:05.986720+0000","last_clean":"2024-09-13T08:46:05.986720+0000","last_became_active":"2024-09-13T08:46:05.975309+0000","last_became_peered":"2024-09-13T08:46:05.975309+0000","last_unstale":"2024-09-13T08:46:05.986720+0000","last_undegraded":"2024-09-13T08:46:05.986720+0000","last_fullsized":"2024-09-13T08:46:05.986720+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950629+0000","last_change":"2024-09-13T08:46:05.980705+0000","last_active":"2024-09-13T08:46:06.950629+0000","last_peered":"2024-09-13T08:46:06.950629+0000","last_clean":"2024-09-13T08:46:06.950629+0000","last_became_active":"2024-09-13T08:46:05.979637+0000","last_became_peered":"2024-09-13T08:46:05.979637+0000","last_unstale":"2024-09-13T08:46:06.950629+0000","last_undegraded":"2024-09-13T08:46:06.950629+0000","last_fullsized":"2024-09-13T08:46:06.950629+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:06.065605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.982920+0000","last_change":"2024-09-13T08:46:05.980018+0000","last_active":"2024-09-13T08:46:05.982920+0000","last_peered":"2024-09-13T08:46:05.982920+0000","last_clean":"2024-09-13T08:46:05.982920+0000","last_became_active":"2024-09-13T08:46:05.979590+0000","last_became_peered":"2024-09-13T08:46:05.979590+0000","last_unstale":"2024-09-13T08:46:05.982920+0000","last_undegraded":"2024-09-13T08:46:05.982920+0000","last_fullsized":"2024-09-13T08:46:05.982920+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948875+0000","last_change":"2024-09-13T08:46:05.971069+0000","last_active":"2024-09-13T08:46:06.948875+0000","last_peered":"2024-09-13T08:46:06.948875+0000","last_clean":"2024-09-13T08:46:06.948875+0000","last_became_active":"2024-09-13T08:46:05.970462+0000","last_became_peered":"2024-09-13T08:46:05.970462+0000","last_unstale":"2024-09-13T08:46:06.948875+0000","last_undegraded":"2024-09-13T08:46:06.948875+0000","last_fullsized":"2024-09-13T08:46:06.948875+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:54.544335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948573+0000","last_change":"2024-09-13T08:46:05.982034+0000","last_active":"2024-09-13T08:46:06.948573+0000","last_peered":"2024-09-13T08:46:06.948573+0000","last_clean":"2024-09-13T08:46:06.948573+0000","last_became_active":"2024-09-13T08:46:05.981533+0000","last_became_peered":"2024-09-13T08:46:05.981533+0000","last_unstale":"2024-09-13T08:46:06.948573+0000","last_undegraded":"2024-09-13T08:46:06.948573+0000","last_fullsized":"2024-09-13T08:46:06.948573+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:43.470156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969533+0000","last_change":"2024-09-13T08:46:05.964100+0000","last_active":"2024-09-13T08:46:05.969533+0000","last_peered":"2024-09-13T08:46:05.969533+0000","last_clean":"2024-09-13T08:46:05.969533+0000","last_became_active":"2024-09-13T08:46:05.963842+0000","last_became_peered":"2024-09-13T08:46:05.963842+0000","last_unstale":"2024-09-13T08:46:05.969533+0000","last_undegraded":"2024-09-13T08:46:05.969533+0000","last_fullsized":"2024-09-13T08:46:05.969533+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:51:54.095289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.986610+0000","last_change":"2024-09-13T08:46:05.974979+0000","last_active":"2024-09-13T08:46:05.986610+0000","last_peered":"2024-09-13T08:46:05.986610+0000","last_clean":"2024-09-13T08:46:05.986610+0000","last_became_active":"2024-09-13T08:46:05.974620+0000","last_became_peered":"2024-09-13T08:46:05.974620+0000","last_unstale":"2024-09-13T08:46:05.986610+0000","last_undegraded":"2024-09-13T08:46:05.986610+0000","last_fullsized":"2024-09-13T08:46:05.986610+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:26.670968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981122+0000","last_change":"2024-09-13T08:46:05.978596+0000","last_active":"2024-09-13T08:46:05.981122+0000","last_peered":"2024-09-13T08:46:05.981122+0000","last_clean":"2024-09-13T08:46:05.981122+0000","last_became_active":"2024-09-13T08:46:05.978014+0000","last_became_peered":"2024-09-13T08:46:05.978014+0000","last_unstale":"2024-09-13T08:46:05.981122+0000","last_undegraded":"2024-09-13T08:46:05.981122+0000","last_fullsized":"2024-09-13T08:46:05.981122+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.983998+0000","last_change":"2024-09-13T08:46:05.978488+0000","last_active":"2024-09-13T08:46:05.983998+0000","last_peered":"2024-09-13T08:46:05.983998+0000","last_clean":"2024-09-13T08:46:05.983998+0000","last_became_active":"2024-09-13T08:46:05.976693+0000","last_became_peered":"2024-09-13T08:46:05.976693+0000","last_unstale":"2024-09-13T08:46:05.983998+0000","last_undegraded":"2024-09-13T08:46:05.983998+0000","last_fullsized":"2024-09-13T08:46:05.983998+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:59.713194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958466+0000","last_change":"2024-09-13T08:46:05.956565+0000","last_active":"2024-09-13T08:46:05.958466+0000","last_peered":"2024-09-13T08:46:05.958466+0000","last_clean":"2024-09-13T08:46:05.958466+0000","last_became_active":"2024-09-13T08:46:05.956157+0000","last_became_peered":"2024-09-13T08:46:05.956157+0000","last_unstale":"2024-09-13T08:46:05.958466+0000","last_undegraded":"2024-09-13T08:46:05.958466+0000","last_fullsized":"2024-09-13T08:46:05.958466+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:10.934759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948951+0000","last_change":"2024-09-13T08:46:05.964483+0000","last_active":"2024-09-13T08:46:06.948951+0000","last_peered":"2024-09-13T08:46:06.948951+0000","last_clean":"2024-09-13T08:46:06.948951+0000","last_became_active":"2024-09-13T08:46:05.964256+0000","last_became_peered":"2024-09-13T08:46:05.964256+0000","last_unstale":"2024-09-13T08:46:06.948951+0000","last_undegraded":"2024-09-13T08:46:06.948951+0000","last_fullsized":"2024-09-13T08:46:06.948951+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:46.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949159+0000","last_change":"2024-09-13T08:46:05.971630+0000","last_active":"2024-09-13T08:46:06.949159+0000","last_peered":"2024-09-13T08:46:06.949159+0000","last_clean":"2024-09-13T08:46:06.949159+0000","last_became_active":"2024-09-13T08:46:05.971352+0000","last_became_peered":"2024-09-13T08:46:05.971352+0000","last_unstale":"2024-09-13T08:46:06.949159+0000","last_undegraded":"2024-09-13T08:46:06.949159+0000","last_fullsized":"2024-09-13T08:46:06.949159+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:48.134772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949506+0000","last_change":"2024-09-13T08:46:05.962568+0000","last_active":"2024-09-13T08:46:06.949506+0000","last_peered":"2024-09-13T08:46:06.949506+0000","last_clean":"2024-09-13T08:46:06.949506+0000","last_became_active":"2024-09-13T08:46:05.957696+0000","last_became_peered":"2024-09-13T08:46:05.957696+0000","last_unstale":"2024-09-13T08:46:06.949506+0000","last_undegraded":"2024-09-13T08:46:06.949506+0000","last_fullsized":"2024-09-13T08:46:06.949506+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:08.697934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"163'4","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.988521+0000","last_change":"2024-09-13T08:46:04.959656+0000","last_active":"2024-09-13T08:46:05.988521+0000","last_peered":"2024-09-13T08:46:05.988521+0000","last_clean":"2024-09-13T08:46:05.988521+0000","last_became_active":"2024-09-13T08:46:04.959376+0000","last_became_peered":"2024-09-13T08:46:04.959376+0000","last_unstale":"2024-09-13T08:46:05.988521+0000","last_undegraded":"2024-09-13T08:46:05.988521+0000","last_fullsized":"2024-09-13T08:46:05.988521+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:52.801573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.985208+0000","last_change":"2024-09-13T08:46:05.984248+0000","last_active":"2024-09-13T08:46:05.985208+0000","last_peered":"2024-09-13T08:46:05.985208+0000","last_clean":"2024-09-13T08:46:05.985208+0000","last_became_active":"2024-09-13T08:46:05.983952+0000","last_became_peered":"2024-09-13T08:46:05.983952+0000","last_unstale":"2024-09-13T08:46:05.985208+0000","last_undegraded":"2024-09-13T08:46:05.985208+0000","last_fullsized":"2024-09-13T08:46:05.985208+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"163'1","reported_seq":14,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.970822+0000","last_change":"2024-09-13T08:46:04.970027+0000","last_active":"2024-09-13T08:46:05.970822+0000","last_peered":"2024-09-13T08:46:05.970822+0000","last_clean":"2024-09-13T08:46:05.970822+0000","last_became_active":"2024-09-13T08:46:04.969527+0000","last_became_peered":"2024-09-13T08:46:04.969527+0000","last_unstale":"2024-09-13T08:46:05.970822+0000","last_undegraded":"2024-09-13T08:46:05.970822+0000","last_fullsized":"2024-09-13T08:46:05.970822+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:54:31.922912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947864+0000","last_change":"2024-09-13T08:46:05.978601+0000","last_active":"2024-09-13T08:46:06.947864+0000","last_peered":"2024-09-13T08:46:06.947864+0000","last_clean":"2024-09-13T08:46:06.947864+0000","last_became_active":"2024-09-13T08:46:05.975598+0000","last_became_peered":"2024-09-13T08:46:05.975598+0000","last_unstale":"2024-09-13T08:46:06.947864+0000","last_undegraded":"2024-09-13T08:46:06.947864+0000","last_fullsized":"2024-09-13T08:46:06.947864+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:30.926174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950195+0000","last_change":"2024-09-13T08:46:04.967771+0000","last_active":"2024-09-13T08:46:06.950195+0000","last_peered":"2024-09-13T08:46:06.950195+0000","last_clean":"2024-09-13T08:46:06.950195+0000","last_became_active":"2024-09-13T08:46:04.966897+0000","last_became_peered":"2024-09-13T08:46:04.966897+0000","last_unstale":"2024-09-13T08:46:06.950195+0000","last_undegraded":"2024-09-13T08:46:06.950195+0000","last_fullsized":"2024-09-13T08:46:06.950195+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:29.919047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948296+0000","last_change":"2024-09-13T08:46:05.979678+0000","last_active":"2024-09-13T08:46:06.948296+0000","last_peered":"2024-09-13T08:46:06.948296+0000","last_clean":"2024-09-13T08:46:06.948296+0000","last_became_active":"2024-09-13T08:46:05.979385+0000","last_became_peered":"2024-09-13T08:46:05.979385+0000","last_unstale":"2024-09-13T08:46:06.948296+0000","last_undegraded":"2024-09-13T08:46:06.948296+0000","last_fullsized":"2024-09-13T08:46:06.948296+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:29.866552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977629+0000","last_change":"2024-09-13T08:46:04.969348+0000","last_active":"2024-09-13T08:46:05.977629+0000","last_peered":"2024-09-13T08:46:05.977629+0000","last_clean":"2024-09-13T08:46:05.977629+0000","last_became_active":"2024-09-13T08:46:04.968942+0000","last_became_peered":"2024-09-13T08:46:04.968942+0000","last_unstale":"2024-09-13T08:46:05.977629+0000","last_undegraded":"2024-09-13T08:46:05.977629+0000","last_fullsized":"2024-09-13T08:46:05.977629+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:54:12.768339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950635+0000","last_change":"2024-09-13T08:46:05.960209+0000","last_active":"2024-09-13T08:46:06.950635+0000","last_peered":"2024-09-13T08:46:06.950635+0000","last_clean":"2024-09-13T08:46:06.950635+0000","last_became_active":"2024-09-13T08:46:05.959911+0000","last_became_peered":"2024-09-13T08:46:05.959911+0000","last_unstale":"2024-09-13T08:46:06.950635+0000","last_undegraded":"2024-09-13T08:46:06.950635+0000","last_fullsized":"2024-09-13T08:46:06.950635+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:28:50.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950126+0000","last_change":"2024-09-13T08:46:04.965445+0000","last_active":"2024-09-13T08:46:06.950126+0000","last_peered":"2024-09-13T08:46:06.950126+0000","last_clean":"2024-09-13T08:46:06.950126+0000","last_became_active":"2024-09-13T08:46:04.964172+0000","last_became_peered":"2024-09-13T08:46:04.964172+0000","last_unstale":"2024-09-13T08:46:06.950126+0000","last_undegraded":"2024-09-13T08:46:06.950126+0000","last_fullsized":"2024-09-13T08:46:06.950126+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:28.859425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950503+0000","last_change":"2024-09-13T08:46:05.970369+0000","last_active":"2024-09-13T08:46:06.950503+0000","last_peered":"2024-09-13T08:46:06.950503+0000","last_clean":"2024-09-13T08:46:06.950503+0000","last_became_active":"2024-09-13T08:46:05.970068+0000","last_became_peered":"2024-09-13T08:46:05.970068+0000","last_unstale":"2024-09-13T08:46:06.950503+0000","last_undegraded":"2024-09-13T08:46:06.950503+0000","last_fullsized":"2024-09-13T08:46:06.950503+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:39.954828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950708+0000","last_change":"2024-09-13T08:46:04.968963+0000","last_active":"2024-09-13T08:46:06.950708+0000","last_peered":"2024-09-13T08:46:06.950708+0000","last_clean":"2024-09-13T08:46:06.950708+0000","last_became_active":"2024-09-13T08:46:04.968673+0000","last_became_peered":"2024-09-13T08:46:04.968673+0000","last_unstale":"2024-09-13T08:46:06.950708+0000","last_undegraded":"2024-09-13T08:46:06.950708+0000","last_fullsized":"2024-09-13T08:46:06.950708+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:32.962876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.961006+0000","last_change":"2024-09-13T08:46:05.959266+0000","last_active":"2024-09-13T08:46:05.961006+0000","last_peered":"2024-09-13T08:46:05.961006+0000","last_clean":"2024-09-13T08:46:05.961006+0000","last_became_active":"2024-09-13T08:46:05.958950+0000","last_became_peered":"2024-09-13T08:46:05.958950+0000","last_unstale":"2024-09-13T08:46:05.961006+0000","last_undegraded":"2024-09-13T08:46:05.961006+0000","last_fullsized":"2024-09-13T08:46:05.961006+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:06.714072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.951701+0000","last_change":"2024-09-13T08:46:04.959351+0000","last_active":"2024-09-13T08:46:05.951701+0000","last_peered":"2024-09-13T08:46:05.951701+0000","last_clean":"2024-09-13T08:46:05.951701+0000","last_became_active":"2024-09-13T08:46:04.958933+0000","last_became_peered":"2024-09-13T08:46:04.958933+0000","last_unstale":"2024-09-13T08:46:05.951701+0000","last_undegraded":"2024-09-13T08:46:05.951701+0000","last_fullsized":"2024-09-13T08:46:05.951701+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:52.801573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987490+0000","last_change":"2024-09-13T08:46:05.982051+0000","last_active":"2024-09-13T08:46:05.987490+0000","last_peered":"2024-09-13T08:46:05.987490+0000","last_clean":"2024-09-13T08:46:05.987490+0000","last_became_active":"2024-09-13T08:46:05.975965+0000","last_became_peered":"2024-09-13T08:46:05.975965+0000","last_unstale":"2024-09-13T08:46:05.987490+0000","last_undegraded":"2024-09-13T08:46:05.987490+0000","last_fullsized":"2024-09-13T08:46:05.987490+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.957931+0000","last_change":"2024-09-13T08:46:04.967778+0000","last_active":"2024-09-13T08:46:05.957931+0000","last_peered":"2024-09-13T08:46:05.957931+0000","last_clean":"2024-09-13T08:46:05.957931+0000","last_became_active":"2024-09-13T08:46:04.967415+0000","last_became_peered":"2024-09-13T08:46:04.967415+0000","last_unstale":"2024-09-13T08:46:05.957931+0000","last_undegraded":"2024-09-13T08:46:05.957931+0000","last_fullsized":"2024-09-13T08:46:05.957931+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:06.738410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979070+0000","last_change":"2024-09-13T08:46:05.976820+0000","last_active":"2024-09-13T08:46:05.979070+0000","last_peered":"2024-09-13T08:46:05.979070+0000","last_clean":"2024-09-13T08:46:05.979070+0000","last_became_active":"2024-09-13T08:46:05.976300+0000","last_became_peered":"2024-09-13T08:46:05.976300+0000","last_unstale":"2024-09-13T08:46:05.979070+0000","last_undegraded":"2024-09-13T08:46:05.979070+0000","last_fullsized":"2024-09-13T08:46:05.979070+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:26.670968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973410+0000","last_change":"2024-09-13T08:46:04.969092+0000","last_active":"2024-09-13T08:46:05.973410+0000","last_peered":"2024-09-13T08:46:05.973410+0000","last_clean":"2024-09-13T08:46:05.973410+0000","last_became_active":"2024-09-13T08:46:04.968791+0000","last_became_peered":"2024-09-13T08:46:04.968791+0000","last_unstale":"2024-09-13T08:46:05.973410+0000","last_undegraded":"2024-09-13T08:46:05.973410+0000","last_fullsized":"2024-09-13T08:46:05.973410+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:57.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972635+0000","last_change":"2024-09-13T08:46:05.965346+0000","last_active":"2024-09-13T08:46:05.972635+0000","last_peered":"2024-09-13T08:46:05.972635+0000","last_clean":"2024-09-13T08:46:05.972635+0000","last_became_active":"2024-09-13T08:46:05.965035+0000","last_became_peered":"2024-09-13T08:46:05.965035+0000","last_unstale":"2024-09-13T08:46:05.972635+0000","last_undegraded":"2024-09-13T08:46:05.972635+0000","last_fullsized":"2024-09-13T08:46:05.972635+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:17:16.009417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"163'3","reported_seq":18,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950327+0000","last_change":"2024-09-13T08:46:04.969855+0000","last_active":"2024-09-13T08:46:06.950327+0000","last_peered":"2024-09-13T08:46:06.950327+0000","last_clean":"2024-09-13T08:46:06.950327+0000","last_became_active":"2024-09-13T08:46:04.968696+0000","last_became_peered":"2024-09-13T08:46:04.968696+0000","last_unstale":"2024-09-13T08:46:06.950327+0000","last_undegraded":"2024-09-13T08:46:06.950327+0000","last_fullsized":"2024-09-13T08:46:06.950327+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:01.929681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987835+0000","last_change":"2024-09-13T08:46:05.987259+0000","last_active":"2024-09-13T08:46:05.987835+0000","last_peered":"2024-09-13T08:46:05.987835+0000","last_clean":"2024-09-13T08:46:05.987835+0000","last_became_active":"2024-09-13T08:46:05.986922+0000","last_became_peered":"2024-09-13T08:46:05.986922+0000","last_unstale":"2024-09-13T08:46:05.987835+0000","last_undegraded":"2024-09-13T08:46:05.987835+0000","last_fullsized":"2024-09-13T08:46:05.987835+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:49:55.852849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"163'6","reported_seq":21,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949450+0000","last_change":"2024-09-13T08:46:04.964670+0000","last_active":"2024-09-13T08:46:06.949450+0000","last_peered":"2024-09-13T08:46:06.949450+0000","last_clean":"2024-09-13T08:46:06.949450+0000","last_became_active":"2024-09-13T08:46:04.963252+0000","last_became_peered":"2024-09-13T08:46:04.963252+0000","last_unstale":"2024-09-13T08:46:06.949450+0000","last_undegraded":"2024-09-13T08:46:06.949450+0000","last_fullsized":"2024-09-13T08:46:06.949450+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.577563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981188+0000","last_change":"2024-09-13T08:46:05.978698+0000","last_active":"2024-09-13T08:46:05.981188+0000","last_peered":"2024-09-13T08:46:05.981188+0000","last_clean":"2024-09-13T08:46:05.981188+0000","last_became_active":"2024-09-13T08:46:05.978445+0000","last_became_peered":"2024-09-13T08:46:05.978445+0000","last_unstale":"2024-09-13T08:46:05.981188+0000","last_undegraded":"2024-09-13T08:46:05.981188+0000","last_fullsized":"2024-09-13T08:46:05.981188+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981904+0000","last_change":"2024-09-13T08:46:04.957718+0000","last_active":"2024-09-13T08:46:05.981904+0000","last_peered":"2024-09-13T08:46:05.981904+0000","last_clean":"2024-09-13T08:46:05.981904+0000","last_became_active":"2024-09-13T08:46:04.957398+0000","last_became_peered":"2024-09-13T08:46:04.957398+0000","last_unstale":"2024-09-13T08:46:05.981904+0000","last_undegraded":"2024-09-13T08:46:05.981904+0000","last_fullsized":"2024-09-13T08:46:05.981904+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:08.111988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972044+0000","last_change":"2024-09-13T08:46:05.963549+0000","last_active":"2024-09-13T08:46:05.972044+0000","last_peered":"2024-09-13T08:46:05.972044+0000","last_clean":"2024-09-13T08:46:05.972044+0000","last_became_active":"2024-09-13T08:46:05.963280+0000","last_became_peered":"2024-09-13T08:46:05.963280+0000","last_unstale":"2024-09-13T08:46:05.972044+0000","last_undegraded":"2024-09-13T08:46:05.972044+0000","last_fullsized":"2024-09-13T08:46:05.972044+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973151+0000","last_change":"2024-09-13T08:46:04.965004+0000","last_active":"2024-09-13T08:46:05.973151+0000","last_peered":"2024-09-13T08:46:05.973151+0000","last_clean":"2024-09-13T08:46:05.973151+0000","last_became_active":"2024-09-13T08:46:04.964507+0000","last_became_peered":"2024-09-13T08:46:04.964507+0000","last_unstale":"2024-09-13T08:46:05.973151+0000","last_undegraded":"2024-09-13T08:46:05.973151+0000","last_fullsized":"2024-09-13T08:46:05.973151+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:18.469821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958190+0000","last_change":"2024-09-13T08:46:05.955400+0000","last_active":"2024-09-13T08:46:05.958190+0000","last_peered":"2024-09-13T08:46:05.958190+0000","last_clean":"2024-09-13T08:46:05.958190+0000","last_became_active":"2024-09-13T08:46:05.955064+0000","last_became_peered":"2024-09-13T08:46:05.955064+0000","last_unstale":"2024-09-13T08:46:05.958190+0000","last_undegraded":"2024-09-13T08:46:05.958190+0000","last_fullsized":"2024-09-13T08:46:05.958190+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:04.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949838+0000","last_change":"2024-09-13T08:46:04.969712+0000","last_active":"2024-09-13T08:46:06.949838+0000","last_peered":"2024-09-13T08:46:06.949838+0000","last_clean":"2024-09-13T08:46:06.949838+0000","last_became_active":"2024-09-13T08:46:04.969436+0000","last_became_peered":"2024-09-13T08:46:04.969436+0000","last_unstale":"2024-09-13T08:46:06.949838+0000","last_undegraded":"2024-09-13T08:46:06.949838+0000","last_fullsized":"2024-09-13T08:46:06.949838+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:09:03.013668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950637+0000","last_change":"2024-09-13T08:46:05.971981+0000","last_active":"2024-09-13T08:46:06.950637+0000","last_peered":"2024-09-13T08:46:06.950637+0000","last_clean":"2024-09-13T08:46:06.950637+0000","last_became_active":"2024-09-13T08:46:05.971680+0000","last_became_peered":"2024-09-13T08:46:05.971680+0000","last_unstale":"2024-09-13T08:46:06.950637+0000","last_undegraded":"2024-09-13T08:46:06.950637+0000","last_fullsized":"2024-09-13T08:46:06.950637+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"163'8","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.985245+0000","last_change":"2024-09-13T08:46:04.969610+0000","last_active":"2024-09-13T08:46:05.985245+0000","last_peered":"2024-09-13T08:46:05.985245+0000","last_clean":"2024-09-13T08:46:05.985245+0000","last_became_active":"2024-09-13T08:46:04.969318+0000","last_became_peered":"2024-09-13T08:46:04.969318+0000","last_unstale":"2024-09-13T08:46:05.985245+0000","last_undegraded":"2024-09-13T08:46:05.985245+0000","last_fullsized":"2024-09-13T08:46:05.985245+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:25.663841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.982990+0000","last_change":"2024-09-13T08:46:05.980699+0000","last_active":"2024-09-13T08:46:05.982990+0000","last_peered":"2024-09-13T08:46:05.982990+0000","last_clean":"2024-09-13T08:46:05.982990+0000","last_became_active":"2024-09-13T08:46:05.979675+0000","last_became_peered":"2024-09-13T08:46:05.979675+0000","last_unstale":"2024-09-13T08:46:05.982990+0000","last_undegraded":"2024-09-13T08:46:05.982990+0000","last_fullsized":"2024-09-13T08:46:05.982990+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950143+0000","last_change":"2024-09-13T08:46:04.968628+0000","last_active":"2024-09-13T08:46:06.950143+0000","last_peered":"2024-09-13T08:46:06.950143+0000","last_clean":"2024-09-13T08:46:06.950143+0000","last_became_active":"2024-09-13T08:46:04.968084+0000","last_became_peered":"2024-09-13T08:46:04.968084+0000","last_unstale":"2024-09-13T08:46:06.950143+0000","last_undegraded":"2024-09-13T08:46:06.950143+0000","last_fullsized":"2024-09-13T08:46:06.950143+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.754569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.959042+0000","last_change":"2024-09-13T08:46:05.957318+0000","last_active":"2024-09-13T08:46:05.959042+0000","last_peered":"2024-09-13T08:46:05.959042+0000","last_clean":"2024-09-13T08:46:05.959042+0000","last_became_active":"2024-09-13T08:46:05.952788+0000","last_became_peered":"2024-09-13T08:46:05.952788+0000","last_unstale":"2024-09-13T08:46:05.959042+0000","last_undegraded":"2024-09-13T08:46:05.959042+0000","last_fullsized":"2024-09-13T08:46:05.959042+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.673880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950262+0000","last_change":"2024-09-13T08:46:04.953330+0000","last_active":"2024-09-13T08:46:06.950262+0000","last_peered":"2024-09-13T08:46:06.950262+0000","last_clean":"2024-09-13T08:46:06.950262+0000","last_became_active":"2024-09-13T08:46:04.952883+0000","last_became_peered":"2024-09-13T08:46:04.952883+0000","last_unstale":"2024-09-13T08:46:06.950262+0000","last_undegraded":"2024-09-13T08:46:06.950262+0000","last_fullsized":"2024-09-13T08:46:06.950262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:28.617454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948065+0000","last_change":"2024-09-13T08:46:05.971314+0000","last_active":"2024-09-13T08:46:06.948065+0000","last_peered":"2024-09-13T08:46:06.948065+0000","last_clean":"2024-09-13T08:46:06.948065+0000","last_became_active":"2024-09-13T08:46:05.970653+0000","last_became_peered":"2024-09-13T08:46:05.970653+0000","last_unstale":"2024-09-13T08:46:06.948065+0000","last_undegraded":"2024-09-13T08:46:06.948065+0000","last_fullsized":"2024-09-13T08:46:06.948065+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:52.842034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"163'1","reported_seq":14,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.976432+0000","last_change":"2024-09-13T08:46:04.962474+0000","last_active":"2024-09-13T08:46:05.976432+0000","last_peered":"2024-09-13T08:46:05.976432+0000","last_clean":"2024-09-13T08:46:05.976432+0000","last_became_active":"2024-09-13T08:46:04.962068+0000","last_became_peered":"2024-09-13T08:46:04.962068+0000","last_unstale":"2024-09-13T08:46:05.976432+0000","last_undegraded":"2024-09-13T08:46:05.976432+0000","last_fullsized":"2024-09-13T08:46:05.976432+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:16.918066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.983799+0000","last_change":"2024-09-13T08:46:05.976559+0000","last_active":"2024-09-13T08:46:05.983799+0000","last_peered":"2024-09-13T08:46:05.983799+0000","last_clean":"2024-09-13T08:46:05.983799+0000","last_became_active":"2024-09-13T08:46:05.976111+0000","last_became_peered":"2024-09-13T08:46:05.976111+0000","last_unstale":"2024-09-13T08:46:05.983799+0000","last_undegraded":"2024-09-13T08:46:05.983799+0000","last_fullsized":"2024-09-13T08:46:05.983799+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:03:27.370693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958405+0000","last_change":"2024-09-13T08:46:04.955393+0000","last_active":"2024-09-13T08:46:05.958405+0000","last_peered":"2024-09-13T08:46:05.958405+0000","last_clean":"2024-09-13T08:46:05.958405+0000","last_became_active":"2024-09-13T08:46:04.954351+0000","last_became_peered":"2024-09-13T08:46:04.954351+0000","last_unstale":"2024-09-13T08:46:05.958405+0000","last_undegraded":"2024-09-13T08:46:05.958405+0000","last_fullsized":"2024-09-13T08:46:05.958405+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:05:45.276045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950299+0000","last_change":"2024-09-13T08:46:05.973090+0000","last_active":"2024-09-13T08:46:06.950299+0000","last_peered":"2024-09-13T08:46:06.950299+0000","last_clean":"2024-09-13T08:46:06.950299+0000","last_became_active":"2024-09-13T08:46:05.972817+0000","last_became_peered":"2024-09-13T08:46:05.972817+0000","last_unstale":"2024-09-13T08:46:06.950299+0000","last_undegraded":"2024-09-13T08:46:06.950299+0000","last_fullsized":"2024-09-13T08:46:06.950299+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:34.985230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950251+0000","last_change":"2024-09-13T08:46:04.964211+0000","last_active":"2024-09-13T08:46:06.950251+0000","last_peered":"2024-09-13T08:46:06.950251+0000","last_clean":"2024-09-13T08:46:06.950251+0000","last_became_active":"2024-09-13T08:46:04.963850+0000","last_became_peered":"2024-09-13T08:46:04.963850+0000","last_unstale":"2024-09-13T08:46:06.950251+0000","last_undegraded":"2024-09-13T08:46:06.950251+0000","last_fullsized":"2024-09-13T08:46:06.950251+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:51.834907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981478+0000","last_change":"2024-09-13T08:46:05.979491+0000","last_active":"2024-09-13T08:46:05.981478+0000","last_peered":"2024-09-13T08:46:05.981478+0000","last_clean":"2024-09-13T08:46:05.981478+0000","last_became_active":"2024-09-13T08:46:05.979198+0000","last_became_peered":"2024-09-13T08:46:05.979198+0000","last_unstale":"2024-09-13T08:46:05.981478+0000","last_undegraded":"2024-09-13T08:46:05.981478+0000","last_fullsized":"2024-09-13T08:46:05.981478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950429+0000","last_change":"2024-09-13T08:46:04.964090+0000","last_active":"2024-09-13T08:46:06.950429+0000","last_peered":"2024-09-13T08:46:06.950429+0000","last_clean":"2024-09-13T08:46:06.950429+0000","last_became_active":"2024-09-13T08:46:04.963155+0000","last_became_peered":"2024-09-13T08:46:04.963155+0000","last_unstale":"2024-09-13T08:46:06.950429+0000","last_undegraded":"2024-09-13T08:46:06.950429+0000","last_fullsized":"2024-09-13T08:46:06.950429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:18:09.679957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.959983+0000","last_change":"2024-09-13T08:46:05.957664+0000","last_active":"2024-09-13T08:46:05.959983+0000","last_peered":"2024-09-13T08:46:05.959983+0000","last_clean":"2024-09-13T08:46:05.959983+0000","last_became_active":"2024-09-13T08:46:05.957112+0000","last_became_peered":"2024-09-13T08:46:05.957112+0000","last_unstale":"2024-09-13T08:46:05.959983+0000","last_undegraded":"2024-09-13T08:46:05.959983+0000","last_fullsized":"2024-09-13T08:46:05.959983+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:51:53.500313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949467+0000","last_change":"2024-09-13T08:46:04.957696+0000","last_active":"2024-09-13T08:46:06.949467+0000","last_peered":"2024-09-13T08:46:06.949467+0000","last_clean":"2024-09-13T08:46:06.949467+0000","last_became_active":"2024-09-13T08:46:04.957325+0000","last_became_peered":"2024-09-13T08:46:04.957325+0000","last_unstale":"2024-09-13T08:46:06.949467+0000","last_undegraded":"2024-09-13T08:46:06.949467+0000","last_fullsized":"2024-09-13T08:46:06.949467+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.386989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977373+0000","last_change":"2024-09-13T08:46:05.974670+0000","last_active":"2024-09-13T08:46:05.977373+0000","last_peered":"2024-09-13T08:46:05.977373+0000","last_clean":"2024-09-13T08:46:05.977373+0000","last_became_active":"2024-09-13T08:46:05.974390+0000","last_became_peered":"2024-09-13T08:46:05.974390+0000","last_unstale":"2024-09-13T08:46:05.977373+0000","last_undegraded":"2024-09-13T08:46:05.977373+0000","last_fullsized":"2024-09-13T08:46:05.977373+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:44.330496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.957487+0000","last_change":"2024-09-13T08:46:04.956148+0000","last_active":"2024-09-13T08:46:05.957487+0000","last_peered":"2024-09-13T08:46:05.957487+0000","last_clean":"2024-09-13T08:46:05.957487+0000","last_became_active":"2024-09-13T08:46:04.955866+0000","last_became_peered":"2024-09-13T08:46:04.955866+0000","last_unstale":"2024-09-13T08:46:05.957487+0000","last_undegraded":"2024-09-13T08:46:05.957487+0000","last_fullsized":"2024-09-13T08:46:05.957487+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:17.889992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987677+0000","last_change":"2024-09-13T08:46:05.985768+0000","last_active":"2024-09-13T08:46:05.987677+0000","last_peered":"2024-09-13T08:46:05.987677+0000","last_clean":"2024-09-13T08:46:05.987677+0000","last_became_active":"2024-09-13T08:46:05.982606+0000","last_became_peered":"2024-09-13T08:46:05.982606+0000","last_unstale":"2024-09-13T08:46:05.987677+0000","last_undegraded":"2024-09-13T08:46:05.987677+0000","last_fullsized":"2024-09-13T08:46:05.987677+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:09.119115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.978776+0000","last_change":"2024-09-13T08:46:04.956637+0000","last_active":"2024-09-13T08:46:05.978776+0000","last_peered":"2024-09-13T08:46:05.978776+0000","last_clean":"2024-09-13T08:46:05.978776+0000","last_became_active":"2024-09-13T08:46:04.956039+0000","last_became_peered":"2024-09-13T08:46:04.956039+0000","last_unstale":"2024-09-13T08:46:05.978776+0000","last_undegraded":"2024-09-13T08:46:05.978776+0000","last_fullsized":"2024-09-13T08:46:05.978776+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:03:32.467054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.988087+0000","last_change":"2024-09-13T08:46:05.985037+0000","last_active":"2024-09-13T08:46:05.988087+0000","last_peered":"2024-09-13T08:46:05.988087+0000","last_clean":"2024-09-13T08:46:05.988087+0000","last_became_active":"2024-09-13T08:46:05.981632+0000","last_became_peered":"2024-09-13T08:46:05.981632+0000","last_unstale":"2024-09-13T08:46:05.988087+0000","last_undegraded":"2024-09-13T08:46:05.988087+0000","last_fullsized":"2024-09-13T08:46:05.988087+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:49:55.852849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969463+0000","last_change":"2024-09-13T08:46:04.968652+0000","last_active":"2024-09-13T08:46:05.969463+0000","last_peered":"2024-09-13T08:46:05.969463+0000","last_clean":"2024-09-13T08:46:05.969463+0000","last_became_active":"2024-09-13T08:46:04.968162+0000","last_became_peered":"2024-09-13T08:46:04.968162+0000","last_unstale":"2024-09-13T08:46:05.969463+0000","last_undegraded":"2024-09-13T08:46:05.969463+0000","last_fullsized":"2024-09-13T08:46:05.969463+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:57.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987080+0000","last_change":"2024-09-13T08:46:05.982162+0000","last_active":"2024-09-13T08:46:05.987080+0000","last_peered":"2024-09-13T08:46:05.987080+0000","last_clean":"2024-09-13T08:46:05.987080+0000","last_became_active":"2024-09-13T08:46:05.981849+0000","last_became_peered":"2024-09-13T08:46:05.981849+0000","last_unstale":"2024-09-13T08:46:05.987080+0000","last_undegraded":"2024-09-13T08:46:05.987080+0000","last_fullsized":"2024-09-13T08:46:05.987080+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972296+0000","last_change":"2024-09-13T08:46:04.964174+0000","last_active":"2024-09-13T08:46:05.972296+0000","last_peered":"2024-09-13T08:46:05.972296+0000","last_clean":"2024-09-13T08:46:05.972296+0000","last_became_active":"2024-09-13T08:46:04.963814+0000","last_became_peered":"2024-09-13T08:46:04.963814+0000","last_unstale":"2024-09-13T08:46:05.972296+0000","last_undegraded":"2024-09-13T08:46:05.972296+0000","last_fullsized":"2024-09-13T08:46:05.972296+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.323368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950357+0000","last_change":"2024-09-13T08:46:05.971261+0000","last_active":"2024-09-13T08:46:06.950357+0000","last_peered":"2024-09-13T08:46:06.950357+0000","last_clean":"2024-09-13T08:46:06.950357+0000","last_became_active":"2024-09-13T08:46:05.970981+0000","last_became_peered":"2024-09-13T08:46:05.970981+0000","last_unstale":"2024-09-13T08:46:06.950357+0000","last_undegraded":"2024-09-13T08:46:06.950357+0000","last_fullsized":"2024-09-13T08:46:06.950357+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:48.409940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949552+0000","last_change":"2024-09-13T08:46:04.970084+0000","last_active":"2024-09-13T08:46:06.949552+0000","last_peered":"2024-09-13T08:46:06.949552+0000","last_clean":"2024-09-13T08:46:06.949552+0000","last_became_active":"2024-09-13T08:46:04.969769+0000","last_became_peered":"2024-09-13T08:46:04.969769+0000","last_unstale":"2024-09-13T08:46:06.949552+0000","last_undegraded":"2024-09-13T08:46:06.949552+0000","last_fullsized":"2024-09-13T08:46:06.949552+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:58:19.862815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947607+0000","last_change":"2024-09-13T08:46:05.964366+0000","last_active":"2024-09-13T08:46:06.947607+0000","last_peered":"2024-09-13T08:46:06.947607+0000","last_clean":"2024-09-13T08:46:06.947607+0000","last_became_active":"2024-09-13T08:46:05.964027+0000","last_became_peered":"2024-09-13T08:46:05.964027+0000","last_unstale":"2024-09-13T08:46:06.947607+0000","last_undegraded":"2024-09-13T08:46:06.947607+0000","last_fullsized":"2024-09-13T08:46:06.947607+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:58:20.869942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947652+0000","last_change":"2024-09-13T08:46:04.966271+0000","last_active":"2024-09-13T08:46:06.947652+0000","last_peered":"2024-09-13T08:46:06.947652+0000","last_clean":"2024-09-13T08:46:06.947652+0000","last_became_active":"2024-09-13T08:46:04.966004+0000","last_became_peered":"2024-09-13T08:46:04.966004+0000","last_unstale":"2024-09-13T08:46:06.947652+0000","last_undegraded":"2024-09-13T08:46:06.947652+0000","last_fullsized":"2024-09-13T08:46:06.947652+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:44.595928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973004+0000","last_change":"2024-09-13T08:46:05.962251+0000","last_active":"2024-09-13T08:46:05.973004+0000","last_peered":"2024-09-13T08:46:05.973004+0000","last_clean":"2024-09-13T08:46:05.973004+0000","last_became_active":"2024-09-13T08:46:05.961939+0000","last_became_peered":"2024-09-13T08:46:05.961939+0000","last_unstale":"2024-09-13T08:46:05.973004+0000","last_undegraded":"2024-09-13T08:46:05.973004+0000","last_fullsized":"2024-09-13T08:46:05.973004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:06:40.752264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947667+0000","last_change":"2024-09-13T08:46:04.963574+0000","last_active":"2024-09-13T08:46:06.947667+0000","last_peered":"2024-09-13T08:46:06.947667+0000","last_clean":"2024-09-13T08:46:06.947667+0000","last_became_active":"2024-09-13T08:46:04.963100+0000","last_became_peered":"2024-09-13T08:46:04.963100+0000","last_unstale":"2024-09-13T08:46:06.947667+0000","last_undegraded":"2024-09-13T08:46:06.947667+0000","last_fullsized":"2024-09-13T08:46:06.947667+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:56.120837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947540+0000","last_change":"2024-09-13T08:46:05.967572+0000","last_active":"2024-09-13T08:46:06.947540+0000","last_peered":"2024-09-13T08:46:06.947540+0000","last_clean":"2024-09-13T08:46:06.947540+0000","last_became_active":"2024-09-13T08:46:05.967267+0000","last_became_peered":"2024-09-13T08:46:05.967267+0000","last_unstale":"2024-09-13T08:46:06.947540+0000","last_undegraded":"2024-09-13T08:46:06.947540+0000","last_fullsized":"2024-09-13T08:46:06.947540+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:33.970003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947439+0000","last_change":"2024-09-13T08:46:04.956931+0000","last_active":"2024-09-13T08:46:06.947439+0000","last_peered":"2024-09-13T08:46:06.947439+0000","last_clean":"2024-09-13T08:46:06.947439+0000","last_became_active":"2024-09-13T08:46:04.956632+0000","last_became_peered":"2024-09-13T08:46:04.956632+0000","last_unstale":"2024-09-13T08:46:06.947439+0000","last_undegraded":"2024-09-13T08:46:06.947439+0000","last_fullsized":"2024-09-13T08:46:06.947439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:54.466875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949626+0000","last_change":"2024-09-13T08:46:05.966289+0000","last_active":"2024-09-13T08:46:06.949626+0000","last_peered":"2024-09-13T08:46:06.949626+0000","last_clean":"2024-09-13T08:46:06.949626+0000","last_became_active":"2024-09-13T08:46:05.966006+0000","last_became_peered":"2024-09-13T08:46:05.966006+0000","last_unstale":"2024-09-13T08:46:06.949626+0000","last_undegraded":"2024-09-13T08:46:06.949626+0000","last_fullsized":"2024-09-13T08:46:06.949626+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:28.532447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948957+0000","last_change":"2024-09-13T08:46:04.956538+0000","last_active":"2024-09-13T08:46:06.948957+0000","last_peered":"2024-09-13T08:46:06.948957+0000","last_clean":"2024-09-13T08:46:06.948957+0000","last_became_active":"2024-09-13T08:46:04.955654+0000","last_became_peered":"2024-09-13T08:46:04.955654+0000","last_unstale":"2024-09-13T08:46:06.948957+0000","last_undegraded":"2024-09-13T08:46:06.948957+0000","last_fullsized":"2024-09-13T08:46:06.948957+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:16.886655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948840+0000","last_change":"2024-09-13T08:46:05.957228+0000","last_active":"2024-09-13T08:46:06.948840+0000","last_peered":"2024-09-13T08:46:06.948840+0000","last_clean":"2024-09-13T08:46:06.948840+0000","last_became_active":"2024-09-13T08:46:05.956933+0000","last_became_peered":"2024-09-13T08:46:05.956933+0000","last_unstale":"2024-09-13T08:46:06.948840+0000","last_undegraded":"2024-09-13T08:46:06.948840+0000","last_fullsized":"2024-09-13T08:46:06.948840+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:07:36.059273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.946546+0000","last_change":"2024-09-13T08:46:04.965870+0000","last_active":"2024-09-13T08:46:05.946546+0000","last_peered":"2024-09-13T08:46:05.946546+0000","last_clean":"2024-09-13T08:46:05.946546+0000","last_became_active":"2024-09-13T08:46:04.965338+0000","last_became_peered":"2024-09-13T08:46:04.965338+0000","last_unstale":"2024-09-13T08:46:05.946546+0000","last_undegraded":"2024-09-13T08:46:05.946546+0000","last_fullsized":"2024-09-13T08:46:05.946546+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:10.888400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":397,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947367+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:06.947367+0000","last_peered":"2024-09-13T08:46:06.947367+0000","last_clean":"2024-09-13T08:46:06.947367+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:06.947367+0000","last_undegraded":"2024-09-13T08:46:06.947367+0000","last_fullsized":"2024-09-13T08:46:06.947367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37372,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334468,"statfs":{"total":96636764160,"available":96598495232,"internally_reserved":0,"allocated":2064384,"data_stored":907612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37352,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334488,"statfs":{"total":96636764160,"available":96598515712,"internally_reserved":0,"allocated":2048000,"data_stored":905978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37936,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333904,"statfs":{"total":96636764160,"available":96597917696,"internally_reserved":0,"allocated":2646016,"data_stored":1496860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542256,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41512,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330328,"statfs":{"total":96636764160,"available":96594255872,"internally_reserved":0,"allocated":2048000,"data_stored":905922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509553,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2060288,"data_stored":907570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2048000,"data_stored":905958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509554,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33768,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338072,"statfs":{"total":96636764160,"available":96602185728,"internally_reserved":0,"allocated":2637824,"data_stored":1496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509552,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":906504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:46:10.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 -- 172.21.15.31:0/18437901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4e4004800 msgr2=0x7fa4e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 --2- 172.21.15.31:0/18437901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4e4004800 0x7fa4e4004c80 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c290 tx=0x7fa4e4005830 comp rx=0 tx=0).stop 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4fe800640 1 -- 172.21.15.31:0/18437901 reap_dead start 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 -- 172.21.15.31:0/18437901 shutdown_connections 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 -- 172.21.15.31:0/18437901 >> 172.21.15.31:0/18437901 conn(0x7fa50006cb00 msgr2=0x7fa50006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 -- 172.21.15.31:0/18437901 shutdown_connections 2024-09-13T08:46:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.402+0000 7fa4ff200640 1 -- 172.21.15.31:0/18437901 wait complete. 2024-09-13T08:46:10.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.403+0000 7fa4ff200640 1 Processor -- start 2024-09-13T08:46:10.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.403+0000 7fa4ff200640 1 -- start start 2024-09-13T08:46:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 0x7fa5001f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 0x7fa500143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa50006f9b0 con 0x7fa5001452d0 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa500070460 con 0x7fa4e4004800 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.404+0000 7fa4ff200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa50006fb30 con 0x7fa500145c90 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37080/0 (socket says 172.21.15.31:37080) 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 -- 172.21.15.31:0/1175168809 learned_addr learned my addr 172.21.15.31:0/1175168809 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4fde00640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 0x7fa5001f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4fe800640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 0x7fa500143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 -- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 msgr2=0x7fa500143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 0x7fa500143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 -- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 msgr2=0x7fa5001f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 0x7fa5001f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 -- 172.21.15.31:0/1175168809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa500158100 con 0x7fa5001452d0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4fe800640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 0x7fa500143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4fde00640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 0x7fa5001f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.405+0000 7fa4f5400640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c3b0 tx=0x7fa4e8006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.406+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8031040 con 0x7fa5001452d0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.406+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4e80035f0 con 0x7fa5001452d0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.406+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8003a00 con 0x7fa5001452d0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.406+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5001583e0 con 0x7fa5001452d0 2024-09-13T08:46:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.406+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa500143520 con 0x7fa5001452d0 2024-09-13T08:46:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.407+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa500152680 con 0x7fa5001452d0 2024-09-13T08:46:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.408+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fa4e8034070 con 0x7fa5001452d0 2024-09-13T08:46:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.409+0000 7fa4f6a00640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 0x7fa4dc07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.409+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa4e80bba70 con 0x7fa5001452d0 2024-09-13T08:46:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.409+0000 7fa4fde00640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 0x7fa4dc07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.410+0000 7fa4fde00640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 0x7fa4dc07b580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fd0 tx=0x7fa4f0009500 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.413+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa4e8083080 con 0x7fa5001452d0 2024-09-13T08:46:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.606+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5000b3520 con 0x7fa4dc0790c0 2024-09-13T08:46:10.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.610+0000 7fa4f6a00640 1 -- 172.21.15.31:0/1175168809 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7fa5000b3520 con 0x7fa4dc0790c0 2024-09-13T08:46:10.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:10.617 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 msgr2=0x7fa4dc07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 0x7fa4dc07b580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fd0 tx=0x7fa4f0009500 comp rx=0 tx=0).stop 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 msgr2=0x7fa500145750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c3b0 tx=0x7fa4e8006d70 comp rx=0 tx=0).stop 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 shutdown_connections 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa4dc0790c0 0x7fa4dc07b580 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa500145c90 0x7fa500143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5001452d0 0x7fa500145750 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.614+0000 7fa4ff200640 1 --2- 172.21.15.31:0/1175168809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e4004800 0x7fa5001f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.615+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 >> 172.21.15.31:0/1175168809 conn(0x7fa50006cb00 msgr2=0x7fa5000c7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.615+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 shutdown_connections 2024-09-13T08:46:10.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.615+0000 7fa4ff200640 1 -- 172.21.15.31:0/1175168809 wait complete. 2024-09-13T08:46:10.633 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-13T08:46:09.011685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287528,"kb_used_data":17192,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754687192,"statfs":{"total":773094113280,"available":772799684608,"internally_reserved":0,"allocated":17604608,"data_stored":8432660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.077756"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977918+0000","last_change":"2024-09-13T08:46:05.975752+0000","last_active":"2024-09-13T08:46:05.977918+0000","last_peered":"2024-09-13T08:46:05.977918+0000","last_clean":"2024-09-13T08:46:05.977918+0000","last_became_active":"2024-09-13T08:46:05.975493+0000","last_became_peered":"2024-09-13T08:46:05.975493+0000","last_unstale":"2024-09-13T08:46:05.977918+0000","last_undegraded":"2024-09-13T08:46:05.977918+0000","last_fullsized":"2024-09-13T08:46:05.977918+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:06.844604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948447+0000","last_change":"2024-09-13T08:46:05.979888+0000","last_active":"2024-09-13T08:46:06.948447+0000","last_peered":"2024-09-13T08:46:06.948447+0000","last_clean":"2024-09-13T08:46:06.948447+0000","last_became_active":"2024-09-13T08:46:05.979607+0000","last_became_peered":"2024-09-13T08:46:05.979607+0000","last_unstale":"2024-09-13T08:46:06.948447+0000","last_undegraded":"2024-09-13T08:46:06.948447+0000","last_fullsized":"2024-09-13T08:46:06.948447+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:47.278648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987594+0000","last_change":"2024-09-13T08:46:05.985405+0000","last_active":"2024-09-13T08:46:05.987594+0000","last_peered":"2024-09-13T08:46:05.987594+0000","last_clean":"2024-09-13T08:46:05.987594+0000","last_became_active":"2024-09-13T08:46:05.984800+0000","last_became_peered":"2024-09-13T08:46:05.984800+0000","last_unstale":"2024-09-13T08:46:05.987594+0000","last_undegraded":"2024-09-13T08:46:05.987594+0000","last_fullsized":"2024-09-13T08:46:05.987594+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.895527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947885+0000","last_change":"2024-09-13T08:46:05.967792+0000","last_active":"2024-09-13T08:46:06.947885+0000","last_peered":"2024-09-13T08:46:06.947885+0000","last_clean":"2024-09-13T08:46:06.947885+0000","last_became_active":"2024-09-13T08:46:05.967483+0000","last_became_peered":"2024-09-13T08:46:05.967483+0000","last_unstale":"2024-09-13T08:46:06.947885+0000","last_undegraded":"2024-09-13T08:46:06.947885+0000","last_fullsized":"2024-09-13T08:46:06.947885+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:11.410024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949902+0000","last_change":"2024-09-13T08:46:05.970808+0000","last_active":"2024-09-13T08:46:06.949902+0000","last_peered":"2024-09-13T08:46:06.949902+0000","last_clean":"2024-09-13T08:46:06.949902+0000","last_became_active":"2024-09-13T08:46:05.970504+0000","last_became_peered":"2024-09-13T08:46:05.970504+0000","last_unstale":"2024-09-13T08:46:06.949902+0000","last_undegraded":"2024-09-13T08:46:06.949902+0000","last_fullsized":"2024-09-13T08:46:06.949902+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:03.960394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949527+0000","last_change":"2024-09-13T08:46:05.962854+0000","last_active":"2024-09-13T08:46:06.949527+0000","last_peered":"2024-09-13T08:46:06.949527+0000","last_clean":"2024-09-13T08:46:06.949527+0000","last_became_active":"2024-09-13T08:46:05.962575+0000","last_became_peered":"2024-09-13T08:46:05.962575+0000","last_unstale":"2024-09-13T08:46:06.949527+0000","last_undegraded":"2024-09-13T08:46:06.949527+0000","last_fullsized":"2024-09-13T08:46:06.949527+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:00:50.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977828+0000","last_change":"2024-09-13T08:46:05.970940+0000","last_active":"2024-09-13T08:46:05.977828+0000","last_peered":"2024-09-13T08:46:05.977828+0000","last_clean":"2024-09-13T08:46:05.977828+0000","last_became_active":"2024-09-13T08:46:05.966346+0000","last_became_peered":"2024-09-13T08:46:05.966346+0000","last_unstale":"2024-09-13T08:46:05.977828+0000","last_undegraded":"2024-09-13T08:46:05.977828+0000","last_fullsized":"2024-09-13T08:46:05.977828+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949004+0000","last_change":"2024-09-13T08:46:05.970763+0000","last_active":"2024-09-13T08:46:06.949004+0000","last_peered":"2024-09-13T08:46:06.949004+0000","last_clean":"2024-09-13T08:46:06.949004+0000","last_became_active":"2024-09-13T08:46:05.970021+0000","last_became_peered":"2024-09-13T08:46:05.970021+0000","last_unstale":"2024-09-13T08:46:06.949004+0000","last_undegraded":"2024-09-13T08:46:06.949004+0000","last_fullsized":"2024-09-13T08:46:06.949004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:01.985808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948400+0000","last_change":"2024-09-13T08:46:05.971432+0000","last_active":"2024-09-13T08:46:06.948400+0000","last_peered":"2024-09-13T08:46:06.948400+0000","last_clean":"2024-09-13T08:46:06.948400+0000","last_became_active":"2024-09-13T08:46:05.970556+0000","last_became_peered":"2024-09-13T08:46:05.970556+0000","last_unstale":"2024-09-13T08:46:06.948400+0000","last_undegraded":"2024-09-13T08:46:06.948400+0000","last_fullsized":"2024-09-13T08:46:06.948400+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:16.584690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979501+0000","last_change":"2024-09-13T08:46:05.977431+0000","last_active":"2024-09-13T08:46:05.979501+0000","last_peered":"2024-09-13T08:46:05.979501+0000","last_clean":"2024-09-13T08:46:05.979501+0000","last_became_active":"2024-09-13T08:46:05.976716+0000","last_became_peered":"2024-09-13T08:46:05.976716+0000","last_unstale":"2024-09-13T08:46:05.979501+0000","last_undegraded":"2024-09-13T08:46:05.979501+0000","last_fullsized":"2024-09-13T08:46:05.979501+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:53.808701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.975840+0000","last_change":"2024-09-13T08:46:05.974464+0000","last_active":"2024-09-13T08:46:05.975840+0000","last_peered":"2024-09-13T08:46:05.975840+0000","last_clean":"2024-09-13T08:46:05.975840+0000","last_became_active":"2024-09-13T08:46:05.974163+0000","last_became_peered":"2024-09-13T08:46:05.974163+0000","last_unstale":"2024-09-13T08:46:05.975840+0000","last_undegraded":"2024-09-13T08:46:05.975840+0000","last_fullsized":"2024-09-13T08:46:05.975840+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:59.885201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969437+0000","last_change":"2024-09-13T08:46:05.960936+0000","last_active":"2024-09-13T08:46:05.969437+0000","last_peered":"2024-09-13T08:46:05.969437+0000","last_clean":"2024-09-13T08:46:05.969437+0000","last_became_active":"2024-09-13T08:46:05.960634+0000","last_became_peered":"2024-09-13T08:46:05.960634+0000","last_unstale":"2024-09-13T08:46:05.969437+0000","last_undegraded":"2024-09-13T08:46:05.969437+0000","last_fullsized":"2024-09-13T08:46:05.969437+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:30:39.469478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979214+0000","last_change":"2024-09-13T08:46:05.977401+0000","last_active":"2024-09-13T08:46:05.979214+0000","last_peered":"2024-09-13T08:46:05.979214+0000","last_clean":"2024-09-13T08:46:05.979214+0000","last_became_active":"2024-09-13T08:46:05.977135+0000","last_became_peered":"2024-09-13T08:46:05.977135+0000","last_unstale":"2024-09-13T08:46:05.979214+0000","last_undegraded":"2024-09-13T08:46:05.979214+0000","last_fullsized":"2024-09-13T08:46:05.979214+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:07.103853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947544+0000","last_change":"2024-09-13T08:46:05.973530+0000","last_active":"2024-09-13T08:46:06.947544+0000","last_peered":"2024-09-13T08:46:06.947544+0000","last_clean":"2024-09-13T08:46:06.947544+0000","last_became_active":"2024-09-13T08:46:05.973253+0000","last_became_peered":"2024-09-13T08:46:05.973253+0000","last_unstale":"2024-09-13T08:46:06.947544+0000","last_undegraded":"2024-09-13T08:46:06.947544+0000","last_fullsized":"2024-09-13T08:46:06.947544+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:13:18.254730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977212+0000","last_change":"2024-09-13T08:46:05.972439+0000","last_active":"2024-09-13T08:46:05.977212+0000","last_peered":"2024-09-13T08:46:05.977212+0000","last_clean":"2024-09-13T08:46:05.977212+0000","last_became_active":"2024-09-13T08:46:05.972034+0000","last_became_peered":"2024-09-13T08:46:05.972034+0000","last_unstale":"2024-09-13T08:46:05.977212+0000","last_undegraded":"2024-09-13T08:46:05.977212+0000","last_fullsized":"2024-09-13T08:46:05.977212+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:19.611048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987547+0000","last_change":"2024-09-13T08:46:05.982404+0000","last_active":"2024-09-13T08:46:05.987547+0000","last_peered":"2024-09-13T08:46:05.987547+0000","last_clean":"2024-09-13T08:46:05.987547+0000","last_became_active":"2024-09-13T08:46:05.982084+0000","last_became_peered":"2024-09-13T08:46:05.982084+0000","last_unstale":"2024-09-13T08:46:05.987547+0000","last_undegraded":"2024-09-13T08:46:05.987547+0000","last_fullsized":"2024-09-13T08:46:05.987547+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950351+0000","last_change":"2024-09-13T08:46:05.978395+0000","last_active":"2024-09-13T08:46:06.950351+0000","last_peered":"2024-09-13T08:46:06.950351+0000","last_clean":"2024-09-13T08:46:06.950351+0000","last_became_active":"2024-09-13T08:46:05.978038+0000","last_became_peered":"2024-09-13T08:46:05.978038+0000","last_unstale":"2024-09-13T08:46:06.950351+0000","last_undegraded":"2024-09-13T08:46:06.950351+0000","last_fullsized":"2024-09-13T08:46:06.950351+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:28.765521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.978189+0000","last_change":"2024-09-13T08:46:05.972336+0000","last_active":"2024-09-13T08:46:05.978189+0000","last_peered":"2024-09-13T08:46:05.978189+0000","last_clean":"2024-09-13T08:46:05.978189+0000","last_became_active":"2024-09-13T08:46:05.972090+0000","last_became_peered":"2024-09-13T08:46:05.972090+0000","last_unstale":"2024-09-13T08:46:05.978189+0000","last_undegraded":"2024-09-13T08:46:05.978189+0000","last_fullsized":"2024-09-13T08:46:05.978189+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987210+0000","last_change":"2024-09-13T08:46:05.975851+0000","last_active":"2024-09-13T08:46:05.987210+0000","last_peered":"2024-09-13T08:46:05.987210+0000","last_clean":"2024-09-13T08:46:05.987210+0000","last_became_active":"2024-09-13T08:46:05.975542+0000","last_became_peered":"2024-09-13T08:46:05.975542+0000","last_unstale":"2024-09-13T08:46:05.987210+0000","last_undegraded":"2024-09-13T08:46:05.987210+0000","last_fullsized":"2024-09-13T08:46:05.987210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949333+0000","last_change":"2024-09-13T08:46:05.968613+0000","last_active":"2024-09-13T08:46:06.949333+0000","last_peered":"2024-09-13T08:46:06.949333+0000","last_clean":"2024-09-13T08:46:06.949333+0000","last_became_active":"2024-09-13T08:46:05.968257+0000","last_became_peered":"2024-09-13T08:46:05.968257+0000","last_unstale":"2024-09-13T08:46:06.949333+0000","last_undegraded":"2024-09-13T08:46:06.949333+0000","last_fullsized":"2024-09-13T08:46:06.949333+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:46.022535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.986720+0000","last_change":"2024-09-13T08:46:05.975824+0000","last_active":"2024-09-13T08:46:05.986720+0000","last_peered":"2024-09-13T08:46:05.986720+0000","last_clean":"2024-09-13T08:46:05.986720+0000","last_became_active":"2024-09-13T08:46:05.975309+0000","last_became_peered":"2024-09-13T08:46:05.975309+0000","last_unstale":"2024-09-13T08:46:05.986720+0000","last_undegraded":"2024-09-13T08:46:05.986720+0000","last_fullsized":"2024-09-13T08:46:05.986720+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950629+0000","last_change":"2024-09-13T08:46:05.980705+0000","last_active":"2024-09-13T08:46:06.950629+0000","last_peered":"2024-09-13T08:46:06.950629+0000","last_clean":"2024-09-13T08:46:06.950629+0000","last_became_active":"2024-09-13T08:46:05.979637+0000","last_became_peered":"2024-09-13T08:46:05.979637+0000","last_unstale":"2024-09-13T08:46:06.950629+0000","last_undegraded":"2024-09-13T08:46:06.950629+0000","last_fullsized":"2024-09-13T08:46:06.950629+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:06.065605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.982920+0000","last_change":"2024-09-13T08:46:05.980018+0000","last_active":"2024-09-13T08:46:05.982920+0000","last_peered":"2024-09-13T08:46:05.982920+0000","last_clean":"2024-09-13T08:46:05.982920+0000","last_became_active":"2024-09-13T08:46:05.979590+0000","last_became_peered":"2024-09-13T08:46:05.979590+0000","last_unstale":"2024-09-13T08:46:05.982920+0000","last_undegraded":"2024-09-13T08:46:05.982920+0000","last_fullsized":"2024-09-13T08:46:05.982920+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948875+0000","last_change":"2024-09-13T08:46:05.971069+0000","last_active":"2024-09-13T08:46:06.948875+0000","last_peered":"2024-09-13T08:46:06.948875+0000","last_clean":"2024-09-13T08:46:06.948875+0000","last_became_active":"2024-09-13T08:46:05.970462+0000","last_became_peered":"2024-09-13T08:46:05.970462+0000","last_unstale":"2024-09-13T08:46:06.948875+0000","last_undegraded":"2024-09-13T08:46:06.948875+0000","last_fullsized":"2024-09-13T08:46:06.948875+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:49:54.544335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948573+0000","last_change":"2024-09-13T08:46:05.982034+0000","last_active":"2024-09-13T08:46:06.948573+0000","last_peered":"2024-09-13T08:46:06.948573+0000","last_clean":"2024-09-13T08:46:06.948573+0000","last_became_active":"2024-09-13T08:46:05.981533+0000","last_became_peered":"2024-09-13T08:46:05.981533+0000","last_unstale":"2024-09-13T08:46:06.948573+0000","last_undegraded":"2024-09-13T08:46:06.948573+0000","last_fullsized":"2024-09-13T08:46:06.948573+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:43.470156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969533+0000","last_change":"2024-09-13T08:46:05.964100+0000","last_active":"2024-09-13T08:46:05.969533+0000","last_peered":"2024-09-13T08:46:05.969533+0000","last_clean":"2024-09-13T08:46:05.969533+0000","last_became_active":"2024-09-13T08:46:05.963842+0000","last_became_peered":"2024-09-13T08:46:05.963842+0000","last_unstale":"2024-09-13T08:46:05.969533+0000","last_undegraded":"2024-09-13T08:46:05.969533+0000","last_fullsized":"2024-09-13T08:46:05.969533+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:51:54.095289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.986610+0000","last_change":"2024-09-13T08:46:05.974979+0000","last_active":"2024-09-13T08:46:05.986610+0000","last_peered":"2024-09-13T08:46:05.986610+0000","last_clean":"2024-09-13T08:46:05.986610+0000","last_became_active":"2024-09-13T08:46:05.974620+0000","last_became_peered":"2024-09-13T08:46:05.974620+0000","last_unstale":"2024-09-13T08:46:05.986610+0000","last_undegraded":"2024-09-13T08:46:05.986610+0000","last_fullsized":"2024-09-13T08:46:05.986610+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:26.670968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981122+0000","last_change":"2024-09-13T08:46:05.978596+0000","last_active":"2024-09-13T08:46:05.981122+0000","last_peered":"2024-09-13T08:46:05.981122+0000","last_clean":"2024-09-13T08:46:05.981122+0000","last_became_active":"2024-09-13T08:46:05.978014+0000","last_became_peered":"2024-09-13T08:46:05.978014+0000","last_unstale":"2024-09-13T08:46:05.981122+0000","last_undegraded":"2024-09-13T08:46:05.981122+0000","last_fullsized":"2024-09-13T08:46:05.981122+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.983998+0000","last_change":"2024-09-13T08:46:05.978488+0000","last_active":"2024-09-13T08:46:05.983998+0000","last_peered":"2024-09-13T08:46:05.983998+0000","last_clean":"2024-09-13T08:46:05.983998+0000","last_became_active":"2024-09-13T08:46:05.976693+0000","last_became_peered":"2024-09-13T08:46:05.976693+0000","last_unstale":"2024-09-13T08:46:05.983998+0000","last_undegraded":"2024-09-13T08:46:05.983998+0000","last_fullsized":"2024-09-13T08:46:05.983998+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:59.713194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958466+0000","last_change":"2024-09-13T08:46:05.956565+0000","last_active":"2024-09-13T08:46:05.958466+0000","last_peered":"2024-09-13T08:46:05.958466+0000","last_clean":"2024-09-13T08:46:05.958466+0000","last_became_active":"2024-09-13T08:46:05.956157+0000","last_became_peered":"2024-09-13T08:46:05.956157+0000","last_unstale":"2024-09-13T08:46:05.958466+0000","last_undegraded":"2024-09-13T08:46:05.958466+0000","last_fullsized":"2024-09-13T08:46:05.958466+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:10.934759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948951+0000","last_change":"2024-09-13T08:46:05.964483+0000","last_active":"2024-09-13T08:46:06.948951+0000","last_peered":"2024-09-13T08:46:06.948951+0000","last_clean":"2024-09-13T08:46:06.948951+0000","last_became_active":"2024-09-13T08:46:05.964256+0000","last_became_peered":"2024-09-13T08:46:05.964256+0000","last_unstale":"2024-09-13T08:46:06.948951+0000","last_undegraded":"2024-09-13T08:46:06.948951+0000","last_fullsized":"2024-09-13T08:46:06.948951+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:46.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949159+0000","last_change":"2024-09-13T08:46:05.971630+0000","last_active":"2024-09-13T08:46:06.949159+0000","last_peered":"2024-09-13T08:46:06.949159+0000","last_clean":"2024-09-13T08:46:06.949159+0000","last_became_active":"2024-09-13T08:46:05.971352+0000","last_became_peered":"2024-09-13T08:46:05.971352+0000","last_unstale":"2024-09-13T08:46:06.949159+0000","last_undegraded":"2024-09-13T08:46:06.949159+0000","last_fullsized":"2024-09-13T08:46:06.949159+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:48.134772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949506+0000","last_change":"2024-09-13T08:46:05.962568+0000","last_active":"2024-09-13T08:46:06.949506+0000","last_peered":"2024-09-13T08:46:06.949506+0000","last_clean":"2024-09-13T08:46:06.949506+0000","last_became_active":"2024-09-13T08:46:05.957696+0000","last_became_peered":"2024-09-13T08:46:05.957696+0000","last_unstale":"2024-09-13T08:46:06.949506+0000","last_undegraded":"2024-09-13T08:46:06.949506+0000","last_fullsized":"2024-09-13T08:46:06.949506+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:08.697934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"163'4","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.988521+0000","last_change":"2024-09-13T08:46:04.959656+0000","last_active":"2024-09-13T08:46:05.988521+0000","last_peered":"2024-09-13T08:46:05.988521+0000","last_clean":"2024-09-13T08:46:05.988521+0000","last_became_active":"2024-09-13T08:46:04.959376+0000","last_became_peered":"2024-09-13T08:46:04.959376+0000","last_unstale":"2024-09-13T08:46:05.988521+0000","last_undegraded":"2024-09-13T08:46:05.988521+0000","last_fullsized":"2024-09-13T08:46:05.988521+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:52.801573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.985208+0000","last_change":"2024-09-13T08:46:05.984248+0000","last_active":"2024-09-13T08:46:05.985208+0000","last_peered":"2024-09-13T08:46:05.985208+0000","last_clean":"2024-09-13T08:46:05.985208+0000","last_became_active":"2024-09-13T08:46:05.983952+0000","last_became_peered":"2024-09-13T08:46:05.983952+0000","last_unstale":"2024-09-13T08:46:05.985208+0000","last_undegraded":"2024-09-13T08:46:05.985208+0000","last_fullsized":"2024-09-13T08:46:05.985208+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"163'1","reported_seq":14,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.970822+0000","last_change":"2024-09-13T08:46:04.970027+0000","last_active":"2024-09-13T08:46:05.970822+0000","last_peered":"2024-09-13T08:46:05.970822+0000","last_clean":"2024-09-13T08:46:05.970822+0000","last_became_active":"2024-09-13T08:46:04.969527+0000","last_became_peered":"2024-09-13T08:46:04.969527+0000","last_unstale":"2024-09-13T08:46:05.970822+0000","last_undegraded":"2024-09-13T08:46:05.970822+0000","last_fullsized":"2024-09-13T08:46:05.970822+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:54:31.922912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947864+0000","last_change":"2024-09-13T08:46:05.978601+0000","last_active":"2024-09-13T08:46:06.947864+0000","last_peered":"2024-09-13T08:46:06.947864+0000","last_clean":"2024-09-13T08:46:06.947864+0000","last_became_active":"2024-09-13T08:46:05.975598+0000","last_became_peered":"2024-09-13T08:46:05.975598+0000","last_unstale":"2024-09-13T08:46:06.947864+0000","last_undegraded":"2024-09-13T08:46:06.947864+0000","last_fullsized":"2024-09-13T08:46:06.947864+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:30.926174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950195+0000","last_change":"2024-09-13T08:46:04.967771+0000","last_active":"2024-09-13T08:46:06.950195+0000","last_peered":"2024-09-13T08:46:06.950195+0000","last_clean":"2024-09-13T08:46:06.950195+0000","last_became_active":"2024-09-13T08:46:04.966897+0000","last_became_peered":"2024-09-13T08:46:04.966897+0000","last_unstale":"2024-09-13T08:46:06.950195+0000","last_undegraded":"2024-09-13T08:46:06.950195+0000","last_fullsized":"2024-09-13T08:46:06.950195+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:29.919047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948296+0000","last_change":"2024-09-13T08:46:05.979678+0000","last_active":"2024-09-13T08:46:06.948296+0000","last_peered":"2024-09-13T08:46:06.948296+0000","last_clean":"2024-09-13T08:46:06.948296+0000","last_became_active":"2024-09-13T08:46:05.979385+0000","last_became_peered":"2024-09-13T08:46:05.979385+0000","last_unstale":"2024-09-13T08:46:06.948296+0000","last_undegraded":"2024-09-13T08:46:06.948296+0000","last_fullsized":"2024-09-13T08:46:06.948296+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:29.866552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977629+0000","last_change":"2024-09-13T08:46:04.969348+0000","last_active":"2024-09-13T08:46:05.977629+0000","last_peered":"2024-09-13T08:46:05.977629+0000","last_clean":"2024-09-13T08:46:05.977629+0000","last_became_active":"2024-09-13T08:46:04.968942+0000","last_became_peered":"2024-09-13T08:46:04.968942+0000","last_unstale":"2024-09-13T08:46:05.977629+0000","last_undegraded":"2024-09-13T08:46:05.977629+0000","last_fullsized":"2024-09-13T08:46:05.977629+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:54:12.768339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950635+0000","last_change":"2024-09-13T08:46:05.960209+0000","last_active":"2024-09-13T08:46:06.950635+0000","last_peered":"2024-09-13T08:46:06.950635+0000","last_clean":"2024-09-13T08:46:06.950635+0000","last_became_active":"2024-09-13T08:46:05.959911+0000","last_became_peered":"2024-09-13T08:46:05.959911+0000","last_unstale":"2024-09-13T08:46:06.950635+0000","last_undegraded":"2024-09-13T08:46:06.950635+0000","last_fullsized":"2024-09-13T08:46:06.950635+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:28:50.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950126+0000","last_change":"2024-09-13T08:46:04.965445+0000","last_active":"2024-09-13T08:46:06.950126+0000","last_peered":"2024-09-13T08:46:06.950126+0000","last_clean":"2024-09-13T08:46:06.950126+0000","last_became_active":"2024-09-13T08:46:04.964172+0000","last_became_peered":"2024-09-13T08:46:04.964172+0000","last_unstale":"2024-09-13T08:46:06.950126+0000","last_undegraded":"2024-09-13T08:46:06.950126+0000","last_fullsized":"2024-09-13T08:46:06.950126+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:47:28.859425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950503+0000","last_change":"2024-09-13T08:46:05.970369+0000","last_active":"2024-09-13T08:46:06.950503+0000","last_peered":"2024-09-13T08:46:06.950503+0000","last_clean":"2024-09-13T08:46:06.950503+0000","last_became_active":"2024-09-13T08:46:05.970068+0000","last_became_peered":"2024-09-13T08:46:05.970068+0000","last_unstale":"2024-09-13T08:46:06.950503+0000","last_undegraded":"2024-09-13T08:46:06.950503+0000","last_fullsized":"2024-09-13T08:46:06.950503+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:39.954828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950708+0000","last_change":"2024-09-13T08:46:04.968963+0000","last_active":"2024-09-13T08:46:06.950708+0000","last_peered":"2024-09-13T08:46:06.950708+0000","last_clean":"2024-09-13T08:46:06.950708+0000","last_became_active":"2024-09-13T08:46:04.968673+0000","last_became_peered":"2024-09-13T08:46:04.968673+0000","last_unstale":"2024-09-13T08:46:06.950708+0000","last_undegraded":"2024-09-13T08:46:06.950708+0000","last_fullsized":"2024-09-13T08:46:06.950708+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:32.962876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.961006+0000","last_change":"2024-09-13T08:46:05.959266+0000","last_active":"2024-09-13T08:46:05.961006+0000","last_peered":"2024-09-13T08:46:05.961006+0000","last_clean":"2024-09-13T08:46:05.961006+0000","last_became_active":"2024-09-13T08:46:05.958950+0000","last_became_peered":"2024-09-13T08:46:05.958950+0000","last_unstale":"2024-09-13T08:46:05.961006+0000","last_undegraded":"2024-09-13T08:46:05.961006+0000","last_fullsized":"2024-09-13T08:46:05.961006+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:06.714072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.951701+0000","last_change":"2024-09-13T08:46:04.959351+0000","last_active":"2024-09-13T08:46:05.951701+0000","last_peered":"2024-09-13T08:46:05.951701+0000","last_clean":"2024-09-13T08:46:05.951701+0000","last_became_active":"2024-09-13T08:46:04.958933+0000","last_became_peered":"2024-09-13T08:46:04.958933+0000","last_unstale":"2024-09-13T08:46:05.951701+0000","last_undegraded":"2024-09-13T08:46:05.951701+0000","last_fullsized":"2024-09-13T08:46:05.951701+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:52.801573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987490+0000","last_change":"2024-09-13T08:46:05.982051+0000","last_active":"2024-09-13T08:46:05.987490+0000","last_peered":"2024-09-13T08:46:05.987490+0000","last_clean":"2024-09-13T08:46:05.987490+0000","last_became_active":"2024-09-13T08:46:05.975965+0000","last_became_peered":"2024-09-13T08:46:05.975965+0000","last_unstale":"2024-09-13T08:46:05.987490+0000","last_undegraded":"2024-09-13T08:46:05.987490+0000","last_fullsized":"2024-09-13T08:46:05.987490+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:29.599131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.957931+0000","last_change":"2024-09-13T08:46:04.967778+0000","last_active":"2024-09-13T08:46:05.957931+0000","last_peered":"2024-09-13T08:46:05.957931+0000","last_clean":"2024-09-13T08:46:05.957931+0000","last_became_active":"2024-09-13T08:46:04.967415+0000","last_became_peered":"2024-09-13T08:46:04.967415+0000","last_unstale":"2024-09-13T08:46:05.957931+0000","last_undegraded":"2024-09-13T08:46:05.957931+0000","last_fullsized":"2024-09-13T08:46:05.957931+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:06.738410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.979070+0000","last_change":"2024-09-13T08:46:05.976820+0000","last_active":"2024-09-13T08:46:05.979070+0000","last_peered":"2024-09-13T08:46:05.979070+0000","last_clean":"2024-09-13T08:46:05.979070+0000","last_became_active":"2024-09-13T08:46:05.976300+0000","last_became_peered":"2024-09-13T08:46:05.976300+0000","last_unstale":"2024-09-13T08:46:05.979070+0000","last_undegraded":"2024-09-13T08:46:05.979070+0000","last_fullsized":"2024-09-13T08:46:05.979070+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:26.670968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973410+0000","last_change":"2024-09-13T08:46:04.969092+0000","last_active":"2024-09-13T08:46:05.973410+0000","last_peered":"2024-09-13T08:46:05.973410+0000","last_clean":"2024-09-13T08:46:05.973410+0000","last_became_active":"2024-09-13T08:46:04.968791+0000","last_became_peered":"2024-09-13T08:46:04.968791+0000","last_unstale":"2024-09-13T08:46:05.973410+0000","last_undegraded":"2024-09-13T08:46:05.973410+0000","last_fullsized":"2024-09-13T08:46:05.973410+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:57.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972635+0000","last_change":"2024-09-13T08:46:05.965346+0000","last_active":"2024-09-13T08:46:05.972635+0000","last_peered":"2024-09-13T08:46:05.972635+0000","last_clean":"2024-09-13T08:46:05.972635+0000","last_became_active":"2024-09-13T08:46:05.965035+0000","last_became_peered":"2024-09-13T08:46:05.965035+0000","last_unstale":"2024-09-13T08:46:05.972635+0000","last_undegraded":"2024-09-13T08:46:05.972635+0000","last_fullsized":"2024-09-13T08:46:05.972635+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:17:16.009417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"163'3","reported_seq":18,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950327+0000","last_change":"2024-09-13T08:46:04.969855+0000","last_active":"2024-09-13T08:46:06.950327+0000","last_peered":"2024-09-13T08:46:06.950327+0000","last_clean":"2024-09-13T08:46:06.950327+0000","last_became_active":"2024-09-13T08:46:04.968696+0000","last_became_peered":"2024-09-13T08:46:04.968696+0000","last_unstale":"2024-09-13T08:46:06.950327+0000","last_undegraded":"2024-09-13T08:46:06.950327+0000","last_fullsized":"2024-09-13T08:46:06.950327+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:01.929681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987835+0000","last_change":"2024-09-13T08:46:05.987259+0000","last_active":"2024-09-13T08:46:05.987835+0000","last_peered":"2024-09-13T08:46:05.987835+0000","last_clean":"2024-09-13T08:46:05.987835+0000","last_became_active":"2024-09-13T08:46:05.986922+0000","last_became_peered":"2024-09-13T08:46:05.986922+0000","last_unstale":"2024-09-13T08:46:05.987835+0000","last_undegraded":"2024-09-13T08:46:05.987835+0000","last_fullsized":"2024-09-13T08:46:05.987835+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:49:55.852849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"163'6","reported_seq":21,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949450+0000","last_change":"2024-09-13T08:46:04.964670+0000","last_active":"2024-09-13T08:46:06.949450+0000","last_peered":"2024-09-13T08:46:06.949450+0000","last_clean":"2024-09-13T08:46:06.949450+0000","last_became_active":"2024-09-13T08:46:04.963252+0000","last_became_peered":"2024-09-13T08:46:04.963252+0000","last_unstale":"2024-09-13T08:46:06.949450+0000","last_undegraded":"2024-09-13T08:46:06.949450+0000","last_fullsized":"2024-09-13T08:46:06.949450+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.577563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981188+0000","last_change":"2024-09-13T08:46:05.978698+0000","last_active":"2024-09-13T08:46:05.981188+0000","last_peered":"2024-09-13T08:46:05.981188+0000","last_clean":"2024-09-13T08:46:05.981188+0000","last_became_active":"2024-09-13T08:46:05.978445+0000","last_became_peered":"2024-09-13T08:46:05.978445+0000","last_unstale":"2024-09-13T08:46:05.981188+0000","last_undegraded":"2024-09-13T08:46:05.981188+0000","last_fullsized":"2024-09-13T08:46:05.981188+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:25:09.259198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981904+0000","last_change":"2024-09-13T08:46:04.957718+0000","last_active":"2024-09-13T08:46:05.981904+0000","last_peered":"2024-09-13T08:46:05.981904+0000","last_clean":"2024-09-13T08:46:05.981904+0000","last_became_active":"2024-09-13T08:46:04.957398+0000","last_became_peered":"2024-09-13T08:46:04.957398+0000","last_unstale":"2024-09-13T08:46:05.981904+0000","last_undegraded":"2024-09-13T08:46:05.981904+0000","last_fullsized":"2024-09-13T08:46:05.981904+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:08.111988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972044+0000","last_change":"2024-09-13T08:46:05.963549+0000","last_active":"2024-09-13T08:46:05.972044+0000","last_peered":"2024-09-13T08:46:05.972044+0000","last_clean":"2024-09-13T08:46:05.972044+0000","last_became_active":"2024-09-13T08:46:05.963280+0000","last_became_peered":"2024-09-13T08:46:05.963280+0000","last_unstale":"2024-09-13T08:46:05.972044+0000","last_undegraded":"2024-09-13T08:46:05.972044+0000","last_fullsized":"2024-09-13T08:46:05.972044+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.075844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973151+0000","last_change":"2024-09-13T08:46:04.965004+0000","last_active":"2024-09-13T08:46:05.973151+0000","last_peered":"2024-09-13T08:46:05.973151+0000","last_clean":"2024-09-13T08:46:05.973151+0000","last_became_active":"2024-09-13T08:46:04.964507+0000","last_became_peered":"2024-09-13T08:46:04.964507+0000","last_unstale":"2024-09-13T08:46:05.973151+0000","last_undegraded":"2024-09-13T08:46:05.973151+0000","last_fullsized":"2024-09-13T08:46:05.973151+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:18.469821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958190+0000","last_change":"2024-09-13T08:46:05.955400+0000","last_active":"2024-09-13T08:46:05.958190+0000","last_peered":"2024-09-13T08:46:05.958190+0000","last_clean":"2024-09-13T08:46:05.958190+0000","last_became_active":"2024-09-13T08:46:05.955064+0000","last_became_peered":"2024-09-13T08:46:05.955064+0000","last_unstale":"2024-09-13T08:46:05.958190+0000","last_undegraded":"2024-09-13T08:46:05.958190+0000","last_fullsized":"2024-09-13T08:46:05.958190+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:04.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949838+0000","last_change":"2024-09-13T08:46:04.969712+0000","last_active":"2024-09-13T08:46:06.949838+0000","last_peered":"2024-09-13T08:46:06.949838+0000","last_clean":"2024-09-13T08:46:06.949838+0000","last_became_active":"2024-09-13T08:46:04.969436+0000","last_became_peered":"2024-09-13T08:46:04.969436+0000","last_unstale":"2024-09-13T08:46:06.949838+0000","last_undegraded":"2024-09-13T08:46:06.949838+0000","last_fullsized":"2024-09-13T08:46:06.949838+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:09:03.013668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950637+0000","last_change":"2024-09-13T08:46:05.971981+0000","last_active":"2024-09-13T08:46:06.950637+0000","last_peered":"2024-09-13T08:46:06.950637+0000","last_clean":"2024-09-13T08:46:06.950637+0000","last_became_active":"2024-09-13T08:46:05.971680+0000","last_became_peered":"2024-09-13T08:46:05.971680+0000","last_unstale":"2024-09-13T08:46:06.950637+0000","last_undegraded":"2024-09-13T08:46:06.950637+0000","last_fullsized":"2024-09-13T08:46:06.950637+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.033016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"163'8","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.985245+0000","last_change":"2024-09-13T08:46:04.969610+0000","last_active":"2024-09-13T08:46:05.985245+0000","last_peered":"2024-09-13T08:46:05.985245+0000","last_clean":"2024-09-13T08:46:05.985245+0000","last_became_active":"2024-09-13T08:46:04.969318+0000","last_became_peered":"2024-09-13T08:46:04.969318+0000","last_unstale":"2024-09-13T08:46:05.985245+0000","last_undegraded":"2024-09-13T08:46:05.985245+0000","last_fullsized":"2024-09-13T08:46:05.985245+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:25.663841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.982990+0000","last_change":"2024-09-13T08:46:05.980699+0000","last_active":"2024-09-13T08:46:05.982990+0000","last_peered":"2024-09-13T08:46:05.982990+0000","last_clean":"2024-09-13T08:46:05.982990+0000","last_became_active":"2024-09-13T08:46:05.979675+0000","last_became_peered":"2024-09-13T08:46:05.979675+0000","last_unstale":"2024-09-13T08:46:05.982990+0000","last_undegraded":"2024-09-13T08:46:05.982990+0000","last_fullsized":"2024-09-13T08:46:05.982990+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"163'1","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950143+0000","last_change":"2024-09-13T08:46:04.968628+0000","last_active":"2024-09-13T08:46:06.950143+0000","last_peered":"2024-09-13T08:46:06.950143+0000","last_clean":"2024-09-13T08:46:06.950143+0000","last_became_active":"2024-09-13T08:46:04.968084+0000","last_became_peered":"2024-09-13T08:46:04.968084+0000","last_unstale":"2024-09-13T08:46:06.950143+0000","last_undegraded":"2024-09-13T08:46:06.950143+0000","last_fullsized":"2024-09-13T08:46:06.950143+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.754569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.959042+0000","last_change":"2024-09-13T08:46:05.957318+0000","last_active":"2024-09-13T08:46:05.959042+0000","last_peered":"2024-09-13T08:46:05.959042+0000","last_clean":"2024-09-13T08:46:05.959042+0000","last_became_active":"2024-09-13T08:46:05.952788+0000","last_became_peered":"2024-09-13T08:46:05.952788+0000","last_unstale":"2024-09-13T08:46:05.959042+0000","last_undegraded":"2024-09-13T08:46:05.959042+0000","last_fullsized":"2024-09-13T08:46:05.959042+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:30.673880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950262+0000","last_change":"2024-09-13T08:46:04.953330+0000","last_active":"2024-09-13T08:46:06.950262+0000","last_peered":"2024-09-13T08:46:06.950262+0000","last_clean":"2024-09-13T08:46:06.950262+0000","last_became_active":"2024-09-13T08:46:04.952883+0000","last_became_peered":"2024-09-13T08:46:04.952883+0000","last_unstale":"2024-09-13T08:46:06.950262+0000","last_undegraded":"2024-09-13T08:46:06.950262+0000","last_fullsized":"2024-09-13T08:46:06.950262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:28.617454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948065+0000","last_change":"2024-09-13T08:46:05.971314+0000","last_active":"2024-09-13T08:46:06.948065+0000","last_peered":"2024-09-13T08:46:06.948065+0000","last_clean":"2024-09-13T08:46:06.948065+0000","last_became_active":"2024-09-13T08:46:05.970653+0000","last_became_peered":"2024-09-13T08:46:05.970653+0000","last_unstale":"2024-09-13T08:46:06.948065+0000","last_undegraded":"2024-09-13T08:46:06.948065+0000","last_fullsized":"2024-09-13T08:46:06.948065+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:52.842034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"163'1","reported_seq":14,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.976432+0000","last_change":"2024-09-13T08:46:04.962474+0000","last_active":"2024-09-13T08:46:05.976432+0000","last_peered":"2024-09-13T08:46:05.976432+0000","last_clean":"2024-09-13T08:46:05.976432+0000","last_became_active":"2024-09-13T08:46:04.962068+0000","last_became_peered":"2024-09-13T08:46:04.962068+0000","last_unstale":"2024-09-13T08:46:05.976432+0000","last_undegraded":"2024-09-13T08:46:05.976432+0000","last_fullsized":"2024-09-13T08:46:05.976432+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:16.918066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.983799+0000","last_change":"2024-09-13T08:46:05.976559+0000","last_active":"2024-09-13T08:46:05.983799+0000","last_peered":"2024-09-13T08:46:05.983799+0000","last_clean":"2024-09-13T08:46:05.983799+0000","last_became_active":"2024-09-13T08:46:05.976111+0000","last_became_peered":"2024-09-13T08:46:05.976111+0000","last_unstale":"2024-09-13T08:46:05.983799+0000","last_undegraded":"2024-09-13T08:46:05.983799+0000","last_fullsized":"2024-09-13T08:46:05.983799+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:03:27.370693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.958405+0000","last_change":"2024-09-13T08:46:04.955393+0000","last_active":"2024-09-13T08:46:05.958405+0000","last_peered":"2024-09-13T08:46:05.958405+0000","last_clean":"2024-09-13T08:46:05.958405+0000","last_became_active":"2024-09-13T08:46:04.954351+0000","last_became_peered":"2024-09-13T08:46:04.954351+0000","last_unstale":"2024-09-13T08:46:05.958405+0000","last_undegraded":"2024-09-13T08:46:05.958405+0000","last_fullsized":"2024-09-13T08:46:05.958405+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:05:45.276045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950299+0000","last_change":"2024-09-13T08:46:05.973090+0000","last_active":"2024-09-13T08:46:06.950299+0000","last_peered":"2024-09-13T08:46:06.950299+0000","last_clean":"2024-09-13T08:46:06.950299+0000","last_became_active":"2024-09-13T08:46:05.972817+0000","last_became_peered":"2024-09-13T08:46:05.972817+0000","last_unstale":"2024-09-13T08:46:06.950299+0000","last_undegraded":"2024-09-13T08:46:06.950299+0000","last_fullsized":"2024-09-13T08:46:06.950299+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:34.985230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"163'2","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950251+0000","last_change":"2024-09-13T08:46:04.964211+0000","last_active":"2024-09-13T08:46:06.950251+0000","last_peered":"2024-09-13T08:46:06.950251+0000","last_clean":"2024-09-13T08:46:06.950251+0000","last_became_active":"2024-09-13T08:46:04.963850+0000","last_became_peered":"2024-09-13T08:46:04.963850+0000","last_unstale":"2024-09-13T08:46:06.950251+0000","last_undegraded":"2024-09-13T08:46:06.950251+0000","last_fullsized":"2024-09-13T08:46:06.950251+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:51.834907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.981478+0000","last_change":"2024-09-13T08:46:05.979491+0000","last_active":"2024-09-13T08:46:05.981478+0000","last_peered":"2024-09-13T08:46:05.981478+0000","last_clean":"2024-09-13T08:46:05.981478+0000","last_became_active":"2024-09-13T08:46:05.979198+0000","last_became_peered":"2024-09-13T08:46:05.979198+0000","last_unstale":"2024-09-13T08:46:05.981478+0000","last_undegraded":"2024-09-13T08:46:05.981478+0000","last_fullsized":"2024-09-13T08:46:05.981478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950429+0000","last_change":"2024-09-13T08:46:04.964090+0000","last_active":"2024-09-13T08:46:06.950429+0000","last_peered":"2024-09-13T08:46:06.950429+0000","last_clean":"2024-09-13T08:46:06.950429+0000","last_became_active":"2024-09-13T08:46:04.963155+0000","last_became_peered":"2024-09-13T08:46:04.963155+0000","last_unstale":"2024-09-13T08:46:06.950429+0000","last_undegraded":"2024-09-13T08:46:06.950429+0000","last_fullsized":"2024-09-13T08:46:06.950429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:18:09.679957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.959983+0000","last_change":"2024-09-13T08:46:05.957664+0000","last_active":"2024-09-13T08:46:05.959983+0000","last_peered":"2024-09-13T08:46:05.959983+0000","last_clean":"2024-09-13T08:46:05.959983+0000","last_became_active":"2024-09-13T08:46:05.957112+0000","last_became_peered":"2024-09-13T08:46:05.957112+0000","last_unstale":"2024-09-13T08:46:05.959983+0000","last_undegraded":"2024-09-13T08:46:05.959983+0000","last_fullsized":"2024-09-13T08:46:05.959983+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:51:53.500313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949467+0000","last_change":"2024-09-13T08:46:04.957696+0000","last_active":"2024-09-13T08:46:06.949467+0000","last_peered":"2024-09-13T08:46:06.949467+0000","last_clean":"2024-09-13T08:46:06.949467+0000","last_became_active":"2024-09-13T08:46:04.957325+0000","last_became_peered":"2024-09-13T08:46:04.957325+0000","last_unstale":"2024-09-13T08:46:06.949467+0000","last_undegraded":"2024-09-13T08:46:06.949467+0000","last_fullsized":"2024-09-13T08:46:06.949467+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.386989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.977373+0000","last_change":"2024-09-13T08:46:05.974670+0000","last_active":"2024-09-13T08:46:05.977373+0000","last_peered":"2024-09-13T08:46:05.977373+0000","last_clean":"2024-09-13T08:46:05.977373+0000","last_became_active":"2024-09-13T08:46:05.974390+0000","last_became_peered":"2024-09-13T08:46:05.974390+0000","last_unstale":"2024-09-13T08:46:05.977373+0000","last_undegraded":"2024-09-13T08:46:05.977373+0000","last_fullsized":"2024-09-13T08:46:05.977373+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:44.330496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.957487+0000","last_change":"2024-09-13T08:46:04.956148+0000","last_active":"2024-09-13T08:46:05.957487+0000","last_peered":"2024-09-13T08:46:05.957487+0000","last_clean":"2024-09-13T08:46:05.957487+0000","last_became_active":"2024-09-13T08:46:04.955866+0000","last_became_peered":"2024-09-13T08:46:04.955866+0000","last_unstale":"2024-09-13T08:46:05.957487+0000","last_undegraded":"2024-09-13T08:46:05.957487+0000","last_fullsized":"2024-09-13T08:46:05.957487+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:17.889992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987677+0000","last_change":"2024-09-13T08:46:05.985768+0000","last_active":"2024-09-13T08:46:05.987677+0000","last_peered":"2024-09-13T08:46:05.987677+0000","last_clean":"2024-09-13T08:46:05.987677+0000","last_became_active":"2024-09-13T08:46:05.982606+0000","last_became_peered":"2024-09-13T08:46:05.982606+0000","last_unstale":"2024-09-13T08:46:05.987677+0000","last_undegraded":"2024-09-13T08:46:05.987677+0000","last_fullsized":"2024-09-13T08:46:05.987677+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:09.119115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"163'2","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.978776+0000","last_change":"2024-09-13T08:46:04.956637+0000","last_active":"2024-09-13T08:46:05.978776+0000","last_peered":"2024-09-13T08:46:05.978776+0000","last_clean":"2024-09-13T08:46:05.978776+0000","last_became_active":"2024-09-13T08:46:04.956039+0000","last_became_peered":"2024-09-13T08:46:04.956039+0000","last_unstale":"2024-09-13T08:46:05.978776+0000","last_undegraded":"2024-09-13T08:46:05.978776+0000","last_fullsized":"2024-09-13T08:46:05.978776+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:03:32.467054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.988087+0000","last_change":"2024-09-13T08:46:05.985037+0000","last_active":"2024-09-13T08:46:05.988087+0000","last_peered":"2024-09-13T08:46:05.988087+0000","last_clean":"2024-09-13T08:46:05.988087+0000","last_became_active":"2024-09-13T08:46:05.981632+0000","last_became_peered":"2024-09-13T08:46:05.981632+0000","last_unstale":"2024-09-13T08:46:05.988087+0000","last_undegraded":"2024-09-13T08:46:05.988087+0000","last_fullsized":"2024-09-13T08:46:05.988087+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:49:55.852849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.969463+0000","last_change":"2024-09-13T08:46:04.968652+0000","last_active":"2024-09-13T08:46:05.969463+0000","last_peered":"2024-09-13T08:46:05.969463+0000","last_clean":"2024-09-13T08:46:05.969463+0000","last_became_active":"2024-09-13T08:46:04.968162+0000","last_became_peered":"2024-09-13T08:46:04.968162+0000","last_unstale":"2024-09-13T08:46:05.969463+0000","last_undegraded":"2024-09-13T08:46:05.969463+0000","last_fullsized":"2024-09-13T08:46:05.969463+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:57.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.987080+0000","last_change":"2024-09-13T08:46:05.982162+0000","last_active":"2024-09-13T08:46:05.987080+0000","last_peered":"2024-09-13T08:46:05.987080+0000","last_clean":"2024-09-13T08:46:05.987080+0000","last_became_active":"2024-09-13T08:46:05.981849+0000","last_became_peered":"2024-09-13T08:46:05.981849+0000","last_unstale":"2024-09-13T08:46:05.987080+0000","last_undegraded":"2024-09-13T08:46:05.987080+0000","last_fullsized":"2024-09-13T08:46:05.987080+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:42.918730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.972296+0000","last_change":"2024-09-13T08:46:04.964174+0000","last_active":"2024-09-13T08:46:05.972296+0000","last_peered":"2024-09-13T08:46:05.972296+0000","last_clean":"2024-09-13T08:46:05.972296+0000","last_became_active":"2024-09-13T08:46:04.963814+0000","last_became_peered":"2024-09-13T08:46:04.963814+0000","last_unstale":"2024-09-13T08:46:05.972296+0000","last_undegraded":"2024-09-13T08:46:05.972296+0000","last_fullsized":"2024-09-13T08:46:05.972296+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:02:43.323368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.950357+0000","last_change":"2024-09-13T08:46:05.971261+0000","last_active":"2024-09-13T08:46:06.950357+0000","last_peered":"2024-09-13T08:46:06.950357+0000","last_clean":"2024-09-13T08:46:06.950357+0000","last_became_active":"2024-09-13T08:46:05.970981+0000","last_became_peered":"2024-09-13T08:46:05.970981+0000","last_unstale":"2024-09-13T08:46:06.950357+0000","last_undegraded":"2024-09-13T08:46:06.950357+0000","last_fullsized":"2024-09-13T08:46:06.950357+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:48.409940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949552+0000","last_change":"2024-09-13T08:46:04.970084+0000","last_active":"2024-09-13T08:46:06.949552+0000","last_peered":"2024-09-13T08:46:06.949552+0000","last_clean":"2024-09-13T08:46:06.949552+0000","last_became_active":"2024-09-13T08:46:04.969769+0000","last_became_peered":"2024-09-13T08:46:04.969769+0000","last_unstale":"2024-09-13T08:46:06.949552+0000","last_undegraded":"2024-09-13T08:46:06.949552+0000","last_fullsized":"2024-09-13T08:46:06.949552+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:58:19.862815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947607+0000","last_change":"2024-09-13T08:46:05.964366+0000","last_active":"2024-09-13T08:46:06.947607+0000","last_peered":"2024-09-13T08:46:06.947607+0000","last_clean":"2024-09-13T08:46:06.947607+0000","last_became_active":"2024-09-13T08:46:05.964027+0000","last_became_peered":"2024-09-13T08:46:05.964027+0000","last_unstale":"2024-09-13T08:46:06.947607+0000","last_undegraded":"2024-09-13T08:46:06.947607+0000","last_fullsized":"2024-09-13T08:46:06.947607+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:58:20.869942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947652+0000","last_change":"2024-09-13T08:46:04.966271+0000","last_active":"2024-09-13T08:46:06.947652+0000","last_peered":"2024-09-13T08:46:06.947652+0000","last_clean":"2024-09-13T08:46:06.947652+0000","last_became_active":"2024-09-13T08:46:04.966004+0000","last_became_peered":"2024-09-13T08:46:04.966004+0000","last_unstale":"2024-09-13T08:46:06.947652+0000","last_undegraded":"2024-09-13T08:46:06.947652+0000","last_fullsized":"2024-09-13T08:46:06.947652+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:23:44.595928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.973004+0000","last_change":"2024-09-13T08:46:05.962251+0000","last_active":"2024-09-13T08:46:05.973004+0000","last_peered":"2024-09-13T08:46:05.973004+0000","last_clean":"2024-09-13T08:46:05.973004+0000","last_became_active":"2024-09-13T08:46:05.961939+0000","last_became_peered":"2024-09-13T08:46:05.961939+0000","last_unstale":"2024-09-13T08:46:05.973004+0000","last_undegraded":"2024-09-13T08:46:05.973004+0000","last_fullsized":"2024-09-13T08:46:05.973004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:06:40.752264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947667+0000","last_change":"2024-09-13T08:46:04.963574+0000","last_active":"2024-09-13T08:46:06.947667+0000","last_peered":"2024-09-13T08:46:06.947667+0000","last_clean":"2024-09-13T08:46:06.947667+0000","last_became_active":"2024-09-13T08:46:04.963100+0000","last_became_peered":"2024-09-13T08:46:04.963100+0000","last_unstale":"2024-09-13T08:46:06.947667+0000","last_undegraded":"2024-09-13T08:46:06.947667+0000","last_fullsized":"2024-09-13T08:46:06.947667+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:56.120837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947540+0000","last_change":"2024-09-13T08:46:05.967572+0000","last_active":"2024-09-13T08:46:06.947540+0000","last_peered":"2024-09-13T08:46:06.947540+0000","last_clean":"2024-09-13T08:46:06.947540+0000","last_became_active":"2024-09-13T08:46:05.967267+0000","last_became_peered":"2024-09-13T08:46:05.967267+0000","last_unstale":"2024-09-13T08:46:06.947540+0000","last_undegraded":"2024-09-13T08:46:06.947540+0000","last_fullsized":"2024-09-13T08:46:06.947540+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:32:33.970003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947439+0000","last_change":"2024-09-13T08:46:04.956931+0000","last_active":"2024-09-13T08:46:06.947439+0000","last_peered":"2024-09-13T08:46:06.947439+0000","last_clean":"2024-09-13T08:46:06.947439+0000","last_became_active":"2024-09-13T08:46:04.956632+0000","last_became_peered":"2024-09-13T08:46:04.956632+0000","last_unstale":"2024-09-13T08:46:06.947439+0000","last_undegraded":"2024-09-13T08:46:06.947439+0000","last_fullsized":"2024-09-13T08:46:06.947439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:07:54.466875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.949626+0000","last_change":"2024-09-13T08:46:05.966289+0000","last_active":"2024-09-13T08:46:06.949626+0000","last_peered":"2024-09-13T08:46:06.949626+0000","last_clean":"2024-09-13T08:46:06.949626+0000","last_became_active":"2024-09-13T08:46:05.966006+0000","last_became_peered":"2024-09-13T08:46:05.966006+0000","last_unstale":"2024-09-13T08:46:06.949626+0000","last_undegraded":"2024-09-13T08:46:06.949626+0000","last_fullsized":"2024-09-13T08:46:06.949626+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:52:28.532447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948957+0000","last_change":"2024-09-13T08:46:04.956538+0000","last_active":"2024-09-13T08:46:06.948957+0000","last_peered":"2024-09-13T08:46:06.948957+0000","last_clean":"2024-09-13T08:46:06.948957+0000","last_became_active":"2024-09-13T08:46:04.955654+0000","last_became_peered":"2024-09-13T08:46:04.955654+0000","last_unstale":"2024-09-13T08:46:06.948957+0000","last_undegraded":"2024-09-13T08:46:06.948957+0000","last_fullsized":"2024-09-13T08:46:06.948957+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:16.886655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.948840+0000","last_change":"2024-09-13T08:46:05.957228+0000","last_active":"2024-09-13T08:46:06.948840+0000","last_peered":"2024-09-13T08:46:06.948840+0000","last_clean":"2024-09-13T08:46:06.948840+0000","last_became_active":"2024-09-13T08:46:05.956933+0000","last_became_peered":"2024-09-13T08:46:05.956933+0000","last_unstale":"2024-09-13T08:46:06.948840+0000","last_undegraded":"2024-09-13T08:46:06.948840+0000","last_fullsized":"2024-09-13T08:46:06.948840+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:04.933940+0000","last_clean_scrub_stamp":"2024-09-13T08:46:04.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:07:36.059273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T08:46:05.946546+0000","last_change":"2024-09-13T08:46:04.965870+0000","last_active":"2024-09-13T08:46:05.946546+0000","last_peered":"2024-09-13T08:46:05.946546+0000","last_clean":"2024-09-13T08:46:05.946546+0000","last_became_active":"2024-09-13T08:46:04.965338+0000","last_became_peered":"2024-09-13T08:46:04.965338+0000","last_unstale":"2024-09-13T08:46:05.946546+0000","last_undegraded":"2024-09-13T08:46:05.946546+0000","last_fullsized":"2024-09-13T08:46:05.946546+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:03.926812+0000","last_clean_scrub_stamp":"2024-09-13T08:46:03.926812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:10.888400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":397,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T08:46:06.947367+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:06.947367+0000","last_peered":"2024-09-13T08:46:06.947367+0000","last_clean":"2024-09-13T08:46:06.947367+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:06.947367+0000","last_undegraded":"2024-09-13T08:46:06.947367+0000","last_fullsized":"2024-09-13T08:46:06.947367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37372,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334468,"statfs":{"total":96636764160,"available":96598495232,"internally_reserved":0,"allocated":2064384,"data_stored":907612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37352,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334488,"statfs":{"total":96636764160,"available":96598515712,"internally_reserved":0,"allocated":2048000,"data_stored":905978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37936,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333904,"statfs":{"total":96636764160,"available":96597917696,"internally_reserved":0,"allocated":2646016,"data_stored":1496860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542256,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41512,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330328,"statfs":{"total":96636764160,"available":96594255872,"internally_reserved":0,"allocated":2048000,"data_stored":905922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509553,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2060288,"data_stored":907570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2048000,"data_stored":905958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509554,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33768,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338072,"statfs":{"total":96636764160,"available":96602185728,"internally_reserved":0,"allocated":2637824,"data_stored":1496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509552,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":906504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:46:10.637 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:46:10.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:46:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 -- 172.21.15.31:0/1580089627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b00cbc80 msgr2=0x7f62b00b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 --2- 172.21.15.31:0/1580089627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b00cbc80 0x7f62b00b29e0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f62b00cfab0 tx=0x7f62a802aa30 comp rx=0 tx=0).stop 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 -- 172.21.15.31:0/1580089627 shutdown_connections 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 --2- 172.21.15.31:0/1580089627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b00cbc80 0x7f62b00b29e0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 --2- 172.21.15.31:0/1580089627 >> v2:172.21.15.45:3300/0 conn(0x7f62b0072560 0x7f62b00cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 -- 172.21.15.31:0/1580089627 >> 172.21.15.31:0/1580089627 conn(0x7f62b00cfbb0 msgr2=0x7f62b00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 -- 172.21.15.31:0/1580089627 shutdown_connections 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.838+0000 7f62b4a00640 1 -- 172.21.15.31:0/1580089627 wait complete. 2024-09-13T08:46:10.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.840+0000 7f62b4a00640 1 Processor -- start 2024-09-13T08:46:10.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.840+0000 7f62b4a00640 1 -- start start 2024-09-13T08:46:10.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62b0072560 0x7f62b00c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62b00cbc80 0x7f62b015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62b006e740 con 0x7f62b015bfc0 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62b006e5c0 con 0x7f62b0072560 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62b4a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f62b00b2f20 con 0x7f62b00cbc80 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37086/0 (socket says 172.21.15.31:37086) 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ae800640 1 -- 172.21.15.31:0/805049613 learned_addr learned my addr 172.21.15.31:0/805049613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ade00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62b0072560 0x7f62b00c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62a5400640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62b00cbc80 0x7f62b015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ae800640 1 -- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62b00cbc80 msgr2=0x7f62b015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.841+0000 7f62ae800640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62b00cbc80 0x7f62b015d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62ae800640 1 -- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62b0072560 msgr2=0x7f62b00c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62ae800640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62b0072560 0x7f62b00c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62ae800640 1 -- 172.21.15.31:0/805049613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b01f8b80 con 0x7f62b015bfc0 2024-09-13T08:46:10.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62ae800640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f62a802b020 tx=0x7f62a8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8047070 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62a8004250 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b01f4ab0 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8033050 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.842+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b01f5070 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.843+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b00b3340 con 0x7f62b015bfc0 2024-09-13T08:46:10.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.845+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f62a804a070 con 0x7f62b015bfc0 2024-09-13T08:46:10.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.846+0000 7f62a6a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 0x7f628c07b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:10.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.846+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f62a80bc9a0 con 0x7f62b015bfc0 2024-09-13T08:46:10.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.846+0000 7f62ade00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 0x7f628c07b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:10.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.847+0000 7f62ade00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 0x7f628c07b5b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006fd0 tx=0x7f62a0009060 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:10.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:10.850+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f62a8083e00 con 0x7f62b015bfc0 2024-09-13T08:46:11.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.119+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f62b01f5240 con 0x7f62b015bfc0 2024-09-13T08:46:11.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.122+0000 7f62a6a00640 1 -- 172.21.15.31:0/805049613 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) ==== 190+0+147 (secure 0 0 0) 0x7f62a8088830 con 0x7f62b015bfc0 2024-09-13T08:46:11.126 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:46:11.126 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:46:11.126 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:46:11.126 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:46:11.126 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 msgr2=0x7f628c07b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 0x7f628c07b5b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006fd0 tx=0x7f62a0009060 comp rx=0 tx=0).stop 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 msgr2=0x7f62b015d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f62a802b020 tx=0x7f62a8003f10 comp rx=0 tx=0).stop 2024-09-13T08:46:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 shutdown_connections 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f628c0790f0 0x7f628c07b5b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b015bfc0 0x7f62b015d6e0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62b00cbc80 0x7f62b015d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 --2- 172.21.15.31:0/805049613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62b0072560 0x7f62b00c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.124+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 >> 172.21.15.31:0/805049613 conn(0x7f62b00cfbb0 msgr2=0x7f62b00ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.125+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 shutdown_connections 2024-09-13T08:46:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.125+0000 7f62b4a00640 1 -- 172.21.15.31:0/805049613 wait complete. 2024-09-13T08:46:11.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.300+0000 7f689be00640 1 -- 172.21.15.31:0/3450702917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68940ca5c0 msgr2=0x7f68940cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.300+0000 7f689be00640 1 --2- 172.21.15.31:0/3450702917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68940ca5c0 0x7f68940cc300 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f6880005fd0 tx=0x7f688002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 -- 172.21.15.31:0/3450702917 shutdown_connections 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 --2- 172.21.15.31:0/3450702917 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 0x7f68940b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 --2- 172.21.15.31:0/3450702917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68940ca5c0 0x7f68940cc300 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 -- 172.21.15.31:0/3450702917 >> 172.21.15.31:0/3450702917 conn(0x7f68940cc910 msgr2=0x7f68940cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 -- 172.21.15.31:0/3450702917 shutdown_connections 2024-09-13T08:46:11.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.301+0000 7f689be00640 1 -- 172.21.15.31:0/3450702917 wait complete. 2024-09-13T08:46:11.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.302+0000 7f689be00640 1 Processor -- start 2024-09-13T08:46:11.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.302+0000 7f689be00640 1 -- start start 2024-09-13T08:46:11.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 0x7f68940b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:11.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68940ca5c0 0x7f68940c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6894073990 con 0x7f689415dfc0 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f689415e4b0 con 0x7f68940ca5c0 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.303+0000 7f689be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6894073b10 con 0x7f68940b1910 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37100/0 (socket says 172.21.15.31:37100) 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 -- 172.21.15.31:0/2338899413 learned_addr learned my addr 172.21.15.31:0/2338899413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6898a00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68940ca5c0 0x7f68940c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899400640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 0x7f68940b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:11.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 -- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 msgr2=0x7f68940b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 0x7f68940b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 -- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68940ca5c0 msgr2=0x7f68940c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68940ca5c0 0x7f68940c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 -- 172.21.15.31:0/2338899413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689409c900 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.304+0000 7f6899e00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f688c00eaa0 tx=0x7f688c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.305+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c00cdb0 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.305+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f688c004c10 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.305+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689409cbe0 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.305+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c005020 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.305+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689409d1a0 con 0x7f689415dfc0 2024-09-13T08:46:11.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.306+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6894074030 con 0x7f689415dfc0 2024-09-13T08:46:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.307+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f688c0051c0 con 0x7f689415dfc0 2024-09-13T08:46:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.308+0000 7f6891600640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 0x7f687407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.308+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f688c09c670 con 0x7f689415dfc0 2024-09-13T08:46:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.308+0000 7f6899400640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 0x7f687407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:11.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.309+0000 7f6899400640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 0x7f687407b580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6888001720 tx=0x7f6888009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:11.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.312+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f688c016030 con 0x7f689415dfc0 2024-09-13T08:46:11.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.544+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68940c9020 con 0x7f689415dfc0 2024-09-13T08:46:11.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.545+0000 7f6891600640 1 -- 172.21.15.31:0/2338899413 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5224 (secure 0 0 0) 0x7f688c0635b0 con 0x7f689415dfc0 2024-09-13T08:46:11.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:11.547 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":109,"btime":"2024-09-13T08:46:08:967593+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:11.547 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 109 2024-09-13T08:46:11.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 msgr2=0x7f687407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 0x7f687407b580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6888001720 tx=0x7f6888009040 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 msgr2=0x7f68940c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f688c00eaa0 tx=0x7f688c00ef60 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 shutdown_connections 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f68740790c0 0x7f687407b580 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689415dfc0 0x7f68940c8ae0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68940ca5c0 0x7f68940c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 --2- 172.21.15.31:0/2338899413 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f68940b1910 0x7f68940b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 >> 172.21.15.31:0/2338899413 conn(0x7f68940cc910 msgr2=0x7f68940b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 shutdown_connections 2024-09-13T08:46:11.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:11.547+0000 7f689be00640 1 -- 172.21.15.31:0/2338899413 wait complete. 2024-09-13T08:46:11.935 INFO:tasks.ceph.mds.d.smithi045.stderr:2024-09-13T08:46:11.934+0000 7ffb36000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:46:11.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:46:05.396638+0000', 'modified': '2024-09-13T08:46:08.400907+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8266': {'gid': 8266, 'name': 'd', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/98622202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6801', 'nonce': 98622202}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 98622202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8266, 'qdb_cluster': [8266]} 2024-09-13T08:46:11.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:46:11.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T08:46:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 -- 172.21.15.31:0/1288675833 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f015be30 msgr2=0x7fa1f00cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/1288675833 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f015be30 0x7fa1f00cdb80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0005fd0 tx=0x7fa1e002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 -- 172.21.15.31:0/1288675833 shutdown_connections 2024-09-13T08:46:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/1288675833 >> v2:172.21.15.45:3300/0 conn(0x7fa1f0073860 0x7fa1f0073d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/1288675833 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f015be30 0x7fa1f00cdb80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 -- 172.21.15.31:0/1288675833 >> 172.21.15.31:0/1288675833 conn(0x7fa1f00b18f0 msgr2=0x7fa1f00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 -- 172.21.15.31:0/1288675833 shutdown_connections 2024-09-13T08:46:12.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.133+0000 7fa1f7a00640 1 -- 172.21.15.31:0/1288675833 wait complete. 2024-09-13T08:46:12.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.134+0000 7fa1f7a00640 1 Processor -- start 2024-09-13T08:46:12.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.135+0000 7fa1f7a00640 1 -- start start 2024-09-13T08:46:12.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 0x7fa1f01f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f015be30 0x7fa1f015d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f00733e0 con 0x7fa1f00ced20 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f00ce9e0 con 0x7fa1f015be30 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f7a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f0074250 con 0x7fa1f0073860 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 0x7fa1f01f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:47422/0 (socket says 172.21.15.31:47422) 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 -- 172.21.15.31:0/902895164 learned_addr learned my addr 172.21.15.31:0/902895164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f015be30 0x7fa1f015d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 -- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f015be30 msgr2=0x7fa1f015d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f015be30 0x7fa1f015d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 -- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 msgr2=0x7fa1f01f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 0x7fa1f01f8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1f5000640 1 -- 172.21.15.31:0/902895164 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f0158f50 con 0x7fa1f0073860 2024-09-13T08:46:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.136+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 0x7fa1f01f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1f5000640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0034040 tx=0x7fa1e00035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1ed600640 1 -- 172.21.15.31:0/902895164 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e003e030 con 0x7fa1f0073860 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1ed600640 1 -- 172.21.15.31:0/902895164 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e0003ae0 con 0x7fa1f0073860 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1ed600640 1 -- 172.21.15.31:0/902895164 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0035040 con 0x7fa1f0073860 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f0154290 con 0x7fa1f0073860 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.137+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f0154850 con 0x7fa1f0073860 2024-09-13T08:46:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.140+0000 7fa1ed600640 1 -- 172.21.15.31:0/902895164 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fa1e0041030 con 0x7fa1f0073860 2024-09-13T08:46:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.141+0000 7fa1ed600640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 0x7fa1cc07b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.141+0000 7fa1ed600640 1 -- 172.21.15.31:0/902895164 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa1e00bc3c0 con 0x7fa1f0073860 2024-09-13T08:46:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.141+0000 7fa1fa9ff740 1 Processor -- start 2024-09-13T08:46:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.141+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 0x7fa1cc07b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.141+0000 7fa1fa9ff740 1 -- start start 2024-09-13T08:46:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 0x7fa1cc07b580 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa1f01584a0 tx=0x7fa1d8009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aaa20 0x563acf5ee390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563acf52d430 con 0x563acf5ef8b0 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563acf51d5d0 con 0x563acf5aaa20 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563acf50ca00 con 0x563acf5aafa0 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1f5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.142+0000 7fa1f5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37120/0 (socket says 172.21.15.31:37120) 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aaa20 0x563acf5ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 -- 172.21.15.31:0/4028513273 learned_addr learned my addr 172.21.15.31:0/4028513273 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 -- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 msgr2=0x563acf5ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5ee8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 -- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aaa20 msgr2=0x563acf5ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aaa20 0x563acf5ee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 -- 172.21.15.31:0/4028513273 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563acf4d5520 con 0x563acf5ef8b0 2024-09-13T08:46:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1f5000640 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00805f0 tx=0x7fa1e0082cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef08e902a7d3698c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1e5600640 1 -- 172.21.15.31:0/4028513273 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0048040 con 0x563acf5ef8b0 2024-09-13T08:46:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1e5600640 1 -- 172.21.15.31:0/4028513273 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e0083380 con 0x563acf5ef8b0 2024-09-13T08:46:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.143+0000 7fa1e5600640 1 -- 172.21.15.31:0/4028513273 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0083820 con 0x563acf5ef8b0 2024-09-13T08:46:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/4028513273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 msgr2=0x563acf5efc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00805f0 tx=0x7fa1e0082cd0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/4028513273 shutdown_connections 2024-09-13T08:46:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5ee8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aaa20 0x563acf5ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/4028513273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5efc90 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/4028513273 >> 172.21.15.31:0/4028513273 conn(0x563acf5a5ff0 msgr2=0x563acf5a9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/4028513273 shutdown_connections 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.144+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/4028513273 wait complete. 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.145+0000 7fa1fa9ff740 1 -- start start 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.145+0000 7fa1fa9ff740 1 Processor -- start 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf5cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5e9fd0 0x563acf5d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563acf4b3330 con 0x563acf5ef8b0 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563acf510810 con 0x563acf5aafa0 2024-09-13T08:46:12.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563acf27d620 con 0x563acf5e9fd0 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1fa9ff740 10 client.0 _ll_register_callbacks cb 0x563acf4b8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37134/0 (socket says 172.21.15.31:37134) 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1e7e00640 1 -- 172.21.15.31:0/436791577 learned_addr learned my addr 172.21.15.31:0/436791577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1f5000640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf5cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.146+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5e9fd0 0x563acf5d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 -- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5e9fd0 msgr2=0x563acf5d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5e9fd0 0x563acf5d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 -- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 msgr2=0x563acf5cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf5cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 -- 172.21.15.31:0/436791577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563acf26e850 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80075c0 tx=0x7fa1d8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f00f19a5202a10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1de000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8013600 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa1d80137a0 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e0083380 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.147+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7fa1d802e050 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 10 client.8964 fetch_fsmap learned FSMap version 109 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) -- 0x563acf2e56b0 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e0083380 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 109) ==== 3502+0+0 (secure 0 0 0) 0x7fa1d8031020 con 0x563acf5ef8b0 2024-09-13T08:46:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 10 client.8964 fetch_fsmap finished waiting for FSMap version 109 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 10 client.8964 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 20 client.8964 populate_metadata read hostname 'smithi031' 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ea580 0x563acf5ff000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 4 client.8964 mds_command: new command op to 8266 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 --> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563acf51f520 con 0x563acf5ea580 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.148+0000 7fa1f5000640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ea580 0x563acf5ff000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.149+0000 7fa1f5000640 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ea580 0x563acf5ff000 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c176331668525d4a server_cookie=2f1eaa63d00f208a in_seq=0 out_seq=0 2024-09-13T08:46:12.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.149+0000 7fa1de000640 10 client.8964 ms_handle_connect on v2:172.21.15.45:6801/98622202 2024-09-13T08:46:12.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.150+0000 7fa1de000640 1 -- 172.21.15.31:0/436791577 <== mds.0 v2:172.21.15.45:6801/98622202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563acf51f520 con 0x563acf5ea580 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.150+0000 7fa1de000640 10 client.8964 handle_command_reply: tid=0 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 client.8964 shutdown 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 msgr2=0x563acf5e9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80075c0 tx=0x7fa1d8000ed0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 shutdown_connections 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5e9fd0 0x563acf5d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ea580 0x563acf5ff000 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf5e9a90 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/436791577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf5cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1de000640 0 client.8964 ms_handle_reset on v2:172.21.15.45:6801/98622202 2024-09-13T08:46:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 >> 172.21.15.31:0/436791577 conn(0x563acf5aaa20 msgr2=0x563acf5d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 shutdown_connections 2024-09-13T08:46:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/436791577 wait complete. 2024-09-13T08:46:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.151+0000 7fa1fa9ff740 20 client.8964 trim_cache size 0 max 16384 2024-09-13T08:46:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.170+0000 7fa1fa9ff740 1 Processor -- start 2024-09-13T08:46:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.170+0000 7fa1fa9ff740 1 -- start start 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5ef8b0 0x563acf601e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f0155f40 con 0x563acf602390 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f015b3c0 con 0x563acf5ef8b0 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f01558f0 con 0x563acf5aafa0 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1f5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1f5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37136/0 (socket says 172.21.15.31:37136) 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.171+0000 7fa1f5a00640 1 -- 172.21.15.31:0/3779466040 learned_addr learned my addr 172.21.15.31:0/3779466040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5000640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5ef8b0 0x563acf601e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 -- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 msgr2=0x563acf5a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 -- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5ef8b0 msgr2=0x563acf601e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5ef8b0 0x563acf601e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 -- 172.21.15.31:0/3779466040 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563acf5f1320 con 0x563acf602390 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5000640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x563acf5d3020 tx=0x7fa1e8004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d830a47962430bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1e4c00640 1 -- 172.21.15.31:0/3779466040 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8020070 con 0x563acf602390 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1e4c00640 1 -- 172.21.15.31:0/3779466040 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e8010630 con 0x563acf602390 2024-09-13T08:46:12.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.172+0000 7fa1e4c00640 1 -- 172.21.15.31:0/3779466040 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8010a00 con 0x563acf602390 2024-09-13T08:46:12.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/3779466040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 msgr2=0x563acf5f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x563acf5d3020 tx=0x7fa1e8004410 comp rx=0 tx=0).stop 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/3779466040 shutdown_connections 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf602390 0x563acf5f1ae0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5ef8b0 0x563acf601e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/3779466040 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf5aafa0 0x563acf5a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/3779466040 >> 172.21.15.31:0/3779466040 conn(0x563acf5aaa20 msgr2=0x563acf601220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/3779466040 shutdown_connections 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/3779466040 wait complete. 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.173+0000 7fa1fa9ff740 1 -- start start 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 Processor -- start 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf670240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf602390 0x563acf672d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563acf5ea400 con 0x563acf5ef8b0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563acf602820 con 0x563acf5aafa0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.174+0000 7fa1fa9ff740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x563acf5eaa90 con 0x563acf602390 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1fa9ff740 10 client.0 _ll_register_callbacks cb 0x7fa1d8007240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37146/0 (socket says 172.21.15.31:37146) 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 -- 172.21.15.31:0/1840318445 learned_addr learned my addr 172.21.15.31:0/1840318445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1f5000640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf670240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1f5a00640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf602390 0x563acf672d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 -- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf602390 msgr2=0x563acf672d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf602390 0x563acf672d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 -- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 msgr2=0x563acf670240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf670240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.175+0000 7fa1e7e00640 1 -- 172.21.15.31:0/1840318445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563acf6054f0 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1e7e00640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80075c0 tx=0x7fa1d8007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cac6c4be5438737 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8022890 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e8010630 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa1d8022a30 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7fa1d8041050 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 10 client.8970 fetch_fsmap learned FSMap version 109 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) -- 0x563acf66f6c0 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e8010630 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 109) ==== 3502+0+0 (secure 0 0 0) 0x7fa1d8044020 con 0x563acf5ef8b0 2024-09-13T08:46:12.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 10 client.8970 fetch_fsmap finished waiting for FSMap version 109 2024-09-13T08:46:12.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 10 client.8970 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T08:46:12.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 20 client.8970 populate_metadata read hostname 'smithi031' 2024-09-13T08:46:12.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ee460 0x563acf5ee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 4 client.8970 mds_command: new command op to 8266 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:46:12.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.176+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 --> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] -- command(tid 0: {"prefix": "status"}) -- 0x563acf51afd0 con 0x563acf5ee460 2024-09-13T08:46:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.177+0000 7fa1f5000640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ee460 0x563acf5ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1f5000640 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ee460 0x563acf5ee840 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cdb227979fd8977 server_cookie=5d47e8dec2f4d3be in_seq=0 out_seq=0 2024-09-13T08:46:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1de000640 10 client.8970 ms_handle_connect on v2:172.21.15.45:6801/98622202 2024-09-13T08:46:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1de000640 1 -- 172.21.15.31:0/1840318445 <== mds.0 v2:172.21.15.45:6801/98622202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x563acf51afd0 con 0x563acf5ee460 2024-09-13T08:46:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1de000640 10 client.8970 handle_command_reply: tid=0 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 client.8970 shutdown 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 msgr2=0x563acf652110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80075c0 tx=0x7fa1d8007730 comp rx=0 tx=0).stop 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 shutdown_connections 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x563acf602390 0x563acf672d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:6801/98622202,v1:172.21.15.45:6835/98622202] conn(0x563acf5ee460 0x563acf5ee840 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563acf5ef8b0 0x563acf652110 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1de000640 0 client.8970 ms_handle_reset on v2:172.21.15.45:6801/98622202 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 --2- 172.21.15.31:0/1840318445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563acf5aafa0 0x563acf670240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 >> 172.21.15.31:0/1840318445 conn(0x563acf5aaa20 msgr2=0x563acf5e91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 shutdown_connections 2024-09-13T08:46:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 1 -- 172.21.15.31:0/1840318445 wait complete. 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.178+0000 7fa1fa9ff740 20 client.8970 trim_cache size 0 max 16384 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 8266, 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:46:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2342365429999997, 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 109, 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 164, 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 164, 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.491440649999999 2024-09-13T08:46:12.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 msgr2=0x7fa1cc07b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 0x7fa1cc07b580 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa1f01584a0 tx=0x7fa1d8009040 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 msgr2=0x7fa1f014c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0034040 tx=0x7fa1e00035f0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 shutdown_connections 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa1cc0790c0 0x7fa1cc07b580 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1f015be30 0x7fa1f015d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f00ced20 0x7fa1f01f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 --2- 172.21.15.31:0/902895164 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa1f0073860 0x7fa1f014c760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 >> 172.21.15.31:0/902895164 conn(0x7fa1f00b18f0 msgr2=0x7fa1f00cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 shutdown_connections 2024-09-13T08:46:12.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.179+0000 7fa1f7a00640 1 -- 172.21.15.31:0/902895164 wait complete. 2024-09-13T08:46:12.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 -- 172.21.15.31:0/1418482733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415bd70 msgr2=0x7f07b40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 --2- 172.21.15.31:0/1418482733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415bd70 0x7f07b40cf850 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005fd0 tx=0x7f07a402a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 -- 172.21.15.31:0/1418482733 shutdown_connections 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 --2- 172.21.15.31:0/1418482733 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415fe20 0x7f07b40cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 --2- 172.21.15.31:0/1418482733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415bd70 0x7f07b40cf850 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 -- 172.21.15.31:0/1418482733 >> 172.21.15.31:0/1418482733 conn(0x7f07b40cc910 msgr2=0x7f07b40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 -- 172.21.15.31:0/1418482733 shutdown_connections 2024-09-13T08:46:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.355+0000 7f07bb800640 1 -- 172.21.15.31:0/1418482733 wait complete. 2024-09-13T08:46:12.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.357+0000 7f07bb800640 1 Processor -- start 2024-09-13T08:46:12.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.357+0000 7f07bb800640 1 -- start start 2024-09-13T08:46:12.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b415bd70 0x7f07b40c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415f9e0 0x7f07b4157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f07b40cec50 con 0x7f07b415fe20 2024-09-13T08:46:12.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07b40cead0 con 0x7f07b415bd70 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07bb800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f07b40cb850 con 0x7f07b415f9e0 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07b9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07b9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37162/0 (socket says 172.21.15.31:37162) 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07b9800640 1 -- 172.21.15.31:0/2308955416 learned_addr learned my addr 172.21.15.31:0/2308955416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07b8e00640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b415bd70 0x7f07b40c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.358+0000 7f07abe00640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415f9e0 0x7f07b4157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 -- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415f9e0 msgr2=0x7f07b4157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415f9e0 0x7f07b4157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 -- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b415bd70 msgr2=0x7f07b40c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b415bd70 0x7f07b40c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 -- 172.21.15.31:0/2308955416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b409ac20 con 0x7f07b415fe20 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b9800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e960 tx=0x7f07ac00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac004d20 con 0x7f07b415fe20 2024-09-13T08:46:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07ac00cb20 con 0x7f07b415fe20 2024-09-13T08:46:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac005020 con 0x7f07b415fe20 2024-09-13T08:46:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b409af00 con 0x7f07b415fe20 2024-09-13T08:46:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.359+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b409b4c0 con 0x7f07b415fe20 2024-09-13T08:46:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.360+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b40cf850 con 0x7f07b415fe20 2024-09-13T08:46:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.361+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f07ac0051c0 con 0x7f07b415fe20 2024-09-13T08:46:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.363+0000 7f07b1600640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 0x7f079007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.363+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f07ac09c4f0 con 0x7f07b415fe20 2024-09-13T08:46:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.363+0000 7f07b8e00640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 0x7f079007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:12.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.364+0000 7f07b8e00640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 0x7f079007b580 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f07a4033000 tx=0x7f07a4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:12.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.367+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f07ac016030 con 0x7f07b415fe20 2024-09-13T08:46:12.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.600+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f07b40ce460 con 0x7f07b415fe20 2024-09-13T08:46:12.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.601+0000 7f07b1600640 1 -- 172.21.15.31:0/2308955416 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v109) ==== 53+0+73 (secure 0 0 0) 0x7f07ac0634b0 con 0x7f07b415fe20 2024-09-13T08:46:12.603 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:46:12.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 msgr2=0x7f079007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 0x7f079007b580 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f07a4033000 tx=0x7f07a4004730 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 msgr2=0x7f07b40b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e960 tx=0x7f07ac00ee20 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 shutdown_connections 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f07900790c0 0x7f079007b580 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07b415fe20 0x7f07b40b47f0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f07b415f9e0 0x7f07b4157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 --2- 172.21.15.31:0/2308955416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b415bd70 0x7f07b40c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 >> 172.21.15.31:0/2308955416 conn(0x7f07b40cc910 msgr2=0x7f07b40af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 shutdown_connections 2024-09-13T08:46:12.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12.603+0000 7f07bb800640 1 -- 172.21.15.31:0/2308955416 wait complete. 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:46:12.621 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:12.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:12.622 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42159sec preferred_lft 42159sec 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:46:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:46:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:12.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:12.661 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:46:12.751 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:46:12.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:12.754 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:46:12.791 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:46:12.791 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:46:12.791 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:46:12.791 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:46:12.791 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:46:12.791 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:46:12.792 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:46:12.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:46:12.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:46:12.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:46:12.792 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:46:12.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:46:12.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:12.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:12.991 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:46:13.027 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:46:13.027 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:46:13.028 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:13.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:13.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:46:13.125 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:13.125 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:13.160 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:13.160 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:13.160 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:13.235 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:46:13.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:13.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:13.364 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:13.364 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:13.392 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:13.393 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:13.393 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:46:13.393 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:46:13.393 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:46:13.395 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:46:13.395 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:46:13.395 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:46:13.395 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:46:13.395 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:46:13.398 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:46:13.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:13.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:13.456 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:46:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:13.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:13.555 INFO:teuthology.orchestra.run.smithi031.stdout:client7481 2024-09-13T08:46:13.556 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:13.556 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:13.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:46:13.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 -- 172.21.15.31:0/622512768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f854415f9e0 msgr2=0x7f854415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 --2- 172.21.15.31:0/622512768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f854415f9e0 0x7f854415fe20 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f854409e090 tx=0x7f852c02a560 comp rx=0 tx=0).stop 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 -- 172.21.15.31:0/622512768 shutdown_connections 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 --2- 172.21.15.31:0/622512768 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f85440ce3c0 0x7f85440ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 --2- 172.21.15.31:0/622512768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f854415f9e0 0x7f854415fe20 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 --2- 172.21.15.31:0/622512768 >> v2:172.21.15.45:3300/0 conn(0x7f854415bd70 0x7f85440cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.792+0000 7f8548e00640 1 -- 172.21.15.31:0/622512768 >> 172.21.15.31:0/622512768 conn(0x7f85440cc910 msgr2=0x7f85440cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.793+0000 7f8548e00640 1 -- 172.21.15.31:0/622512768 shutdown_connections 2024-09-13T08:46:13.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.793+0000 7f8548e00640 1 -- 172.21.15.31:0/622512768 wait complete. 2024-09-13T08:46:13.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.794+0000 7f8548e00640 1 Processor -- start 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.794+0000 7f8548e00640 1 -- start start 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 0x7f85440c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 0x7f85440c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f85440cb9d0 con 0x7f85440ce3c0 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85440cb850 con 0x7f854415f9e0 2024-09-13T08:46:13.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.795+0000 7f8548e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f85440cbc50 con 0x7f854415bd70 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37178/0 (socket says 172.21.15.31:37178) 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 -- 172.21.15.31:0/2423210639 learned_addr learned my addr 172.21.15.31:0/2423210639 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8542800640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 0x7f85440c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8539400640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 0x7f85440c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 -- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 msgr2=0x7f85440c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 0x7f85440c28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 -- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 msgr2=0x7f85440c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 0x7f85440c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 -- 172.21.15.31:0/2423210639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85441f49c0 con 0x7f85440ce3c0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8539400640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 0x7f85440c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8542800640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 0x7f85440c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f8541e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f853400d980 tx=0x7f853400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.796+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534017070 con 0x7f85440ce3c0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.797+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8534009d20 con 0x7f85440ce3c0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.797+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340106c0 con 0x7f85440ce3c0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.797+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85441f4ca0 con 0x7f85440ce3c0 2024-09-13T08:46:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.797+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85441f4670 con 0x7f85440ce3c0 2024-09-13T08:46:13.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.798+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85440cf850 con 0x7f85440ce3c0 2024-09-13T08:46:13.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.799+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f8534010860 con 0x7f85440ce3c0 2024-09-13T08:46:13.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.800+0000 7f853aa00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 0x7f852007b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.800+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f853409bfb0 con 0x7f85440ce3c0 2024-09-13T08:46:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.800+0000 7f8539400640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 0x7f852007b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.801+0000 7f8539400640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 0x7f852007b580 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f852c033000 tx=0x7f852c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:13.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:13.804+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f85340635c0 con 0x7f85440ce3c0 2024-09-13T08:46:14.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.046+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f85440745d0 con 0x7f85440ce3c0 2024-09-13T08:46:14.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.046+0000 7f853aa00640 1 -- 172.21.15.31:0/2423210639 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v109) ==== 53+0+73 (secure 0 0 0) 0x7f8534062ec0 con 0x7f85440ce3c0 2024-09-13T08:46:14.049 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:46:14.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 msgr2=0x7f852007b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:14.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 0x7f852007b580 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f852c033000 tx=0x7f852c004730 comp rx=0 tx=0).stop 2024-09-13T08:46:14.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 msgr2=0x7f854409d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:14.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f853400d980 tx=0x7f853400de40 comp rx=0 tx=0).stop 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 shutdown_connections 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f85200790c0 0x7f852007b580 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854415f9e0 0x7f85440c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f854415bd70 0x7f85440c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 --2- 172.21.15.31:0/2423210639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85440ce3c0 0x7f854409d580 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 >> 172.21.15.31:0/2423210639 conn(0x7f85440cc910 msgr2=0x7f854415e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 shutdown_connections 2024-09-13T08:46:14.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:14.049+0000 7f8548e00640 1 -- 172.21.15.31:0/2423210639 wait complete. 2024-09-13T08:46:14.066 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:46:14.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:46:14.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:46:14.066 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:46:14.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:14.067 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:46:14.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:46:14.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:46:14.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:14.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.067 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:46:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:46:14.108 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:14.109 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:14.110 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42162sec preferred_lft 42162sec 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:14.111 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:46:14.112 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:46:14.113 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:46:14.114 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:14.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.114 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:46:14.207 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:46:14.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.210 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:46:14.248 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:46:14.249 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:46:14.249 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:46:14.249 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:46:14.249 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:46:14.250 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:46:14.250 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:46:14.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:46:14.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:46:14.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:46:14.251 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:46:14.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:46:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:14.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.447 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:46:14.484 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:46:14.484 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:46:14.484 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:14.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:46:14.580 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:14.580 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:14.616 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:14.616 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:14.691 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:46:14.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:14.820 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:14.821 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:14.848 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:14.848 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:14.848 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:46:14.848 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:46:14.849 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:46:14.851 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:46:14.851 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:46:14.851 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:46:14.851 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:46:14.851 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:46:14.854 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:46:14.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:14.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:14.910 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:46:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:14.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:15.000 INFO:teuthology.orchestra.run.smithi045.stdout:client8395 2024-09-13T08:46:15.001 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:15.001 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:15.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 -- 172.21.15.31:0/3608141948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac15bfc0 msgr2=0x7feeac072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 --2- 172.21.15.31:0/3608141948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac15bfc0 0x7feeac072560 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7feeac0cfab0 tx=0x7fee9c02a5f0 comp rx=0 tx=0).stop 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 -- 172.21.15.31:0/3608141948 shutdown_connections 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 --2- 172.21.15.31:0/3608141948 >> v2:172.21.15.45:3300/0 conn(0x7feeac06fa60 0x7feeac0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 --2- 172.21.15.31:0/3608141948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac15bfc0 0x7feeac072560 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 -- 172.21.15.31:0/3608141948 >> 172.21.15.31:0/3608141948 conn(0x7feeac0cfbb0 msgr2=0x7feeac0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 -- 172.21.15.31:0/3608141948 shutdown_connections 2024-09-13T08:46:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.212+0000 7feeb1200640 1 -- 172.21.15.31:0/3608141948 wait complete. 2024-09-13T08:46:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.214+0000 7feeb1200640 1 Processor -- start 2024-09-13T08:46:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.214+0000 7feeb1200640 1 -- start start 2024-09-13T08:46:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feeac15bfc0 0x7feeac145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac146150 0x7feeac1f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feeac0b3270 con 0x7feeac06fa60 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feeac0b30f0 con 0x7feeac146150 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feeb1200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7feeac0b33f0 con 0x7feeac15bfc0 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feea9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.215+0000 7feea9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59064/0 (socket says 172.21.15.31:59064) 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 -- 172.21.15.31:0/2360325591 learned_addr learned my addr 172.21.15.31:0/2360325591 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea1400640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feeac15bfc0 0x7feeac145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feeaa800640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac146150 0x7feeac1f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 -- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feeac15bfc0 msgr2=0x7feeac145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feeac15bfc0 0x7feeac145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 -- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac146150 msgr2=0x7feeac1f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac146150 0x7feeac1f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 -- 172.21.15.31:0/2360325591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac1f6c70 con 0x7feeac06fa60 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea9e00640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fee9c034040 tx=0x7fee9c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.216+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c03e030 con 0x7feeac06fa60 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.217+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee9c003e00 con 0x7feeac06fa60 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.217+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c035040 con 0x7feeac06fa60 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.217+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac1e4690 con 0x7feeac06fa60 2024-09-13T08:46:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.217+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac0c6300 con 0x7feeac06fa60 2024-09-13T08:46:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.218+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeac0cbf20 con 0x7feeac06fa60 2024-09-13T08:46:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.219+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fee9c041030 con 0x7feeac06fa60 2024-09-13T08:46:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.220+0000 7feea2a00640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 0x7fee8807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.220+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fee9c0bc2c0 con 0x7feeac06fa60 2024-09-13T08:46:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.220+0000 7feea1400640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 0x7fee8807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.221+0000 7feea1400640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 0x7fee8807b5b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feeac15b8b0 tx=0x7fee9400b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:15.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.224+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fee9c083280 con 0x7feeac06fa60 2024-09-13T08:46:15.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:15.459+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7feeac0cb310 con 0x7feeac06fa60 2024-09-13T08:46:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.003+0000 7feea2a00640 1 -- 172.21.15.31:0/2360325591 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v110) ==== 136+0+0 (secure 0 0 0) 0x7fee9c087cb0 con 0x7feeac06fa60 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 msgr2=0x7fee8807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 0x7fee8807b5b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feeac15b8b0 tx=0x7fee9400b390 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 msgr2=0x7feeac0c5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fee9c034040 tx=0x7fee9c003910 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 shutdown_connections 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fee880790f0 0x7fee8807b5b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac146150 0x7feeac1f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7feeac15bfc0 0x7feeac145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 --2- 172.21.15.31:0/2360325591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeac06fa60 0x7feeac0c5dc0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.005+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 >> 172.21.15.31:0/2360325591 conn(0x7feeac0cfbb0 msgr2=0x7feeac15ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:16.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.006+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 shutdown_connections 2024-09-13T08:46:16.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.006+0000 7feeb1200640 1 -- 172.21.15.31:0/2360325591 wait complete. 2024-09-13T08:46:16.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:46:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 -- 172.21.15.31:0/2739637600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 msgr2=0x7f62440cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 --2- 172.21.15.31:0/2739637600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440cdb80 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f6238005fd0 tx=0x7f623802a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 -- 172.21.15.31:0/2739637600 shutdown_connections 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 --2- 172.21.15.31:0/2739637600 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6244073860 0x7f6244073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 --2- 172.21.15.31:0/2739637600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440cdb80 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 -- 172.21.15.31:0/2739637600 >> 172.21.15.31:0/2739637600 conn(0x7f62440b18f0 msgr2=0x7f62440b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 -- 172.21.15.31:0/2739637600 shutdown_connections 2024-09-13T08:46:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.188+0000 7f624c200640 1 -- 172.21.15.31:0/2739637600 wait complete. 2024-09-13T08:46:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.190+0000 7f624c200640 1 Processor -- start 2024-09-13T08:46:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.190+0000 7f624c200640 1 -- start start 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6244073860 0x7f624414d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62440ced20 0x7f6244156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62440733e0 con 0x7f624415be30 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62440ce9e0 con 0x7f6244073860 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624c200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6244074230 con 0x7f62440ced20 2024-09-13T08:46:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59076/0 (socket says 172.21.15.31:59076) 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 -- 172.21.15.31:0/4105979350 learned_addr learned my addr 172.21.15.31:0/4105979350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f6248e00640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62440ced20 0x7f6244156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f6249800640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6244073860 0x7f624414d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 -- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62440ced20 msgr2=0x7f6244156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62440ced20 0x7f6244156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 -- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6244073860 msgr2=0x7f624414d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6244073860 0x7f624414d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.191+0000 7f624a200640 1 -- 172.21.15.31:0/4105979350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6244156e90 con 0x7f624415be30 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f624a200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d750 tx=0x7f623c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c017070 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f623c009cc0 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c00de60 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6244157170 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.192+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6244158c20 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.193+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6244158e20 con 0x7f624415be30 2024-09-13T08:46:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.195+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f623c005280 con 0x7f624415be30 2024-09-13T08:46:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.195+0000 7f6241600640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 0x7f622407b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.195+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f623c09bd80 con 0x7f624415be30 2024-09-13T08:46:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.196+0000 7f6249800640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 0x7f622407b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.196+0000 7f6249800640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 0x7f622407b5b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6238034000 tx=0x7f6238004ab0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.201+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6244158e20 con 0x7f624415be30 2024-09-13T08:46:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.419+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f6244074720 con 0x7f62240790f0 2024-09-13T08:46:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.421+0000 7f6241600640 1 -- 172.21.15.31:0/4105979350 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f6244074720 con 0x7f62240790f0 2024-09-13T08:46:16.421 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:46:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:46:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:16.422 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:46:16.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 msgr2=0x7f622407b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 0x7f622407b5b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6238034000 tx=0x7f6238004ab0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 msgr2=0x7f62440c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d750 tx=0x7f623c00dc10 comp rx=0 tx=0).stop 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 shutdown_connections 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f62240790f0 0x7f622407b5b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f624415be30 0x7f62440c9fc0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f62440ced20 0x7f6244156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 --2- 172.21.15.31:0/4105979350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6244073860 0x7f624414d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 >> 172.21.15.31:0/4105979350 conn(0x7f62440b18f0 msgr2=0x7f62440b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 shutdown_connections 2024-09-13T08:46:16.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.423+0000 7f624c200640 1 -- 172.21.15.31:0/4105979350 wait complete. 2024-09-13T08:46:16.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:46:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.602+0000 7ff4ac600640 1 -- 172.21.15.31:0/3279487310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 msgr2=0x7ff4a40cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.602+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3279487310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a40cb740 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff4a406e410 tx=0x7ff49002a540 comp rx=0 tx=0).stop 2024-09-13T08:46:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 -- 172.21.15.31:0/3279487310 shutdown_connections 2024-09-13T08:46:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3279487310 >> v2:172.21.15.45:3300/0 conn(0x7ff4a40cbc80 0x7ff4a40b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3279487310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a40cb740 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 -- 172.21.15.31:0/3279487310 >> 172.21.15.31:0/3279487310 conn(0x7ff4a40cfbb0 msgr2=0x7ff4a40cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:16.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 -- 172.21.15.31:0/3279487310 shutdown_connections 2024-09-13T08:46:16.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.603+0000 7ff4ac600640 1 -- 172.21.15.31:0/3279487310 wait complete. 2024-09-13T08:46:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.604+0000 7ff4ac600640 1 Processor -- start 2024-09-13T08:46:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.604+0000 7ff4ac600640 1 -- start start 2024-09-13T08:46:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 0x7ff4a4152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 0x7ff4a40c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a406e740 con 0x7ff4a4072560 2024-09-13T08:46:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a406e5c0 con 0x7ff4a415bfc0 2024-09-13T08:46:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.605+0000 7ff4ac600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a40b2f20 con 0x7ff4a40cbc80 2024-09-13T08:46:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59094/0 (socket says 172.21.15.31:59094) 2024-09-13T08:46:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 -- 172.21.15.31:0/3140115532 learned_addr learned my addr 172.21.15.31:0/3140115532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9200640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 0x7ff4a4152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 -- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 msgr2=0x7ff4a4152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4aa600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 0x7ff4a40c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 0x7ff4a4152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 -- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 msgr2=0x7ff4a40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 0x7ff4a40c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 -- 172.21.15.31:0/3140115532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a41f5380 con 0x7ff4a4072560 2024-09-13T08:46:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9200640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 0x7ff4a4152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4aa600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 0x7ff4a40c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.606+0000 7ff4a9c00640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c3a0 tx=0x7ff49800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.607+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498011790 con 0x7ff4a4072560 2024-09-13T08:46:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.607+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff498017070 con 0x7ff4a4072560 2024-09-13T08:46:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.607+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498011cc0 con 0x7ff4a4072560 2024-09-13T08:46:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.607+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a41f5660 con 0x7ff4a4072560 2024-09-13T08:46:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.607+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a4155bf0 con 0x7ff4a4072560 2024-09-13T08:46:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.608+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a40b3340 con 0x7ff4a4072560 2024-09-13T08:46:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.609+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7ff49801c460 con 0x7ff4a4072560 2024-09-13T08:46:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.610+0000 7ff4a1600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 0x7ff48407b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.610+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7ff49809ba00 con 0x7ff4a4072560 2024-09-13T08:46:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.610+0000 7ff4a9200640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 0x7ff48407b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.611+0000 7ff4a9200640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 0x7ff48407b5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff490034000 tx=0x7ff490004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.614+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff498063010 con 0x7ff4a4072560 2024-09-13T08:46:16.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.810+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff4a40cb310 con 0x7ff4a4072560 2024-09-13T08:46:16.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.810+0000 7ff4a1600640 1 -- 172.21.15.31:0/3140115532 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7ff498062910 con 0x7ff4a4072560 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 msgr2=0x7ff48407b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 0x7ff48407b5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff490034000 tx=0x7ff490004730 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 msgr2=0x7ff4a41f8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c3a0 tx=0x7ff49800c860 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 shutdown_connections 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff4840790f0 0x7ff48407b5b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4a415bfc0 0x7ff4a40c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff4a40cbc80 0x7ff4a4152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 --2- 172.21.15.31:0/3140115532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a4072560 0x7ff4a41f8c90 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 >> 172.21.15.31:0/3140115532 conn(0x7ff4a40cfbb0 msgr2=0x7ff4a40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 shutdown_connections 2024-09-13T08:46:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:16.813+0000 7ff4ac600640 1 -- 172.21.15.31:0/3140115532 wait complete. 2024-09-13T08:46:16.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:16.830 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol98440/043b064b-f76c-4d52-bb79-26b4d09ea412) 2024-09-13T08:46:16.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:16.895 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:46:16.932 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:46:16.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:16.933 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol98440/043b064b-f76c-4d52-bb79-26b4d09ea412) 2024-09-13T08:46:17.036 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:17.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol98440/.meta 2024-09-13T08:46:17.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:17.097 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp50380/subvol26421/e072a254-b1b5-487f-802b-66255764fab9) 2024-09-13T08:46:17.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:17.199 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:46:17.235 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:46:17.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:17.236 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp50380/subvol26421/e072a254-b1b5-487f-802b-66255764fab9) 2024-09-13T08:46:17.338 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:17.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp50380/subvol26421/.meta 2024-09-13T08:46:17.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol98440 2024-09-13T08:46:17.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 -- 172.21.15.31:0/764244439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 msgr2=0x7fd3cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:17.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 --2- 172.21.15.31:0/764244439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3cc004c80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0001830 tx=0x7fd3cc005830 comp rx=0 tx=0).stop 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3e9c00640 1 -- 172.21.15.31:0/764244439 reap_dead start 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 -- 172.21.15.31:0/764244439 shutdown_connections 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 -- 172.21.15.31:0/764244439 >> 172.21.15.31:0/764244439 conn(0x7fd3e40b1520 msgr2=0x7fd3e40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 -- 172.21.15.31:0/764244439 shutdown_connections 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.600+0000 7fd3ea600640 1 -- 172.21.15.31:0/764244439 wait complete. 2024-09-13T08:46:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.602+0000 7fd3ea600640 1 Processor -- start 2024-09-13T08:46:17.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.602+0000 7fd3ea600640 1 -- start start 2024-09-13T08:46:17.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:17.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3e40c4120 0x7fd3e41f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3e41f8010 0x7fd3e41f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4164910 con 0x7fd3cc004800 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4164790 con 0x7fd3e41f8010 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3ea600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e4164a90 con 0x7fd3e40c4120 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59114/0 (socket says 172.21.15.31:59114) 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e9200640 1 -- 172.21.15.31:0/1434917708 learned_addr learned my addr 172.21.15.31:0/1434917708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e9c00640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3e41f8010 0x7fd3e41f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e3e00640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3e40c4120 0x7fd3e41f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.603+0000 7fd3e9200640 1 -- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3e40c4120 msgr2=0x7fd3e41f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e9200640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3e40c4120 0x7fd3e41f7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e9200640 1 -- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3e41f8010 msgr2=0x7fd3e41f8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:17.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e9200640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3e41f8010 0x7fd3e41f8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e9200640 1 -- 172.21.15.31:0/1434917708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e40c45a0 con 0x7fd3cc004800 2024-09-13T08:46:17.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e9200640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006fa0 tx=0x7fd3d8003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:17.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800f760 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3d8016020 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e40c4810 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800fbc0 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.604+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e40c4dd0 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.605+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e4159b80 con 0x7fd3cc004800 2024-09-13T08:46:17.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.606+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fd3d8011430 con 0x7fd3cc004800 2024-09-13T08:46:17.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.607+0000 7fd3e0c00640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 0x7fd3c407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:17.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.608+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fd3d809c6a0 con 0x7fd3cc004800 2024-09-13T08:46:17.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.608+0000 7fd3e3e00640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 0x7fd3c407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:17.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.608+0000 7fd3e3e00640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 0x7fd3c407b580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0032000 tx=0x7fd3d0001360 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:17.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.612+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd3d8020020 con 0x7fd3cc004800 2024-09-13T08:46:17.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.835+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol98440", "target": ["mon-mgr", ""]}) -- 0x7fd3e40cc3e0 con 0x7fd3c40790c0 2024-09-13T08:46:17.842 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:17.841+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:17.842 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:17.841+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:17.842 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:17.841+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:17.842 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:17.841+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:17.843 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:17.841+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:17.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.900+0000 7fd3e0c00640 1 -- 172.21.15.31:0/1434917708 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7fd3e40cc3e0 con 0x7fd3c40790c0 2024-09-13T08:46:17.901 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/subvol98440/043b064b-f76c-4d52-bb79-26b4d09ea412 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 msgr2=0x7fd3c407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 0x7fd3c407b580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0032000 tx=0x7fd3d0001360 comp rx=0 tx=0).stop 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 msgr2=0x7fd3e40c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006fa0 tx=0x7fd3d8003a80 comp rx=0 tx=0).stop 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 shutdown_connections 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3c40790c0 0x7fd3c407b580 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3e41f8010 0x7fd3e41f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3e40c4120 0x7fd3e41f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.902+0000 7fd3ea600640 1 --2- 172.21.15.31:0/1434917708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3cc004800 0x7fd3e40c3be0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.903+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 >> 172.21.15.31:0/1434917708 conn(0x7fd3e40b1520 msgr2=0x7fd3e40b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.903+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 shutdown_connections 2024-09-13T08:46:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:17.903+0000 7fd3ea600640 1 -- 172.21.15.31:0/1434917708 wait complete. 2024-09-13T08:46:17.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol26421 subvol_grp50380 2024-09-13T08:46:18.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 -- 172.21.15.31:0/3624451826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 msgr2=0x7f53b8072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 --2- 172.21.15.31:0/3624451826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8072560 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009830 tx=0x7f53ac02a510 comp rx=0 tx=0).stop 2024-09-13T08:46:18.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 -- 172.21.15.31:0/3624451826 shutdown_connections 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 --2- 172.21.15.31:0/3624451826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8072560 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 -- 172.21.15.31:0/3624451826 >> 172.21.15.31:0/3624451826 conn(0x7f53b80cfbb0 msgr2=0x7f53b80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 -- 172.21.15.31:0/3624451826 shutdown_connections 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.087+0000 7f53bfe00640 1 -- 172.21.15.31:0/3624451826 wait complete. 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.089+0000 7f53bfe00640 1 Processor -- start 2024-09-13T08:46:18.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.089+0000 7f53bfe00640 1 -- start start 2024-09-13T08:46:18.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 0x7f53b81f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 0x7f53b80c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53b806e730 con 0x7f53b815bfc0 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53b80cc060 con 0x7f53b806fa60 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bfe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f53b80b30f0 con 0x7f53b80c3c10 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59144/0 (socket says 172.21.15.31:59144) 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 -- 172.21.15.31:0/973772094 learned_addr learned my addr 172.21.15.31:0/973772094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bd400640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 0x7f53b81f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bde00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 0x7f53b80c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 -- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 msgr2=0x7f53b80c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 0x7f53b80c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 -- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 msgr2=0x7f53b81f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 0x7f53b81f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bca00640 1 -- 172.21.15.31:0/973772094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b81f8de0 con 0x7f53b815bfc0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bd400640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 0x7f53b81f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.090+0000 7f53bde00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 0x7f53b80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53bca00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c430 tx=0x7f53a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4011790 con 0x7f53b815bfc0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53a4017070 con 0x7f53b815bfc0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4011d00 con 0x7f53b815bfc0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b81f90c0 con 0x7f53b815bfc0 2024-09-13T08:46:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.091+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b80c6db0 con 0x7f53b815bfc0 2024-09-13T08:46:18.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.092+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b80c6fb0 con 0x7f53b815bfc0 2024-09-13T08:46:18.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.093+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7f53a401c460 con 0x7f53b815bfc0 2024-09-13T08:46:18.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.094+0000 7f53b5600640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 0x7f539807b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.094+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f53a409bad0 con 0x7f53b815bfc0 2024-09-13T08:46:18.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.095+0000 7f53bd400640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 0x7f539807b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.095+0000 7f53bd400640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 0x7f539807b5b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f53ac034000 tx=0x7f53ac004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:18.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.099+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f53b80c6fb0 con 0x7f53b815bfc0 2024-09-13T08:46:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.324+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol26421", "group_name": "subvol_grp50380", "target": ["mon-mgr", ""]}) -- 0x7f53b80c32f0 con 0x7f53980790f0 2024-09-13T08:46:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.378+0000 7f53b5600640 1 -- 172.21.15.31:0/973772094 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7f53b80c32f0 con 0x7f53980790f0 2024-09-13T08:46:18.379 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp50380/subvol26421/e072a254-b1b5-487f-802b-66255764fab9 2024-09-13T08:46:18.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 msgr2=0x7f539807b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 0x7f539807b5b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f53ac034000 tx=0x7f53ac004730 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 msgr2=0x7f53b8147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c430 tx=0x7f53a400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 shutdown_connections 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53980790f0 0x7f539807b5b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53b80c3c10 0x7f53b80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53b815bfc0 0x7f53b8147e30 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 --2- 172.21.15.31:0/973772094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b806fa60 0x7f53b81f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.380+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 >> 172.21.15.31:0/973772094 conn(0x7f53b80cfbb0 msgr2=0x7f53b80cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.381+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 shutdown_connections 2024-09-13T08:46:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.381+0000 7f53bfe00640 1 -- 172.21.15.31:0/973772094 wait complete. 2024-09-13T08:46:18.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:18.398 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol98440/.meta) 2024-09-13T08:46:18.455 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:46:18.456 INFO:teuthology.orchestra.run.smithi031.stdout:version = 2 2024-09-13T08:46:18.456 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:46:18.456 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/_nogroup/subvol98440/043b064b-f76c-4d52-bb79-26b4d09ea412 2024-09-13T08:46:18.456 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:46:18.456 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:18.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:18.459 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp50380/subvol26421/.meta) 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout:version = 2 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/subvol_grp50380/subvol26421/e072a254-b1b5-487f-802b-66255764fab9 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:46:18.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:18.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol98440 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 -- 172.21.15.31:0/59110682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd41617d0 msgr2=0x7fadd40b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 --2- 172.21.15.31:0/59110682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd41617d0 0x7fadd40b3c10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fadc0005fd0 tx=0x7fadc002a540 comp rx=0 tx=0).stop 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 -- 172.21.15.31:0/59110682 shutdown_connections 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 --2- 172.21.15.31:0/59110682 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd40d4930 0x7fadd40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 --2- 172.21.15.31:0/59110682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd41617d0 0x7fadd40b3c10 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 -- 172.21.15.31:0/59110682 >> 172.21.15.31:0/59110682 conn(0x7fadd40d3e10 msgr2=0x7fadd40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 -- 172.21.15.31:0/59110682 shutdown_connections 2024-09-13T08:46:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.682+0000 7fadd3e00640 1 -- 172.21.15.31:0/59110682 wait complete. 2024-09-13T08:46:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.683+0000 7fadd3e00640 1 Processor -- start 2024-09-13T08:46:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.684+0000 7fadd3e00640 1 -- start start 2024-09-13T08:46:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.684+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.684+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd41613f0 0x7fadd41f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd41617d0 0x7fadd40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40b1940 con 0x7fadd40d4930 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd3e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40b17c0 con 0x7fadd41613f0 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd3e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fadd40b1ac0 con 0x7fadd41617d0 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59162/0 (socket says 172.21.15.31:59162) 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 -- 172.21.15.31:0/2876132677 learned_addr learned my addr 172.21.15.31:0/2876132677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2000640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd41613f0 0x7fadd41f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd3400640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd41617d0 0x7fadd40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 -- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd41617d0 msgr2=0x7fadd40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd41617d0 0x7fadd40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 -- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd41613f0 msgr2=0x7fadd41f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd41613f0 0x7fadd41f4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.685+0000 7fadd2a00640 1 -- 172.21.15.31:0/2876132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd414ad20 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadd2a00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c450 tx=0x7fadc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40052d0 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadc4005470 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4012870 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd41f53d0 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.686+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd414a2e0 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.687+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd40b1070 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.688+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fadc40035f0 con 0x7fadd40d4930 2024-09-13T08:46:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.689+0000 7fadcaa00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 0x7fadb407b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.689+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fadc409b6d0 con 0x7fadd40d4930 2024-09-13T08:46:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.689+0000 7fadd2000640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 0x7fadb407b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.690+0000 7fadd2000640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 0x7fadb407b5b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fadc0033000 tx=0x7fadc0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:18.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.693+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fadd41f53d0 con 0x7fadd40d4930 2024-09-13T08:46:18.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.915+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol98440", "target": ["mon-mgr", ""]}) -- 0x7fadd40d1eb0 con 0x7fadb40790f0 2024-09-13T08:46:18.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.930+0000 7fadcaa00640 1 -- 172.21.15.31:0/2876132677 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fadd40d1eb0 con 0x7fadb40790f0 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 msgr2=0x7fadb407b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 0x7fadb407b5b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fadc0033000 tx=0x7fadc0004730 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 msgr2=0x7fadd414a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c450 tx=0x7fadc400c910 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 shutdown_connections 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb40790f0 0x7fadb407b5b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fadd41617d0 0x7fadd40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadd41613f0 0x7fadd41f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 --2- 172.21.15.31:0/2876132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40d4930 0x7fadd414a7e0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.933+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 >> 172.21.15.31:0/2876132677 conn(0x7fadd40d3e10 msgr2=0x7fadd41ea670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.934+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 shutdown_connections 2024-09-13T08:46:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:18.934+0000 7fadd3e00640 1 -- 172.21.15.31:0/2876132677 wait complete. 2024-09-13T08:46:18.938 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.937+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.938 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.937+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.938 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.937+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.939 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.937+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.939 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.937+0000 7fe16d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol26421 subvol_grp50380 2024-09-13T08:46:18.958 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.956+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.959 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.956+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.959 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.956+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.959 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.956+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:18.959 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:18.956+0000 7fe16ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 -- 172.21.15.31:0/1387355495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa868004020 msgr2=0x7fa868004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 --2- 172.21.15.31:0/1387355495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa868004020 0x7fa868004800 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa870009b50 tx=0x7fa868005830 comp rx=0 tx=0).stop 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa883e00640 1 -- 172.21.15.31:0/1387355495 reap_dead start 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 -- 172.21.15.31:0/1387355495 shutdown_connections 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 -- 172.21.15.31:0/1387355495 >> 172.21.15.31:0/1387355495 conn(0x7fa8840d1c00 msgr2=0x7fa8840d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 -- 172.21.15.31:0/1387355495 shutdown_connections 2024-09-13T08:46:19.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.120+0000 7fa889200640 1 -- 172.21.15.31:0/1387355495 wait complete. 2024-09-13T08:46:19.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.121+0000 7fa889200640 1 Processor -- start 2024-09-13T08:46:19.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 -- start start 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa868004020 0x7fa88406e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 0x7fa88415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840d0610 con 0x7fa8841f2160 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840d0490 con 0x7fa8841f4f50 2024-09-13T08:46:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.122+0000 7fa889200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa88415f820 con 0x7fa868004020 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59208/0 (socket says 172.21.15.31:59208) 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 -- 172.21.15.31:0/4125318719 learned_addr learned my addr 172.21.15.31:0/4125318719 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa883400640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa868004020 0x7fa88406e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa883e00640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 0x7fa88415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 -- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa868004020 msgr2=0x7fa88406e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa868004020 0x7fa88406e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 -- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 msgr2=0x7fa88415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 0x7fa88415b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 -- 172.21.15.31:0/4125318719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88406dd90 con 0x7fa8841f2160 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa883e00640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 0x7fa88415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:19.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.123+0000 7fa882a00640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa870036040 tx=0x7fa870033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.124+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870038040 con 0x7fa8841f2160 2024-09-13T08:46:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.124+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa870004230 con 0x7fa8841f2160 2024-09-13T08:46:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.124+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870033860 con 0x7fa8841f2160 2024-09-13T08:46:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.124+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8841f5530 con 0x7fa8841f2160 2024-09-13T08:46:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.124+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa88406e050 con 0x7fa8841f2160 2024-09-13T08:46:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.125+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8840af600 con 0x7fa8841f2160 2024-09-13T08:46:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.126+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 102052+0+0 (secure 0 0 0) 0x7fa87002b8a0 con 0x7fa8841f2160 2024-09-13T08:46:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.127+0000 7fa87b400640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 0x7fa86407b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.127+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fa8700bc060 con 0x7fa8841f2160 2024-09-13T08:46:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.127+0000 7fa883400640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 0x7fa86407b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:19.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.128+0000 7fa883400640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 0x7fa86407b580 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa874001cd0 tx=0x7fa874006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:19.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.132+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa8700830b0 con 0x7fa8841f2160 2024-09-13T08:46:19.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:19.352+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol26421", "group_name": "subvol_grp50380", "target": ["mon-mgr", ""]}) -- 0x7fa8841f5a50 con 0x7fa8640790c0 2024-09-13T08:46:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.020+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fa87007f320 con 0x7fa8841f2160 2024-09-13T08:46:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.704+0000 7fa87b400640 1 -- 172.21.15.31:0/4125318719 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa874016070 con 0x7fa8640790c0 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 msgr2=0x7fa86407b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 0x7fa86407b580 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa874001cd0 tx=0x7fa874006040 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 msgr2=0x7fa8841f4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa870036040 tx=0x7fa870033670 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 shutdown_connections 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa8640790c0 0x7fa86407b580 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8841f4f50 0x7fa88415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8841f2160 0x7fa8841f4a10 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 --2- 172.21.15.31:0/4125318719 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa868004020 0x7fa88406e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.707+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 >> 172.21.15.31:0/4125318719 conn(0x7fa8840d1c00 msgr2=0x7fa88415cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:20.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.708+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 shutdown_connections 2024-09-13T08:46:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20.708+0000 7fa889200640 1 -- 172.21.15.31:0/4125318719 wait complete. 2024-09-13T08:46:20.731 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:46:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:20.823 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:46:20.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:20.832 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:46:20.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:20.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:21.027 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:46:21.027 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:21.027 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:21.027 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:21.063 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:21.063 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:21.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:46:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 -- 172.21.15.31:0/1314420315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb40ced20 msgr2=0x7f7bb40ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/1314420315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb40ced20 0x7f7bb40ce290 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009830 tx=0x7f7ba002a510 comp rx=0 tx=0).stop 2024-09-13T08:46:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 -- 172.21.15.31:0/1314420315 shutdown_connections 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/1314420315 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb4073860 0x7f7bb4073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/1314420315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb40ced20 0x7f7bb40ce290 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.340+0000 7f7bbbc00640 1 -- 172.21.15.31:0/1314420315 >> 172.21.15.31:0/1314420315 conn(0x7f7bb40b18f0 msgr2=0x7f7bb40b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.341+0000 7f7bbbc00640 1 -- 172.21.15.31:0/1314420315 shutdown_connections 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.341+0000 7f7bbbc00640 1 -- 172.21.15.31:0/1314420315 wait complete. 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.342+0000 7f7bbbc00640 1 Processor -- start 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.342+0000 7f7bbbc00640 1 -- start start 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb4073860 0x7f7bb41f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb40ced20 0x7f7bb4157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb40733e0 con 0x7f7bb415be30 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb40ce9e0 con 0x7f7bb4073860 2024-09-13T08:46:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.343+0000 7f7bbbc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb4074230 con 0x7f7bb40ced20 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59220/0 (socket says 172.21.15.31:59220) 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 -- 172.21.15.31:0/3917825853 learned_addr learned my addr 172.21.15.31:0/3917825853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb3e00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb40ced20 0x7f7bb4157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9200640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb4073860 0x7f7bb41f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 -- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb40ced20 msgr2=0x7f7bb4157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb40ced20 0x7f7bb4157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 -- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb4073860 msgr2=0x7f7bb41f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb4073860 0x7f7bb41f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 -- 172.21.15.31:0/3917825853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb41f4810 con 0x7f7bb415be30 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.344+0000 7f7bb9c00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00ead0 tx=0x7f7bac00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.345+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac00cde0 con 0x7f7bb415be30 2024-09-13T08:46:21.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.345+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bac004c10 con 0x7f7bb415be30 2024-09-13T08:46:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.345+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb415c6b0 con 0x7f7bb415be30 2024-09-13T08:46:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.345+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac005020 con 0x7f7bb415be30 2024-09-13T08:46:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.345+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb41f4ad0 con 0x7f7bb415be30 2024-09-13T08:46:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.346+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb4074720 con 0x7f7bb415be30 2024-09-13T08:46:21.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.347+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f7bac0051c0 con 0x7f7bb415be30 2024-09-13T08:46:21.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.348+0000 7f7bb0c00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 0x7f7b9407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:21.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.348+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f7bac099da0 con 0x7f7bb415be30 2024-09-13T08:46:21.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.348+0000 7f7bb9200640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 0x7f7b9407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:21.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.349+0000 7f7bb9200640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 0x7f7b9407b730 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4000f60 tx=0x7f7ba4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:21.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.352+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7bac016030 con 0x7f7bb415be30 2024-09-13T08:46:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.560+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7bb40b25f0 con 0x7f7bb415be30 2024-09-13T08:46:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.560+0000 7f7bb0c00640 1 -- 172.21.15.31:0/3917825853 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) ==== 80+0+0 (secure 0 0 0) 0x7f7bac06a070 con 0x7f7bb415be30 2024-09-13T08:46:21.561 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 msgr2=0x7f7b9407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 0x7f7b9407b730 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4000f60 tx=0x7f7ba4009040 comp rx=0 tx=0).stop 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 msgr2=0x7f7bb415ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00ead0 tx=0x7f7bac00ef90 comp rx=0 tx=0).stop 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 shutdown_connections 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7b94079270 0x7f7b9407b730 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bb415be30 0x7f7bb415ce70 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7bb40ced20 0x7f7bb4157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 --2- 172.21.15.31:0/3917825853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bb4073860 0x7f7bb41f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 >> 172.21.15.31:0/3917825853 conn(0x7f7bb40b18f0 msgr2=0x7f7bb415f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 shutdown_connections 2024-09-13T08:46:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:21.563+0000 7f7bbbc00640 1 -- 172.21.15.31:0/3917825853 wait complete. 2024-09-13T08:46:21.580 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:46:21.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:21.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:21.679 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:46:21.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:21.679 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:46:21.721 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:21.721 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:21.721 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:21.760 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:21.760 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:21.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:46:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 -- 172.21.15.31:0/3252657532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e7815be30 msgr2=0x7f9e780cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/3252657532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e7815be30 0x7f9e780cdb80 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f9e7809e090 tx=0x7f9e6802aa10 comp rx=0 tx=0).stop 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 -- 172.21.15.31:0/3252657532 shutdown_connections 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/3252657532 >> v2:172.21.15.45:3300/0 conn(0x7f9e780ced20 0x7f9e780ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/3252657532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e7815be30 0x7f9e780cdb80 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 -- 172.21.15.31:0/3252657532 >> 172.21.15.31:0/3252657532 conn(0x7f9e780b18f0 msgr2=0x7f9e780b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 -- 172.21.15.31:0/3252657532 shutdown_connections 2024-09-13T08:46:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.004+0000 7f9e7fe00640 1 -- 172.21.15.31:0/3252657532 wait complete. 2024-09-13T08:46:22.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.006+0000 7f9e7fe00640 1 Processor -- start 2024-09-13T08:46:22.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.006+0000 7f9e7fe00640 1 -- start start 2024-09-13T08:46:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e780ced20 0x7f9e7809d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 0x7f9e7815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e780749f0 con 0x7f9e780ce600 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78074870 con 0x7f9e7815be30 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7fe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9e78160160 con 0x7f9e780ced20 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59250/0 (socket says 172.21.15.31:59250) 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.007+0000 7f9e7d400640 1 -- 172.21.15.31:0/2459519593 learned_addr learned my addr 172.21.15.31:0/2459519593 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 -- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e780ced20 msgr2=0x7f9e7809d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e780ced20 0x7f9e7809d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7de00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 0x7f9e7815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 -- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 msgr2=0x7f9e7815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 0x7f9e7815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 -- 172.21.15.31:0/2459519593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e78157d20 con 0x7f9e780ce600 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7de00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 0x7f9e7815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7d400640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009d60 tx=0x7f9e68003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68033050 con 0x7f9e780ce600 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e68004550 con 0x7f9e780ce600 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68038450 con 0x7f9e780ce600 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.008+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e780c99b0 con 0x7f9e780ce600 2024-09-13T08:46:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.009+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e780c9f70 con 0x7f9e780ce600 2024-09-13T08:46:22.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.010+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e781605c0 con 0x7f9e780ce600 2024-09-13T08:46:22.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.011+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f9e68036070 con 0x7f9e780ce600 2024-09-13T08:46:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.012+0000 7f9e75600640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 0x7f9e5807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.012+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f9e680ba6a0 con 0x7f9e780ce600 2024-09-13T08:46:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.013+0000 7f9e7ca00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 0x7f9e5807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:22.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.013+0000 7f9e7ca00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 0x7f9e5807b730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9e781f83d0 tx=0x7f9e64005c70 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:22.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.017+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9e68084760 con 0x7f9e780ce600 2024-09-13T08:46:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.225+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9e780b25f0 con 0x7f9e780ce600 2024-09-13T08:46:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.225+0000 7f9e75600640 1 -- 172.21.15.31:0/2459519593 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) ==== 80+0+0 (secure 0 0 0) 0x7f9e68084060 con 0x7f9e780ce600 2024-09-13T08:46:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:46:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 msgr2=0x7f9e5807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 0x7f9e5807b730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9e781f83d0 tx=0x7f9e64005c70 comp rx=0 tx=0).stop 2024-09-13T08:46:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 msgr2=0x7f9e780cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009d60 tx=0x7f9e68003f90 comp rx=0 tx=0).stop 2024-09-13T08:46:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 shutdown_connections 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e58079270 0x7f9e5807b730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7815be30 0x7f9e7815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e780ced20 0x7f9e7809d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 --2- 172.21.15.31:0/2459519593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e780ce600 0x7f9e780cb930 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 >> 172.21.15.31:0/2459519593 conn(0x7f9e780b18f0 msgr2=0x7f9e780b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 shutdown_connections 2024-09-13T08:46:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.228+0000 7f9e7fe00640 1 -- 172.21.15.31:0/2459519593 wait complete. 2024-09-13T08:46:22.245 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:46:22.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:22.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:22.313 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:46:22.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:22.313 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:46:22.351 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:22.351 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:22.351 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:22.387 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:46:22.387 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:22.388 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:22.388 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:22.427 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:46:22.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.587+0000 7f1e75e00640 1 -- 172.21.15.31:0/2399140748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015bd70 msgr2=0x7f1e700cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.587+0000 7f1e75e00640 1 --2- 172.21.15.31:0/2399140748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015bd70 0x7f1e700cf850 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1e60009830 tx=0x7f1e6002a540 comp rx=0 tx=0).stop 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.587+0000 7f1e75e00640 1 -- 172.21.15.31:0/2399140748 shutdown_connections 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.587+0000 7f1e75e00640 1 --2- 172.21.15.31:0/2399140748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015bd70 0x7f1e700cf850 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.588+0000 7f1e75e00640 1 -- 172.21.15.31:0/2399140748 >> 172.21.15.31:0/2399140748 conn(0x7f1e700cc910 msgr2=0x7f1e700cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:22.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.588+0000 7f1e75e00640 1 -- 172.21.15.31:0/2399140748 shutdown_connections 2024-09-13T08:46:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.588+0000 7f1e75e00640 1 -- 172.21.15.31:0/2399140748 wait complete. 2024-09-13T08:46:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.589+0000 7f1e75e00640 1 Processor -- start 2024-09-13T08:46:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.589+0000 7f1e75e00640 1 -- start start 2024-09-13T08:46:22.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015f9e0 0x7f1e700b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1e7015fe20 0x7f1e700cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e700cec50 con 0x7f1e7015bd70 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e700cead0 con 0x7f1e7015f9e0 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.590+0000 7f1e75e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f1e700cb850 con 0x7f1e7015fe20 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59262/0 (socket says 172.21.15.31:59262) 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 -- 172.21.15.31:0/1601066289 learned_addr learned my addr 172.21.15.31:0/1601066289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 -- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1e7015fe20 msgr2=0x7f1e700cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1e7015fe20 0x7f1e700cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 -- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015f9e0 msgr2=0x7f1e700b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015f9e0 0x7f1e700b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 -- 172.21.15.31:0/1601066289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7006e8b0 con 0x7f1e7015bd70 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e6ea00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002ab30 tx=0x7f1e60003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005c90 con 0x7f1e7015bd70 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.591+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e60005020 con 0x7f1e7015bd70 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.592+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005430 con 0x7f1e7015bd70 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.592+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7006eb90 con 0x7f1e7015bd70 2024-09-13T08:46:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.592+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7006f150 con 0x7f1e7015bd70 2024-09-13T08:46:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.593+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e700cf850 con 0x7f1e7015bd70 2024-09-13T08:46:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.594+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f1e60039040 con 0x7f1e7015bd70 2024-09-13T08:46:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.595+0000 7f1e67e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 0x7f1e4c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.595+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f1e600ba3b0 con 0x7f1e7015bd70 2024-09-13T08:46:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.595+0000 7f1e66000640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 0x7f1e4c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.596+0000 7f1e66000640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 0x7f1e4c07b730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1e700ca910 tx=0x7f1e580078f0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:22.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.599+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1e60040030 con 0x7f1e7015bd70 2024-09-13T08:46:22.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.832+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e700ca940 con 0x7f1e7015bd70 2024-09-13T08:46:22.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.833+0000 7f1e67e00640 1 -- 172.21.15.31:0/1601066289 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5224 (secure 0 0 0) 0x7f1e60084250 con 0x7f1e7015bd70 2024-09-13T08:46:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:22.834 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":110,"btime":"2024-09-13T08:46:15:998581+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:22.834 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 110 2024-09-13T08:46:22.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 msgr2=0x7f1e4c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 0x7f1e4c07b730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1e700ca910 tx=0x7f1e580078f0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 msgr2=0x7f1e70070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:22.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002ab30 tx=0x7f1e60003c10 comp rx=0 tx=0).stop 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 shutdown_connections 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1e4c079270 0x7f1e4c07b730 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f1e7015fe20 0x7f1e700cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7015f9e0 0x7f1e700b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 --2- 172.21.15.31:0/1601066289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e7015bd70 0x7f1e70070020 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 >> 172.21.15.31:0/1601066289 conn(0x7f1e700cc910 msgr2=0x7f1e700acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 shutdown_connections 2024-09-13T08:46:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:22.835+0000 7f1e75e00640 1 -- 172.21.15.31:0/1601066289 wait complete. 2024-09-13T08:46:22.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 -- 172.21.15.31:0/1063970071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ce8004800 msgr2=0x7f4ce8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 --2- 172.21.15.31:0/1063970071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ce8004800 0x7f4ce8004c80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009b50 tx=0x7f4ce8005830 comp rx=0 tx=0).stop 2024-09-13T08:46:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 -- 172.21.15.31:0/1063970071 shutdown_connections 2024-09-13T08:46:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 --2- 172.21.15.31:0/1063970071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ce8004800 0x7f4ce8004c80 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 -- 172.21.15.31:0/1063970071 >> 172.21.15.31:0/1063970071 conn(0x7f4d040cc910 msgr2=0x7f4d040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:23.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 -- 172.21.15.31:0/1063970071 shutdown_connections 2024-09-13T08:46:23.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.013+0000 7f4d08e00640 1 -- 172.21.15.31:0/1063970071 wait complete. 2024-09-13T08:46:23.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.014+0000 7f4d08e00640 1 Processor -- start 2024-09-13T08:46:23.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 -- start start 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ce8004800 0x7f4d040b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0409d460 0x7f4d040b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d040cb9d0 con 0x7f4d040b4f50 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d040cb850 con 0x7f4d0409d460 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.015+0000 7f4d08e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4d040ce3c0 con 0x7f4ce8004800 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59288/0 (socket says 172.21.15.31:59288) 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 -- 172.21.15.31:0/3849615126 learned_addr learned my addr 172.21.15.31:0/3849615126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d01e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ce8004800 0x7f4d040b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4cf9400640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0409d460 0x7f4d040b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 -- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ce8004800 msgr2=0x7f4d040b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ce8004800 0x7f4d040b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 -- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0409d460 msgr2=0x7f4d040b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0409d460 0x7f4d040b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.016+0000 7f4d02800640 1 -- 172.21.15.31:0/3849615126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0415e320 con 0x7f4d040b4f50 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4d02800640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4d040cf330 tx=0x7f4cfc0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc017070 con 0x7f4d040b4f50 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cfc0108e0 con 0x7f4d040b4f50 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc010d30 con 0x7f4d040b4f50 2024-09-13T08:46:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d0409b010 con 0x7f4d040b4f50 2024-09-13T08:46:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.017+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d040b53d0 con 0x7f4d040b4f50 2024-09-13T08:46:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.018+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d040b35b0 con 0x7f4d040b4f50 2024-09-13T08:46:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.019+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f4cfc01a070 con 0x7f4d040b4f50 2024-09-13T08:46:23.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.020+0000 7f4cfaa00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 0x7f4ce007b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.020+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f4cfc099160 con 0x7f4d040b4f50 2024-09-13T08:46:23.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.020+0000 7f4d01e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 0x7f4ce007b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.021+0000 7f4d01e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 0x7f4ce007b760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4000fb0 tx=0x7f4cf4006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.024+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4d040b35b0 con 0x7f4d040b4f50 2024-09-13T08:46:23.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.256+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d040704a0 con 0x7f4d040b4f50 2024-09-13T08:46:23.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.257+0000 7f4cfaa00640 1 -- 172.21.15.31:0/3849615126 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5224 (secure 0 0 0) 0x7f4d040704a0 con 0x7f4d040b4f50 2024-09-13T08:46:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:23.258 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":110,"btime":"2024-09-13T08:46:15:998581+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:23.258 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 110 2024-09-13T08:46:23.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 msgr2=0x7f4ce007b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 0x7f4ce007b760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4000fb0 tx=0x7f4cf4006040 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 msgr2=0x7f4d041469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4d040cf330 tx=0x7f4cfc0019b0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 shutdown_connections 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4ce00792a0 0x7f4ce007b760 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d040b4f50 0x7f4d041469b0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0409d460 0x7f4d040b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 --2- 172.21.15.31:0/3849615126 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4ce8004800 0x7f4d040b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.259+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 >> 172.21.15.31:0/3849615126 conn(0x7f4d040cc910 msgr2=0x7f4d040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.260+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 shutdown_connections 2024-09-13T08:46:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.260+0000 7f4d08e00640 1 -- 172.21.15.31:0/3849615126 wait complete. 2024-09-13T08:46:23.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f8000640 1 -- 172.21.15.31:0/1338119753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4d8003c60 msgr2=0x7fe4d8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f8000640 1 --2- 172.21.15.31:0/1338119753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4d8003c60 0x7fe4d8004060 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00b6000 tx=0x7fe4d8005830 comp rx=0 tx=0).stop 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f6000640 1 -- 172.21.15.31:0/1338119753 reap_dead start 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f8000640 1 -- 172.21.15.31:0/1338119753 shutdown_connections 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f8000640 1 -- 172.21.15.31:0/1338119753 >> 172.21.15.31:0/1338119753 conn(0x7fe4f00cc910 msgr2=0x7fe4f00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.437+0000 7fe4f8000640 1 -- 172.21.15.31:0/1338119753 shutdown_connections 2024-09-13T08:46:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.438+0000 7fe4f8000640 1 -- 172.21.15.31:0/1338119753 wait complete. 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.439+0000 7fe4f8000640 1 Processor -- start 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.439+0000 7fe4f8000640 1 -- start start 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4d8003c60 0x7fe4f00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00b4a10 0x7fe4f00b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f00cc300 con 0x7fe4f00b55c0 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f00740a0 con 0x7fe4f00b4a10 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f8000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f00cc480 con 0x7fe4d8003c60 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59298/0 (socket says 172.21.15.31:59298) 2024-09-13T08:46:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.440+0000 7fe4f6000640 1 -- 172.21.15.31:0/3939102017 learned_addr learned my addr 172.21.15.31:0/3939102017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f4c00640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00b4a10 0x7fe4f00b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f5600640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4d8003c60 0x7fe4f00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 -- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4d8003c60 msgr2=0x7fe4f00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4d8003c60 0x7fe4f00b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 -- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00b4a10 msgr2=0x7fe4f00b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00b4a10 0x7fe4f00b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 -- 172.21.15.31:0/3939102017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f00b5b20 con 0x7fe4f00b55c0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f6000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00ceec0 tx=0x7fe4e8001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011840 con 0x7fe4f00b55c0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4e8018070 con 0x7fe4f00b55c0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f01f8780 con 0x7fe4f00b55c0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011c60 con 0x7fe4f00b55c0 2024-09-13T08:46:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.441+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f01f8d40 con 0x7fe4f00b55c0 2024-09-13T08:46:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.443+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f00b1e80 con 0x7fe4f00b55c0 2024-09-13T08:46:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.444+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fe4e801d460 con 0x7fe4f00b55c0 2024-09-13T08:46:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.445+0000 7fe4ed600640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 0x7fe4d007b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.445+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fe4e8099a90 con 0x7fe4f00b55c0 2024-09-13T08:46:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.445+0000 7fe4f5600640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 0x7fe4d007b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.446+0000 7fe4f5600640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 0x7fe4d007b760 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4034000 tx=0x7fe4e402d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.449+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fe4e80638f0 con 0x7fe4f00b55c0 2024-09-13T08:46:23.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.643+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4f00af5d0 con 0x7fe4f00b55c0 2024-09-13T08:46:23.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.644+0000 7fe4ed600640 1 -- 172.21.15.31:0/3939102017 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+16702 (secure 0 0 0) 0x7fe4e80631f0 con 0x7fe4f00b55c0 2024-09-13T08:46:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:23.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":164,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:06.943667+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:03.200753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:04.357008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.646+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 msgr2=0x7fe4d007b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.646+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 0x7fe4d007b760 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4034000 tx=0x7fe4e402d040 comp rx=0 tx=0).stop 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.646+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 msgr2=0x7fe4f009ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.646+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00ceec0 tx=0x7fe4e8001800 comp rx=0 tx=0).stop 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 shutdown_connections 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe4d00792a0 0x7fe4d007b760 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4f00b55c0 0x7fe4f009ca20 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00b4a10 0x7fe4f00b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 --2- 172.21.15.31:0/3939102017 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe4d8003c60 0x7fe4f00b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 >> 172.21.15.31:0/3939102017 conn(0x7fe4f00cc910 msgr2=0x7fe4f00b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 shutdown_connections 2024-09-13T08:46:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.647+0000 7fe4f8000640 1 -- 172.21.15.31:0/3939102017 wait complete. 2024-09-13T08:46:23.664 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:46:23.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 -- 172.21.15.31:0/145276133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff384004d40 msgr2=0x7ff3840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 --2- 172.21.15.31:0/145276133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff384004d40 0x7ff3840051c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff394009ed0 tx=0x7ff384005830 comp rx=0 tx=0).stop 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a1e00640 1 -- 172.21.15.31:0/145276133 reap_dead start 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 -- 172.21.15.31:0/145276133 shutdown_connections 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 -- 172.21.15.31:0/145276133 >> 172.21.15.31:0/145276133 conn(0x7ff39c06cb00 msgr2=0x7ff39c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 -- 172.21.15.31:0/145276133 shutdown_connections 2024-09-13T08:46:23.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.826+0000 7ff3a2800640 1 -- 172.21.15.31:0/145276133 wait complete. 2024-09-13T08:46:23.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.828+0000 7ff3a2800640 1 Processor -- start 2024-09-13T08:46:23.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.828+0000 7ff3a2800640 1 -- start start 2024-09-13T08:46:23.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff384004d40 0x7ff39c1f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 0x7ff39c154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c152680 con 0x7ff39c156570 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0703b0 con 0x7ff384004d40 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a2800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff39c152800 con 0x7ff39c153ff0 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59312/0 (socket says 172.21.15.31:59312) 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a0a00640 1 -- 172.21.15.31:0/1560491076 learned_addr learned my addr 172.21.15.31:0/1560491076 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a1400640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff384004d40 0x7ff39c1f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.829+0000 7ff3a1e00640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 0x7ff39c154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 -- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 msgr2=0x7ff39c154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 0x7ff39c154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 -- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff384004d40 msgr2=0x7ff39c1f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff384004d40 0x7ff39c1f5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 -- 172.21.15.31:0/1560491076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c154cb0 con 0x7ff39c156570 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a1e00640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 0x7ff39c154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a0a00640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7ff388011b10 tx=0x7ff388009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38800fc80 con 0x7ff39c156570 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff38800fe20 con 0x7ff39c156570 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39c154f90 con 0x7ff39c156570 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.830+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388004280 con 0x7ff39c156570 2024-09-13T08:46:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.831+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff39c155550 con 0x7ff39c156570 2024-09-13T08:46:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.832+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39c06f9b0 con 0x7ff39c156570 2024-09-13T08:46:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.833+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7ff3880035f0 con 0x7ff39c156570 2024-09-13T08:46:23.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.834+0000 7ff399600640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 0x7ff37c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.834+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7ff388099ea0 con 0x7ff39c156570 2024-09-13T08:46:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.834+0000 7ff3a1400640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 0x7ff37c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:23.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.835+0000 7ff3a1400640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 0x7ff37c07b730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff390000fb0 tx=0x7ff390006040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:23.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:23.838+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff3880636f0 con 0x7ff39c156570 2024-09-13T08:46:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.074+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff39c06f1b0 con 0x7ff39c156570 2024-09-13T08:46:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.075+0000 7ff399600640 1 -- 172.21.15.31:0/1560491076 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5224 (secure 0 0 0) 0x7ff388068120 con 0x7ff39c156570 2024-09-13T08:46:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:24.076 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":110,"btime":"2024-09-13T08:46:15:998581+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 110 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 msgr2=0x7ff37c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 0x7ff37c07b730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff390000fb0 tx=0x7ff390006040 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 msgr2=0x7ff39c1569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7ff388011b10 tx=0x7ff388009f10 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 shutdown_connections 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff37c079270 0x7ff37c07b730 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff39c153ff0 0x7ff39c154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39c156570 0x7ff39c1569f0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.077+0000 7ff3a2800640 1 --2- 172.21.15.31:0/1560491076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff384004d40 0x7ff39c1f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.078+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 >> 172.21.15.31:0/1560491076 conn(0x7ff39c06cb00 msgr2=0x7ff39c0c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.078+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 shutdown_connections 2024-09-13T08:46:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.078+0000 7ff3a2800640 1 -- 172.21.15.31:0/1560491076 wait complete. 2024-09-13T08:46:24.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:24.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 -- 172.21.15.31:0/3763404529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2e8004e60 msgr2=0x7fc2e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 --2- 172.21.15.31:0/3763404529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2e8004e60 0x7fc2e800b9c0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8001dc0 tx=0x7fc2e8005830 comp rx=0 tx=0).stop 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc306200640 1 -- 172.21.15.31:0/3763404529 reap_dead start 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 -- 172.21.15.31:0/3763404529 shutdown_connections 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 -- 172.21.15.31:0/3763404529 >> 172.21.15.31:0/3763404529 conn(0x7fc3000cc910 msgr2=0x7fc3000cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 -- 172.21.15.31:0/3763404529 shutdown_connections 2024-09-13T08:46:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.257+0000 7fc308200640 1 -- 172.21.15.31:0/3763404529 wait complete. 2024-09-13T08:46:24.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.259+0000 7fc308200640 1 Processor -- start 2024-09-13T08:46:24.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.259+0000 7fc308200640 1 -- start start 2024-09-13T08:46:24.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 0x7fc30009d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 0x7fc3000ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001600f0 con 0x7fc30015d7d0 2024-09-13T08:46:24.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000cb850 con 0x7fc30015a3d0 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc308200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc30009be60 con 0x7fc2e8004e60 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc306200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc306200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59320/0 (socket says 172.21.15.31:59320) 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.260+0000 7fc306200640 1 -- 172.21.15.31:0/802372322 learned_addr learned my addr 172.21.15.31:0/802372322 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc305800640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 0x7fc30009d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc304e00640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 0x7fc3000ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 -- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 msgr2=0x7fc30009d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 0x7fc30009d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 -- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 msgr2=0x7fc3000ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 0x7fc3000ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 -- 172.21.15.31:0/802372322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30015a180 con 0x7fc30015d7d0 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc305800640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 0x7fc30009d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:24.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc304e00640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 0x7fc3000ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc306200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8001eb0 tx=0x7fc2f8006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80062b0 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f8006450 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8006840 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.261+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30009c0b0 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.262+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30015c490 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.263+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000ce5d0 con 0x7fc30015d7d0 2024-09-13T08:46:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.264+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fc2f8003a30 con 0x7fc30015d7d0 2024-09-13T08:46:24.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.265+0000 7fc2fd600640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 0x7fc2e007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.265+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7fc2f80b94e0 con 0x7fc30015d7d0 2024-09-13T08:46:24.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.265+0000 7fc305800640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 0x7fc2e007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.266+0000 7fc305800640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 0x7fc2e007b730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000fb0 tx=0x7fc2f4006720 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:24.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.269+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc3000ce5d0 con 0x7fc30015d7d0 2024-09-13T08:46:24.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.508+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3000cc3f0 con 0x7fc30015d7d0 2024-09-13T08:46:24.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.508+0000 7fc2fd600640 1 -- 172.21.15.31:0/802372322 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5224 (secure 0 0 0) 0x7fc2f8083380 con 0x7fc30015d7d0 2024-09-13T08:46:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:24.509 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":110,"btime":"2024-09-13T08:46:15:998581+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:05.396638+0000","modified":"2024-09-13T08:46:08.400907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8266":{"gid":8266,"name":"d","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/98622202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":98622202},{"type":"v1","addr":"172.21.15.45:6835","nonce":98622202}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8266,"qdb_cluster":[8266]},"id":12}]} 2024-09-13T08:46:24.510 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 110 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 msgr2=0x7fc2e007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 0x7fc2e007b730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000fb0 tx=0x7fc2f4006720 comp rx=0 tx=0).stop 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 msgr2=0x7fc30015e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8001eb0 tx=0x7fc2f8006ca0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 shutdown_connections 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079270 0x7fc2e007b730 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc30015d7d0 0x7fc30015e070 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30015a3d0 0x7fc3000ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 --2- 172.21.15.31:0/802372322 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8004e60 0x7fc30009d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 >> 172.21.15.31:0/802372322 conn(0x7fc3000cc910 msgr2=0x7fc3000acf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 shutdown_connections 2024-09-13T08:46:24.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.511+0000 7fc308200640 1 -- 172.21.15.31:0/802372322 wait complete. 2024-09-13T08:46:24.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 -- 172.21.15.31:0/2045724842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 msgr2=0x7f33c80cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 --2- 172.21.15.31:0/2045724842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80cf850 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009830 tx=0x7f33bc02a510 comp rx=0 tx=0).stop 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 -- 172.21.15.31:0/2045724842 shutdown_connections 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 --2- 172.21.15.31:0/2045724842 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c815f9e0 0x7f33c815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 --2- 172.21.15.31:0/2045724842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80cf850 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 -- 172.21.15.31:0/2045724842 >> 172.21.15.31:0/2045724842 conn(0x7f33c80cc910 msgr2=0x7f33c80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 -- 172.21.15.31:0/2045724842 shutdown_connections 2024-09-13T08:46:24.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.686+0000 7f33d0000640 1 -- 172.21.15.31:0/2045724842 wait complete. 2024-09-13T08:46:24.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.688+0000 7f33d0000640 1 Processor -- start 2024-09-13T08:46:24.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.688+0000 7f33d0000640 1 -- start start 2024-09-13T08:46:24.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 0x7f33c8159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33c815f9e0 0x7f33c8159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33c80cb9d0 con 0x7f33c815bd70 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33c80cb850 con 0x7f33c815f9e0 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33d0000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f33c80cc360 con 0x7f33c80ce3c0 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33ccc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33ccc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59330/0 (socket says 172.21.15.31:59330) 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33ccc00640 1 -- 172.21.15.31:0/1730313952 learned_addr learned my addr 172.21.15.31:0/1730313952 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33cd600640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 0x7f33c8159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.689+0000 7f33ce000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33c815f9e0 0x7f33c8159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 -- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 msgr2=0x7f33c8159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 0x7f33c8159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 -- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33c815f9e0 msgr2=0x7f33c8159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33c815f9e0 0x7f33c8159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 -- 172.21.15.31:0/1730313952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c8159c90 con 0x7f33c815bd70 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33cd600640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 0x7f33c8159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33ccc00640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d700 tx=0x7f33b400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4017070 con 0x7f33c815bd70 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33b4004310 con 0x7f33c815bd70 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4009a80 con 0x7f33c815bd70 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c80b4580 con 0x7f33c815bd70 2024-09-13T08:46:24.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.690+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c815de70 con 0x7f33c815bd70 2024-09-13T08:46:24.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.691+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c815d9b0 con 0x7f33c815bd70 2024-09-13T08:46:24.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.693+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f33b4009c20 con 0x7f33c815bd70 2024-09-13T08:46:24.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.693+0000 7f33c5600640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 0x7f33a807b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:24.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.694+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f33b4099570 con 0x7f33c815bd70 2024-09-13T08:46:24.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.694+0000 7f33cd600640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 0x7f33a807b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:24.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.694+0000 7f33cd600640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 0x7f33a807b760 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f33bc034000 tx=0x7f33bc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:24.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.698+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f33c815d9b0 con 0x7f33c815bd70 2024-09-13T08:46:24.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.894+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33c815ff90 con 0x7f33c815bd70 2024-09-13T08:46:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.895+0000 7f33c5600640 1 -- 172.21.15.31:0/1730313952 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+16702 (secure 0 0 0) 0x7f33b4063390 con 0x7f33c815bd70 2024-09-13T08:46:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:24.898 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":164,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:06.943667+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:03.200753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:04.357008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.897+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 msgr2=0x7f33a807b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.897+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 0x7f33a807b760 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f33bc034000 tx=0x7f33bc004730 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.897+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 msgr2=0x7f33c80b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.897+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d700 tx=0x7f33b400dbc0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 shutdown_connections 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f33a80792a0 0x7f33a807b760 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33c815f9e0 0x7f33c8159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33c815bd70 0x7f33c80b3e30 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 --2- 172.21.15.31:0/1730313952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f33c80ce3c0 0x7f33c8159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 >> 172.21.15.31:0/1730313952 conn(0x7f33c80cc910 msgr2=0x7f33c80ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 shutdown_connections 2024-09-13T08:46:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:24.898+0000 7f33d0000640 1 -- 172.21.15.31:0/1730313952 wait complete. 2024-09-13T08:46:24.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:46:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 -- 172.21.15.31:0/1941689927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 msgr2=0x7f69b40d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 --2- 172.21.15.31:0/1941689927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40d0790 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009830 tx=0x7f69a802a5b0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 -- 172.21.15.31:0/1941689927 shutdown_connections 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 --2- 172.21.15.31:0/1941689927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40d0790 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 -- 172.21.15.31:0/1941689927 >> 172.21.15.31:0/1941689927 conn(0x7f69b40d3f50 msgr2=0x7f69b40d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 -- 172.21.15.31:0/1941689927 shutdown_connections 2024-09-13T08:46:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.080+0000 7f69bac00640 1 -- 172.21.15.31:0/1941689927 wait complete. 2024-09-13T08:46:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.082+0000 7f69bac00640 1 Processor -- start 2024-09-13T08:46:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.082+0000 7f69bac00640 1 -- start start 2024-09-13T08:46:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69b40d0cd0 0x7f69b40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69b40d14a0 con 0x7f69b40cfa70 2024-09-13T08:46:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69b40d1320 con 0x7f69b415e160 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69bac00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f69b40d1620 con 0x7f69b40d0cd0 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69b9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69b9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44054/0 (socket says 172.21.15.31:44054) 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69b9800640 1 -- 172.21.15.31:0/773209252 learned_addr learned my addr 172.21.15.31:0/773209252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69b8e00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69b40d0cd0 0x7f69b40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.083+0000 7f69ba200640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 -- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69b40d0cd0 msgr2=0x7f69b40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69b40d0cd0 0x7f69b40ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 -- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 msgr2=0x7f69b40cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 -- 172.21.15.31:0/773209252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b40cdf50 con 0x7f69b40cfa70 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69ba200640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b9800640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f69a8000c00 tx=0x7f69a8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8031d60 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69a8033e10 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8033310 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.084+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b40b8e30 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.085+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b40b93f0 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.086+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b40c9490 con 0x7f69b40cfa70 2024-09-13T08:46:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.087+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f69a80334b0 con 0x7f69b40cfa70 2024-09-13T08:46:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.087+0000 7f69b1600640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 0x7f699407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.088+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5753+0+0 (secure 0 0 0) 0x7f69a80ba690 con 0x7f69b40cfa70 2024-09-13T08:46:25.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.088+0000 7f69b8e00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 0x7f699407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.088+0000 7f69b8e00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 0x7f699407b730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69b40cc580 tx=0x7f69a0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:25.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.092+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f69a803f030 con 0x7f69b40cfa70 2024-09-13T08:46:25.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.327+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f69b40b9730 con 0x7f69b40cfa70 2024-09-13T08:46:25.328 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:46:25.328+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:46:25.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.330+0000 7f69b1600640 1 -- 172.21.15.31:0/773209252 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) ==== 161+0+0 (secure 0 0 0) 0x7f69a8084530 con 0x7f69b40cfa70 2024-09-13T08:46:25.331 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:46:25.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 msgr2=0x7f699407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 0x7f699407b730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69b40cc580 tx=0x7f69a0009040 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 msgr2=0x7f69b406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f69a8000c00 tx=0x7f69a8033c20 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 shutdown_connections 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6994079270 0x7f699407b730 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b415e160 0x7f69b40cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f69b40d0cd0 0x7f69b40ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 --2- 172.21.15.31:0/773209252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69b40cfa70 0x7f69b406e580 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 >> 172.21.15.31:0/773209252 conn(0x7f69b40d3f50 msgr2=0x7f69b40c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 shutdown_connections 2024-09-13T08:46:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.333+0000 7f69bac00640 1 -- 172.21.15.31:0/773209252 wait complete. 2024-09-13T08:46:25.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:46:25.404 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.514+0000 7fd1de400640 1 -- 172.21.15.31:0/1156741657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 msgr2=0x7fd1d80d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.514+0000 7fd1de400640 1 --2- 172.21.15.31:0/1156741657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80d0690 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80724f0 tx=0x7fd1c802a560 comp rx=0 tx=0).stop 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 -- 172.21.15.31:0/1156741657 shutdown_connections 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 --2- 172.21.15.31:0/1156741657 >> v2:172.21.15.45:3300/0 conn(0x7fd1d80cb570 0x7fd1d80cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 --2- 172.21.15.31:0/1156741657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80d0690 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 -- 172.21.15.31:0/1156741657 >> 172.21.15.31:0/1156741657 conn(0x7fd1d80cfb70 msgr2=0x7fd1d80cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 -- 172.21.15.31:0/1156741657 shutdown_connections 2024-09-13T08:46:25.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.515+0000 7fd1de400640 1 -- 172.21.15.31:0/1156741657 wait complete. 2024-09-13T08:46:25.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.516+0000 7fd1de400640 1 Processor -- start 2024-09-13T08:46:25.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.516+0000 7fd1de400640 1 -- start start 2024-09-13T08:46:25.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1d80cb570 0x7fd1d81522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d806f810 con 0x7fd1d815c040 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d806f690 con 0x7fd1d8072560 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1de400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d806f990 con 0x7fd1d80cb570 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.517+0000 7fd1d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:42638/0 (socket says 172.21.15.31:42638) 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7400640 1 -- 172.21.15.31:0/2506297350 learned_addr learned my addr 172.21.15.31:0/2506297350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44074/0 (socket says 172.21.15.31:44074) 2024-09-13T08:46:25.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1cea00640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1d80cb570 0x7fd1d81522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 -- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1d80cb570 msgr2=0x7fd1d81522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1d80cb570 0x7fd1d81522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 -- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 msgr2=0x7fd1d80ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:25.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 -- 172.21.15.31:0/2506297350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d80aea10 con 0x7fd1d815c040 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.518+0000 7fd1d7e00640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000cae0 tx=0x7fd1d000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.519+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000ae30 con 0x7fd1d815c040 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.519+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1d000f560 con 0x7fd1d815c040 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.519+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000f9b0 con 0x7fd1d815c040 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.519+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d80aecf0 con 0x7fd1d815c040 2024-09-13T08:46:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.519+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d80b0b50 con 0x7fd1d815c040 2024-09-13T08:46:25.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.520+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d806fe30 con 0x7fd1d815c040 2024-09-13T08:46:25.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.521+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fd1d000fb50 con 0x7fd1d815c040 2024-09-13T08:46:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.522+0000 7fd1cfe00640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 0x7fd1b407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.523+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5839+0+0 (secure 0 0 0) 0x7fd1d0099df0 con 0x7fd1d815c040 2024-09-13T08:46:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.523+0000 7fd1d7400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 0x7fd1b407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.523+0000 7fd1d7400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 0x7fd1b407b730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80096c0 tx=0x7fd1c8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:25.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.527+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd1d0014030 con 0x7fd1d815c040 2024-09-13T08:46:25.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:25.762+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd1d80b0d20 con 0x7fd1d815c040 2024-09-13T08:46:26.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.332+0000 7fd1cfe00640 1 -- 172.21.15.31:0/2506297350 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v112) ==== 104+0+0 (secure 0 0 0) 0x7fd1d0063640 con 0x7fd1d815c040 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 msgr2=0x7fd1b407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 0x7fd1b407b730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80096c0 tx=0x7fd1c8004730 comp rx=0 tx=0).stop 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 msgr2=0x7fd1d80cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000cae0 tx=0x7fd1d000a250 comp rx=0 tx=0).stop 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 shutdown_connections 2024-09-13T08:46:26.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd1b4079270 0x7fd1b407b730 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1d815c040 0x7fd1d80cd950 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd1d80cb570 0x7fd1d81522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 --2- 172.21.15.31:0/2506297350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1d8072560 0x7fd1d80ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 >> 172.21.15.31:0/2506297350 conn(0x7fd1d80cfb70 msgr2=0x7fd1d80cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 shutdown_connections 2024-09-13T08:46:26.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.335+0000 7fd1de400640 1 -- 172.21.15.31:0/2506297350 wait complete. 2024-09-13T08:46:26.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:46:26.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 -- 172.21.15.31:0/914337407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b4004d40 msgr2=0x7f15b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:26.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 --2- 172.21.15.31:0/914337407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b4004d40 0x7f15b40051c0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009f00 tx=0x7f15b4005830 comp rx=0 tx=0).stop 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d1800640 1 -- 172.21.15.31:0/914337407 reap_dead start 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 -- 172.21.15.31:0/914337407 shutdown_connections 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 -- 172.21.15.31:0/914337407 >> 172.21.15.31:0/914337407 conn(0x7f15cc0cfbb0 msgr2=0x7f15cc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 -- 172.21.15.31:0/914337407 shutdown_connections 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.520+0000 7f15d3800640 1 -- 172.21.15.31:0/914337407 wait complete. 2024-09-13T08:46:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.521+0000 7f15d3800640 1 Processor -- start 2024-09-13T08:46:26.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 -- start start 2024-09-13T08:46:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15b4004d40 0x7f15cc0c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15cc0c9310 0x7f15cc0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc06efd0 con 0x7f15cc0ab470 2024-09-13T08:46:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc06ee50 con 0x7f15cc0c9310 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.522+0000 7f15d3800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc06f150 con 0x7f15b4004d40 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44088/0 (socket says 172.21.15.31:44088) 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 -- 172.21.15.31:0/977433653 learned_addr learned my addr 172.21.15.31:0/977433653 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d0e00640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15b4004d40 0x7f15cc0c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15cbe00640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15cc0c9310 0x7f15cc0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 -- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15b4004d40 msgr2=0x7f15cc0c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15b4004d40 0x7f15cc0c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 -- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15cc0c9310 msgr2=0x7f15cc0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15cc0c9310 0x7f15cc0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 -- 172.21.15.31:0/977433653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc0ab910 con 0x7f15cc0ab470 2024-09-13T08:46:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.523+0000 7f15d1800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009f00 tx=0x7f15c4033dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.524+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4043070 con 0x7f15cc0ab470 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.524+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15c4003c60 con 0x7f15cc0ab470 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.524+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4038040 con 0x7f15cc0ab470 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.524+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc0a96d0 con 0x7f15cc0ab470 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.524+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc0c71c0 con 0x7f15cc0ab470 2024-09-13T08:46:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.525+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15cc0cd780 con 0x7f15cc0ab470 2024-09-13T08:46:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.526+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f15c4004290 con 0x7f15cc0ab470 2024-09-13T08:46:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.528+0000 7f15c8c00640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 0x7f15ac07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.528+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5839+0+0 (secure 0 0 0) 0x7f15c40b95a0 con 0x7f15cc0ab470 2024-09-13T08:46:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.528+0000 7f15d0e00640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 0x7f15ac07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.529+0000 7f15d0e00640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 0x7f15ac07b730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f15c0001240 tx=0x7f15c0001110 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:26.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.532+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f15c4082df0 con 0x7f15cc0ab470 2024-09-13T08:46:26.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:26.725+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f15cc0d02a0 con 0x7f15cc0ab470 2024-09-13T08:46:27.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.337+0000 7f15c8c00640 1 -- 172.21.15.31:0/977433653 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v166) ==== 181+0+0 (secure 0 0 0) 0x7f15c4087820 con 0x7f15cc0ab470 2024-09-13T08:46:27.338 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.340+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 msgr2=0x7f15ac07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 0x7f15ac07b730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f15c0001240 tx=0x7f15c0001110 comp rx=0 tx=0).stop 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 msgr2=0x7f15cc156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009f00 tx=0x7f15c4033dc0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 shutdown_connections 2024-09-13T08:46:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f15ac079270 0x7f15ac07b730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15cc0ab470 0x7f15cc156dc0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15cc0c9310 0x7f15cc0aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 --2- 172.21.15.31:0/977433653 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f15b4004d40 0x7f15cc0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 >> 172.21.15.31:0/977433653 conn(0x7f15cc0cfbb0 msgr2=0x7f15cc0ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 shutdown_connections 2024-09-13T08:46:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.341+0000 7f15d3800640 1 -- 172.21.15.31:0/977433653 wait complete. 2024-09-13T08:46:27.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 -- 172.21.15.31:0/2513107698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41613f0 msgr2=0x7faeb40b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 --2- 172.21.15.31:0/2513107698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41613f0 0x7faeb40b3500 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faea4009830 tx=0x7faea402a480 comp rx=0 tx=0).stop 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 -- 172.21.15.31:0/2513107698 shutdown_connections 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 --2- 172.21.15.31:0/2513107698 >> v2:172.21.15.45:3300/0 conn(0x7faeb40d4930 0x7faeb40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 --2- 172.21.15.31:0/2513107698 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41613f0 0x7faeb40b3500 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.531+0000 7faeb9c00640 1 -- 172.21.15.31:0/2513107698 >> 172.21.15.31:0/2513107698 conn(0x7faeb40d3e10 msgr2=0x7faeb40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.532+0000 7faeb9c00640 1 -- 172.21.15.31:0/2513107698 shutdown_connections 2024-09-13T08:46:27.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.532+0000 7faeb9c00640 1 -- 172.21.15.31:0/2513107698 wait complete. 2024-09-13T08:46:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.533+0000 7faeb9c00640 1 Processor -- start 2024-09-13T08:46:27.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.533+0000 7faeb9c00640 1 -- start start 2024-09-13T08:46:27.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeb40d4930 0x7faeb415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:27.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:27.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41617d0 0x7faeb40cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faeb40b1940 con 0x7faeb41613f0 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faeb40b17c0 con 0x7faeb40d4930 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeb9c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7faeb40b1ac0 con 0x7faeb41617d0 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.534+0000 7faeab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44092/0 (socket says 172.21.15.31:44092) 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 -- 172.21.15.31:0/3320975480 learned_addr learned my addr 172.21.15.31:0/3320975480 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeb9200640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41617d0 0x7faeb40cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:27.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeb3e00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeb40d4930 0x7faeb415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:27.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 -- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41617d0 msgr2=0x7faeb40cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:27.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41617d0 0x7faeb40cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 -- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeb40d4930 msgr2=0x7faeb415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:27.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeb40d4930 0x7faeb415d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 -- 172.21.15.31:0/3320975480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb415d8f0 con 0x7faeb41613f0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.535+0000 7faeab400640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c430 tx=0x7fae9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.536+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c011790 con 0x7faeb41613f0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.536+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb40c8540 con 0x7faeb41613f0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.536+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae9c017070 con 0x7faeb41613f0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.536+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c011cc0 con 0x7faeb41613f0 2024-09-13T08:46:27.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.536+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb40cb360 con 0x7faeb41613f0 2024-09-13T08:46:27.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.537+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb40b1050 con 0x7faeb41613f0 2024-09-13T08:46:27.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.538+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fae9c01c460 con 0x7faeb41613f0 2024-09-13T08:46:27.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.539+0000 7faeb0c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 0x7fae9007b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:27.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.539+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 5401+0+0 (secure 0 0 0) 0x7fae9c098fd0 con 0x7faeb41613f0 2024-09-13T08:46:27.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.539+0000 7faeb3e00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 0x7fae9007b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:27.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.540+0000 7faeb3e00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 0x7fae9007b760 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faea4034000 tx=0x7faea4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:27.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.543+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fae9c021020 con 0x7faeb41613f0 2024-09-13T08:46:27.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:27.740+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7faeb40d2750 con 0x7faeb41613f0 2024-09-13T08:46:28.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.339+0000 7faeb0c00640 1 -- 172.21.15.31:0/3320975480 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v167) ==== 169+0+0 (secure 0 0 0) 0x7faeb415d8f0 con 0x7faeb41613f0 2024-09-13T08:46:28.341 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 msgr2=0x7fae9007b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 0x7fae9007b760 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faea4034000 tx=0x7faea4004730 comp rx=0 tx=0).stop 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 msgr2=0x7faeb40cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c430 tx=0x7fae9c00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 shutdown_connections 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae900792a0 0x7fae9007b760 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faeb41617d0 0x7faeb40cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faeb41613f0 0x7faeb40cf760 secure :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c430 tx=0x7fae9c00c8f0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 --2- 172.21.15.31:0/3320975480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeb40d4930 0x7faeb415d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.343+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 >> 172.21.15.31:0/3320975480 conn(0x7faeb40d3e10 msgr2=0x7faeb415e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.344+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 shutdown_connections 2024-09-13T08:46:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.344+0000 7faeb9c00640 1 -- 172.21.15.31:0/3320975480 wait complete. 2024-09-13T08:46:28.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 -- 172.21.15.31:0/1717706383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c15be30 msgr2=0x7fae1c0cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 --2- 172.21.15.31:0/1717706383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c15be30 0x7fae1c0cdb80 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0b5020 tx=0x7fae1002a530 comp rx=0 tx=0).stop 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 -- 172.21.15.31:0/1717706383 shutdown_connections 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 --2- 172.21.15.31:0/1717706383 >> v2:172.21.15.45:3300/0 conn(0x7fae1c073860 0x7fae1c073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 --2- 172.21.15.31:0/1717706383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c15be30 0x7fae1c0cdb80 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.528+0000 7fae24800640 1 -- 172.21.15.31:0/1717706383 >> 172.21.15.31:0/1717706383 conn(0x7fae1c0b18f0 msgr2=0x7fae1c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.529+0000 7fae24800640 1 -- 172.21.15.31:0/1717706383 shutdown_connections 2024-09-13T08:46:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.529+0000 7fae24800640 1 -- 172.21.15.31:0/1717706383 wait complete. 2024-09-13T08:46:28.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.530+0000 7fae24800640 1 Processor -- start 2024-09-13T08:46:28.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.530+0000 7fae24800640 1 -- start start 2024-09-13T08:46:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 0x7fae1c09cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae1c15be30 0x7fae1c09d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c0733e0 con 0x7fae1c073860 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c0ce9e0 con 0x7fae1c0ced20 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae24800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fae1c0741f0 con 0x7fae1c15be30 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae21e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae21e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44108/0 (socket says 172.21.15.31:44108) 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.531+0000 7fae21e00640 1 -- 172.21.15.31:0/970597113 learned_addr learned my addr 172.21.15.31:0/970597113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21400640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 0x7fae1c09cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae22800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae1c15be30 0x7fae1c09d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 -- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae1c15be30 msgr2=0x7fae1c09d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae1c15be30 0x7fae1c09d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 -- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 msgr2=0x7fae1c09cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 0x7fae1c09cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 -- 172.21.15.31:0/970597113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c1551a0 con 0x7fae1c073860 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21400640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 0x7fae1c09cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae21e00640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fae10009920 tx=0x7fae10033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10031d60 con 0x7fae1c073860 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae10033e10 con 0x7fae1c073860 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10005260 con 0x7fae1c073860 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.532+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c154ec0 con 0x7fae1c073860 2024-09-13T08:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.533+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c1f4200 con 0x7fae1c073860 2024-09-13T08:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.534+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1c073cb0 con 0x7fae1c073860 2024-09-13T08:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.535+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fae10005400 con 0x7fae1c073860 2024-09-13T08:46:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.536+0000 7fae16000640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 0x7fae0007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.537+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7fae100ba570 con 0x7fae1c073860 2024-09-13T08:46:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.537+0000 7fae21400640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 0x7fae0007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.537+0000 7fae21400640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 0x7fae0007b730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fae1c157c10 tx=0x7fae0c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.541+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fae100c2020 con 0x7fae1c073860 2024-09-13T08:46:28.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.737+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fae1c0b25f0 con 0x7fae1c073860 2024-09-13T08:46:28.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.740+0000 7fae16000640 1 -- 172.21.15.31:0/970597113 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7fae1003f030 con 0x7fae1c073860 2024-09-13T08:46:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 msgr2=0x7fae0007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 0x7fae0007b730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fae1c157c10 tx=0x7fae0c009040 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 msgr2=0x7fae1c1527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fae10009920 tx=0x7fae10033c20 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 shutdown_connections 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fae00079270 0x7fae0007b730 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fae1c15be30 0x7fae1c09d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae1c0ced20 0x7fae1c09cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 --2- 172.21.15.31:0/970597113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae1c073860 0x7fae1c1527f0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 >> 172.21.15.31:0/970597113 conn(0x7fae1c0b18f0 msgr2=0x7fae1c0af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.743+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 shutdown_connections 2024-09-13T08:46:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.744+0000 7fae24800640 1 -- 172.21.15.31:0/970597113 wait complete. 2024-09-13T08:46:28.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-13T08:46:28.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 -- 172.21.15.31:0/327241871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 msgr2=0x7fbc440ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 --2- 172.21.15.31:0/327241871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc440ce3c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fbc440ca470 tx=0x7fbc2c02a540 comp rx=0 tx=0).stop 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 -- 172.21.15.31:0/327241871 shutdown_connections 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 --2- 172.21.15.31:0/327241871 >> v2:172.21.15.45:3300/0 conn(0x7fbc440ce900 0x7fbc440cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 --2- 172.21.15.31:0/327241871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc440ce3c0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 -- 172.21.15.31:0/327241871 >> 172.21.15.31:0/327241871 conn(0x7fbc440cc910 msgr2=0x7fbc440cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 -- 172.21.15.31:0/327241871 shutdown_connections 2024-09-13T08:46:28.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.930+0000 7fbc4b600640 1 -- 172.21.15.31:0/327241871 wait complete. 2024-09-13T08:46:28.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.932+0000 7fbc4b600640 1 Processor -- start 2024-09-13T08:46:28.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.932+0000 7fbc4b600640 1 -- start start 2024-09-13T08:46:28.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 0x7fbc441f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 0x7fbc4415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc441600f0 con 0x7fbc440cf850 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbc440cb850 con 0x7fbc4415bd70 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc4b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4409b890 con 0x7fbc440ce900 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc3be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc3be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44118/0 (socket says 172.21.15.31:44118) 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc3be00640 1 -- 172.21.15.31:0/3852985075 learned_addr learned my addr 172.21.15.31:0/3852985075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc49600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 0x7fbc4415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.933+0000 7fbc48c00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 0x7fbc441f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 -- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 msgr2=0x7fbc441f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 0x7fbc441f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 -- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 msgr2=0x7fbc4415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 0x7fbc4415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 -- 172.21.15.31:0/3852985075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4409d5a0 con 0x7fbc440cf850 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc49600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 0x7fbc4415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc48c00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 0x7fbc441f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc3be00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009920 tx=0x7fbc2c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c040070 con 0x7fbc440cf850 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc2c005310 con 0x7fbc440cf850 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c005c90 con 0x7fbc440cf850 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4415d460 con 0x7fbc440cf850 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.934+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4415da20 con 0x7fbc440cf850 2024-09-13T08:46:28.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.936+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc440ce3c0 con 0x7fbc440cf850 2024-09-13T08:46:28.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.937+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fbc2c039040 con 0x7fbc440cf850 2024-09-13T08:46:28.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.938+0000 7fbc41600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 0x7fbc2007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:28.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.938+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7fbc2c0ba590 con 0x7fbc440cf850 2024-09-13T08:46:28.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.938+0000 7fbc48c00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 0x7fbc2007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:28.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.939+0000 7fbc48c00640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 0x7fbc2007b730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007a50 tx=0x7fbc34009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:28.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:28.942+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fbc4409d5a0 con 0x7fbc440cf850 2024-09-13T08:46:29.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.211+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7fbc440cb490 con 0x7fbc440cf850 2024-09-13T08:46:29.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.348+0000 7fbc41600640 1 -- 172.21.15.31:0/3852985075 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v516) ==== 145+0+0 (secure 0 0 0) 0x7fbc2c084650 con 0x7fbc440cf850 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 msgr2=0x7fbc2007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 0x7fbc2007b730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007a50 tx=0x7fbc34009040 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 msgr2=0x7fbc4409d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009920 tx=0x7fbc2c032c10 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 shutdown_connections 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fbc20079270 0x7fbc2007b730 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc4415bd70 0x7fbc4415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc440cf850 0x7fbc4409d060 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 --2- 172.21.15.31:0/3852985075 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fbc440ce900 0x7fbc441f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 >> 172.21.15.31:0/3852985075 conn(0x7fbc440cc910 msgr2=0x7fbc440ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.350+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 shutdown_connections 2024-09-13T08:46:29.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.351+0000 7fbc4b600640 1 -- 172.21.15.31:0/3852985075 wait complete. 2024-09-13T08:46:29.369 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-13T08:46:29.369 INFO:tasks.cephfs_test_runner:Starting test: test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate unparseable text .meta file on legacy subvol root doesn't break the system 2024-09-13T08:46:29.369 INFO:tasks.cephfs_test_runner:test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-13T08:46:29.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 -- 172.21.15.31:0/2305479412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c15dfc0 msgr2=0x7f479c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 --2- 172.21.15.31:0/2305479412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c15dfc0 0x7f479c0cc300 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f479c070b80 tx=0x7f478c02a560 comp rx=0 tx=0).stop 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 -- 172.21.15.31:0/2305479412 shutdown_connections 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 --2- 172.21.15.31:0/2305479412 >> v2:172.21.15.45:3300/0 conn(0x7f479c0ca5c0 0x7f479c0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 --2- 172.21.15.31:0/2305479412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c15dfc0 0x7f479c0cc300 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 -- 172.21.15.31:0/2305479412 >> 172.21.15.31:0/2305479412 conn(0x7f479c0cc910 msgr2=0x7f479c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 -- 172.21.15.31:0/2305479412 shutdown_connections 2024-09-13T08:46:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.529+0000 7f47a3a00640 1 -- 172.21.15.31:0/2305479412 wait complete. 2024-09-13T08:46:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.531+0000 7f47a3a00640 1 Processor -- start 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.531+0000 7f47a3a00640 1 -- start start 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 0x7f479c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 0x7f479c0b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f479c074010 con 0x7f479c0ca5c0 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f479c073e90 con 0x7f479c0b1910 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f47a3a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f479c09af50 con 0x7f479c15dfc0 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f4793e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f4793e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44134/0 (socket says 172.21.15.31:44134) 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.532+0000 7f4793e00640 1 -- 172.21.15.31:0/1254551197 learned_addr learned my addr 172.21.15.31:0/1254551197 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f47a1000640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 0x7f479c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f47a1a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 0x7f479c0b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 -- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 msgr2=0x7f479c0b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 0x7f479c0b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 -- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 msgr2=0x7f479c14f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 0x7f479c14f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 -- 172.21.15.31:0/1254551197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c1e8d50 con 0x7f479c0ca5c0 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f47a1000640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 0x7f479c14f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f47a1a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 0x7f479c0b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4793e00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f478400b4a0 tx=0x7f478400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784014070 con 0x7f479c0ca5c0 2024-09-13T08:46:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4784010700 con 0x7f479c0ca5c0 2024-09-13T08:46:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784010b50 con 0x7f479c0ca5c0 2024-09-13T08:46:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.533+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c1e8a70 con 0x7f479c0ca5c0 2024-09-13T08:46:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.534+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c1f8890 con 0x7f479c0ca5c0 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.534+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479c0b2130 con 0x7f479c0ca5c0 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.536+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f4784017070 con 0x7f479c0ca5c0 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.537+0000 7f4799600640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 0x7f477807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.537+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7f4784098ea0 con 0x7f479c0ca5c0 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.537+0000 7f47a1000640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 0x7f477807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.538+0000 7f47a1000640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 0x7f477807b730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f478c033000 tx=0x7f478c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:29.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.542+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4784062d40 con 0x7f479c0ca5c0 2024-09-13T08:46:29.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.738+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f479c0c69d0 con 0x7f479c0ca5c0 2024-09-13T08:46:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.739+0000 7f4799600640 1 -- 172.21.15.31:0/1254551197 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) ==== 74+0+13057 (secure 0 0 0) 0x7f4784062640 con 0x7f479c0ca5c0 2024-09-13T08:46:29.740 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:29.740 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":167,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:28.337987+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/98622202":"2024-09-14T08:46:25.328403+0000","172.21.15.45:6801/98622202":"2024-09-14T08:46:25.328403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.741+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 msgr2=0x7f477807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.741+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 0x7f477807b730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f478c033000 tx=0x7f478c004730 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 msgr2=0x7f479c09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f478400b4a0 tx=0x7f478400b960 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 shutdown_connections 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4778079270 0x7f477807b730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f479c15dfc0 0x7f479c0b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f479c0ca5c0 0x7f479c09aa10 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 --2- 172.21.15.31:0/1254551197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0b1910 0x7f479c14f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 >> 172.21.15.31:0/1254551197 conn(0x7f479c0cc910 msgr2=0x7f479c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 shutdown_connections 2024-09-13T08:46:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.742+0000 7f47a3a00640 1 -- 172.21.15.31:0/1254551197 wait complete. 2024-09-13T08:46:29.759 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T08:36:59.061847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T08:46:29.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.924+0000 7fca45000640 1 -- 172.21.15.31:0/78422231 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 msgr2=0x7fca400d48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.924+0000 7fca45000640 1 --2- 172.21.15.31:0/78422231 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400d48d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009890 tx=0x7fca2c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.924+0000 7fca45000640 1 -- 172.21.15.31:0/78422231 shutdown_connections 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.924+0000 7fca45000640 1 --2- 172.21.15.31:0/78422231 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400d48d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.924+0000 7fca45000640 1 -- 172.21.15.31:0/78422231 >> 172.21.15.31:0/78422231 conn(0x7fca400d3db0 msgr2=0x7fca400d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.925+0000 7fca45000640 1 -- 172.21.15.31:0/78422231 shutdown_connections 2024-09-13T08:46:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.925+0000 7fca45000640 1 -- 172.21.15.31:0/78422231 wait complete. 2024-09-13T08:46:29.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.926+0000 7fca45000640 1 Processor -- start 2024-09-13T08:46:29.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.926+0000 7fca45000640 1 -- start start 2024-09-13T08:46:29.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400b9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 0x7fca4015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca40169160 con 0x7fca400d1f90 2024-09-13T08:46:29.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca400b2e40 con 0x7fca400d2370 2024-09-13T08:46:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca45000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fca401692e0 con 0x7fca400b48b0 2024-09-13T08:46:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca3ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca3ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44138/0 (socket says 172.21.15.31:44138) 2024-09-13T08:46:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.927+0000 7fca3ea00640 1 -- 172.21.15.31:0/3411012332 learned_addr learned my addr 172.21.15.31:0/3411012332 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3f400640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3fe00640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 0x7fca4015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 -- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 msgr2=0x7fca400b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400b9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 -- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 msgr2=0x7fca4015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 0x7fca4015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 -- 172.21.15.31:0/3411012332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4015cef0 con 0x7fca400d1f90 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3f400640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3fe00640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 0x7fca4015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca3ea00640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fca2c02abc0 tx=0x7fca2c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c040070 con 0x7fca400d1f90 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca2c005310 con 0x7fca400d1f90 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.928+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca4015d1d0 con 0x7fca400d1f90 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.929+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0047a0 con 0x7fca400d1f90 2024-09-13T08:46:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.929+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca4015b530 con 0x7fca400d1f90 2024-09-13T08:46:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.930+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca400d48d0 con 0x7fca400d1f90 2024-09-13T08:46:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.931+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fca2c039040 con 0x7fca400d1f90 2024-09-13T08:46:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.932+0000 7fca37400640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 0x7fca2007b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.932+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7fca2c0ba550 con 0x7fca400d1f90 2024-09-13T08:46:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.932+0000 7fca3f400640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 0x7fca2007b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:29.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.933+0000 7fca3f400640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 0x7fca2007b760 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fca30007fd0 tx=0x7fca30009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:29.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:29.936+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fca2c084590 con 0x7fca400d1f90 2024-09-13T08:46:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.132+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fca400b5890 con 0x7fca400d1f90 2024-09-13T08:46:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.132+0000 7fca37400640 1 -- 172.21.15.31:0/3411012332 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) ==== 93+0+10 (secure 0 0 0) 0x7fca2c083e90 con 0x7fca400d1f90 2024-09-13T08:46:30.134 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 msgr2=0x7fca2007b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 0x7fca2007b760 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fca30007fd0 tx=0x7fca30009040 comp rx=0 tx=0).stop 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 msgr2=0x7fca400b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fca2c02abc0 tx=0x7fca2c032c10 comp rx=0 tx=0).stop 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 shutdown_connections 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fca200792a0 0x7fca2007b760 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca400d2370 0x7fca4015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca400d1f90 0x7fca400b8a10 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 --2- 172.21.15.31:0/3411012332 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fca400b48b0 0x7fca400b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 >> 172.21.15.31:0/3411012332 conn(0x7fca400d3db0 msgr2=0x7fca400b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:30.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 shutdown_connections 2024-09-13T08:46:30.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.135+0000 7fca45000640 1 -- 172.21.15.31:0/3411012332 wait complete. 2024-09-13T08:46:30.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:46:30.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 -- 172.21.15.31:0/2848276547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 msgr2=0x7f283806f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 --2- 172.21.15.31:0/2848276547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f283806f4e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2830005fd0 tx=0x7f283002a810 comp rx=0 tx=0).stop 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 -- 172.21.15.31:0/2848276547 shutdown_connections 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 --2- 172.21.15.31:0/2848276547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f283806f4e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 --2- 172.21.15.31:0/2848276547 >> v2:172.21.15.45:3300/0 conn(0x7f283815ad60 0x7f283815f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 -- 172.21.15.31:0/2848276547 >> 172.21.15.31:0/2848276547 conn(0x7f28380cc910 msgr2=0x7f28380cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 -- 172.21.15.31:0/2848276547 shutdown_connections 2024-09-13T08:46:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.313+0000 7f283d800640 1 -- 172.21.15.31:0/2848276547 wait complete. 2024-09-13T08:46:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.315+0000 7f283d800640 1 Processor -- start 2024-09-13T08:46:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.315+0000 7f283d800640 1 -- start start 2024-09-13T08:46:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f283815b140 0x7f2838155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f28381562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f283806fec0 con 0x7f283815ad60 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f283806fd40 con 0x7f283815b140 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f283d800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f28380ca4e0 con 0x7f283815b5b0 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f2836a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f2836a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44150/0 (socket says 172.21.15.31:44150) 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f2836a00640 1 -- 172.21.15.31:0/3666301018 learned_addr learned my addr 172.21.15.31:0/3666301018 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f2837400640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f28381562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.316+0000 7f282e000640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f283815b140 0x7f2838155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 -- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 msgr2=0x7f28381562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f28381562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 -- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f283815b140 msgr2=0x7f2838155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f283815b140 0x7f2838155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 -- 172.21.15.31:0/3666301018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28380b5990 con 0x7f283815ad60 2024-09-13T08:46:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f2836a00640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9c0 tx=0x7f282800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282800cde0 con 0x7f283815ad60 2024-09-13T08:46:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2828004020 con 0x7f283815ad60 2024-09-13T08:46:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2838157d80 con 0x7f283815ad60 2024-09-13T08:46:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828004450 con 0x7f283815ad60 2024-09-13T08:46:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.317+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28380c6f60 con 0x7f283815ad60 2024-09-13T08:46:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.318+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283815f3d0 con 0x7f283815ad60 2024-09-13T08:46:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.319+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f28280035f0 con 0x7f283815ad60 2024-09-13T08:46:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.321+0000 7f282f400640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 0x7f281407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.321+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7f2828099d40 con 0x7f283815ad60 2024-09-13T08:46:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.321+0000 7f282e000640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 0x7f281407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.322+0000 7f282e000640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 0x7f281407b730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2838159d80 tx=0x7f2820009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:30.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.325+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2828063590 con 0x7f283815ad60 2024-09-13T08:46:30.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:30.600+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f283806f800 con 0x7f283815ad60 2024-09-13T08:46:31.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.355+0000 7f282f400640 1 -- 172.21.15.31:0/3666301018 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v518) ==== 166+0+0 (secure 0 0 0) 0x7f2828067fc0 con 0x7f283815ad60 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 msgr2=0x7f281407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 0x7f281407b730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2838159d80 tx=0x7f2820009040 comp rx=0 tx=0).stop 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 msgr2=0x7f28381e9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9c0 tx=0x7f282800ee80 comp rx=0 tx=0).stop 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 shutdown_connections 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2814079270 0x7f281407b730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f283815b5b0 0x7f28381562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f283815b140 0x7f2838155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 --2- 172.21.15.31:0/3666301018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283815ad60 0x7f28381e9040 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 >> 172.21.15.31:0/3666301018 conn(0x7f28380cc910 msgr2=0x7f28380cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:31.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 shutdown_connections 2024-09-13T08:46:31.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.360+0000 7f283d800640 1 -- 172.21.15.31:0/3666301018 wait complete. 2024-09-13T08:46:31.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-13T08:46:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.545+0000 7f522e400640 1 -- 172.21.15.31:0/2584754385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 msgr2=0x7f522806f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.545+0000 7f522e400640 1 --2- 172.21.15.31:0/2584754385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f9b0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f5218005fd0 tx=0x7f521802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 -- 172.21.15.31:0/2584754385 shutdown_connections 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 --2- 172.21.15.31:0/2584754385 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 0x7f5228152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 --2- 172.21.15.31:0/2584754385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f9b0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 -- 172.21.15.31:0/2584754385 >> 172.21.15.31:0/2584754385 conn(0x7f522806cb00 msgr2=0x7f522806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 -- 172.21.15.31:0/2584754385 shutdown_connections 2024-09-13T08:46:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.546+0000 7f522e400640 1 -- 172.21.15.31:0/2584754385 wait complete. 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.547+0000 7f522e400640 1 Processor -- start 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.547+0000 7f522e400640 1 -- start start 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 0x7f522814d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52280ccb00 0x7f52281f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5228152bc0 con 0x7f52280cc720 2024-09-13T08:46:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52280703b0 con 0x7f52280ccb00 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.548+0000 7f522e400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5228152d40 con 0x7f52280b52d0 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44160/0 (socket says 172.21.15.31:44160) 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 -- 172.21.15.31:0/1145777742 learned_addr learned my addr 172.21.15.31:0/1145777742 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f522d000640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 0x7f522814d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f522da00640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52280ccb00 0x7f52281f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 -- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 msgr2=0x7f522814d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 0x7f522814d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 -- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52280ccb00 msgr2=0x7f52281f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52280ccb00 0x7f52281f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.549+0000 7f521fe00640 1 -- 172.21.15.31:0/1145777742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522814c7e0 con 0x7f52280cc720 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f521fe00640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f521000c3a0 tx=0x7f521000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210017070 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52100053c0 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522814ca50 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210003850 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.550+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522814d2f0 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.551+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52280d08c0 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.552+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f521001a070 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.553+0000 7f5225600640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 0x7f520407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.553+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7f5210098750 con 0x7f52280cc720 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.553+0000 7f522d000640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 0x7f520407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.554+0000 7f522d000640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 0x7f520407b730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5218033000 tx=0x7f5218004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.558+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f52280d08c0 con 0x7f52280cc720 2024-09-13T08:46:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.752+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f522814cc20 con 0x7f52280cc720 2024-09-13T08:46:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.753+0000 7f5225600640 1 -- 172.21.15.31:0/1145777742 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+1998 (secure 0 0 0) 0x7f5210061fa0 con 0x7f52280cc720 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:mon advanced mon_op_complaint_time 120 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 3636.621475 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3979.938246 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 3898.402592 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3808.834239 2024-09-13T08:46:31.754 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5596.259827 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4042.059652 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4014.528896 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3994.757860 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds 20/20 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_mds_balancer 20/20 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced debug_ms 2/2 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_frag true 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_debug_scatterstat true 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds dev mds_verify_scatter true 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_mon_op_timeout 900 2024-09-13T08:46:31.755 INFO:teuthology.orchestra.run.smithi031.stdout:mds advanced rados_osd_op_timeout 900 2024-09-13T08:46:31.756 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced client_mount_timeout 600 2024-09-13T08:46:31.756 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_client 20/20 2024-09-13T08:46:31.756 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced debug_ms 1/1 2024-09-13T08:46:31.756 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_mon_op_timeout 900 2024-09-13T08:46:31.756 INFO:teuthology.orchestra.run.smithi031.stdout:client advanced rados_osd_op_timeout 900 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 msgr2=0x7f520407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 0x7f520407b730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5218033000 tx=0x7f5218004730 comp rx=0 tx=0).stop 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 msgr2=0x7f522806f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f521000c3a0 tx=0x7f521000c860 comp rx=0 tx=0).stop 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 shutdown_connections 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5204079270 0x7f520407b730 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52280ccb00 0x7f52281f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52280cc720 0x7f522806f5b0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 --2- 172.21.15.31:0/1145777742 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f52280b52d0 0x7f522814d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 >> 172.21.15.31:0/1145777742 conn(0x7f522806cb00 msgr2=0x7f522815eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 shutdown_connections 2024-09-13T08:46:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.755+0000 7f522e400640 1 -- 172.21.15.31:0/1145777742 wait complete. 2024-09-13T08:46:31.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 -- 172.21.15.31:0/2065942838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060815bc00 msgr2=0x7f0608073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 --2- 172.21.15.31:0/2065942838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060815bc00 0x7f0608073ac0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f06080af400 tx=0x7f05fc02a4f0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 -- 172.21.15.31:0/2065942838 shutdown_connections 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 --2- 172.21.15.31:0/2065942838 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f06080b2820 0x7f060806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 --2- 172.21.15.31:0/2065942838 >> v2:172.21.15.45:3300/0 conn(0x7f06080cc3e0 0x7f06080cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 --2- 172.21.15.31:0/2065942838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f060815bc00 0x7f0608073ac0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 -- 172.21.15.31:0/2065942838 >> 172.21.15.31:0/2065942838 conn(0x7f06080b1520 msgr2=0x7f06080b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 -- 172.21.15.31:0/2065942838 shutdown_connections 2024-09-13T08:46:31.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.935+0000 7f060f200640 1 -- 172.21.15.31:0/2065942838 wait complete. 2024-09-13T08:46:31.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.937+0000 7f060f200640 1 Processor -- start 2024-09-13T08:46:31.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.937+0000 7f060f200640 1 -- start start 2024-09-13T08:46:31.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080cc3e0 0x7f06080c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f060815bc00 0x7f06081f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06080caad0 con 0x7f06080b2820 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06080ca950 con 0x7f06080cc3e0 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f06080cac50 con 0x7f060815bc00 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44182/0 (socket says 172.21.15.31:44182) 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060de00640 1 -- 172.21.15.31:0/1300223985 learned_addr learned my addr 172.21.15.31:0/1300223985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060d400640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080cc3e0 0x7f06080c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.938+0000 7f060e800640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f060815bc00 0x7f06081f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 -- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f060815bc00 msgr2=0x7f06081f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f060815bc00 0x7f06081f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 -- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080cc3e0 msgr2=0x7f06080c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080cc3e0 0x7f06080c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:31.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 -- 172.21.15.31:0/1300223985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06080c5da0 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060de00640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f05fc034040 tx=0x7f05fc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc03e030 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05fc003e00 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc035040 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.939+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06080c6080 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.940+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06081f49a0 con 0x7f06080b2820 2024-09-13T08:46:31.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.941+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06080cc830 con 0x7f06080b2820 2024-09-13T08:46:31.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.942+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f05fc041030 con 0x7f06080b2820 2024-09-13T08:46:31.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.943+0000 7f0602000640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 0x7f05ec07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:31.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.943+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7f05fc0b9a30 con 0x7f06080b2820 2024-09-13T08:46:31.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.943+0000 7f060d400640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 0x7f05ec07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:31.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.944+0000 7f060d400640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 0x7f05ec07b730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0608159860 tx=0x7f05f800b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:31.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:31.947+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f05fc083240 con 0x7f06080b2820 2024-09-13T08:46:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.143+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f06081f3f80 con 0x7f06080b2820 2024-09-13T08:46:32.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.147+0000 7f0602000640 1 -- 172.21.15.31:0/1300223985 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f05fc087c70 con 0x7f06080b2820 2024-09-13T08:46:32.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 msgr2=0x7f05ec07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 0x7f05ec07b730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0608159860 tx=0x7f05f800b390 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 msgr2=0x7f0608159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f05fc034040 tx=0x7f05fc003910 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 shutdown_connections 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f05ec079270 0x7f05ec07b730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f060815bc00 0x7f06081f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06080cc3e0 0x7f06080c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 --2- 172.21.15.31:0/1300223985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06080b2820 0x7f0608159320 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.149+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 >> 172.21.15.31:0/1300223985 conn(0x7f06080b1520 msgr2=0x7f06080b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.150+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 shutdown_connections 2024-09-13T08:46:32.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.150+0000 7f060f200640 1 -- 172.21.15.31:0/1300223985 wait complete. 2024-09-13T08:46:32.166 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:46:32.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:32.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:32.361 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:46:32.361 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:32.362 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:32.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:46:32.398 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:32.398 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:32.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:46:32.436 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:32.436 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:32.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:46:32.472 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:32.472 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:32.513 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:46:32.513 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T08:46:32.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 -- 172.21.15.31:0/75259271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec15dfc0 msgr2=0x7f6dec0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 --2- 172.21.15.31:0/75259271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec15dfc0 0x7f6dec0cc300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009860 tx=0x7f6ddc02a5a0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 -- 172.21.15.31:0/75259271 shutdown_connections 2024-09-13T08:46:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 --2- 172.21.15.31:0/75259271 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec15dfc0 0x7f6dec0cc300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 -- 172.21.15.31:0/75259271 >> 172.21.15.31:0/75259271 conn(0x7f6dec0cc910 msgr2=0x7f6dec0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:32.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 -- 172.21.15.31:0/75259271 shutdown_connections 2024-09-13T08:46:32.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.673+0000 7f6df3a00640 1 -- 172.21.15.31:0/75259271 wait complete. 2024-09-13T08:46:32.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.674+0000 7f6df3a00640 1 Processor -- start 2024-09-13T08:46:32.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.675+0000 7f6df3a00640 1 -- start start 2024-09-13T08:46:32.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.675+0000 7f6df3a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:32.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df3a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0ca5c0 0x7f6dec0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:32.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df3a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 0x7f6dec14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df3a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec0b1da0 con 0x7f6dec0b1910 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df3a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec0740a0 con 0x7f6dec15dfc0 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df3a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec0b1f20 con 0x7f6dec0ca5c0 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44186/0 (socket says 172.21.15.31:44186) 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 -- 172.21.15.31:0/431445951 learned_addr learned my addr 172.21.15.31:0/431445951 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 0x7f6dec14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6de3e00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0ca5c0 0x7f6dec0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:32.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 -- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0ca5c0 msgr2=0x7f6dec0c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0ca5c0 0x7f6dec0c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 -- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 msgr2=0x7f6dec14f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 0x7f6dec14f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1000640 1 -- 172.21.15.31:0/431445951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec15b4b0 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.676+0000 7f6df1a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 0x7f6dec14f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6df1000640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009980 tx=0x7f6ddc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc005c90 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ddc031e50 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec1f8460 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc005260 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.677+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1f8a20 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.678+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec0cc300 con 0x7f6dec0b1910 2024-09-13T08:46:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.679+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f6ddc005400 con 0x7f6dec0b1910 2024-09-13T08:46:32.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.680+0000 7f6de9600640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 0x7f6dc807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:32.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.680+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7f6ddc0ba3a0 con 0x7f6dec0b1910 2024-09-13T08:46:32.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.681+0000 7f6de3e00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 0x7f6dc807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:32.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.681+0000 7f6de3e00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 0x7f6dc807b730 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4007a70 tx=0x7f6dd4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:32.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.685+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6ddc03f030 con 0x7f6dec0b1910 2024-09-13T08:46:32.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.919+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dec0afd90 con 0x7f6dec0b1910 2024-09-13T08:46:32.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.920+0000 7f6de9600640 1 -- 172.21.15.31:0/431445951 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+3913 (secure 0 0 0) 0x7f6ddc0842a0 con 0x7f6dec0b1910 2024-09-13T08:46:32.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:32.921 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":112,"btime":"2024-09-13T08:46:26:330762+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7385,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/882911493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":882911493},{"type":"v1","addr":"172.21.15.45:6834","nonce":882911493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/1175919670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1175919670},{"type":"v1","addr":"172.21.15.31:6834","nonce":1175919670}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3172949155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3172949155},{"type":"v1","addr":"172.21.15.31:6835","nonce":3172949155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8497,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1132875491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":1132875491},{"type":"v1","addr":"172.21.15.45:6835","nonce":1132875491}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[]} 2024-09-13T08:46:32.921 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 112 2024-09-13T08:46:32.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 msgr2=0x7f6dc807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 0x7f6dc807b730 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4007a70 tx=0x7f6dd4009040 comp rx=0 tx=0).stop 2024-09-13T08:46:32.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 msgr2=0x7f6dec0c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:32.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009980 tx=0x7f6ddc033c20 comp rx=0 tx=0).stop 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 shutdown_connections 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6dc8079270 0x7f6dc807b730 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec15dfc0 0x7f6dec14f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6dec0ca5c0 0x7f6dec0c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 --2- 172.21.15.31:0/431445951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dec0b1910 0x7f6dec0c8040 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 >> 172.21.15.31:0/431445951 conn(0x7f6dec0cc910 msgr2=0x7f6dec0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 shutdown_connections 2024-09-13T08:46:32.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:32.922+0000 7f6df3a00640 1 -- 172.21.15.31:0/431445951 wait complete. 2024-09-13T08:46:32.941 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T08:46:32.941 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T08:46:32.941 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:46:32.941 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:32.941 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T08:46:32.941 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T08:46:32.942 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:46:32.942 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:32.942 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T08:46:32.942 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T08:46:32.942 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:46:32.943 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:32.943 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T08:46:32.943 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T08:46:32.943 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:46:32.943 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:46:32.956 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:46:32.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T08:46:32.957 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:46:32.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T08:46:32.960 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:46:32.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T08:46:32.962 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:46:32.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T08:46:33.001 INFO:tasks.ceph.mds.a:Started 2024-09-13T08:46:33.001 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:33.003 INFO:tasks.ceph.mds.c:Started 2024-09-13T08:46:33.003 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:33.005 INFO:tasks.ceph.mds.d:Started 2024-09-13T08:46:33.005 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:33.008 INFO:tasks.ceph.mds.b:Started 2024-09-13T08:46:33.008 DEBUG:teuthology.parallel:result is None 2024-09-13T08:46:33.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T08:46:33.131 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-13T08:46:33.134 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:46:33.142 INFO:tasks.ceph.mds.b.smithi045.stdout:starting mds.b at 2024-09-13T08:46:33.142 INFO:tasks.ceph.mds.d.smithi045.stdout:starting mds.d at 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 -- 172.21.15.31:0/1793605173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc0ce530 msgr2=0x7fa2cc070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/1793605173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc070f00 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4005fd0 tx=0x7fa2c402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 -- 172.21.15.31:0/1793605173 shutdown_connections 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/1793605173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc070f00 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/1793605173 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 0x7fa2cc0cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 -- 172.21.15.31:0/1793605173 >> 172.21.15.31:0/1793605173 conn(0x7fa2cc0d0f70 msgr2=0x7fa2cc0d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:33.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 -- 172.21.15.31:0/1793605173 shutdown_connections 2024-09-13T08:46:33.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.181+0000 7fa2d1c00640 1 -- 172.21.15.31:0/1793605173 wait complete. 2024-09-13T08:46:33.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.183+0000 7fa2d1c00640 1 Processor -- start 2024-09-13T08:46:33.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.183+0000 7fa2d1c00640 1 -- start start 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 0x7fa2cc159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06cc80 con 0x7fa2cc072080 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06cb00 con 0x7fa2cc0ce530 2024-09-13T08:46:33.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2d1c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa2cc06ce00 con 0x7fa2cc15f010 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2caa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2caa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44236/0 (socket says 172.21.15.31:44236) 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2caa00640 1 -- 172.21.15.31:0/2224529547 learned_addr learned my addr 172.21.15.31:0/2224529547 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2cb400640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 0x7fa2cc159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.184+0000 7fa2c2000640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 -- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 msgr2=0x7fa2cc159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 0x7fa2cc159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 -- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2cc0ce530 msgr2=0x7fa2cc0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc0afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 -- 172.21.15.31:0/2224529547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc0ae8d0 con 0x7fa2cc072080 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2caa00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c730 tx=0x7fa2bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc016020 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2bc00a930 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2cc0aebb0 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc00ad80 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.185+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2cc0af170 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.186+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2cc0af880 con 0x7fa2cc072080 2024-09-13T08:46:33.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.188+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fa2bc019020 con 0x7fa2cc072080 2024-09-13T08:46:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.188+0000 7fa2c3400640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 0x7fa2a807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.189+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4987+0+0 (secure 0 0 0) 0x7fa2bc099390 con 0x7fa2cc072080 2024-09-13T08:46:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.189+0000 7fa2c2000640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 0x7fa2a807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.189+0000 7fa2c2000640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 0x7fa2a807b730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc1f8310 tx=0x7fa2b4005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.193+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa2bc020020 con 0x7fa2cc072080 2024-09-13T08:46:33.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:33.406+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa2cc06d2b0 con 0x7fa2cc072080 2024-09-13T08:46:34.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.378+0000 7fa2c3400640 1 -- 172.21.15.31:0/2224529547 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v172) ==== 97+0+0 (secure 0 0 0) 0x7fa2cc15f450 con 0x7fa2cc072080 2024-09-13T08:46:34.379 INFO:teuthology.orchestra.run.smithi031.stderr: removed all blocklist entries 2024-09-13T08:46:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 msgr2=0x7fa2a807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 0x7fa2a807b730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc1f8310 tx=0x7fa2b4005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 msgr2=0x7fa2cc1f8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c730 tx=0x7fa2bc00cbf0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 shutdown_connections 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa2a8079270 0x7fa2a807b730 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa2cc15f010 0x7fa2cc159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2cc0ce530 0x7fa2cc0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 --2- 172.21.15.31:0/2224529547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2cc072080 0x7fa2cc1f8dc0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.381+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 >> 172.21.15.31:0/2224529547 conn(0x7fa2cc0d0f70 msgr2=0x7fa2cc160610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.382+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 shutdown_connections 2024-09-13T08:46:34.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.382+0000 7fa2d1c00640 1 -- 172.21.15.31:0/2224529547 wait complete. 2024-09-13T08:46:34.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T08:46:34.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 -- 172.21.15.31:0/3750459669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 msgr2=0x7f56e40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 --2- 172.21.15.31:0/3750459669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e40d2b90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f56e40cf6c0 tx=0x7f56d402a510 comp rx=0 tx=0).stop 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 -- 172.21.15.31:0/3750459669 shutdown_connections 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 --2- 172.21.15.31:0/3750459669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e40d2b90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 -- 172.21.15.31:0/3750459669 >> 172.21.15.31:0/3750459669 conn(0x7f56e409d190 msgr2=0x7f56e40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 -- 172.21.15.31:0/3750459669 shutdown_connections 2024-09-13T08:46:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.566+0000 7f56e8a00640 1 -- 172.21.15.31:0/3750459669 wait complete. 2024-09-13T08:46:34.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.568+0000 7f56e8a00640 1 Processor -- start 2024-09-13T08:46:34.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.568+0000 7f56e8a00640 1 -- start start 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56e40b4b60 0x7f56e40b7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4162dc0 con 0x7f56e41620c0 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4169380 con 0x7f56e40d27b0 2024-09-13T08:46:34.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e8a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f56e40b8250 con 0x7f56e40b4b60 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44252/0 (socket says 172.21.15.31:44252) 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e3e00640 1 -- 172.21.15.31:0/2759485177 learned_addr learned my addr 172.21.15.31:0/2759485177 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56dbe00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.569+0000 7f56e3400640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56e40b4b60 0x7f56e40b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 -- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56e40b4b60 msgr2=0x7f56e40b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56e40b4b60 0x7f56e40b7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 -- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 msgr2=0x7f56e415e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e415e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 -- 172.21.15.31:0/2759485177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e40b8530 con 0x7f56e41620c0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e3e00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c730 tx=0x7f56dc00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc003b70 con 0x7f56e41620c0 2024-09-13T08:46:34.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56dc003d10 con 0x7f56e41620c0 2024-09-13T08:46:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc0052d0 con 0x7f56e41620c0 2024-09-13T08:46:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e40b6f30 con 0x7f56e41620c0 2024-09-13T08:46:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.570+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e40c8290 con 0x7f56e41620c0 2024-09-13T08:46:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.571+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e40d2b90 con 0x7f56e41620c0 2024-09-13T08:46:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.573+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f56dc005470 con 0x7f56e41620c0 2024-09-13T08:46:34.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.573+0000 7f56db400640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 0x7f56c007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:34.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.574+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4901+0+0 (secure 0 0 0) 0x7f56dc099270 con 0x7f56e41620c0 2024-09-13T08:46:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.574+0000 7f56e3400640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 0x7f56c007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.574+0000 7f56e3400640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 0x7f56c007b730 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f56d4034000 tx=0x7f56d4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:34.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.578+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f56dc063110 con 0x7f56e41620c0 2024-09-13T08:46:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.844+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f56e40b7100 con 0x7f56e41620c0 2024-09-13T08:46:34.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.844+0000 7f56db400640 1 -- 172.21.15.31:0/2759485177 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) ==== 80+0+5304 (secure 0 0 0) 0x7f56dc062a10 con 0x7f56e41620c0 2024-09-13T08:46:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:34.845 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_dump": [ 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.a", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm4xfbLxAA4maM2MWCCdHnskX+Y6su9Q==", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.b", 2024-09-13T08:46:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmGQD4BxAAk+OR00d9/DC+OEP0PwgYSg==", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.c", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNm5V6qORAAFih2fj63M4co+1kbTHr7Dw==", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mds.d", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmZvTRERAATNWhuOJfY6xelvGt6Xalxw==", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.0", 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB8+eNm4fhEKRAA/DRn2SQ5iGRsbBjUB26sYQ==", 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.849 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.1", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB/+eNmzXNvMRAAwcNQUcS/BS64RtfRYcpEVA==", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.2", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCD+eNm1URmDRAAk9l243hO9KGeJJPZCar7EQ==", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.850 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.3", 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCG+eNmb5jsGBAAU2k0AYKQcWxc8ERWi7C8Rw==", 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.851 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.4", 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCK+eNmvR63LhAAmB8ksa6QMAOYVDMGUSk1hQ==", 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.5", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCN+eNmNJkNORAAu/KppbMvlEFcAD8J6cf9uQ==", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.6", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCR+eNmIFFCBxAANLndK4Yst/y7gqEBusnQhw==", 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.853 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "osd.7", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQCU+eNm7QKxDxAAck/yV7GRPNc6Hev10AvNDA==", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow profile osd", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile osd", 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.0", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw==", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow r", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.1", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmuDXJJxAAq4J8YM3j2Y0IClLFs5/djw==", 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.855 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.2", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNmhRQ5IRAA1WH7Jf65BV564dIQCd5etw==", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:34.856 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.3", 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB5+eNm/a9GLhAArMZYVugo6eaNIwMjG6+zcA==", 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow", 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow r", 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow rw", 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow rwx" 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "client.admin", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmCNxnABAALx5E13JlQeus2Y8Snhu89g==", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": "allow *", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow *", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.x", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmkooeKRAAA1f/XUHJHwVmQ1j3OeGyXA==", 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.858 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "entity": "mgr.y", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "AQB4+eNmSDn1IRAAvessPMwBUuLaNQwROcL80A==", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": { 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": "allow *", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": "allow profile mgr", 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "allow *" 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:34.860 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-13T08:46:34.860 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 msgr2=0x7f56c007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 0x7f56c007b730 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f56d4034000 tx=0x7f56d4004730 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 msgr2=0x7f56e415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c730 tx=0x7f56dc00cb30 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 shutdown_connections 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f56c0079270 0x7f56c007b730 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56e41620c0 0x7f56e415e9e0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56e40d27b0 0x7f56e415e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 --2- 172.21.15.31:0/2759485177 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f56e40b4b60 0x7f56e40b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 >> 172.21.15.31:0/2759485177 conn(0x7f56e409d190 msgr2=0x7f56e40b56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 shutdown_connections 2024-09-13T08:46:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:34.847+0000 7f56e8a00640 1 -- 172.21.15.31:0/2759485177 wait complete. 2024-09-13T08:46:34.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 -- 172.21.15.31:0/3124098255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0ccb00 msgr2=0x7f5c3c152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 --2- 172.21.15.31:0/3124098255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0ccb00 0x7f5c3c152680 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0cec50 tx=0x7f5c3402aac0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 -- 172.21.15.31:0/3124098255 shutdown_connections 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 --2- 172.21.15.31:0/3124098255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0ccb00 0x7f5c3c152680 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 --2- 172.21.15.31:0/3124098255 >> v2:172.21.15.45:3300/0 conn(0x7f5c3c0b52d0 0x7f5c3c06f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.024+0000 7f5c40a00640 1 -- 172.21.15.31:0/3124098255 >> 172.21.15.31:0/3124098255 conn(0x7f5c3c06cb00 msgr2=0x7f5c3c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.025+0000 7f5c40a00640 1 -- 172.21.15.31:0/3124098255 shutdown_connections 2024-09-13T08:46:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.025+0000 7f5c40a00640 1 -- 172.21.15.31:0/3124098255 wait complete. 2024-09-13T08:46:35.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.026+0000 7f5c40a00640 1 Processor -- start 2024-09-13T08:46:35.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.026+0000 7f5c40a00640 1 -- start start 2024-09-13T08:46:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c0cc720 0x7f5c3c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c3c0ccb00 0x7f5c3c1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c152bc0 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c070460 con 0x7f5c3c0cc720 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c40a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3c152d40 con 0x7f5c3c0ccb00 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c3b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c3b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49042/0 (socket says 172.21.15.31:49042) 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c3b400640 1 -- 172.21.15.31:0/963212825 learned_addr learned my addr 172.21.15.31:0/963212825 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.027+0000 7f5c32a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c0cc720 0x7f5c3c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3be00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c3c0ccb00 0x7f5c3c1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 -- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c3c0ccb00 msgr2=0x7f5c3c1501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c3c0ccb00 0x7f5c3c1501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 -- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c0cc720 msgr2=0x7f5c3c155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c0cc720 0x7f5c3c155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 -- 172.21.15.31:0/963212825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c155590 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c3b400640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00c450 tx=0x7f5c2c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0052d0 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c2c005470 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c012870 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.028+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c1f5bf0 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.029+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c3c1f6490 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.030+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3c06e960 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.031+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f5c2c0035f0 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.032+0000 7f5c33e00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 0x7f5c1807b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.032+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4901+0+0 (secure 0 0 0) 0x7f5c2c098e40 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.032+0000 7f5c32a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 0x7f5c1807b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.032+0000 7f5c32a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 0x7f5c1807b760 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c1548c0 tx=0x7f5c2400b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:35.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.036+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5c2c062690 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.230+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5c3c06e680 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.230+0000 7f5c33e00640 1 -- 172.21.15.31:0/963212825 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v172) ==== 84+0+65 (secure 0 0 0) 0x7f5c2c0670c0 con 0x7f5c3c0b52d0 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:35.232 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 msgr2=0x7f5c1807b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 0x7f5c1807b760 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c1548c0 tx=0x7f5c2400b390 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 msgr2=0x7f5c3c15f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00c450 tx=0x7f5c2c00c910 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 shutdown_connections 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c180792a0 0x7f5c1807b760 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c3c0ccb00 0x7f5c3c1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c0cc720 0x7f5c3c155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 --2- 172.21.15.31:0/963212825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c3c0b52d0 0x7f5c3c15f000 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 >> 172.21.15.31:0/963212825 conn(0x7f5c3c06cb00 msgr2=0x7f5c3c06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:35.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 shutdown_connections 2024-09-13T08:46:35.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.233+0000 7f5c40a00640 1 -- 172.21.15.31:0/963212825 wait complete. 2024-09-13T08:46:35.249 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T08:46:35.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T08:46:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 -- 172.21.15.31:0/4110206531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c06fa60 msgr2=0x7f5c9c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/4110206531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c06fa60 0x7f5c9c0cb740 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c069dd0 tx=0x7f5c8402a540 comp rx=0 tx=0).stop 2024-09-13T08:46:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 -- 172.21.15.31:0/4110206531 shutdown_connections 2024-09-13T08:46:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/4110206531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c06fa60 0x7f5c9c0cb740 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/4110206531 >> v2:172.21.15.45:3300/0 conn(0x7f5c9c15bfc0 0x7f5c9c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 -- 172.21.15.31:0/4110206531 >> 172.21.15.31:0/4110206531 conn(0x7f5c9c0cfbb0 msgr2=0x7f5c9c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 -- 172.21.15.31:0/4110206531 shutdown_connections 2024-09-13T08:46:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.416+0000 7f5ca1e00640 1 -- 172.21.15.31:0/4110206531 wait complete. 2024-09-13T08:46:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.417+0000 7f5ca1e00640 1 Processor -- start 2024-09-13T08:46:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.418+0000 7f5ca1e00640 1 -- start start 2024-09-13T08:46:35.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.418+0000 7f5ca1e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c9c06fa60 0x7f5c9c1f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.418+0000 7f5ca1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5ca1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c9c155af0 0x7f5c9c149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5ca1e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c0b3270 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5ca1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c0b30f0 con 0x7f5c9c155af0 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5ca1e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9c0b33f0 con 0x7f5c9c06fa60 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49060/0 (socket says 172.21.15.31:49060) 2024-09-13T08:46:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 -- 172.21.15.31:0/3887600381 learned_addr learned my addr 172.21.15.31:0/3887600381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c9b400640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c9c155af0 0x7f5c9c149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c9c06fa60 0x7f5c9c1f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 -- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c9c06fa60 msgr2=0x7f5c9c1f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c9c06fa60 0x7f5c9c1f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 -- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c9c155af0 msgr2=0x7f5c9c149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c9c155af0 0x7f5c9c149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.419+0000 7f5c92000640 1 -- 172.21.15.31:0/3887600381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c1563b0 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5c92000640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009780 tx=0x7f5c84032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84040070 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c84005310 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c840047a0 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9c156690 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.420+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9c1490f0 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.421+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9c148ee0 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.423+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f5c84039040 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.424+0000 7f5c93e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 0x7f5c7807b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:35.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.424+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4901+0+0 (secure 0 0 0) 0x7f5c840ba570 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.424+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 0x7f5c7807b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:35.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.424+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 0x7f5c7807b760 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00bb00 tx=0x7f5c8c005e30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:35.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.428+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f5c840c3080 con 0x7f5c9c15bfc0 2024-09-13T08:46:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:35.625+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5c9c0cbf20 con 0x7f5c9c15bfc0 2024-09-13T08:46:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.387+0000 7f5c93e00640 1 -- 172.21.15.31:0/3887600381 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v173) ==== 138+0+0 (secure 0 0 0) 0x7f5c84084630 con 0x7f5c9c15bfc0 2024-09-13T08:46:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-09-13T08:46:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.391+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 msgr2=0x7f5c7807b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.391+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 0x7f5c7807b760 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00bb00 tx=0x7f5c8c005e30 comp rx=0 tx=0).stop 2024-09-13T08:46:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.391+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 msgr2=0x7f5c9c152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.391+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009780 tx=0x7f5c84032c10 comp rx=0 tx=0).stop 2024-09-13T08:46:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 shutdown_connections 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f5c780792a0 0x7f5c7807b760 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c9c155af0 0x7f5c9c149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9c15bfc0 0x7f5c9c152d40 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 --2- 172.21.15.31:0/3887600381 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f5c9c06fa60 0x7f5c9c1f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 >> 172.21.15.31:0/3887600381 conn(0x7f5c9c0cfbb0 msgr2=0x7f5c9c0b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.392+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 shutdown_connections 2024-09-13T08:46:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.393+0000 7f5ca1e00640 1 -- 172.21.15.31:0/3887600381 wait complete. 2024-09-13T08:46:36.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 -- 172.21.15.31:0/3790805531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 msgr2=0x7f2c3c072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 --2- 172.21.15.31:0/3790805531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c072560 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009830 tx=0x7f2c2c02a5b0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 -- 172.21.15.31:0/3790805531 shutdown_connections 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 --2- 172.21.15.31:0/3790805531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c072560 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 -- 172.21.15.31:0/3790805531 >> 172.21.15.31:0/3790805531 conn(0x7f2c3c0cfbb0 msgr2=0x7f2c3c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 -- 172.21.15.31:0/3790805531 shutdown_connections 2024-09-13T08:46:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.589+0000 7f2c43a00640 1 -- 172.21.15.31:0/3790805531 wait complete. 2024-09-13T08:46:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.590+0000 7f2c43a00640 1 Processor -- start 2024-09-13T08:46:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.590+0000 7f2c43a00640 1 -- start start 2024-09-13T08:46:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c06fa60 0x7f2c3c0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 0x7f2c3c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c0cbfd0 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c0cbe50 con 0x7f2c3c06fa60 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.591+0000 7f2c43a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2c3c0b30f0 con 0x7f2c3c0c95a0 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49086/0 (socket says 172.21.15.31:49086) 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 -- 172.21.15.31:0/731715898 learned_addr learned my addr 172.21.15.31:0/731715898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c41000640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c06fa60 0x7f2c3c0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c41a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 0x7f2c3c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 -- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 msgr2=0x7f2c3c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 0x7f2c3c15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 -- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c06fa60 msgr2=0x7f2c3c0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c06fa60 0x7f2c3c0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 -- 172.21.15.31:0/731715898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3c0c8500 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c41a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 0x7f2c3c15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.592+0000 7f2c33e00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c500 tx=0x7f2c2400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.593+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24016020 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.593+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c24012920 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.593+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3c0c87e0 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.593+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24012d90 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.593+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c3c0c8da0 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.594+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3c072560 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.595+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f2c24019020 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.596+0000 7f2c39600640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 0x7f2c1807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:36.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.596+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5271+0+0 (secure 0 0 0) 0x7f2c240995f0 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.596+0000 7f2c41000640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 0x7f2c1807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:36.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.597+0000 7f2c41000640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 0x7f2c1807b730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c034000 tx=0x7f2c2c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:36.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.601+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2c24020020 con 0x7f2c3c15bfc0 2024-09-13T08:46:36.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:36.799+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2c3c0cb310 con 0x7f2c3c15bfc0 2024-09-13T08:46:37.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.395+0000 7f2c39600640 1 -- 172.21.15.31:0/731715898 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v174) ==== 170+0+0 (secure 0 0 0) 0x7f2c24062dc0 con 0x7f2c3c15bfc0 2024-09-13T08:46:37.396 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 msgr2=0x7f2c1807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 0x7f2c1807b730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c034000 tx=0x7f2c2c004730 comp rx=0 tx=0).stop 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 msgr2=0x7f2c3c15c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c500 tx=0x7f2c2400c9c0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 shutdown_connections 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2c18079270 0x7f2c1807b730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c3c0c95a0 0x7f2c3c15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c3c15bfc0 0x7f2c3c15c4f0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 --2- 172.21.15.31:0/731715898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c06fa60 0x7f2c3c0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 >> 172.21.15.31:0/731715898 conn(0x7f2c3c0cfbb0 msgr2=0x7f2c3c158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 shutdown_connections 2024-09-13T08:46:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.398+0000 7f2c43a00640 1 -- 172.21.15.31:0/731715898 wait complete. 2024-09-13T08:46:37.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T08:46:37.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:46:37.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:46:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 -- 172.21.15.31:0/2648722375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c161480 msgr2=0x7f956c0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 --2- 172.21.15.31:0/2648722375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c161480 0x7f956c0cd780 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9560009830 tx=0x7f956002a530 comp rx=0 tx=0).stop 2024-09-13T08:46:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 -- 172.21.15.31:0/2648722375 shutdown_connections 2024-09-13T08:46:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 --2- 172.21.15.31:0/2648722375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c161480 0x7f956c0cd780 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 -- 172.21.15.31:0/2648722375 >> 172.21.15.31:0/2648722375 conn(0x7f956c0cfbb0 msgr2=0x7f956c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 -- 172.21.15.31:0/2648722375 shutdown_connections 2024-09-13T08:46:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.594+0000 7f9573800640 1 -- 172.21.15.31:0/2648722375 wait complete. 2024-09-13T08:46:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.596+0000 7f9573800640 1 Processor -- start 2024-09-13T08:46:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.596+0000 7f9573800640 1 -- start start 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 0x7f956c0b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 0x7f956c0b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f956c0cbdc0 con 0x7f956c161480 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f956c0cbc40 con 0x7f956c0cc190 2024-09-13T08:46:37.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9573800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f956c0cbf40 con 0x7f956c0cb380 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9571800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 0x7f956c0b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9570e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f956be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 0x7f956c0b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9570e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:55468/0 (socket says 172.21.15.31:55468) 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.597+0000 7f9570e00640 1 -- 172.21.15.31:0/644787678 learned_addr learned my addr 172.21.15.31:0/644787678 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 -- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 msgr2=0x7f956c1f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 -- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 msgr2=0x7f956c0b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 0x7f956c0b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 -- 172.21.15.31:0/644787678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c0c3620 con 0x7f956c161480 2024-09-13T08:46:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9570e00640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f956be00640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 0x7f956c0b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9571800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 0x7f956c0b0f00 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f956400c6c0 tx=0x7f956400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9564003b70 con 0x7f956c161480 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9564003d10 con 0x7f956c161480 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9564005330 con 0x7f956c161480 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c0b3ad0 con 0x7f956c161480 2024-09-13T08:46:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.598+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c15ef10 con 0x7f956c161480 2024-09-13T08:46:37.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.599+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f956c0cde30 con 0x7f956c161480 2024-09-13T08:46:37.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.601+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f95640054d0 con 0x7f956c161480 2024-09-13T08:46:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.602+0000 7f9568c00640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 0x7f954c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.602+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5653+0+0 (secure 0 0 0) 0x7f9564099c40 con 0x7f956c161480 2024-09-13T08:46:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.602+0000 7f9570e00640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 0x7f954c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.603+0000 7f9570e00640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 0x7f954c07b730 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9560009830 tx=0x7f9560004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.605+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f956c0c3620 con 0x7f956c161480 2024-09-13T08:46:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:37.845+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f956c0b5b30 con 0x7f956c161480 2024-09-13T08:46:38.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.397+0000 7f9568c00640 1 -- 172.21.15.31:0/644787678 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v115) ==== 174+0+0 (secure 0 0 0) 0x7f956406c020 con 0x7f956c161480 2024-09-13T08:46:38.398 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 26 and data pool 27 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 msgr2=0x7f954c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 0x7f954c07b730 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9560009830 tx=0x7f9560004730 comp rx=0 tx=0).stop 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 msgr2=0x7f956c0b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 0x7f956c0b0f00 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f956400c6c0 tx=0x7f956400cb80 comp rx=0 tx=0).stop 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 shutdown_connections 2024-09-13T08:46:38.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f954c079270 0x7f954c07b730 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c161480 0x7f956c0b0f00 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f956c0cc190 0x7f956c0b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 --2- 172.21.15.31:0/644787678 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f956c0cb380 0x7f956c1f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.401+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 >> 172.21.15.31:0/644787678 conn(0x7f956c0cfbb0 msgr2=0x7f956c0ad890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.402+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 shutdown_connections 2024-09-13T08:46:38.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.402+0000 7f9573800640 1 -- 172.21.15.31:0/644787678 wait complete. 2024-09-13T08:46:38.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 -- 172.21.15.31:0/1185538582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415bd70 msgr2=0x7fec240cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 --2- 172.21.15.31:0/1185538582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415bd70 0x7fec240cf850 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fec240ca470 tx=0x7fec1402a5f0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 -- 172.21.15.31:0/1185538582 shutdown_connections 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 --2- 172.21.15.31:0/1185538582 >> v2:172.21.15.45:3300/0 conn(0x7fec2415fe20 0x7fec240cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 --2- 172.21.15.31:0/1185538582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415bd70 0x7fec240cf850 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 -- 172.21.15.31:0/1185538582 >> 172.21.15.31:0/1185538582 conn(0x7fec240cc910 msgr2=0x7fec240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 -- 172.21.15.31:0/1185538582 shutdown_connections 2024-09-13T08:46:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.603+0000 7fec29400640 1 -- 172.21.15.31:0/1185538582 wait complete. 2024-09-13T08:46:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.605+0000 7fec29400640 1 Processor -- start 2024-09-13T08:46:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.605+0000 7fec29400640 1 -- start start 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec2415bd70 0x7fec240c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec2415fe20 0x7fec24155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec240cec50 con 0x7fec2415f9e0 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec240cead0 con 0x7fec2415bd70 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec29400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fec240cb850 con 0x7fec2415fe20 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec19e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec19e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49104/0 (socket says 172.21.15.31:49104) 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.606+0000 7fec19e00640 1 -- 172.21.15.31:0/4254330521 learned_addr learned my addr 172.21.15.31:0/4254330521 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec22800640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec2415bd70 0x7fec240c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec23200640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec2415fe20 0x7fec24155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 -- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec2415fe20 msgr2=0x7fec24155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec2415fe20 0x7fec24155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 -- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec2415bd70 msgr2=0x7fec240c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec2415bd70 0x7fec240c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 -- 172.21.15.31:0/4254330521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2406f410 con 0x7fec2415f9e0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec19e00640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c450 tx=0x7fec0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c0052d0 con 0x7fec2415f9e0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec0c005470 con 0x7fec2415f9e0 2024-09-13T08:46:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c012870 con 0x7fec2415f9e0 2024-09-13T08:46:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.607+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec240c4ce0 con 0x7fec2415f9e0 2024-09-13T08:46:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.608+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec240c4260 con 0x7fec2415f9e0 2024-09-13T08:46:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.609+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec240cf850 con 0x7fec2415f9e0 2024-09-13T08:46:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.610+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fec0c0035f0 con 0x7fec2415f9e0 2024-09-13T08:46:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.611+0000 7fec1b400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 0x7fec0007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.611+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 5753+0+0 (secure 0 0 0) 0x7fec0c098f00 con 0x7fec2415f9e0 2024-09-13T08:46:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.611+0000 7fec22800640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 0x7fec0007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.612+0000 7fec22800640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 0x7fec0007b730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fec14033000 tx=0x7fec14004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.615+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fec240c4ce0 con 0x7fec2415f9e0 2024-09-13T08:46:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.809+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec240ce740 con 0x7fec2415f9e0 2024-09-13T08:46:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.810+0000 7fec1b400640 1 -- 172.21.15.31:0/4254330521 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) ==== 74+0+16707 (secure 0 0 0) 0x7fec0c062750 con 0x7fec2415f9e0 2024-09-13T08:46:38.811 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:38.811 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":175,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:37.846126+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:35.636566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:36.809469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 msgr2=0x7fec0007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 0x7fec0007b730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fec14033000 tx=0x7fec14004730 comp rx=0 tx=0).stop 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 msgr2=0x7fec2406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c450 tx=0x7fec0c00c910 comp rx=0 tx=0).stop 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 shutdown_connections 2024-09-13T08:46:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fec00079270 0x7fec0007b730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fec2415fe20 0x7fec24155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec2415f9e0 0x7fec2406eed0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 --2- 172.21.15.31:0/4254330521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec2415bd70 0x7fec240c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 >> 172.21.15.31:0/4254330521 conn(0x7fec240cc910 msgr2=0x7fec240acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 shutdown_connections 2024-09-13T08:46:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.812+0000 7fec29400640 1 -- 172.21.15.31:0/4254330521 wait complete. 2024-09-13T08:46:38.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:38.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 -- 172.21.15.31:0/79417824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a480ce900 msgr2=0x7f9a480cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 --2- 172.21.15.31:0/79417824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a480ce900 0x7f9a480cbc50 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f9a40005fd0 tx=0x7f9a4002aa90 comp rx=0 tx=0).stop 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 -- 172.21.15.31:0/79417824 shutdown_connections 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 --2- 172.21.15.31:0/79417824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a480ce900 0x7f9a480cbc50 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 --2- 172.21.15.31:0/79417824 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 0x7f9a480ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 -- 172.21.15.31:0/79417824 >> 172.21.15.31:0/79417824 conn(0x7f9a480cc910 msgr2=0x7f9a480cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 -- 172.21.15.31:0/79417824 shutdown_connections 2024-09-13T08:46:38.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.990+0000 7f9a4e800640 1 -- 172.21.15.31:0/79417824 wait complete. 2024-09-13T08:46:38.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.992+0000 7f9a4e800640 1 Processor -- start 2024-09-13T08:46:38.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.992+0000 7f9a4e800640 1 -- start start 2024-09-13T08:46:38.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a480ce900 0x7f9a480c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 0x7f9a480b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a481600f0 con 0x7f9a4815bd70 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a480cb850 con 0x7f9a480ce900 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a4e800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9a480b5c00 con 0x7f9a480cf850 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a47e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a47e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49114/0 (socket says 172.21.15.31:49114) 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a47e00640 1 -- 172.21.15.31:0/4204593701 learned_addr learned my addr 172.21.15.31:0/4204593701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a47400640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a480ce900 0x7f9a480c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.993+0000 7f9a46a00640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 0x7f9a480b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:38.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 -- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 msgr2=0x7f9a480b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 0x7f9a480b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 -- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a480ce900 msgr2=0x7f9a480c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:38.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a480ce900 0x7f9a480c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:38.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 -- 172.21.15.31:0/4204593701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a48153920 con 0x7f9a4815bd70 2024-09-13T08:46:38.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a47e00640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006010 tx=0x7f9a40003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40046070 con 0x7f9a4815bd70 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a40004040 con 0x7f9a4815bd70 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40004430 con 0x7f9a4815bd70 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a480b0eb0 con 0x7f9a4815bd70 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.994+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a480b5640 con 0x7f9a4815bd70 2024-09-13T08:46:38.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.995+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a480cc270 con 0x7f9a4815bd70 2024-09-13T08:46:38.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.997+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f9a40049070 con 0x7f9a4815bd70 2024-09-13T08:46:38.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.998+0000 7f9a3f400640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 0x7f9a2807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:38.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.998+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 5753+0+0 (secure 0 0 0) 0x7f9a400ba290 con 0x7f9a4815bd70 2024-09-13T08:46:38.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.998+0000 7f9a47400640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 0x7f9a2807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:39.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:38.999+0000 7f9a47400640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 0x7f9a2807b730 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9a380015e0 tx=0x7f9a38009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:39.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.003+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9a40032030 con 0x7f9a4815bd70 2024-09-13T08:46:39.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.200+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a48072880 con 0x7f9a4815bd70 2024-09-13T08:46:39.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.201+0000 7f9a3f400640 1 -- 172.21.15.31:0/4204593701 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) ==== 74+0+16707 (secure 0 0 0) 0x7f9a40084130 con 0x7f9a4815bd70 2024-09-13T08:46:39.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:39.203 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":175,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:37.846126+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:35.636566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:36.809469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 msgr2=0x7f9a2807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 0x7f9a2807b730 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9a380015e0 tx=0x7f9a38009040 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 msgr2=0x7f9a4809caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006010 tx=0x7f9a40003ce0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 shutdown_connections 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9a28079270 0x7f9a2807b730 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a4815bd70 0x7f9a4809caf0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9a480cf850 0x7f9a480b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 --2- 172.21.15.31:0/4204593701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a480ce900 0x7f9a480c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 >> 172.21.15.31:0/4204593701 conn(0x7f9a480cc910 msgr2=0x7f9a480ae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:39.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 shutdown_connections 2024-09-13T08:46:39.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.204+0000 7f9a4e800640 1 -- 172.21.15.31:0/4204593701 wait complete. 2024-09-13T08:46:39.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T08:46:39.379 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-13T08:46:39.377+0000 7f3257000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T08:46:39.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 -- 172.21.15.31:0/1350735212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e00cbc80 msgr2=0x7f97e00b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 --2- 172.21.15.31:0/1350735212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e00cbc80 0x7f97e00b29e0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc02a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 -- 172.21.15.31:0/1350735212 shutdown_connections 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 --2- 172.21.15.31:0/1350735212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e00cbc80 0x7f97e00b29e0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 --2- 172.21.15.31:0/1350735212 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e0072560 0x7f97e00cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 -- 172.21.15.31:0/1350735212 >> 172.21.15.31:0/1350735212 conn(0x7f97e00cfbb0 msgr2=0x7f97e00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 -- 172.21.15.31:0/1350735212 shutdown_connections 2024-09-13T08:46:39.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.383+0000 7f97e8800640 1 -- 172.21.15.31:0/1350735212 wait complete. 2024-09-13T08:46:39.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.385+0000 7f97e8800640 1 Processor -- start 2024-09-13T08:46:39.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.385+0000 7f97e8800640 1 -- start start 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e00cbc80 0x7f97e01f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97e015bfc0 0x7f97e00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97e006e740 con 0x7f97e0072560 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97e006e5c0 con 0x7f97e015bfc0 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e8800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f97e00b2f20 con 0x7f97e00cbc80 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:39.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49140/0 (socket says 172.21.15.31:49140) 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 -- 172.21.15.31:0/1522017313 learned_addr learned my addr 172.21.15.31:0/1522017313 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5400640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e00cbc80 0x7f97e01f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e6800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97e015bfc0 0x7f97e00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 -- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e00cbc80 msgr2=0x7f97e01f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e00cbc80 0x7f97e01f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 -- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97e015bfc0 msgr2=0x7f97e00c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.386+0000 7f97e5e00640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97e015bfc0 0x7f97e00c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97e5e00640 1 -- 172.21.15.31:0/1522017313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e00c9d60 con 0x7f97e0072560 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97e5e00640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c430 tx=0x7f97d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4011790 con 0x7f97e0072560 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97d4017070 con 0x7f97e0072560 2024-09-13T08:46:39.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4011ce0 con 0x7f97e0072560 2024-09-13T08:46:39.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e00c6ed0 con 0x7f97e0072560 2024-09-13T08:46:39.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.387+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e00ce610 con 0x7f97e0072560 2024-09-13T08:46:39.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.388+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e00c6c60 con 0x7f97e0072560 2024-09-13T08:46:39.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.390+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f97d401c460 con 0x7f97e0072560 2024-09-13T08:46:39.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.390+0000 7f97da000640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 0x7f97c407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:39.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.391+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 5753+0+0 (secure 0 0 0) 0x7f97d4099220 con 0x7f97e0072560 2024-09-13T08:46:39.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.391+0000 7f97e5400640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 0x7f97c407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:39.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.391+0000 7f97e5400640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 0x7f97c407b730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97e0153300 tx=0x7f97d0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.395+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f97d40630c0 con 0x7f97e0072560 2024-09-13T08:46:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:39.636+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f97e00b33b0 con 0x7f97e0072560 2024-09-13T08:46:40.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.424+0000 7f97da000640 1 -- 172.21.15.31:0/1522017313 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v117) ==== 118+0+0 (secure 0 0 0) 0x7f97d40629c0 con 0x7f97e0072560 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.427+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 msgr2=0x7f97c407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.427+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 0x7f97c407b730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97e0153300 tx=0x7f97d0005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.427+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 msgr2=0x7f97e0149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.427+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c430 tx=0x7f97d400c8f0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 shutdown_connections 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f97c4079270 0x7f97c407b730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97e015bfc0 0x7f97e00c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f97e00cbc80 0x7f97e01f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 --2- 172.21.15.31:0/1522017313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072560 0x7f97e0149110 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 >> 172.21.15.31:0/1522017313 conn(0x7f97e00cfbb0 msgr2=0x7f97e00ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 shutdown_connections 2024-09-13T08:46:40.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.428+0000 7f97e8800640 1 -- 172.21.15.31:0/1522017313 wait complete. 2024-09-13T08:46:40.447 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T08:46:40.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 -- 172.21.15.31:0/657975216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7815bfc0 msgr2=0x7f0a78072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 --2- 172.21.15.31:0/657975216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7815bfc0 0x7f0a78072560 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0a780cfab0 tx=0x7f0a6c02a560 comp rx=0 tx=0).stop 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 -- 172.21.15.31:0/657975216 shutdown_connections 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 --2- 172.21.15.31:0/657975216 >> v2:172.21.15.45:3300/0 conn(0x7f0a7806fa60 0x7f0a780cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 --2- 172.21.15.31:0/657975216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7815bfc0 0x7f0a78072560 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 -- 172.21.15.31:0/657975216 >> 172.21.15.31:0/657975216 conn(0x7f0a780cfbb0 msgr2=0x7f0a780cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 -- 172.21.15.31:0/657975216 shutdown_connections 2024-09-13T08:46:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.614+0000 7f0a7f600640 1 -- 172.21.15.31:0/657975216 wait complete. 2024-09-13T08:46:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.616+0000 7f0a7f600640 1 Processor -- start 2024-09-13T08:46:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.616+0000 7f0a7f600640 1 -- start start 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7806fa60 0x7f0a781f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a7815bfc0 0x7f0a780c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a780b3270 con 0x7f0a78147400 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a780b30f0 con 0x7f0a7806fa60 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7f600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0a780b33f0 con 0x7f0a7815bfc0 2024-09-13T08:46:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49148/0 (socket says 172.21.15.31:49148) 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7d600640 1 -- 172.21.15.31:0/604685617 learned_addr learned my addr 172.21.15.31:0/604685617 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.617+0000 7f0a7cc00640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7806fa60 0x7f0a781f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a77e00640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a7815bfc0 0x7f0a780c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 -- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a7815bfc0 msgr2=0x7f0a780c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a7815bfc0 0x7f0a780c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 -- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7806fa60 msgr2=0x7f0a781f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7806fa60 0x7f0a781f49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 -- 172.21.15.31:0/604685617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a78147dc0 con 0x7f0a78147400 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a7d600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c780 tx=0x7f0a7000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70017070 con 0x7f0a78147400 2024-09-13T08:46:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a70003c80 con 0x7f0a78147400 2024-09-13T08:46:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.618+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a700052d0 con 0x7f0a78147400 2024-09-13T08:46:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.619+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781480a0 con 0x7f0a78147400 2024-09-13T08:46:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.619+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a780c4b40 con 0x7f0a78147400 2024-09-13T08:46:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.620+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a781588c0 con 0x7f0a78147400 2024-09-13T08:46:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.621+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f0a70005470 con 0x7f0a78147400 2024-09-13T08:46:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.622+0000 7f0a74c00640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 0x7f0a5807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.622+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f0a700988a0 con 0x7f0a78147400 2024-09-13T08:46:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.623+0000 7f0a7cc00640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 0x7f0a5807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.623+0000 7f0a7cc00640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 0x7f0a5807b730 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c033000 tx=0x7f0a6c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.627+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0a781588c0 con 0x7f0a78147400 2024-09-13T08:46:40.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:40.862+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0a780cb310 con 0x7f0a78147400 2024-09-13T08:46:41.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.430+0000 7f0a74c00640 1 -- 172.21.15.31:0/604685617 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v118) ==== 122+0+0 (secure 0 0 0) 0x7f0a700620f0 con 0x7f0a78147400 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 msgr2=0x7f0a5807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 0x7f0a5807b730 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c033000 tx=0x7f0a6c004730 comp rx=0 tx=0).stop 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 msgr2=0x7f0a780c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c780 tx=0x7f0a7000cb80 comp rx=0 tx=0).stop 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 shutdown_connections 2024-09-13T08:46:41.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0a58079270 0x7f0a5807b730 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a78147400 0x7f0a780c6c80 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0a7815bfc0 0x7f0a780c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 --2- 172.21.15.31:0/604685617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7806fa60 0x7f0a781f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 >> 172.21.15.31:0/604685617 conn(0x7f0a780cfbb0 msgr2=0x7f0a780b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 shutdown_connections 2024-09-13T08:46:41.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.433+0000 7f0a7f600640 1 -- 172.21.15.31:0/604685617 wait complete. 2024-09-13T08:46:41.452 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T08:46:41.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:41.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 -- 172.21.15.31:0/3497817706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 msgr2=0x7f6a9c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 --2- 172.21.15.31:0/3497817706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6a9c004010 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009b80 tx=0x7f6a9c005830 comp rx=0 tx=0).stop 2024-09-13T08:46:41.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abd000640 1 -- 172.21.15.31:0/3497817706 reap_dead start 2024-09-13T08:46:41.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 -- 172.21.15.31:0/3497817706 shutdown_connections 2024-09-13T08:46:41.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 -- 172.21.15.31:0/3497817706 >> 172.21.15.31:0/3497817706 conn(0x7f6ab806cb00 msgr2=0x7f6ab806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:41.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 -- 172.21.15.31:0/3497817706 shutdown_connections 2024-09-13T08:46:41.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.616+0000 7f6abda00640 1 -- 172.21.15.31:0/3497817706 wait complete. 2024-09-13T08:46:41.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.618+0000 7f6abda00640 1 Processor -- start 2024-09-13T08:46:41.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.618+0000 7f6abda00640 1 -- start start 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ab814ead0 0x7f6ab814ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 0x7f6ab80b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab806f9b0 con 0x7f6a9c003c10 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab80703b0 con 0x7f6ab814ead0 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abda00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab806fb30 con 0x7f6ab81525c0 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49160/0 (socket says 172.21.15.31:49160) 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 -- 172.21.15.31:0/3076022846 learned_addr learned my addr 172.21.15.31:0/3076022846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6aaf400640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ab814ead0 0x7f6ab814ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abd000640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 0x7f6ab80b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:41.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 -- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 msgr2=0x7f6ab80b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 0x7f6ab80b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 -- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ab814ead0 msgr2=0x7f6ab814ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ab814ead0 0x7f6ab814ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6ab7e00640 1 -- 172.21.15.31:0/3076022846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab814c760 con 0x7f6a9c003c10 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.619+0000 7f6abd000640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 0x7f6ab80b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6ab7e00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f6aa802b4f0 tx=0x7f6aa8033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8038040 con 0x7f6a9c003c10 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6aa8004020 con 0x7f6a9c003c10 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab814efe0 con 0x7f6a9c003c10 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8004450 con 0x7f6a9c003c10 2024-09-13T08:46:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.620+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab814f5a0 con 0x7f6a9c003c10 2024-09-13T08:46:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.621+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab80b4bc0 con 0x7f6a9c003c10 2024-09-13T08:46:41.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.622+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f6aa8045020 con 0x7f6a9c003c10 2024-09-13T08:46:41.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.623+0000 7f6ab4c00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 0x7f6a9407b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:41.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.623+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f6aa80b9df0 con 0x7f6a9c003c10 2024-09-13T08:46:41.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.623+0000 7f6aaf400640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 0x7f6a9407b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:41.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.624+0000 7f6aaf400640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 0x7f6a9407b760 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ab806d4b0 tx=0x7f6aa0006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:41.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.628+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6aa8083c90 con 0x7f6a9c003c10 2024-09-13T08:46:41.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.863+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ab80b39e0 con 0x7f6a9c003c10 2024-09-13T08:46:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.864+0000 7f6ab4c00640 1 -- 172.21.15.31:0/3076022846 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5230 (secure 0 0 0) 0x7f6aa8083590 con 0x7f6a9c003c10 2024-09-13T08:46:41.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:41.865 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":118,"btime":"2024-09-13T08:46:41:425334+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 118 2024-09-13T08:46:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 msgr2=0x7f6a9407b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 0x7f6a9407b760 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ab806d4b0 tx=0x7f6aa0006b10 comp rx=0 tx=0).stop 2024-09-13T08:46:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 msgr2=0x7f6ab814c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f6aa802b4f0 tx=0x7f6aa8033670 comp rx=0 tx=0).stop 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 shutdown_connections 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6a940792a0 0x7f6a9407b760 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ab81525c0 0x7f6ab80b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ab814ead0 0x7f6ab814ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 --2- 172.21.15.31:0/3076022846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9c003c10 0x7f6ab814c220 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 >> 172.21.15.31:0/3076022846 conn(0x7f6ab806cb00 msgr2=0x7f6ab815be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 shutdown_connections 2024-09-13T08:46:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:41.866+0000 7f6abda00640 1 -- 172.21.15.31:0/3076022846 wait complete. 2024-09-13T08:46:41.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.043+0000 7f8e13000640 1 -- 172.21.15.31:0/720860623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 msgr2=0x7f8e0c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.043+0000 7f8e13000640 1 --2- 172.21.15.31:0/720860623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0cc300 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f8e04005fd0 tx=0x7f8e0402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.043+0000 7f8e13000640 1 -- 172.21.15.31:0/720860623 shutdown_connections 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.043+0000 7f8e13000640 1 --2- 172.21.15.31:0/720860623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0cc300 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.043+0000 7f8e13000640 1 -- 172.21.15.31:0/720860623 >> 172.21.15.31:0/720860623 conn(0x7f8e0c0cc910 msgr2=0x7f8e0c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.044+0000 7f8e13000640 1 -- 172.21.15.31:0/720860623 shutdown_connections 2024-09-13T08:46:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.044+0000 7f8e13000640 1 -- 172.21.15.31:0/720860623 wait complete. 2024-09-13T08:46:42.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.045+0000 7f8e13000640 1 Processor -- start 2024-09-13T08:46:42.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.045+0000 7f8e13000640 1 -- start start 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 0x7f8e0c09c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e0c15dfc0 0x7f8e0c09cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c073b40 con 0x7f8e0c0b2020 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c0739c0 con 0x7f8e0c0ca5c0 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e13000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0c073cc0 con 0x7f8e0c15dfc0 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49176/0 (socket says 172.21.15.31:49176) 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.046+0000 7f8e0be00640 1 -- 172.21.15.31:0/3315762134 learned_addr learned my addr 172.21.15.31:0/3315762134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e03400640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 0x7f8e0c09c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e11000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e0c15dfc0 0x7f8e0c09cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 -- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e0c15dfc0 msgr2=0x7f8e0c09cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e0c15dfc0 0x7f8e0c09cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 -- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 msgr2=0x7f8e0c09c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 0x7f8e0c09c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 -- 172.21.15.31:0/3315762134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0c0c8a40 con 0x7f8e0c0b2020 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e03400640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 0x7f8e0c09c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e0be00640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b3f0 tx=0x7f8dfc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc015020 con 0x7f8e0c0b2020 2024-09-13T08:46:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dfc011920 con 0x7f8e0c0b2020 2024-09-13T08:46:42.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0c0c8200 con 0x7f8e0c0b2020 2024-09-13T08:46:42.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc011d70 con 0x7f8e0c0b2020 2024-09-13T08:46:42.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.047+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0c0aecb0 con 0x7f8e0c0b2020 2024-09-13T08:46:42.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.048+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0c09cc90 con 0x7f8e0c0b2020 2024-09-13T08:46:42.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.050+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f8dfc018020 con 0x7f8e0c0b2020 2024-09-13T08:46:42.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.051+0000 7f8e08c00640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 0x7f8de807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.051+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f8dfc099150 con 0x7f8e0c0b2020 2024-09-13T08:46:42.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.051+0000 7f8e03400640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 0x7f8de807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.051+0000 7f8e03400640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 0x7f8de807b730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c1f2c40 tx=0x7f8df4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.055+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8dfc01f030 con 0x7f8e0c0b2020 2024-09-13T08:46:42.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.250+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e0c0cc300 con 0x7f8e0c0b2020 2024-09-13T08:46:42.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.251+0000 7f8e08c00640 1 -- 172.21.15.31:0/3315762134 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) ==== 74+0+16694 (secure 0 0 0) 0x7f8e0c0c8200 con 0x7f8e0c0b2020 2024-09-13T08:46:42.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:42.253 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:39.411099+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:35.636566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:36.809469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 msgr2=0x7f8de807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 0x7f8de807b730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c1f2c40 tx=0x7f8df4009040 comp rx=0 tx=0).stop 2024-09-13T08:46:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 msgr2=0x7f8e0c0ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b3f0 tx=0x7f8dfc00b8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 shutdown_connections 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f8de8079270 0x7f8de807b730 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8e0c15dfc0 0x7f8e0c09cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0c0ca5c0 0x7f8e0c09c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 --2- 172.21.15.31:0/3315762134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c0b2020 0x7f8e0c0ae770 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 >> 172.21.15.31:0/3315762134 conn(0x7f8e0c0cc910 msgr2=0x7f8e0c15d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 shutdown_connections 2024-09-13T08:46:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.254+0000 7f8e13000640 1 -- 172.21.15.31:0/3315762134 wait complete. 2024-09-13T08:46:42.274 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:46:42.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:46:42.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 -- 172.21.15.31:0/914406544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 msgr2=0x7f6580167d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 --2- 172.21.15.31:0/914406544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f6580167d80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f658006a4f0 tx=0x7f657802a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 -- 172.21.15.31:0/914406544 shutdown_connections 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 --2- 172.21.15.31:0/914406544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f6580167d80 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 --2- 172.21.15.31:0/914406544 >> v2:172.21.15.45:3300/0 conn(0x7f658015e310 0x7f65800cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 -- 172.21.15.31:0/914406544 >> 172.21.15.31:0/914406544 conn(0x7f65800d4070 msgr2=0x7f65800d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 -- 172.21.15.31:0/914406544 shutdown_connections 2024-09-13T08:46:42.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.435+0000 7f6587400640 1 -- 172.21.15.31:0/914406544 wait complete. 2024-09-13T08:46:42.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.437+0000 7f6587400640 1 Processor -- start 2024-09-13T08:46:42.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.437+0000 7f6587400640 1 -- start start 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65800b9d30 0x7f6580158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658015e310 0x7f658015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65801684d0 con 0x7f65800d1190 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65800d4ea0 con 0x7f658015e310 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6587400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6580168650 con 0x7f65800b9d30 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f657fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f657fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49198/0 (socket says 172.21.15.31:49198) 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f657fe00640 1 -- 172.21.15.31:0/3833593725 learned_addr learned my addr 172.21.15.31:0/3833593725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6584a00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65800b9d30 0x7f6580158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.438+0000 7f6585400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658015e310 0x7f658015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 -- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65800b9d30 msgr2=0x7f6580158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65800b9d30 0x7f6580158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 -- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658015e310 msgr2=0x7f658015cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658015e310 0x7f658015cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 -- 172.21.15.31:0/3833593725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65801f4f60 con 0x7f65800d1190 2024-09-13T08:46:42.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657fe00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f656c00c450 tx=0x7f656c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c016020 con 0x7f65800d1190 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f656c012920 con 0x7f65800d1190 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c012d70 con 0x7f65800d1190 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65801f5220 con 0x7f65800d1190 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.439+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65801f57e0 con 0x7f65800d1190 2024-09-13T08:46:42.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.441+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6580069ab0 con 0x7f65800d1190 2024-09-13T08:46:42.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.442+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f656c019020 con 0x7f65800d1190 2024-09-13T08:46:42.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.443+0000 7f657cc00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 0x7f656007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.443+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f656c0993b0 con 0x7f65800d1190 2024-09-13T08:46:42.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.443+0000 7f6584a00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 0x7f656007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.444+0000 7f6584a00640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 0x7f656007b730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6574009c00 tx=0x7f6574005f30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.447+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6580069ab0 con 0x7f65800d1190 2024-09-13T08:46:42.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.638+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65800cf690 con 0x7f6560079270 2024-09-13T08:46:42.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.642+0000 7f657cc00640 1 -- 172.21.15.31:0/3833593725 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240274 (secure 0 0 0) 0x7f65800cf690 con 0x7f6560079270 2024-09-13T08:46:42.646 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:42.647 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 msgr2=0x7f656007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 0x7f656007b730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6574009c00 tx=0x7f6574005f30 comp rx=0 tx=0).stop 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 msgr2=0x7f658015c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f656c00c450 tx=0x7f656c00c910 comp rx=0 tx=0).stop 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 shutdown_connections 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6560079270 0x7f656007b730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658015e310 0x7f658015cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65800d1190 0x7f658015c820 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 --2- 172.21.15.31:0/3833593725 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f65800b9d30 0x7f6580158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.646+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 >> 172.21.15.31:0/3833593725 conn(0x7f65800d4070 msgr2=0x7f65800b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.647+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 shutdown_connections 2024-09-13T08:46:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.647+0000 7f6587400640 1 -- 172.21.15.31:0/3833593725 wait complete. 2024-09-13T08:46:42.663 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-13T08:46:41.017844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":288412,"kb_used_data":18092,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754686308,"statfs":{"total":773094113280,"available":772798779392,"internally_reserved":0,"allocated":18526208,"data_stored":8845775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.629088"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.397908+0000","last_change":"2024-09-13T08:46:36.397908+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.397908+0000","last_undegraded":"2024-09-13T08:46:36.397908+0000","last_fullsized":"2024-09-13T08:46:36.397908+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.393493+0000","last_change":"2024-09-13T08:46:36.393493+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.393493+0000","last_undegraded":"2024-09-13T08:46:36.393493+0000","last_fullsized":"2024-09-13T08:46:36.393493+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392760+0000","last_change":"2024-09-13T08:46:36.392760+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392760+0000","last_undegraded":"2024-09-13T08:46:36.392760+0000","last_fullsized":"2024-09-13T08:46:36.392760+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395672+0000","last_change":"2024-09-13T08:46:36.395672+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395672+0000","last_undegraded":"2024-09-13T08:46:36.395672+0000","last_fullsized":"2024-09-13T08:46:36.395672+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395311+0000","last_change":"2024-09-13T08:46:36.395311+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395311+0000","last_undegraded":"2024-09-13T08:46:36.395311+0000","last_fullsized":"2024-09-13T08:46:36.395311+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392709+0000","last_change":"2024-09-13T08:46:36.392709+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392709+0000","last_undegraded":"2024-09-13T08:46:36.392709+0000","last_fullsized":"2024-09-13T08:46:36.392709+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.396455+0000","last_change":"2024-09-13T08:46:36.396455+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.396455+0000","last_undegraded":"2024-09-13T08:46:36.396455+0000","last_fullsized":"2024-09-13T08:46:36.396455+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391778+0000","last_change":"2024-09-13T08:46:36.391778+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391778+0000","last_undegraded":"2024-09-13T08:46:36.391778+0000","last_fullsized":"2024-09-13T08:46:36.391778+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.396221+0000","last_change":"2024-09-13T08:46:36.396221+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.396221+0000","last_undegraded":"2024-09-13T08:46:36.396221+0000","last_fullsized":"2024-09-13T08:46:36.396221+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391359+0000","last_change":"2024-09-13T08:46:36.391359+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391359+0000","last_undegraded":"2024-09-13T08:46:36.391359+0000","last_fullsized":"2024-09-13T08:46:36.391359+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.394404+0000","last_change":"2024-09-13T08:46:36.394404+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.394404+0000","last_undegraded":"2024-09-13T08:46:36.394404+0000","last_fullsized":"2024-09-13T08:46:36.394404+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.394896+0000","last_change":"2024-09-13T08:46:36.394896+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.394896+0000","last_undegraded":"2024-09-13T08:46:36.394896+0000","last_fullsized":"2024-09-13T08:46:36.394896+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392227+0000","last_change":"2024-09-13T08:46:36.392227+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392227+0000","last_undegraded":"2024-09-13T08:46:36.392227+0000","last_fullsized":"2024-09-13T08:46:36.392227+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392188+0000","last_change":"2024-09-13T08:46:36.392188+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392188+0000","last_undegraded":"2024-09-13T08:46:36.392188+0000","last_fullsized":"2024-09-13T08:46:36.392188+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392525+0000","last_change":"2024-09-13T08:46:36.392525+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392525+0000","last_undegraded":"2024-09-13T08:46:36.392525+0000","last_fullsized":"2024-09-13T08:46:36.392525+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395553+0000","last_change":"2024-09-13T08:46:36.395553+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395553+0000","last_undegraded":"2024-09-13T08:46:36.395553+0000","last_fullsized":"2024-09-13T08:46:36.395553+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.394568+0000","last_change":"2024-09-13T08:46:36.394568+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.394568+0000","last_undegraded":"2024-09-13T08:46:36.394568+0000","last_fullsized":"2024-09-13T08:46:36.394568+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.398168+0000","last_change":"2024-09-13T08:46:36.398168+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.398168+0000","last_undegraded":"2024-09-13T08:46:36.398168+0000","last_fullsized":"2024-09-13T08:46:36.398168+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.396043+0000","last_change":"2024-09-13T08:46:36.396043+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.396043+0000","last_undegraded":"2024-09-13T08:46:36.396043+0000","last_fullsized":"2024-09-13T08:46:36.396043+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.398303+0000","last_change":"2024-09-13T08:46:36.398303+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.398303+0000","last_undegraded":"2024-09-13T08:46:36.398303+0000","last_fullsized":"2024-09-13T08:46:36.398303+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391918+0000","last_change":"2024-09-13T08:46:36.391918+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391918+0000","last_undegraded":"2024-09-13T08:46:36.391918+0000","last_fullsized":"2024-09-13T08:46:36.391918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391398+0000","last_change":"2024-09-13T08:46:36.391398+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391398+0000","last_undegraded":"2024-09-13T08:46:36.391398+0000","last_fullsized":"2024-09-13T08:46:36.391398+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391507+0000","last_change":"2024-09-13T08:46:36.391507+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391507+0000","last_undegraded":"2024-09-13T08:46:36.391507+0000","last_fullsized":"2024-09-13T08:46:36.391507+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391993+0000","last_change":"2024-09-13T08:46:36.391993+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391993+0000","last_undegraded":"2024-09-13T08:46:36.391993+0000","last_fullsized":"2024-09-13T08:46:36.391993+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392164+0000","last_change":"2024-09-13T08:46:36.392164+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392164+0000","last_undegraded":"2024-09-13T08:46:36.392164+0000","last_fullsized":"2024-09-13T08:46:36.392164+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395683+0000","last_change":"2024-09-13T08:46:36.395683+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395683+0000","last_undegraded":"2024-09-13T08:46:36.395683+0000","last_fullsized":"2024-09-13T08:46:36.395683+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395429+0000","last_change":"2024-09-13T08:46:36.395429+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395429+0000","last_undegraded":"2024-09-13T08:46:36.395429+0000","last_fullsized":"2024-09-13T08:46:36.395429+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.396095+0000","last_change":"2024-09-13T08:46:36.396095+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.396095+0000","last_undegraded":"2024-09-13T08:46:36.396095+0000","last_fullsized":"2024-09-13T08:46:36.396095+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.391428+0000","last_change":"2024-09-13T08:46:36.391428+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.391428+0000","last_undegraded":"2024-09-13T08:46:36.391428+0000","last_fullsized":"2024-09-13T08:46:36.391428+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.397897+0000","last_change":"2024-09-13T08:46:36.397897+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.397897+0000","last_undegraded":"2024-09-13T08:46:36.397897+0000","last_fullsized":"2024-09-13T08:46:36.397897+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.395978+0000","last_change":"2024-09-13T08:46:36.395978+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.395978+0000","last_undegraded":"2024-09-13T08:46:36.395978+0000","last_fullsized":"2024-09-13T08:46:36.395978+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T08:46:37.388566+0000","last_change":"2024-09-13T08:46:37.388566+0000","last_active":"2024-09-13T08:46:37.388566+0000","last_peered":"2024-09-13T08:46:37.388566+0000","last_clean":"2024-09-13T08:46:37.388566+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:37.388566+0000","last_undegraded":"2024-09-13T08:46:37.388566+0000","last_fullsized":"2024-09-13T08:46:37.388566+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":1,"reported_epoch":173,"state":"creating+peering","last_fresh":"2024-09-13T08:46:36.392220+0000","last_change":"2024-09-13T08:46:36.392220+0000","last_active":"2024-09-13T08:46:36.380475+0000","last_peered":"2024-09-13T08:46:36.380475+0000","last_clean":"2024-09-13T08:46:36.380475+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T08:46:36.392220+0000","last_undegraded":"2024-09-13T08:46:36.392220+0000","last_fullsized":"2024-09-13T08:46:36.392220+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":413,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T08:46:36.397665+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:36.397665+0000","last_peered":"2024-09-13T08:46:36.397665+0000","last_clean":"2024-09-13T08:46:36.397665+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:36.397665+0000","last_undegraded":"2024-09-13T08:46:36.397665+0000","last_fullsized":"2024-09-13T08:46:36.397665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37476,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2174976,"data_stored":961793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2162688,"data_stored":955932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509559,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38056,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333784,"statfs":{"total":96636764160,"available":96597794816,"internally_reserved":0,"allocated":2768896,"data_stored":1552161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2174976,"data_stored":961793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":955932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":955932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509560,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33880,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2756608,"data_stored":1546300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":955932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:46:42.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.862+0000 7f7835000640 1 -- 172.21.15.31:0/861616648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300d4110 msgr2=0x7f78300b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.862+0000 7f7835000640 1 --2- 172.21.15.31:0/861616648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300d4110 0x7f78300b0f80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f7828005fd0 tx=0x7f782802aa90 comp rx=0 tx=0).stop 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 -- 172.21.15.31:0/861616648 shutdown_connections 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 --2- 172.21.15.31:0/861616648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300d4110 0x7f78300b0f80 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 --2- 172.21.15.31:0/861616648 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f78300b47f0 0x7f78300d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 -- 172.21.15.31:0/861616648 >> 172.21.15.31:0/861616648 conn(0x7f783008d8b0 msgr2=0x7f783009d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 -- 172.21.15.31:0/861616648 shutdown_connections 2024-09-13T08:46:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.863+0000 7f7835000640 1 -- 172.21.15.31:0/861616648 wait complete. 2024-09-13T08:46:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.864+0000 7f7835000640 1 Processor -- start 2024-09-13T08:46:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.864+0000 7f7835000640 1 -- start start 2024-09-13T08:46:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78300d4110 0x7f7830162340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f783015dfa0 0x7f78300a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78300b1990 con 0x7f78300b47f0 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78300b1810 con 0x7f78300d4110 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f7835000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f78300b1b10 con 0x7f783015dfa0 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.865+0000 7f782f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49218/0 (socket says 172.21.15.31:49218) 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 -- 172.21.15.31:0/2405603950 learned_addr learned my addr 172.21.15.31:0/2405603950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782fe00640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f783015dfa0 0x7f78300a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782ea00640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78300d4110 0x7f7830162340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 -- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f783015dfa0 msgr2=0x7f78300a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f783015dfa0 0x7f78300a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 -- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78300d4110 msgr2=0x7f7830162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78300d4110 0x7f7830162340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 -- 172.21.15.31:0/2405603950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78300a1b70 con 0x7f78300b47f0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f782f400640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f782000c3c0 tx=0x7f782000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.866+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820017070 con 0x7f78300b47f0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.867+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78200035f0 con 0x7f78300b47f0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.867+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820003a80 con 0x7f78300b47f0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.867+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783009e8f0 con 0x7f78300b47f0 2024-09-13T08:46:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.867+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783009eeb0 con 0x7f78300b47f0 2024-09-13T08:46:42.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.868+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783009e650 con 0x7f78300b47f0 2024-09-13T08:46:42.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.869+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f782001a070 con 0x7f78300b47f0 2024-09-13T08:46:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.870+0000 7f7827400640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 0x7f781007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.870+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f78200986e0 con 0x7f78300b47f0 2024-09-13T08:46:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.870+0000 7f782ea00640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 0x7f781007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.871+0000 7f782ea00640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 0x7f781007b730 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f78300b92d0 tx=0x7f781c005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:42.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:42.874+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f78300a1b70 con 0x7f78300b47f0 2024-09-13T08:46:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.066+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78300d4cf0 con 0x7f7810079270 2024-09-13T08:46:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.070+0000 7f7827400640 1 -- 172.21.15.31:0/2405603950 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252482 (secure 0 0 0) 0x7f78300d4cf0 con 0x7f7810079270 2024-09-13T08:46:43.074 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 msgr2=0x7f781007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 0x7f781007b730 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f78300b92d0 tx=0x7f781c005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 msgr2=0x7f783015cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f782000c3c0 tx=0x7f782000c880 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 shutdown_connections 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7810079270 0x7f781007b730 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f783015dfa0 0x7f78300a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78300d4110 0x7f7830162340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 --2- 172.21.15.31:0/2405603950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78300b47f0 0x7f783015cfc0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.074+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 >> 172.21.15.31:0/2405603950 conn(0x7f783008d8b0 msgr2=0x7f78300d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.075+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 shutdown_connections 2024-09-13T08:46:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.075+0000 7f7835000640 1 -- 172.21.15.31:0/2405603950 wait complete. 2024-09-13T08:46:43.092 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-13T08:46:43.018145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":288852,"kb_used_data":18504,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754685868,"statfs":{"total":773094113280,"available":772798328832,"internally_reserved":0,"allocated":18948096,"data_stored":9035756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001071"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581392+0000","last_change":"2024-09-13T08:46:38.427043+0000","last_active":"2024-09-13T08:46:39.581392+0000","last_peered":"2024-09-13T08:46:39.581392+0000","last_clean":"2024-09-13T08:46:39.581392+0000","last_became_active":"2024-09-13T08:46:38.426581+0000","last_became_peered":"2024-09-13T08:46:38.426581+0000","last_unstale":"2024-09-13T08:46:39.581392+0000","last_undegraded":"2024-09-13T08:46:39.581392+0000","last_fullsized":"2024-09-13T08:46:39.581392+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:18:17.419221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581393+0000","last_change":"2024-09-13T08:46:38.432215+0000","last_active":"2024-09-13T08:46:39.581393+0000","last_peered":"2024-09-13T08:46:39.581393+0000","last_clean":"2024-09-13T08:46:39.581393+0000","last_became_active":"2024-09-13T08:46:38.431380+0000","last_became_peered":"2024-09-13T08:46:38.431380+0000","last_unstale":"2024-09-13T08:46:39.581393+0000","last_undegraded":"2024-09-13T08:46:39.581393+0000","last_fullsized":"2024-09-13T08:46:39.581393+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:20.599854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422350+0000","last_change":"2024-09-13T08:46:38.425286+0000","last_active":"2024-09-13T08:46:39.422350+0000","last_peered":"2024-09-13T08:46:39.422350+0000","last_clean":"2024-09-13T08:46:39.422350+0000","last_became_active":"2024-09-13T08:46:38.424163+0000","last_became_peered":"2024-09-13T08:46:38.424163+0000","last_unstale":"2024-09-13T08:46:39.422350+0000","last_undegraded":"2024-09-13T08:46:39.422350+0000","last_fullsized":"2024-09-13T08:46:39.422350+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:37.679483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581378+0000","last_change":"2024-09-13T08:46:38.428357+0000","last_active":"2024-09-13T08:46:39.581378+0000","last_peered":"2024-09-13T08:46:39.581378+0000","last_clean":"2024-09-13T08:46:39.581378+0000","last_became_active":"2024-09-13T08:46:38.428147+0000","last_became_peered":"2024-09-13T08:46:38.428147+0000","last_unstale":"2024-09-13T08:46:39.581378+0000","last_undegraded":"2024-09-13T08:46:39.581378+0000","last_fullsized":"2024-09-13T08:46:39.581378+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:30:25.622138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422571+0000","last_change":"2024-09-13T08:46:38.423001+0000","last_active":"2024-09-13T08:46:39.422571+0000","last_peered":"2024-09-13T08:46:39.422571+0000","last_clean":"2024-09-13T08:46:39.422571+0000","last_became_active":"2024-09-13T08:46:38.422708+0000","last_became_peered":"2024-09-13T08:46:38.422708+0000","last_unstale":"2024-09-13T08:46:39.422571+0000","last_undegraded":"2024-09-13T08:46:39.422571+0000","last_fullsized":"2024-09-13T08:46:39.422571+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:30.984965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581787+0000","last_change":"2024-09-13T08:46:38.425979+0000","last_active":"2024-09-13T08:46:39.581787+0000","last_peered":"2024-09-13T08:46:39.581787+0000","last_clean":"2024-09-13T08:46:39.581787+0000","last_became_active":"2024-09-13T08:46:38.425613+0000","last_became_peered":"2024-09-13T08:46:38.425613+0000","last_unstale":"2024-09-13T08:46:39.581787+0000","last_undegraded":"2024-09-13T08:46:39.581787+0000","last_fullsized":"2024-09-13T08:46:39.581787+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:57:49.773675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580848+0000","last_change":"2024-09-13T08:46:38.424751+0000","last_active":"2024-09-13T08:46:39.580848+0000","last_peered":"2024-09-13T08:46:39.580848+0000","last_clean":"2024-09-13T08:46:39.580848+0000","last_became_active":"2024-09-13T08:46:38.424473+0000","last_became_peered":"2024-09-13T08:46:38.424473+0000","last_unstale":"2024-09-13T08:46:39.580848+0000","last_undegraded":"2024-09-13T08:46:39.580848+0000","last_fullsized":"2024-09-13T08:46:39.580848+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:17.143023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421080+0000","last_change":"2024-09-13T08:46:38.413769+0000","last_active":"2024-09-13T08:46:39.421080+0000","last_peered":"2024-09-13T08:46:39.421080+0000","last_clean":"2024-09-13T08:46:39.421080+0000","last_became_active":"2024-09-13T08:46:38.413459+0000","last_became_peered":"2024-09-13T08:46:38.413459+0000","last_unstale":"2024-09-13T08:46:39.421080+0000","last_undegraded":"2024-09-13T08:46:39.421080+0000","last_fullsized":"2024-09-13T08:46:39.421080+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:11.684214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580968+0000","last_change":"2024-09-13T08:46:38.422478+0000","last_active":"2024-09-13T08:46:39.580968+0000","last_peered":"2024-09-13T08:46:39.580968+0000","last_clean":"2024-09-13T08:46:39.580968+0000","last_became_active":"2024-09-13T08:46:38.422216+0000","last_became_peered":"2024-09-13T08:46:38.422216+0000","last_unstale":"2024-09-13T08:46:39.580968+0000","last_undegraded":"2024-09-13T08:46:39.580968+0000","last_fullsized":"2024-09-13T08:46:39.580968+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:25.025668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.423063+0000","last_change":"2024-09-13T08:46:38.414795+0000","last_active":"2024-09-13T08:46:39.423063+0000","last_peered":"2024-09-13T08:46:39.423063+0000","last_clean":"2024-09-13T08:46:39.423063+0000","last_became_active":"2024-09-13T08:46:38.414440+0000","last_became_peered":"2024-09-13T08:46:38.414440+0000","last_unstale":"2024-09-13T08:46:39.423063+0000","last_undegraded":"2024-09-13T08:46:39.423063+0000","last_fullsized":"2024-09-13T08:46:39.423063+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:56:43.383101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581472+0000","last_change":"2024-09-13T08:46:38.425453+0000","last_active":"2024-09-13T08:46:39.581472+0000","last_peered":"2024-09-13T08:46:39.581472+0000","last_clean":"2024-09-13T08:46:39.581472+0000","last_became_active":"2024-09-13T08:46:38.425090+0000","last_became_peered":"2024-09-13T08:46:38.425090+0000","last_unstale":"2024-09-13T08:46:39.581472+0000","last_undegraded":"2024-09-13T08:46:39.581472+0000","last_fullsized":"2024-09-13T08:46:39.581472+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:23.729071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420335+0000","last_change":"2024-09-13T08:46:38.418685+0000","last_active":"2024-09-13T08:46:39.420335+0000","last_peered":"2024-09-13T08:46:39.420335+0000","last_clean":"2024-09-13T08:46:39.420335+0000","last_became_active":"2024-09-13T08:46:38.418397+0000","last_became_peered":"2024-09-13T08:46:38.418397+0000","last_unstale":"2024-09-13T08:46:39.420335+0000","last_undegraded":"2024-09-13T08:46:39.420335+0000","last_fullsized":"2024-09-13T08:46:39.420335+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:32.339828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580453+0000","last_change":"2024-09-13T08:46:38.434001+0000","last_active":"2024-09-13T08:46:39.580453+0000","last_peered":"2024-09-13T08:46:39.580453+0000","last_clean":"2024-09-13T08:46:39.580453+0000","last_became_active":"2024-09-13T08:46:38.433692+0000","last_became_peered":"2024-09-13T08:46:38.433692+0000","last_unstale":"2024-09-13T08:46:39.580453+0000","last_undegraded":"2024-09-13T08:46:39.580453+0000","last_fullsized":"2024-09-13T08:46:39.580453+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:17:43.389386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580720+0000","last_change":"2024-09-13T08:46:38.434363+0000","last_active":"2024-09-13T08:46:39.580720+0000","last_peered":"2024-09-13T08:46:39.580720+0000","last_clean":"2024-09-13T08:46:39.580720+0000","last_became_active":"2024-09-13T08:46:38.433529+0000","last_became_peered":"2024-09-13T08:46:38.433529+0000","last_unstale":"2024-09-13T08:46:39.580720+0000","last_undegraded":"2024-09-13T08:46:39.580720+0000","last_fullsized":"2024-09-13T08:46:39.580720+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:22:22.836521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581067+0000","last_change":"2024-09-13T08:46:38.423739+0000","last_active":"2024-09-13T08:46:39.581067+0000","last_peered":"2024-09-13T08:46:39.581067+0000","last_clean":"2024-09-13T08:46:39.581067+0000","last_became_active":"2024-09-13T08:46:38.423452+0000","last_became_peered":"2024-09-13T08:46:38.423452+0000","last_unstale":"2024-09-13T08:46:39.581067+0000","last_undegraded":"2024-09-13T08:46:39.581067+0000","last_fullsized":"2024-09-13T08:46:39.581067+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:13:27.321096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580552+0000","last_change":"2024-09-13T08:46:38.419567+0000","last_active":"2024-09-13T08:46:39.580552+0000","last_peered":"2024-09-13T08:46:39.580552+0000","last_clean":"2024-09-13T08:46:39.580552+0000","last_became_active":"2024-09-13T08:46:38.417852+0000","last_became_peered":"2024-09-13T08:46:38.417852+0000","last_unstale":"2024-09-13T08:46:39.580552+0000","last_undegraded":"2024-09-13T08:46:39.580552+0000","last_fullsized":"2024-09-13T08:46:39.580552+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:17.160887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581612+0000","last_change":"2024-09-13T08:46:38.432523+0000","last_active":"2024-09-13T08:46:39.581612+0000","last_peered":"2024-09-13T08:46:39.581612+0000","last_clean":"2024-09-13T08:46:39.581612+0000","last_became_active":"2024-09-13T08:46:38.432212+0000","last_became_peered":"2024-09-13T08:46:38.432212+0000","last_unstale":"2024-09-13T08:46:39.581612+0000","last_undegraded":"2024-09-13T08:46:39.581612+0000","last_fullsized":"2024-09-13T08:46:39.581612+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:22.642667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.423083+0000","last_change":"2024-09-13T08:46:38.426780+0000","last_active":"2024-09-13T08:46:39.423083+0000","last_peered":"2024-09-13T08:46:39.423083+0000","last_clean":"2024-09-13T08:46:39.423083+0000","last_became_active":"2024-09-13T08:46:38.424792+0000","last_became_peered":"2024-09-13T08:46:38.424792+0000","last_unstale":"2024-09-13T08:46:39.423083+0000","last_undegraded":"2024-09-13T08:46:39.423083+0000","last_fullsized":"2024-09-13T08:46:39.423083+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:58.544299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422483+0000","last_change":"2024-09-13T08:46:38.421764+0000","last_active":"2024-09-13T08:46:39.422483+0000","last_peered":"2024-09-13T08:46:39.422483+0000","last_clean":"2024-09-13T08:46:39.422483+0000","last_became_active":"2024-09-13T08:46:38.421462+0000","last_became_peered":"2024-09-13T08:46:38.421462+0000","last_unstale":"2024-09-13T08:46:39.422483+0000","last_undegraded":"2024-09-13T08:46:39.422483+0000","last_fullsized":"2024-09-13T08:46:39.422483+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:39.585318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422764+0000","last_change":"2024-09-13T08:46:38.414522+0000","last_active":"2024-09-13T08:46:39.422764+0000","last_peered":"2024-09-13T08:46:39.422764+0000","last_clean":"2024-09-13T08:46:39.422764+0000","last_became_active":"2024-09-13T08:46:38.414200+0000","last_became_peered":"2024-09-13T08:46:38.414200+0000","last_unstale":"2024-09-13T08:46:39.422764+0000","last_undegraded":"2024-09-13T08:46:39.422764+0000","last_fullsized":"2024-09-13T08:46:39.422764+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:25.569809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420340+0000","last_change":"2024-09-13T08:46:38.424580+0000","last_active":"2024-09-13T08:46:39.420340+0000","last_peered":"2024-09-13T08:46:39.420340+0000","last_clean":"2024-09-13T08:46:39.420340+0000","last_became_active":"2024-09-13T08:46:38.424241+0000","last_became_peered":"2024-09-13T08:46:38.424241+0000","last_unstale":"2024-09-13T08:46:39.420340+0000","last_undegraded":"2024-09-13T08:46:39.420340+0000","last_fullsized":"2024-09-13T08:46:39.420340+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:58.544299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581279+0000","last_change":"2024-09-13T08:46:38.420031+0000","last_active":"2024-09-13T08:46:39.581279+0000","last_peered":"2024-09-13T08:46:39.581279+0000","last_clean":"2024-09-13T08:46:39.581279+0000","last_became_active":"2024-09-13T08:46:38.416648+0000","last_became_peered":"2024-09-13T08:46:38.416648+0000","last_unstale":"2024-09-13T08:46:39.581279+0000","last_undegraded":"2024-09-13T08:46:39.581279+0000","last_fullsized":"2024-09-13T08:46:39.581279+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:43.449666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580599+0000","last_change":"2024-09-13T08:46:38.427099+0000","last_active":"2024-09-13T08:46:39.580599+0000","last_peered":"2024-09-13T08:46:39.580599+0000","last_clean":"2024-09-13T08:46:39.580599+0000","last_became_active":"2024-09-13T08:46:38.426386+0000","last_became_peered":"2024-09-13T08:46:38.426386+0000","last_unstale":"2024-09-13T08:46:39.580599+0000","last_undegraded":"2024-09-13T08:46:39.580599+0000","last_fullsized":"2024-09-13T08:46:39.580599+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:03.128507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580990+0000","last_change":"2024-09-13T08:46:38.416218+0000","last_active":"2024-09-13T08:46:39.580990+0000","last_peered":"2024-09-13T08:46:39.580990+0000","last_clean":"2024-09-13T08:46:39.580990+0000","last_became_active":"2024-09-13T08:46:38.415849+0000","last_became_peered":"2024-09-13T08:46:38.415849+0000","last_unstale":"2024-09-13T08:46:39.580990+0000","last_undegraded":"2024-09-13T08:46:39.580990+0000","last_fullsized":"2024-09-13T08:46:39.580990+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:50.420267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581215+0000","last_change":"2024-09-13T08:46:38.432579+0000","last_active":"2024-09-13T08:46:39.581215+0000","last_peered":"2024-09-13T08:46:39.581215+0000","last_clean":"2024-09-13T08:46:39.581215+0000","last_became_active":"2024-09-13T08:46:38.432267+0000","last_became_peered":"2024-09-13T08:46:38.432267+0000","last_unstale":"2024-09-13T08:46:39.581215+0000","last_undegraded":"2024-09-13T08:46:39.581215+0000","last_fullsized":"2024-09-13T08:46:39.581215+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:32.167821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422144+0000","last_change":"2024-09-13T08:46:38.423774+0000","last_active":"2024-09-13T08:46:39.422144+0000","last_peered":"2024-09-13T08:46:39.422144+0000","last_clean":"2024-09-13T08:46:39.422144+0000","last_became_active":"2024-09-13T08:46:38.423455+0000","last_became_peered":"2024-09-13T08:46:38.423455+0000","last_unstale":"2024-09-13T08:46:39.422144+0000","last_undegraded":"2024-09-13T08:46:39.422144+0000","last_fullsized":"2024-09-13T08:46:39.422144+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:33.896142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420383+0000","last_change":"2024-09-13T08:46:38.419511+0000","last_active":"2024-09-13T08:46:39.420383+0000","last_peered":"2024-09-13T08:46:39.420383+0000","last_clean":"2024-09-13T08:46:39.420383+0000","last_became_active":"2024-09-13T08:46:38.419260+0000","last_became_peered":"2024-09-13T08:46:38.419260+0000","last_unstale":"2024-09-13T08:46:39.420383+0000","last_undegraded":"2024-09-13T08:46:39.420383+0000","last_fullsized":"2024-09-13T08:46:39.420383+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:39.558480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421725+0000","last_change":"2024-09-13T08:46:38.429892+0000","last_active":"2024-09-13T08:46:39.421725+0000","last_peered":"2024-09-13T08:46:39.421725+0000","last_clean":"2024-09-13T08:46:39.421725+0000","last_became_active":"2024-09-13T08:46:38.427579+0000","last_became_peered":"2024-09-13T08:46:38.427579+0000","last_unstale":"2024-09-13T08:46:39.421725+0000","last_undegraded":"2024-09-13T08:46:39.421725+0000","last_fullsized":"2024-09-13T08:46:39.421725+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:55:31.198016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581123+0000","last_change":"2024-09-13T08:46:38.420343+0000","last_active":"2024-09-13T08:46:39.581123+0000","last_peered":"2024-09-13T08:46:39.581123+0000","last_clean":"2024-09-13T08:46:39.581123+0000","last_became_active":"2024-09-13T08:46:38.420084+0000","last_became_peered":"2024-09-13T08:46:38.420084+0000","last_unstale":"2024-09-13T08:46:39.581123+0000","last_undegraded":"2024-09-13T08:46:39.581123+0000","last_fullsized":"2024-09-13T08:46:39.581123+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:01.559277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580113+0000","last_change":"2024-09-13T08:46:38.432912+0000","last_active":"2024-09-13T08:46:39.580113+0000","last_peered":"2024-09-13T08:46:39.580113+0000","last_clean":"2024-09-13T08:46:39.580113+0000","last_became_active":"2024-09-13T08:46:38.432013+0000","last_became_peered":"2024-09-13T08:46:38.432013+0000","last_unstale":"2024-09-13T08:46:39.580113+0000","last_undegraded":"2024-09-13T08:46:39.580113+0000","last_fullsized":"2024-09-13T08:46:39.580113+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:37.341212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421521+0000","last_change":"2024-09-13T08:46:38.410131+0000","last_active":"2024-09-13T08:46:39.421521+0000","last_peered":"2024-09-13T08:46:39.421521+0000","last_clean":"2024-09-13T08:46:39.421521+0000","last_became_active":"2024-09-13T08:46:38.409815+0000","last_became_peered":"2024-09-13T08:46:38.409815+0000","last_unstale":"2024-09-13T08:46:39.421521+0000","last_undegraded":"2024-09-13T08:46:39.421521+0000","last_fullsized":"2024-09-13T08:46:39.421521+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:31.857143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580459+0000","last_change":"2024-09-13T08:46:38.423627+0000","last_active":"2024-09-13T08:46:39.580459+0000","last_peered":"2024-09-13T08:46:39.580459+0000","last_clean":"2024-09-13T08:46:39.580459+0000","last_became_active":"2024-09-13T08:46:38.423280+0000","last_became_peered":"2024-09-13T08:46:38.423280+0000","last_unstale":"2024-09-13T08:46:39.580459+0000","last_undegraded":"2024-09-13T08:46:39.580459+0000","last_fullsized":"2024-09-13T08:46:39.580459+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:55:20.781666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420543+0000","last_change":"2024-09-13T08:46:38.408137+0000","last_active":"2024-09-13T08:46:39.420543+0000","last_peered":"2024-09-13T08:46:39.420543+0000","last_clean":"2024-09-13T08:46:39.420543+0000","last_became_active":"2024-09-13T08:46:38.407356+0000","last_became_peered":"2024-09-13T08:46:38.407356+0000","last_unstale":"2024-09-13T08:46:39.420543+0000","last_undegraded":"2024-09-13T08:46:39.420543+0000","last_fullsized":"2024-09-13T08:46:39.420543+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:17.160887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"175'4","reported_seq":19,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420565+0000","last_change":"2024-09-13T08:46:37.422469+0000","last_active":"2024-09-13T08:46:39.420565+0000","last_peered":"2024-09-13T08:46:39.420565+0000","last_clean":"2024-09-13T08:46:39.420565+0000","last_became_active":"2024-09-13T08:46:37.422198+0000","last_became_peered":"2024-09-13T08:46:37.422198+0000","last_unstale":"2024-09-13T08:46:39.420565+0000","last_undegraded":"2024-09-13T08:46:39.420565+0000","last_fullsized":"2024-09-13T08:46:39.420565+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:49.371729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581309+0000","last_change":"2024-09-13T08:46:38.432703+0000","last_active":"2024-09-13T08:46:39.581309+0000","last_peered":"2024-09-13T08:46:39.581309+0000","last_clean":"2024-09-13T08:46:39.581309+0000","last_became_active":"2024-09-13T08:46:38.430285+0000","last_became_peered":"2024-09-13T08:46:38.430285+0000","last_unstale":"2024-09-13T08:46:39.581309+0000","last_undegraded":"2024-09-13T08:46:39.581309+0000","last_fullsized":"2024-09-13T08:46:39.581309+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:27.810171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1e","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581419+0000","last_change":"2024-09-13T08:46:37.416773+0000","last_active":"2024-09-13T08:46:39.581419+0000","last_peered":"2024-09-13T08:46:39.581419+0000","last_clean":"2024-09-13T08:46:39.581419+0000","last_became_active":"2024-09-13T08:46:37.416505+0000","last_became_peered":"2024-09-13T08:46:37.416505+0000","last_unstale":"2024-09-13T08:46:39.581419+0000","last_undegraded":"2024-09-13T08:46:39.581419+0000","last_fullsized":"2024-09-13T08:46:39.581419+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:41.932160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422550+0000","last_change":"2024-09-13T08:46:38.430073+0000","last_active":"2024-09-13T08:46:39.422550+0000","last_peered":"2024-09-13T08:46:39.422550+0000","last_clean":"2024-09-13T08:46:39.422550+0000","last_became_active":"2024-09-13T08:46:38.429759+0000","last_became_peered":"2024-09-13T08:46:38.429759+0000","last_unstale":"2024-09-13T08:46:39.422550+0000","last_undegraded":"2024-09-13T08:46:39.422550+0000","last_fullsized":"2024-09-13T08:46:39.422550+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:41:24.673510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421812+0000","last_change":"2024-09-13T08:46:37.417020+0000","last_active":"2024-09-13T08:46:39.421812+0000","last_peered":"2024-09-13T08:46:39.421812+0000","last_clean":"2024-09-13T08:46:39.421812+0000","last_became_active":"2024-09-13T08:46:37.416711+0000","last_became_peered":"2024-09-13T08:46:37.416711+0000","last_unstale":"2024-09-13T08:46:39.421812+0000","last_undegraded":"2024-09-13T08:46:39.421812+0000","last_fullsized":"2024-09-13T08:46:39.421812+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:19:59.183401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422624+0000","last_change":"2024-09-13T08:46:38.420689+0000","last_active":"2024-09-13T08:46:39.422624+0000","last_peered":"2024-09-13T08:46:39.422624+0000","last_clean":"2024-09-13T08:46:39.422624+0000","last_became_active":"2024-09-13T08:46:38.419382+0000","last_became_peered":"2024-09-13T08:46:38.419382+0000","last_unstale":"2024-09-13T08:46:39.422624+0000","last_undegraded":"2024-09-13T08:46:39.422624+0000","last_fullsized":"2024-09-13T08:46:39.422624+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:54.532495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1c","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580935+0000","last_change":"2024-09-13T08:46:37.421345+0000","last_active":"2024-09-13T08:46:39.580935+0000","last_peered":"2024-09-13T08:46:39.580935+0000","last_clean":"2024-09-13T08:46:39.580935+0000","last_became_active":"2024-09-13T08:46:37.421097+0000","last_became_peered":"2024-09-13T08:46:37.421097+0000","last_unstale":"2024-09-13T08:46:39.580935+0000","last_undegraded":"2024-09-13T08:46:39.580935+0000","last_fullsized":"2024-09-13T08:46:39.580935+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:36.487900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580665+0000","last_change":"2024-09-13T08:46:38.427756+0000","last_active":"2024-09-13T08:46:39.580665+0000","last_peered":"2024-09-13T08:46:39.580665+0000","last_clean":"2024-09-13T08:46:39.580665+0000","last_became_active":"2024-09-13T08:46:38.425572+0000","last_became_peered":"2024-09-13T08:46:38.425572+0000","last_unstale":"2024-09-13T08:46:39.580665+0000","last_undegraded":"2024-09-13T08:46:39.580665+0000","last_fullsized":"2024-09-13T08:46:39.580665+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:31:11.924104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581592+0000","last_change":"2024-09-13T08:46:37.419246+0000","last_active":"2024-09-13T08:46:39.581592+0000","last_peered":"2024-09-13T08:46:39.581592+0000","last_clean":"2024-09-13T08:46:39.581592+0000","last_became_active":"2024-09-13T08:46:37.418981+0000","last_became_peered":"2024-09-13T08:46:37.418981+0000","last_unstale":"2024-09-13T08:46:39.581592+0000","last_undegraded":"2024-09-13T08:46:39.581592+0000","last_fullsized":"2024-09-13T08:46:39.581592+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:03.241709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580805+0000","last_change":"2024-09-13T08:46:38.429192+0000","last_active":"2024-09-13T08:46:39.580805+0000","last_peered":"2024-09-13T08:46:39.580805+0000","last_clean":"2024-09-13T08:46:39.580805+0000","last_became_active":"2024-09-13T08:46:38.428913+0000","last_became_peered":"2024-09-13T08:46:38.428913+0000","last_unstale":"2024-09-13T08:46:39.580805+0000","last_undegraded":"2024-09-13T08:46:39.580805+0000","last_fullsized":"2024-09-13T08:46:39.580805+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:10.844195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421001+0000","last_change":"2024-09-13T08:46:37.407629+0000","last_active":"2024-09-13T08:46:39.421001+0000","last_peered":"2024-09-13T08:46:39.421001+0000","last_clean":"2024-09-13T08:46:39.421001+0000","last_became_active":"2024-09-13T08:46:37.407324+0000","last_became_peered":"2024-09-13T08:46:37.407324+0000","last_unstale":"2024-09-13T08:46:39.421001+0000","last_undegraded":"2024-09-13T08:46:39.421001+0000","last_fullsized":"2024-09-13T08:46:39.421001+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:18:58.213638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580481+0000","last_change":"2024-09-13T08:46:38.430012+0000","last_active":"2024-09-13T08:46:39.580481+0000","last_peered":"2024-09-13T08:46:39.580481+0000","last_clean":"2024-09-13T08:46:39.580481+0000","last_became_active":"2024-09-13T08:46:38.429725+0000","last_became_peered":"2024-09-13T08:46:38.429725+0000","last_unstale":"2024-09-13T08:46:39.580481+0000","last_undegraded":"2024-09-13T08:46:39.580481+0000","last_fullsized":"2024-09-13T08:46:39.580481+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:22.024790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580048+0000","last_change":"2024-09-13T08:46:37.408161+0000","last_active":"2024-09-13T08:46:39.580048+0000","last_peered":"2024-09-13T08:46:39.580048+0000","last_clean":"2024-09-13T08:46:39.580048+0000","last_became_active":"2024-09-13T08:46:37.407863+0000","last_became_peered":"2024-09-13T08:46:37.407863+0000","last_unstale":"2024-09-13T08:46:39.580048+0000","last_undegraded":"2024-09-13T08:46:39.580048+0000","last_fullsized":"2024-09-13T08:46:39.580048+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:50.923483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580995+0000","last_change":"2024-09-13T08:46:38.432750+0000","last_active":"2024-09-13T08:46:39.580995+0000","last_peered":"2024-09-13T08:46:39.580995+0000","last_clean":"2024-09-13T08:46:39.580995+0000","last_became_active":"2024-09-13T08:46:38.432463+0000","last_became_peered":"2024-09-13T08:46:38.432463+0000","last_unstale":"2024-09-13T08:46:39.580995+0000","last_undegraded":"2024-09-13T08:46:39.580995+0000","last_fullsized":"2024-09-13T08:46:39.580995+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:17:48.464043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.423024+0000","last_change":"2024-09-13T08:46:37.418769+0000","last_active":"2024-09-13T08:46:39.423024+0000","last_peered":"2024-09-13T08:46:39.423024+0000","last_clean":"2024-09-13T08:46:39.423024+0000","last_became_active":"2024-09-13T08:46:37.418294+0000","last_became_peered":"2024-09-13T08:46:37.418294+0000","last_unstale":"2024-09-13T08:46:39.423024+0000","last_undegraded":"2024-09-13T08:46:39.423024+0000","last_fullsized":"2024-09-13T08:46:39.423024+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:59.790794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580737+0000","last_change":"2024-09-13T08:46:38.425788+0000","last_active":"2024-09-13T08:46:39.580737+0000","last_peered":"2024-09-13T08:46:39.580737+0000","last_clean":"2024-09-13T08:46:39.580737+0000","last_became_active":"2024-09-13T08:46:38.425486+0000","last_became_peered":"2024-09-13T08:46:38.425486+0000","last_unstale":"2024-09-13T08:46:39.580737+0000","last_undegraded":"2024-09-13T08:46:39.580737+0000","last_fullsized":"2024-09-13T08:46:39.580737+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:36:47.257859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581057+0000","last_change":"2024-09-13T08:46:37.419286+0000","last_active":"2024-09-13T08:46:39.581057+0000","last_peered":"2024-09-13T08:46:39.581057+0000","last_clean":"2024-09-13T08:46:39.581057+0000","last_became_active":"2024-09-13T08:46:37.419014+0000","last_became_peered":"2024-09-13T08:46:37.419014+0000","last_unstale":"2024-09-13T08:46:39.581057+0000","last_undegraded":"2024-09-13T08:46:39.581057+0000","last_fullsized":"2024-09-13T08:46:39.581057+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:28:50.304359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580214+0000","last_change":"2024-09-13T08:46:38.414513+0000","last_active":"2024-09-13T08:46:39.580214+0000","last_peered":"2024-09-13T08:46:39.580214+0000","last_clean":"2024-09-13T08:46:39.580214+0000","last_became_active":"2024-09-13T08:46:38.413739+0000","last_became_peered":"2024-09-13T08:46:38.413739+0000","last_unstale":"2024-09-13T08:46:39.580214+0000","last_undegraded":"2024-09-13T08:46:39.580214+0000","last_fullsized":"2024-09-13T08:46:39.580214+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:32:32.358542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"175'3","reported_seq":18,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421894+0000","last_change":"2024-09-13T08:46:37.421790+0000","last_active":"2024-09-13T08:46:39.421894+0000","last_peered":"2024-09-13T08:46:39.421894+0000","last_clean":"2024-09-13T08:46:39.421894+0000","last_became_active":"2024-09-13T08:46:37.421398+0000","last_became_peered":"2024-09-13T08:46:37.421398+0000","last_unstale":"2024-09-13T08:46:39.421894+0000","last_undegraded":"2024-09-13T08:46:39.421894+0000","last_fullsized":"2024-09-13T08:46:39.421894+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:25.941346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421819+0000","last_change":"2024-09-13T08:46:38.420362+0000","last_active":"2024-09-13T08:46:39.421819+0000","last_peered":"2024-09-13T08:46:39.421819+0000","last_clean":"2024-09-13T08:46:39.421819+0000","last_became_active":"2024-09-13T08:46:38.419915+0000","last_became_peered":"2024-09-13T08:46:38.419915+0000","last_unstale":"2024-09-13T08:46:39.421819+0000","last_undegraded":"2024-09-13T08:46:39.421819+0000","last_fullsized":"2024-09-13T08:46:39.421819+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:44.948332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"175'6","reported_seq":21,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580112+0000","last_change":"2024-09-13T08:46:37.423980+0000","last_active":"2024-09-13T08:46:39.580112+0000","last_peered":"2024-09-13T08:46:39.580112+0000","last_clean":"2024-09-13T08:46:39.580112+0000","last_became_active":"2024-09-13T08:46:37.423715+0000","last_became_peered":"2024-09-13T08:46:37.423715+0000","last_unstale":"2024-09-13T08:46:39.580112+0000","last_undegraded":"2024-09-13T08:46:39.580112+0000","last_fullsized":"2024-09-13T08:46:39.580112+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:29:23.482401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580513+0000","last_change":"2024-09-13T08:46:38.424113+0000","last_active":"2024-09-13T08:46:39.580513+0000","last_peered":"2024-09-13T08:46:39.580513+0000","last_clean":"2024-09-13T08:46:39.580513+0000","last_became_active":"2024-09-13T08:46:38.423835+0000","last_became_peered":"2024-09-13T08:46:38.423835+0000","last_unstale":"2024-09-13T08:46:39.580513+0000","last_undegraded":"2024-09-13T08:46:39.580513+0000","last_fullsized":"2024-09-13T08:46:39.580513+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:41.813067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.13","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580023+0000","last_change":"2024-09-13T08:46:37.422671+0000","last_active":"2024-09-13T08:46:39.580023+0000","last_peered":"2024-09-13T08:46:39.580023+0000","last_clean":"2024-09-13T08:46:39.580023+0000","last_became_active":"2024-09-13T08:46:37.421944+0000","last_became_peered":"2024-09-13T08:46:37.421944+0000","last_unstale":"2024-09-13T08:46:39.580023+0000","last_undegraded":"2024-09-13T08:46:39.580023+0000","last_fullsized":"2024-09-13T08:46:39.580023+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:47.683750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422821+0000","last_change":"2024-09-13T08:46:38.433002+0000","last_active":"2024-09-13T08:46:39.422821+0000","last_peered":"2024-09-13T08:46:39.422821+0000","last_clean":"2024-09-13T08:46:39.422821+0000","last_became_active":"2024-09-13T08:46:38.430267+0000","last_became_peered":"2024-09-13T08:46:38.430267+0000","last_unstale":"2024-09-13T08:46:39.422821+0000","last_undegraded":"2024-09-13T08:46:39.422821+0000","last_fullsized":"2024-09-13T08:46:39.422821+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:39.033995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580367+0000","last_change":"2024-09-13T08:46:37.416111+0000","last_active":"2024-09-13T08:46:39.580367+0000","last_peered":"2024-09-13T08:46:39.580367+0000","last_clean":"2024-09-13T08:46:39.580367+0000","last_became_active":"2024-09-13T08:46:37.415842+0000","last_became_peered":"2024-09-13T08:46:37.415842+0000","last_unstale":"2024-09-13T08:46:39.580367+0000","last_undegraded":"2024-09-13T08:46:39.580367+0000","last_fullsized":"2024-09-13T08:46:39.580367+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:30.985519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421025+0000","last_change":"2024-09-13T08:46:38.416583+0000","last_active":"2024-09-13T08:46:39.421025+0000","last_peered":"2024-09-13T08:46:39.421025+0000","last_clean":"2024-09-13T08:46:39.421025+0000","last_became_active":"2024-09-13T08:46:38.416323+0000","last_became_peered":"2024-09-13T08:46:38.416323+0000","last_unstale":"2024-09-13T08:46:39.421025+0000","last_undegraded":"2024-09-13T08:46:39.421025+0000","last_fullsized":"2024-09-13T08:46:39.421025+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:12:11.533267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421144+0000","last_change":"2024-09-13T08:46:37.413895+0000","last_active":"2024-09-13T08:46:39.421144+0000","last_peered":"2024-09-13T08:46:39.421144+0000","last_clean":"2024-09-13T08:46:39.421144+0000","last_became_active":"2024-09-13T08:46:37.413535+0000","last_became_peered":"2024-09-13T08:46:37.413535+0000","last_unstale":"2024-09-13T08:46:39.421144+0000","last_undegraded":"2024-09-13T08:46:39.421144+0000","last_fullsized":"2024-09-13T08:46:39.421144+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:23:29.398299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581482+0000","last_change":"2024-09-13T08:46:38.432244+0000","last_active":"2024-09-13T08:46:39.581482+0000","last_peered":"2024-09-13T08:46:39.581482+0000","last_clean":"2024-09-13T08:46:39.581482+0000","last_became_active":"2024-09-13T08:46:38.431922+0000","last_became_peered":"2024-09-13T08:46:38.431922+0000","last_unstale":"2024-09-13T08:46:39.581482+0000","last_undegraded":"2024-09-13T08:46:39.581482+0000","last_fullsized":"2024-09-13T08:46:39.581482+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:48:25.497267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.10","version":"175'8","reported_seq":23,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421437+0000","last_change":"2024-09-13T08:46:37.405329+0000","last_active":"2024-09-13T08:46:39.421437+0000","last_peered":"2024-09-13T08:46:39.421437+0000","last_clean":"2024-09-13T08:46:39.421437+0000","last_became_active":"2024-09-13T08:46:37.405041+0000","last_became_peered":"2024-09-13T08:46:37.405041+0000","last_unstale":"2024-09-13T08:46:39.421437+0000","last_undegraded":"2024-09-13T08:46:39.421437+0000","last_fullsized":"2024-09-13T08:46:39.421437+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:10:55.448889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420565+0000","last_change":"2024-09-13T08:46:38.428550+0000","last_active":"2024-09-13T08:46:39.420565+0000","last_peered":"2024-09-13T08:46:39.420565+0000","last_clean":"2024-09-13T08:46:39.420565+0000","last_became_active":"2024-09-13T08:46:38.428216+0000","last_became_peered":"2024-09-13T08:46:38.428216+0000","last_unstale":"2024-09-13T08:46:39.420565+0000","last_undegraded":"2024-09-13T08:46:39.420565+0000","last_fullsized":"2024-09-13T08:46:39.420565+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:42:10.844195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.f","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581879+0000","last_change":"2024-09-13T08:46:37.419417+0000","last_active":"2024-09-13T08:46:39.581879+0000","last_peered":"2024-09-13T08:46:39.581879+0000","last_clean":"2024-09-13T08:46:39.581879+0000","last_became_active":"2024-09-13T08:46:37.416845+0000","last_became_peered":"2024-09-13T08:46:37.416845+0000","last_unstale":"2024-09-13T08:46:39.581879+0000","last_undegraded":"2024-09-13T08:46:39.581879+0000","last_fullsized":"2024-09-13T08:46:39.581879+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:03:17.394656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581413+0000","last_change":"2024-09-13T08:46:38.426486+0000","last_active":"2024-09-13T08:46:39.581413+0000","last_peered":"2024-09-13T08:46:39.581413+0000","last_clean":"2024-09-13T08:46:39.581413+0000","last_became_active":"2024-09-13T08:46:38.426151+0000","last_became_peered":"2024-09-13T08:46:38.426151+0000","last_unstale":"2024-09-13T08:46:39.581413+0000","last_undegraded":"2024-09-13T08:46:39.581413+0000","last_fullsized":"2024-09-13T08:46:39.581413+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:11.682276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580505+0000","last_change":"2024-09-13T08:46:37.419385+0000","last_active":"2024-09-13T08:46:39.580505+0000","last_peered":"2024-09-13T08:46:39.580505+0000","last_clean":"2024-09-13T08:46:39.580505+0000","last_became_active":"2024-09-13T08:46:37.419120+0000","last_became_peered":"2024-09-13T08:46:37.419120+0000","last_unstale":"2024-09-13T08:46:39.580505+0000","last_undegraded":"2024-09-13T08:46:39.580505+0000","last_fullsized":"2024-09-13T08:46:39.580505+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:51.781211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422838+0000","last_change":"2024-09-13T08:46:38.415400+0000","last_active":"2024-09-13T08:46:39.422838+0000","last_peered":"2024-09-13T08:46:39.422838+0000","last_clean":"2024-09-13T08:46:39.422838+0000","last_became_active":"2024-09-13T08:46:38.415111+0000","last_became_peered":"2024-09-13T08:46:38.415111+0000","last_unstale":"2024-09-13T08:46:39.422838+0000","last_undegraded":"2024-09-13T08:46:39.422838+0000","last_fullsized":"2024-09-13T08:46:39.422838+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:07:13.206891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.d","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580979+0000","last_change":"2024-09-13T08:46:37.418591+0000","last_active":"2024-09-13T08:46:39.580979+0000","last_peered":"2024-09-13T08:46:39.580979+0000","last_clean":"2024-09-13T08:46:39.580979+0000","last_became_active":"2024-09-13T08:46:37.417921+0000","last_became_peered":"2024-09-13T08:46:37.417921+0000","last_unstale":"2024-09-13T08:46:39.580979+0000","last_undegraded":"2024-09-13T08:46:39.580979+0000","last_fullsized":"2024-09-13T08:46:39.580979+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:21.016698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581585+0000","last_change":"2024-09-13T08:46:38.422324+0000","last_active":"2024-09-13T08:46:39.581585+0000","last_peered":"2024-09-13T08:46:39.581585+0000","last_clean":"2024-09-13T08:46:39.581585+0000","last_became_active":"2024-09-13T08:46:38.422072+0000","last_became_peered":"2024-09-13T08:46:38.422072+0000","last_unstale":"2024-09-13T08:46:39.581585+0000","last_undegraded":"2024-09-13T08:46:39.581585+0000","last_fullsized":"2024-09-13T08:46:39.581585+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:13.940854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581257+0000","last_change":"2024-09-13T08:46:37.417059+0000","last_active":"2024-09-13T08:46:39.581257+0000","last_peered":"2024-09-13T08:46:39.581257+0000","last_clean":"2024-09-13T08:46:39.581257+0000","last_became_active":"2024-09-13T08:46:37.416791+0000","last_became_peered":"2024-09-13T08:46:37.416791+0000","last_unstale":"2024-09-13T08:46:39.581257+0000","last_undegraded":"2024-09-13T08:46:39.581257+0000","last_fullsized":"2024-09-13T08:46:39.581257+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:12:10.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422140+0000","last_change":"2024-09-13T08:46:38.410389+0000","last_active":"2024-09-13T08:46:39.422140+0000","last_peered":"2024-09-13T08:46:39.422140+0000","last_clean":"2024-09-13T08:46:39.422140+0000","last_became_active":"2024-09-13T08:46:38.410088+0000","last_became_peered":"2024-09-13T08:46:38.410088+0000","last_unstale":"2024-09-13T08:46:39.422140+0000","last_undegraded":"2024-09-13T08:46:39.422140+0000","last_fullsized":"2024-09-13T08:46:39.422140+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:17.615776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580376+0000","last_change":"2024-09-13T08:46:37.416654+0000","last_active":"2024-09-13T08:46:39.580376+0000","last_peered":"2024-09-13T08:46:39.580376+0000","last_clean":"2024-09-13T08:46:39.580376+0000","last_became_active":"2024-09-13T08:46:37.416397+0000","last_became_peered":"2024-09-13T08:46:37.416397+0000","last_unstale":"2024-09-13T08:46:39.580376+0000","last_undegraded":"2024-09-13T08:46:39.580376+0000","last_fullsized":"2024-09-13T08:46:39.580376+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:32:31.350451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581354+0000","last_change":"2024-09-13T08:46:38.417805+0000","last_active":"2024-09-13T08:46:39.581354+0000","last_peered":"2024-09-13T08:46:39.581354+0000","last_clean":"2024-09-13T08:46:39.581354+0000","last_became_active":"2024-09-13T08:46:38.417483+0000","last_became_peered":"2024-09-13T08:46:38.417483+0000","last_unstale":"2024-09-13T08:46:39.581354+0000","last_undegraded":"2024-09-13T08:46:39.581354+0000","last_fullsized":"2024-09-13T08:46:39.581354+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:27.004106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421269+0000","last_change":"2024-09-13T08:46:37.417292+0000","last_active":"2024-09-13T08:46:39.421269+0000","last_peered":"2024-09-13T08:46:39.421269+0000","last_clean":"2024-09-13T08:46:39.421269+0000","last_became_active":"2024-09-13T08:46:37.417014+0000","last_became_peered":"2024-09-13T08:46:37.417014+0000","last_unstale":"2024-09-13T08:46:39.421269+0000","last_undegraded":"2024-09-13T08:46:39.421269+0000","last_fullsized":"2024-09-13T08:46:39.421269+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:23.191403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.579984+0000","last_change":"2024-09-13T08:46:38.428546+0000","last_active":"2024-09-13T08:46:39.579984+0000","last_peered":"2024-09-13T08:46:39.579984+0000","last_clean":"2024-09-13T08:46:39.579984+0000","last_became_active":"2024-09-13T08:46:38.428265+0000","last_became_peered":"2024-09-13T08:46:38.428265+0000","last_unstale":"2024-09-13T08:46:39.579984+0000","last_undegraded":"2024-09-13T08:46:39.579984+0000","last_fullsized":"2024-09-13T08:46:39.579984+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:55:05.384666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421589+0000","last_change":"2024-09-13T08:46:37.416218+0000","last_active":"2024-09-13T08:46:39.421589+0000","last_peered":"2024-09-13T08:46:39.421589+0000","last_clean":"2024-09-13T08:46:39.421589+0000","last_became_active":"2024-09-13T08:46:37.415692+0000","last_became_peered":"2024-09-13T08:46:37.415692+0000","last_unstale":"2024-09-13T08:46:39.421589+0000","last_undegraded":"2024-09-13T08:46:39.421589+0000","last_fullsized":"2024-09-13T08:46:39.421589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:21.777851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580820+0000","last_change":"2024-09-13T08:46:38.434383+0000","last_active":"2024-09-13T08:46:39.580820+0000","last_peered":"2024-09-13T08:46:39.580820+0000","last_clean":"2024-09-13T08:46:39.580820+0000","last_became_active":"2024-09-13T08:46:38.433877+0000","last_became_peered":"2024-09-13T08:46:38.433877+0000","last_unstale":"2024-09-13T08:46:39.580820+0000","last_undegraded":"2024-09-13T08:46:39.580820+0000","last_fullsized":"2024-09-13T08:46:39.580820+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:26.549916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421555+0000","last_change":"2024-09-13T08:46:37.413752+0000","last_active":"2024-09-13T08:46:39.421555+0000","last_peered":"2024-09-13T08:46:39.421555+0000","last_clean":"2024-09-13T08:46:39.421555+0000","last_became_active":"2024-09-13T08:46:37.413458+0000","last_became_peered":"2024-09-13T08:46:37.413458+0000","last_unstale":"2024-09-13T08:46:39.421555+0000","last_undegraded":"2024-09-13T08:46:39.421555+0000","last_fullsized":"2024-09-13T08:46:39.421555+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:14.370423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420779+0000","last_change":"2024-09-13T08:46:38.428872+0000","last_active":"2024-09-13T08:46:39.420779+0000","last_peered":"2024-09-13T08:46:39.420779+0000","last_clean":"2024-09-13T08:46:39.420779+0000","last_became_active":"2024-09-13T08:46:38.428263+0000","last_became_peered":"2024-09-13T08:46:38.428263+0000","last_unstale":"2024-09-13T08:46:39.420779+0000","last_undegraded":"2024-09-13T08:46:39.420779+0000","last_fullsized":"2024-09-13T08:46:39.420779+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:17.143023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"175'2","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421153+0000","last_change":"2024-09-13T08:46:37.408348+0000","last_active":"2024-09-13T08:46:39.421153+0000","last_peered":"2024-09-13T08:46:39.421153+0000","last_clean":"2024-09-13T08:46:39.421153+0000","last_became_active":"2024-09-13T08:46:37.408044+0000","last_became_peered":"2024-09-13T08:46:37.408044+0000","last_unstale":"2024-09-13T08:46:39.421153+0000","last_undegraded":"2024-09-13T08:46:39.421153+0000","last_fullsized":"2024-09-13T08:46:39.421153+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:10.536485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581729+0000","last_change":"2024-09-13T08:46:38.432497+0000","last_active":"2024-09-13T08:46:39.581729+0000","last_peered":"2024-09-13T08:46:39.581729+0000","last_clean":"2024-09-13T08:46:39.581729+0000","last_became_active":"2024-09-13T08:46:38.431763+0000","last_became_peered":"2024-09-13T08:46:38.431763+0000","last_unstale":"2024-09-13T08:46:39.581729+0000","last_undegraded":"2024-09-13T08:46:39.581729+0000","last_fullsized":"2024-09-13T08:46:39.581729+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:15.953523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.6","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581472+0000","last_change":"2024-09-13T08:46:37.423431+0000","last_active":"2024-09-13T08:46:39.581472+0000","last_peered":"2024-09-13T08:46:39.581472+0000","last_clean":"2024-09-13T08:46:39.581472+0000","last_became_active":"2024-09-13T08:46:37.421906+0000","last_became_peered":"2024-09-13T08:46:37.421906+0000","last_unstale":"2024-09-13T08:46:39.581472+0000","last_undegraded":"2024-09-13T08:46:39.581472+0000","last_fullsized":"2024-09-13T08:46:39.581472+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:49.881425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580956+0000","last_change":"2024-09-13T08:46:38.422052+0000","last_active":"2024-09-13T08:46:39.580956+0000","last_peered":"2024-09-13T08:46:39.580956+0000","last_clean":"2024-09-13T08:46:39.580956+0000","last_became_active":"2024-09-13T08:46:38.421783+0000","last_became_peered":"2024-09-13T08:46:38.421783+0000","last_unstale":"2024-09-13T08:46:39.580956+0000","last_undegraded":"2024-09-13T08:46:39.580956+0000","last_fullsized":"2024-09-13T08:46:39.580956+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:17.990224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581661+0000","last_change":"2024-09-13T08:46:37.420437+0000","last_active":"2024-09-13T08:46:39.581661+0000","last_peered":"2024-09-13T08:46:39.581661+0000","last_clean":"2024-09-13T08:46:39.581661+0000","last_became_active":"2024-09-13T08:46:37.420120+0000","last_became_peered":"2024-09-13T08:46:37.420120+0000","last_unstale":"2024-09-13T08:46:39.581661+0000","last_undegraded":"2024-09-13T08:46:39.581661+0000","last_fullsized":"2024-09-13T08:46:39.581661+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:32.296605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420455+0000","last_change":"2024-09-13T08:46:38.425001+0000","last_active":"2024-09-13T08:46:39.420455+0000","last_peered":"2024-09-13T08:46:39.420455+0000","last_clean":"2024-09-13T08:46:39.420455+0000","last_became_active":"2024-09-13T08:46:38.424758+0000","last_became_peered":"2024-09-13T08:46:38.424758+0000","last_unstale":"2024-09-13T08:46:39.420455+0000","last_undegraded":"2024-09-13T08:46:39.420455+0000","last_fullsized":"2024-09-13T08:46:39.420455+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:45:51.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581261+0000","last_change":"2024-09-13T08:46:37.420206+0000","last_active":"2024-09-13T08:46:39.581261+0000","last_peered":"2024-09-13T08:46:39.581261+0000","last_clean":"2024-09-13T08:46:39.581261+0000","last_became_active":"2024-09-13T08:46:37.419884+0000","last_became_peered":"2024-09-13T08:46:37.419884+0000","last_unstale":"2024-09-13T08:46:39.581261+0000","last_undegraded":"2024-09-13T08:46:39.581261+0000","last_fullsized":"2024-09-13T08:46:39.581261+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:30.849052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581515+0000","last_change":"2024-09-13T08:46:38.431099+0000","last_active":"2024-09-13T08:46:39.581515+0000","last_peered":"2024-09-13T08:46:39.581515+0000","last_clean":"2024-09-13T08:46:39.581515+0000","last_became_active":"2024-09-13T08:46:38.430820+0000","last_became_peered":"2024-09-13T08:46:38.430820+0000","last_unstale":"2024-09-13T08:46:39.581515+0000","last_undegraded":"2024-09-13T08:46:39.581515+0000","last_fullsized":"2024-09-13T08:46:39.581515+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:34.140685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580676+0000","last_change":"2024-09-13T08:46:37.420964+0000","last_active":"2024-09-13T08:46:39.580676+0000","last_peered":"2024-09-13T08:46:39.580676+0000","last_clean":"2024-09-13T08:46:39.580676+0000","last_became_active":"2024-09-13T08:46:37.420470+0000","last_became_peered":"2024-09-13T08:46:37.420470+0000","last_unstale":"2024-09-13T08:46:39.580676+0000","last_undegraded":"2024-09-13T08:46:39.580676+0000","last_fullsized":"2024-09-13T08:46:39.580676+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:57:07.804612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581227+0000","last_change":"2024-09-13T08:46:38.417364+0000","last_active":"2024-09-13T08:46:39.581227+0000","last_peered":"2024-09-13T08:46:39.581227+0000","last_clean":"2024-09-13T08:46:39.581227+0000","last_became_active":"2024-09-13T08:46:38.417109+0000","last_became_peered":"2024-09-13T08:46:38.417109+0000","last_unstale":"2024-09-13T08:46:39.581227+0000","last_undegraded":"2024-09-13T08:46:39.581227+0000","last_fullsized":"2024-09-13T08:46:39.581227+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:04.287507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422157+0000","last_change":"2024-09-13T08:46:37.418548+0000","last_active":"2024-09-13T08:46:39.422157+0000","last_peered":"2024-09-13T08:46:39.422157+0000","last_clean":"2024-09-13T08:46:39.422157+0000","last_became_active":"2024-09-13T08:46:37.418221+0000","last_became_peered":"2024-09-13T08:46:37.418221+0000","last_unstale":"2024-09-13T08:46:39.422157+0000","last_undegraded":"2024-09-13T08:46:39.422157+0000","last_fullsized":"2024-09-13T08:46:39.422157+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:23.857444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.421363+0000","last_change":"2024-09-13T08:46:38.416459+0000","last_active":"2024-09-13T08:46:39.421363+0000","last_peered":"2024-09-13T08:46:39.421363+0000","last_clean":"2024-09-13T08:46:39.421363+0000","last_became_active":"2024-09-13T08:46:38.416154+0000","last_became_peered":"2024-09-13T08:46:38.416154+0000","last_unstale":"2024-09-13T08:46:39.421363+0000","last_undegraded":"2024-09-13T08:46:39.421363+0000","last_fullsized":"2024-09-13T08:46:39.421363+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:35.778045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.420044+0000","last_change":"2024-09-13T08:46:37.407330+0000","last_active":"2024-09-13T08:46:39.420044+0000","last_peered":"2024-09-13T08:46:39.420044+0000","last_clean":"2024-09-13T08:46:39.420044+0000","last_became_active":"2024-09-13T08:46:37.406558+0000","last_became_peered":"2024-09-13T08:46:37.406558+0000","last_unstale":"2024-09-13T08:46:39.420044+0000","last_undegraded":"2024-09-13T08:46:39.420044+0000","last_fullsized":"2024-09-13T08:46:39.420044+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:50.343655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581019+0000","last_change":"2024-09-13T08:46:38.434113+0000","last_active":"2024-09-13T08:46:39.581019+0000","last_peered":"2024-09-13T08:46:39.581019+0000","last_clean":"2024-09-13T08:46:39.581019+0000","last_became_active":"2024-09-13T08:46:38.433542+0000","last_became_peered":"2024-09-13T08:46:38.433542+0000","last_unstale":"2024-09-13T08:46:39.581019+0000","last_undegraded":"2024-09-13T08:46:39.581019+0000","last_fullsized":"2024-09-13T08:46:39.581019+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:03:59.825320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580138+0000","last_change":"2024-09-13T08:46:37.412900+0000","last_active":"2024-09-13T08:46:39.580138+0000","last_peered":"2024-09-13T08:46:39.580138+0000","last_clean":"2024-09-13T08:46:39.580138+0000","last_became_active":"2024-09-13T08:46:37.412113+0000","last_became_peered":"2024-09-13T08:46:37.412113+0000","last_unstale":"2024-09-13T08:46:39.580138+0000","last_undegraded":"2024-09-13T08:46:39.580138+0000","last_fullsized":"2024-09-13T08:46:39.580138+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:29.874792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.422063+0000","last_change":"2024-09-13T08:46:38.424237+0000","last_active":"2024-09-13T08:46:39.422063+0000","last_peered":"2024-09-13T08:46:39.422063+0000","last_clean":"2024-09-13T08:46:39.422063+0000","last_became_active":"2024-09-13T08:46:38.422667+0000","last_became_peered":"2024-09-13T08:46:38.422667+0000","last_unstale":"2024-09-13T08:46:39.422063+0000","last_undegraded":"2024-09-13T08:46:39.422063+0000","last_fullsized":"2024-09-13T08:46:39.422063+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:37.388566+0000","last_clean_scrub_stamp":"2024-09-13T08:46:37.388566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:04.249801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"175'1","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.581258+0000","last_change":"2024-09-13T08:46:37.420207+0000","last_active":"2024-09-13T08:46:39.581258+0000","last_peered":"2024-09-13T08:46:39.581258+0000","last_clean":"2024-09-13T08:46:39.581258+0000","last_became_active":"2024-09-13T08:46:37.419931+0000","last_became_peered":"2024-09-13T08:46:37.419931+0000","last_unstale":"2024-09-13T08:46:39.581258+0000","last_undegraded":"2024-09-13T08:46:39.581258+0000","last_fullsized":"2024-09-13T08:46:39.581258+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:46:36.380475+0000","last_clean_scrub_stamp":"2024-09-13T08:46:36.380475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:10.599722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'79","reported_seq":419,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T08:46:39.580225+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:39.580225+0000","last_peered":"2024-09-13T08:46:39.580225+0000","last_clean":"2024-09-13T08:46:39.580225+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:39.580225+0000","last_undegraded":"2024-09-13T08:46:39.580225+0000","last_fullsized":"2024-09-13T08:46:39.580225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509560,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37520,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334320,"statfs":{"total":96636764160,"available":96598343680,"internally_reserved":0,"allocated":2215936,"data_stored":981297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509560,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37516,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334324,"statfs":{"total":96636764160,"available":96598347776,"internally_reserved":0,"allocated":2215936,"data_stored":981345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509560,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38108,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333732,"statfs":{"total":96636764160,"available":96597741568,"internally_reserved":0,"allocated":2822144,"data_stored":1572371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542263,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41680,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330160,"statfs":{"total":96636764160,"available":96594083840,"internally_reserved":0,"allocated":2215936,"data_stored":981365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509560,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2224128,"data_stored":982887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509560,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33376,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338464,"statfs":{"total":96636764160,"available":96602587136,"internally_reserved":0,"allocated":2232320,"data_stored":982991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509561,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33932,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337908,"statfs":{"total":96636764160,"available":96602017792,"internally_reserved":0,"allocated":2805760,"data_stored":1571643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509559,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33352,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338488,"statfs":{"total":96636764160,"available":96602611712,"internally_reserved":0,"allocated":2215936,"data_stored":981857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:46:43.096 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:46:43.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T08:46:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 -- 172.21.15.31:0/3631207360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc15bfc0 msgr2=0x7f98cc072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 --2- 172.21.15.31:0/3631207360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc15bfc0 0x7f98cc072560 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f98cc06e2f0 tx=0x7f98bc02a600 comp rx=0 tx=0).stop 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 -- 172.21.15.31:0/3631207360 shutdown_connections 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 --2- 172.21.15.31:0/3631207360 >> v2:172.21.15.45:3300/0 conn(0x7f98cc06fec0 0x7f98cc0b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 --2- 172.21.15.31:0/3631207360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc15bfc0 0x7f98cc072560 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 -- 172.21.15.31:0/3631207360 >> 172.21.15.31:0/3631207360 conn(0x7f98cc0cfbb0 msgr2=0x7f98cc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 -- 172.21.15.31:0/3631207360 shutdown_connections 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.300+0000 7f98d1000640 1 -- 172.21.15.31:0/3631207360 wait complete. 2024-09-13T08:46:43.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.301+0000 7f98d1000640 1 Processor -- start 2024-09-13T08:46:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.302+0000 7f98d1000640 1 -- start start 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98cc06fa60 0x7f98cc0c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f98cc15bfc0 0x7f98cc1f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc0cbfd0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc0cbe50 con 0x7f98cc06fa60 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98d1000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f98cc0b2f20 con 0x7f98cc15bfc0 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49232/0 (socket says 172.21.15.31:49232) 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 -- 172.21.15.31:0/1500157714 learned_addr learned my addr 172.21.15.31:0/1500157714 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98c9400640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f98cc15bfc0 0x7f98cc1f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98c9e00640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98cc06fa60 0x7f98cc0c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 -- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f98cc15bfc0 msgr2=0x7f98cc1f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f98cc15bfc0 0x7f98cc1f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 -- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98cc06fa60 msgr2=0x7f98cc0c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98cc06fa60 0x7f98cc0c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.303+0000 7f98ca800640 1 -- 172.21.15.31:0/1500157714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98cc1f5810 con 0x7f98cc1f52a0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98ca800640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009fd0 tx=0x7f98c400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4004d20 con 0x7f98cc1f52a0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98c400cac0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005020 con 0x7f98cc1f52a0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98cc1f5af0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.304+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98cc0cf5e0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.305+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98cc0b33d0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.306+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f98c40051c0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.307+0000 7f98c2000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 0x7f98ac07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.307+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f98c4099dd0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.308+0000 7f98c9e00640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 0x7f98ac07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.308+0000 7f98c9e00640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 0x7f98ac07b730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f98bc033000 tx=0x7f98bc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:43.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.311+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f98c4016030 con 0x7f98cc1f52a0 2024-09-13T08:46:43.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.584+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f98cc0c9fc0 con 0x7f98cc1f52a0 2024-09-13T08:46:43.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.587+0000 7f98c2000640 1 -- 172.21.15.31:0/1500157714 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) ==== 190+0+147 (secure 0 0 0) 0x7f98c4063c70 con 0x7f98cc1f52a0 2024-09-13T08:46:43.588 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-13T08:46:43.588 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB5+eNm65eiGhAAZV5fflEfCJ6gxeXKF5v+Uw== 2024-09-13T08:46:43.588 INFO:teuthology.orchestra.run.smithi031.stdout: caps mds = "allow" 2024-09-13T08:46:43.589 INFO:teuthology.orchestra.run.smithi031.stdout: caps mon = "allow r" 2024-09-13T08:46:43.589 INFO:teuthology.orchestra.run.smithi031.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-13T08:46:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:updated caps for client.0 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 msgr2=0x7f98ac07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 0x7f98ac07b730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f98bc033000 tx=0x7f98bc004730 comp rx=0 tx=0).stop 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 msgr2=0x7f98cc1e8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009fd0 tx=0x7f98c400ee60 comp rx=0 tx=0).stop 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 shutdown_connections 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f98ac079270 0x7f98ac07b730 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98cc1f52a0 0x7f98cc1e8850 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f98cc15bfc0 0x7f98cc1f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 --2- 172.21.15.31:0/1500157714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98cc06fa60 0x7f98cc0c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 >> 172.21.15.31:0/1500157714 conn(0x7f98cc0cfbb0 msgr2=0x7f98cc15e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:43.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 shutdown_connections 2024-09-13T08:46:43.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.590+0000 7f98d1000640 1 -- 172.21.15.31:0/1500157714 wait complete. 2024-09-13T08:46:43.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:43.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 -- 172.21.15.31:0/2970446138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 msgr2=0x7f87f80cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 --2- 172.21.15.31:0/2970446138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f80cf850 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005fd0 tx=0x7f87e802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 -- 172.21.15.31:0/2970446138 shutdown_connections 2024-09-13T08:46:43.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 --2- 172.21.15.31:0/2970446138 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 0x7f87f815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 --2- 172.21.15.31:0/2970446138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f80cf850 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 -- 172.21.15.31:0/2970446138 >> 172.21.15.31:0/2970446138 conn(0x7f87f80cc910 msgr2=0x7f87f80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:43.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 -- 172.21.15.31:0/2970446138 shutdown_connections 2024-09-13T08:46:43.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.766+0000 7f87fd200640 1 -- 172.21.15.31:0/2970446138 wait complete. 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.767+0000 7f87fd200640 1 Processor -- start 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 -- start start 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f80ce3c0 0x7f87f80b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 0x7f87f806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f87f80cb9d0 con 0x7f87f815bd70 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87f80cb850 con 0x7f87f80ce3c0 2024-09-13T08:46:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.768+0000 7f87fd200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f87f80cc360 con 0x7f87f815f9e0 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49250/0 (socket says 172.21.15.31:49250) 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 -- 172.21.15.31:0/3983203328 learned_addr learned my addr 172.21.15.31:0/3983203328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87f6800640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 0x7f87f806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87f5e00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f80ce3c0 0x7f87f80b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 -- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 msgr2=0x7f87f806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 0x7f87f806ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 -- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f80ce3c0 msgr2=0x7f87f80b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f80ce3c0 0x7f87f80b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 -- 172.21.15.31:0/3983203328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f81570e0 con 0x7f87f815bd70 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.769+0000 7f87efe00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f87e000d6d0 tx=0x7f87e000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.770+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0017070 con 0x7f87f815bd70 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.770+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87e0004310 con 0x7f87f815bd70 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.770+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0009a80 con 0x7f87f815bd70 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.770+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f80b3ab0 con 0x7f87f815bd70 2024-09-13T08:46:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.770+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f80b4070 con 0x7f87f815bd70 2024-09-13T08:46:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.771+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f80cf850 con 0x7f87f815bd70 2024-09-13T08:46:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.772+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f87e00035f0 con 0x7f87f815bd70 2024-09-13T08:46:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.773+0000 7f87ee000640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 0x7f87d407b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.773+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f87e0099460 con 0x7f87f815bd70 2024-09-13T08:46:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.773+0000 7f87f5e00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 0x7f87d407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.774+0000 7f87f5e00640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 0x7f87d407b730 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f87e8033000 tx=0x7f87e8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:43.777+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f87e0062ed0 con 0x7f87f815bd70 2024-09-13T08:46:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.015+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87f8072950 con 0x7f87f815bd70 2024-09-13T08:46:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.016+0000 7f87ee000640 1 -- 172.21.15.31:0/3983203328 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5230 (secure 0 0 0) 0x7f87e0067900 con 0x7f87f815bd70 2024-09-13T08:46:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:44.017 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":118,"btime":"2024-09-13T08:46:41:425334+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 118 2024-09-13T08:46:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 msgr2=0x7f87d407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 0x7f87d407b730 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f87e8033000 tx=0x7f87e8004730 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 msgr2=0x7f87f806e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f87e000d6d0 tx=0x7f87e000db90 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 shutdown_connections 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f87d4079270 0x7f87d407b730 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f87f815f9e0 0x7f87f806ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f815bd70 0x7f87f806e550 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 --2- 172.21.15.31:0/3983203328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f80ce3c0 0x7f87f80b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.018+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 >> 172.21.15.31:0/3983203328 conn(0x7f87f80cc910 msgr2=0x7f87f80acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.019+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 shutdown_connections 2024-09-13T08:46:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.019+0000 7f87fd200640 1 -- 172.21.15.31:0/3983203328 wait complete. 2024-09-13T08:46:44.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 118, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T08:46:37.846092+0000', 'modified': '2024-09-13T08:46:40.863828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9198': {'gid': 9198, 'name': 'c', 'rank': 0, 'incarnation': 115, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.31:6835/2620315837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6833', 'nonce': 2620315837}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 2620315837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9198, 'qdb_cluster': [9198]} 2024-09-13T08:46:44.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T08:46:44.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 -- 172.21.15.31:0/2982641341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 msgr2=0x7f198c0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 --2- 172.21.15.31:0/2982641341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c0cdf70 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f1974005fd0 tx=0x7f197402a540 comp rx=0 tx=0).stop 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 -- 172.21.15.31:0/2982641341 shutdown_connections 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 --2- 172.21.15.31:0/2982641341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c0cdf70 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 --2- 172.21.15.31:0/2982641341 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 -- 172.21.15.31:0/2982641341 >> 172.21.15.31:0/2982641341 conn(0x7f198c0cc7a0 msgr2=0x7f198c0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 -- 172.21.15.31:0/2982641341 shutdown_connections 2024-09-13T08:46:44.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.602+0000 7f198b200640 1 -- 172.21.15.31:0/2982641341 wait complete. 2024-09-13T08:46:44.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.603+0000 7f198b200640 1 Processor -- start 2024-09-13T08:46:44.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 -- start start 2024-09-13T08:46:44.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f198c0ce4b0 0x7f198c144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f198c070040 con 0x7f198c06e670 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f198c06fec0 con 0x7f198c0ce4b0 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.604+0000 7f198b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f198c0701c0 con 0x7f198c15bb70 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49278/0 (socket says 172.21.15.31:49278) 2024-09-13T08:46:44.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 -- 172.21.15.31:0/184508201 learned_addr learned my addr 172.21.15.31:0/184508201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f198a800640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 -- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 msgr2=0x7f198c145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1981400640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f198c0ce4b0 0x7f198c144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c145220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 -- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f198c0ce4b0 msgr2=0x7f198c144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f198c0ce4b0 0x7f198c144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 -- 172.21.15.31:0/184508201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c15eb50 con 0x7f198c06e670 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f198a800640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.605+0000 7f1989e00640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f197c00da50 tx=0x7f197c00df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.606+0000 7f1982a00640 1 -- 172.21.15.31:0/184508201 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c017070 con 0x7f198c06e670 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.606+0000 7f1982a00640 1 -- 172.21.15.31:0/184508201 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197c010460 con 0x7f198c06e670 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.606+0000 7f1982a00640 1 -- 172.21.15.31:0/184508201 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c0108b0 con 0x7f198c06e670 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.606+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c15c440 con 0x7f198c06e670 2024-09-13T08:46:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.606+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c15ca00 con 0x7f198c06e670 2024-09-13T08:46:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.608+0000 7f1982a00640 1 -- 172.21.15.31:0/184508201 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f197c01a070 con 0x7f198c06e670 2024-09-13T08:46:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.609+0000 7f1982a00640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 0x7f196807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.609+0000 7f1982a00640 1 -- 172.21.15.31:0/184508201 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f197c099960 con 0x7f198c06e670 2024-09-13T08:46:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.609+0000 7f1993aaf740 1 Processor -- start 2024-09-13T08:46:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.609+0000 7f1981400640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 0x7f196807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.609+0000 7f1993aaf740 1 -- start start 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1981400640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 0x7f196807b730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1974033000 tx=0x7f1974004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72ce1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72c9d190 0x564e72ce0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564e72c1f430 con 0x564e72c9cc10 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564e72c0f5d0 con 0x564e72c9d190 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.610+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x564e72bfea00 con 0x564e72ce1aa0 2024-09-13T08:46:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49282/0 (socket says 172.21.15.31:49282) 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 -- 172.21.15.31:0/3067767976 learned_addr learned my addr 172.21.15.31:0/3067767976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f198a800640 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72c9d190 0x564e72ce0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1989e00640 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72ce1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 -- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 msgr2=0x564e72ce1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72ce1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 -- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72c9d190 msgr2=0x564e72ce0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72c9d190 0x564e72ce0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 -- 172.21.15.31:0/3067767976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e72bc7520 con 0x564e72c9cc10 2024-09-13T08:46:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.611+0000 7f1981400640 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1974009850 tx=0x7f197403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fe3a8c71758ca6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f197a600640 1 -- 172.21.15.31:0/3067767976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974035050 con 0x564e72c9cc10 2024-09-13T08:46:44.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f197a600640 1 -- 172.21.15.31:0/3067767976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197404c070 con 0x564e72c9cc10 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f197a600640 1 -- 172.21.15.31:0/3067767976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19740476a0 con 0x564e72c9cc10 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 -- 172.21.15.31:0/3067767976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 msgr2=0x564e72ce0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1974009850 tx=0x7f197403b000 comp rx=0 tx=0).stop 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 -- 172.21.15.31:0/3067767976 shutdown_connections 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72c9d190 0x564e72ce0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9cc10 0x564e72ce0580 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 --2- 172.21.15.31:0/3067767976 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72ce1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 -- 172.21.15.31:0/3067767976 >> 172.21.15.31:0/3067767976 conn(0x564e72c981e0 msgr2=0x564e72c9bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 -- 172.21.15.31:0/3067767976 shutdown_connections 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.612+0000 7f1993aaf740 1 -- 172.21.15.31:0/3067767976 wait complete. 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.613+0000 7f1993aaf740 1 -- start start 2024-09-13T08:46:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.613+0000 7f1993aaf740 1 Processor -- start 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72cf1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72cb0810 0x564e72cb0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564e72ba5330 con 0x564e72c9d190 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564e72c02810 con 0x564e72cb0810 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x564e7296f620 con 0x564e72ce1aa0 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.614+0000 7f1993aaf740 10 client.0 _ll_register_callbacks cb 0x564e72baa420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49292/0 (socket says 172.21.15.31:49292) 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 -- 172.21.15.31:0/1938455480 learned_addr learned my addr 172.21.15.31:0/1938455480 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f198a800640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72cb0810 0x564e72cb0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1981400640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72cf1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 -- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 msgr2=0x564e72cf1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72cf1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 -- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72cb0810 msgr2=0x564e72cb0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72cb0810 0x564e72cb0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 -- 172.21.15.31:0/1938455480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e72960850 con 0x564e72c9d190 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1989e00640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f197c062e00 tx=0x7f197c062990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28ee562dcb30e07 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f196ea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c021040 con 0x564e72c9d190 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f197c089180 con 0x564e72c9d190 2024-09-13T08:46:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.615+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197404c070 con 0x564e72c9d190 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f197c067be0 con 0x564e72c9d190 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 10 client.9339 fetch_fsmap learned FSMap version 118 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x564e729d76b0 con 0x564e72c9d190 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197404c070 con 0x564e72c9d190 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 118) ==== 3502+0+0 (secure 0 0 0) 0x7f197c09c0c0 con 0x564e72c9d190 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 10 client.9339 fetch_fsmap finished waiting for FSMap version 118 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 10 client.9339 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 20 client.9339 populate_metadata read hostname 'smithi031' 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72cb4040 0x564e72cddfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 4 client.9339 mds_command: new command op to 9198 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 --> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e72c11520 con 0x564e72cb4040 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.616+0000 7f1981400640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72cb4040 0x564e72cddfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.617+0000 7f1981400640 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72cb4040 0x564e72cddfb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd56614beec9f46 server_cookie=4e99994051d3a55e in_seq=0 out_seq=0 2024-09-13T08:46:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.617+0000 7f196ea00640 10 client.9339 ms_handle_connect on v2:172.21.15.31:6833/2620315837 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f196ea00640 1 -- 172.21.15.31:0/1938455480 <== mds.0 v2:172.21.15.31:6833/2620315837 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564e72c11520 con 0x564e72cb4040 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f196ea00640 10 client.9339 handle_command_reply: tid=0 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 client.9339 shutdown 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 msgr2=0x564e72cc53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f197c062e00 tx=0x7f197c062990 comp rx=0 tx=0).stop 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 shutdown_connections 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72cb4040 0x564e72cddfb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72cb0810 0x564e72cb0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f196ea00640 0 client.9339 ms_handle_reset on v2:172.21.15.31:6833/2620315837 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72ce1aa0 0x564e72cf1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 --2- 172.21.15.31:0/1938455480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72cc53f0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 >> 172.21.15.31:0/1938455480 conn(0x564e72c9cc10 msgr2=0x564e72cda600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 shutdown_connections 2024-09-13T08:46:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 1 -- 172.21.15.31:0/1938455480 wait complete. 2024-09-13T08:46:44.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.618+0000 7f1993aaf740 20 client.9339 trim_cache size 0 max 16384 2024-09-13T08:46:44.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.637+0000 7f1993aaf740 1 Processor -- start 2024-09-13T08:46:44.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 -- start start 2024-09-13T08:46:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72cb0810 0x564e72cde7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72c95390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f198c200040 con 0x564e72c9d190 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f198c0b5480 con 0x564e72ce1aa0 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.638+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f198c1f87d0 con 0x564e72cb0810 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49308/0 (socket says 172.21.15.31:49308) 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 -- 172.21.15.31:0/4278852281 learned_addr learned my addr 172.21.15.31:0/4278852281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f198a800640 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72c95390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1981400640 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72cb0810 0x564e72cde7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 -- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72cb0810 msgr2=0x564e72cde7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72cb0810 0x564e72cde7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 -- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 msgr2=0x564e72c95390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72c95390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 -- 172.21.15.31:0/4278852281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e72ce3510 con 0x564e72c9d190 2024-09-13T08:46:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1989e00640 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f197c099bd0 tx=0x7f197c099b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04094889eb166f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1979c00640 1 -- 172.21.15.31:0/4278852281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c021070 con 0x564e72c9d190 2024-09-13T08:46:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1979c00640 1 -- 172.21.15.31:0/4278852281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197c07c020 con 0x564e72c9d190 2024-09-13T08:46:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.639+0000 7f1979c00640 1 -- 172.21.15.31:0/4278852281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c062ed0 con 0x564e72c9d190 2024-09-13T08:46:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 -- 172.21.15.31:0/4278852281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 msgr2=0x564e72ce1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f197c099bd0 tx=0x7f197c099b60 comp rx=0 tx=0).stop 2024-09-13T08:46:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 -- 172.21.15.31:0/4278852281 shutdown_connections 2024-09-13T08:46:44.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72c95390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72cb0810 0x564e72cde7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 --2- 172.21.15.31:0/4278852281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72c9d190 0x564e72ce1080 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 -- 172.21.15.31:0/4278852281 >> 172.21.15.31:0/4278852281 conn(0x564e72c9cc10 msgr2=0x564e72c97170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 -- 172.21.15.31:0/4278852281 shutdown_connections 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.640+0000 7f1993aaf740 1 -- 172.21.15.31:0/4278852281 wait complete. 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.641+0000 7f1993aaf740 1 -- start start 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.641+0000 7f1993aaf740 1 Processor -- start 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72c9d190 0x564e72d5b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72cc5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564e72cb3ec0 con 0x564e72cb0810 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564e72cc2580 con 0x564e72ce1aa0 2024-09-13T08:46:44.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x564e72c9b950 con 0x564e72c9d190 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1993aaf740 10 client.0 _ll_register_callbacks cb 0x7f197c062870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1981400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1981400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49324/0 (socket says 172.21.15.31:49324) 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1981400640 1 -- 172.21.15.31:0/687780387 learned_addr learned my addr 172.21.15.31:0/687780387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f1989e00640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72c9d190 0x564e72d5b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.642+0000 7f198a800640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72cc5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 -- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72c9d190 msgr2=0x564e72d5b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72c9d190 0x564e72d5b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 -- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 msgr2=0x564e72cc5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72cc5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 -- 172.21.15.31:0/687780387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e72d5b5a0 con 0x564e72cb0810 2024-09-13T08:46:44.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1981400640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f19740099f0 tx=0x7f19740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb72b95ae405f2c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f196ea00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974035040 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f197403b660 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197c07c020 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f1974062050 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1993aaf740 10 client.9345 fetch_fsmap learned FSMap version 118 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.643+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x564e72d64bc0 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197c07c020 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 118) ==== 3502+0+0 (secure 0 0 0) 0x7f1974065020 con 0x564e72cb0810 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 10 client.9345 fetch_fsmap finished waiting for FSMap version 118 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 10 client.9345 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 20 client.9345 populate_metadata read hostname 'smithi031' 2024-09-13T08:46:44.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72ce04a0 0x564e72ce3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 4 client.9345 mds_command: new command op to 9198 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 --> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] -- command(tid 0: {"prefix": "status"}) -- 0x564e72c0cfd0 con 0x564e72ce04a0 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1989e00640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72ce04a0 0x564e72ce3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f1989e00640 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72ce04a0 0x564e72ce3880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ec2fd34ac0cabd server_cookie=8d1914091e82c1e0 in_seq=0 out_seq=0 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.644+0000 7f196ea00640 10 client.9345 ms_handle_connect on v2:172.21.15.31:6833/2620315837 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f196ea00640 1 -- 172.21.15.31:0/687780387 <== mds.0 v2:172.21.15.31:6833/2620315837 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x564e72c0cfd0 con 0x564e72ce04a0 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f196ea00640 10 client.9345 handle_command_reply: tid=0 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 client.9345 shutdown 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 msgr2=0x564e72ceeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f19740099f0 tx=0x7f19740046c0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 shutdown_connections 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:6833/2620315837,v1:172.21.15.31:6835/2620315837] conn(0x564e72ce04a0 0x564e72ce3880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e72ce1aa0 0x564e72cc5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e72cb0810 0x564e72ceeef0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 --2- 172.21.15.31:0/687780387 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x564e72c9d190 0x564e72d5b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f196ea00640 0 client.9345 ms_handle_reset on v2:172.21.15.31:6833/2620315837 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 >> 172.21.15.31:0/687780387 conn(0x564e72c9cc10 msgr2=0x564e72cc1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 shutdown_connections 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 1 -- 172.21.15.31:0/687780387 wait complete. 2024-09-13T08:46:44.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.645+0000 7f1993aaf740 20 client.9345 trim_cache size 0 max 16384 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "0251b593-a02a-47fa-b938-c065b7284822", 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 9198, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 6.2467754370000002, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 118, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 176, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 176, 2024-09-13T08:46:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 11.506568129 2024-09-13T08:46:44.653 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:46:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 msgr2=0x7f196807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 0x7f196807b730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1974033000 tx=0x7f1974004730 comp rx=0 tx=0).stop 2024-09-13T08:46:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 msgr2=0x7f198c1f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f197c00da50 tx=0x7f197c00df10 comp rx=0 tx=0).stop 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 shutdown_connections 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1968079270 0x7f196807b730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f198c15bb70 0x7f198c145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f198c0ce4b0 0x7f198c144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 --2- 172.21.15.31:0/184508201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f198c06e670 0x7f198c1f3d00 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 >> 172.21.15.31:0/184508201 conn(0x7f198c0cc7a0 msgr2=0x7f198c158030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 shutdown_connections 2024-09-13T08:46:44.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.646+0000 7f198b200640 1 -- 172.21.15.31:0/184508201 wait complete. 2024-09-13T08:46:44.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:46:44.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 -- 172.21.15.31:0/183021268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 msgr2=0x7f09400cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 --2- 172.21.15.31:0/183021268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400cc800 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0938005fd0 tx=0x7f093802ab30 comp rx=0 tx=0).stop 2024-09-13T08:46:44.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 -- 172.21.15.31:0/183021268 shutdown_connections 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 --2- 172.21.15.31:0/183021268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400cc800 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 -- 172.21.15.31:0/183021268 >> 172.21.15.31:0/183021268 conn(0x7f094006cb00 msgr2=0x7f094006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 -- 172.21.15.31:0/183021268 shutdown_connections 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.823+0000 7f093f200640 1 -- 172.21.15.31:0/183021268 wait complete. 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.825+0000 7f093f200640 1 Processor -- start 2024-09-13T08:46:44.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.825+0000 7f093f200640 1 -- start start 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09400caa00 0x7f09400c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09400cf2f0 0x7f09401576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09400ce5d0 con 0x7f09400cc3e0 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f09400ce450 con 0x7f09400caa00 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093f200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f09400ce750 con 0x7f09400cf2f0 2024-09-13T08:46:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35102/0 (socket says 172.21.15.31:35102) 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 -- 172.21.15.31:0/1908569790 learned_addr learned my addr 172.21.15.31:0/1908569790 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093de00640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09400caa00 0x7f09400c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f093e800640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09400cf2f0 0x7f09401576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 -- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09400cf2f0 msgr2=0x7f09401576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09400cf2f0 0x7f09401576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 -- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09400caa00 msgr2=0x7f09400c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09400caa00 0x7f09400c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.826+0000 7f0935400640 1 -- 172.21.15.31:0/1908569790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09400c8100 con 0x7f09400cc3e0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f0935400640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f092800c400 tx=0x7f092800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928017070 con 0x7f09400cc3e0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09280053c0 con 0x7f09400cc3e0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09401f5130 con 0x7f09400cc3e0 2024-09-13T08:46:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928003850 con 0x7f09400cc3e0 2024-09-13T08:46:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.827+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09401f56f0 con 0x7f09400cc3e0 2024-09-13T08:46:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.828+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09400cb440 con 0x7f09400cc3e0 2024-09-13T08:46:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.829+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f092801a070 con 0x7f09400cc3e0 2024-09-13T08:46:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.830+0000 7f0936a00640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 0x7f091c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.831+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f0928098830 con 0x7f09400cc3e0 2024-09-13T08:46:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.831+0000 7f093de00640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 0x7f091c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:44.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.831+0000 7f093de00640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 0x7f091c07b730 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0940158020 tx=0x7f0930009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:44.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:44.835+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f09400cb440 con 0x7f09400cc3e0 2024-09-13T08:46:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.070+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f09400cec00 con 0x7f09400cc3e0 2024-09-13T08:46:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.071+0000 7f0936a00640 1 -- 172.21.15.31:0/1908569790 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v118) ==== 53+0+73 (secure 0 0 0) 0x7f09280626d0 con 0x7f09400cc3e0 2024-09-13T08:46:45.072 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:46:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 msgr2=0x7f091c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 0x7f091c07b730 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0940158020 tx=0x7f0930009040 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 msgr2=0x7f09400c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f092800c400 tx=0x7f092800c8c0 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 shutdown_connections 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f091c079270 0x7f091c07b730 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f09400cf2f0 0x7f09401576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09400cc3e0 0x7f09400c93d0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 --2- 172.21.15.31:0/1908569790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09400caa00 0x7f09400c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 >> 172.21.15.31:0/1908569790 conn(0x7f094006cb00 msgr2=0x7f09400afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 shutdown_connections 2024-09-13T08:46:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45.073+0000 7f093f200640 1 -- 172.21.15.31:0/1908569790 wait complete. 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:46:45.091 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:45.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.092 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.128 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-13T08:46:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42127sec preferred_lft 42127sec 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::1433:35ff:fe0b:58b4/64 scope link 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 8a:8b:0e:11:ea:ac brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::888b:eff:fe11:eaac/64 scope link 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 1a:de:6c:04:b5:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::18de:6cff:fe04:b5c2/64 scope link 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:45.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.133 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:46:45.222 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-13T08:46:45.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.225 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-13T08:46:45.261 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-13T08:46:45.261 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:46:45.262 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:46:45.262 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:46:45.262 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:46:45.262 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-13T08:46:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:46:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:45.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.464 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:46:45.500 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:46:45.500 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:46:45.500 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:45.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-13T08:46:45.597 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:45.597 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:45.632 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T08:46:45.633 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:45.633 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:45.708 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T08:46:45.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:45.837 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:45.837 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:45.864 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:45.864 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:45.864 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:46:45.864 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:46:45.865 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:46:45.866 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T08:46:45.866 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:46:45.866 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:46:45.866 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:46:45.866 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:46:45.869 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:46:45.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:45.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:45.926 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:46:46.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:46.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:46.023 INFO:teuthology.orchestra.run.smithi031.stdout:client7721 2024-09-13T08:46:46.023 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:46.023 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:46.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T08:46:46.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 -- 172.21.15.31:0/190964914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815fe20 msgr2=0x7fece80cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:46.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 --2- 172.21.15.31:0/190964914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815fe20 0x7fece80cbc50 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fece0005fd0 tx=0x7fece002aa90 comp rx=0 tx=0).stop 2024-09-13T08:46:46.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 -- 172.21.15.31:0/190964914 shutdown_connections 2024-09-13T08:46:46.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 --2- 172.21.15.31:0/190964914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815fe20 0x7fece80cbc50 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 --2- 172.21.15.31:0/190964914 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815bd70 0x7fece80cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 -- 172.21.15.31:0/190964914 >> 172.21.15.31:0/190964914 conn(0x7fece80cc910 msgr2=0x7fece80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 -- 172.21.15.31:0/190964914 shutdown_connections 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.260+0000 7fecefe00640 1 -- 172.21.15.31:0/190964914 wait complete. 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.262+0000 7fecefe00640 1 Processor -- start 2024-09-13T08:46:46.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.262+0000 7fecefe00640 1 -- start start 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815f9e0 0x7fece8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece815fe20 0x7fece80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fece80cec50 con 0x7fece815bd70 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fece80cead0 con 0x7fece815fe20 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecefe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fece80cb850 con 0x7fece815f9e0 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:46.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35128/0 (socket says 172.21.15.31:35128) 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 -- 172.21.15.31:0/3993688594 learned_addr learned my addr 172.21.15.31:0/3993688594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fecede00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece815fe20 0x7fece80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7fececa00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815f9e0 0x7fece8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 -- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815f9e0 msgr2=0x7fece8155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815f9e0 0x7fece8155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 -- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece815fe20 msgr2=0x7fece80c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.263+0000 7feced400640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece815fe20 0x7fece80c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7feced400640 1 -- 172.21.15.31:0/3993688594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece8156920 con 0x7fece815bd70 2024-09-13T08:46:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7feced400640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00d980 tx=0x7fecdc00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc017030 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecdc009b20 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc004020 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece80c3a40 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.264+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece80c4000 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.265+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece80cf850 con 0x7fece815bd70 2024-09-13T08:46:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.267+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fecdc0041c0 con 0x7fece815bd70 2024-09-13T08:46:46.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.267+0000 7fece5600640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 0x7fecc807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:46.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.268+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7fecdc0994b0 con 0x7fece815bd70 2024-09-13T08:46:46.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.268+0000 7fececa00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 0x7fecc807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.268+0000 7fececa00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 0x7fecc807b730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fece8157b90 tx=0x7fecd4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.272+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fecdc0633b0 con 0x7fece815bd70 2024-09-13T08:46:46.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.507+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fece80c36e0 con 0x7fece815bd70 2024-09-13T08:46:46.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.508+0000 7fece5600640 1 -- 172.21.15.31:0/3993688594 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v118) ==== 53+0+73 (secure 0 0 0) 0x7fecdc062cb0 con 0x7fece815bd70 2024-09-13T08:46:46.509 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 msgr2=0x7fecc807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 0x7fecc807b730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fece8157b90 tx=0x7fecd4009040 comp rx=0 tx=0).stop 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 msgr2=0x7fece80c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00d980 tx=0x7fecdc00de40 comp rx=0 tx=0).stop 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 shutdown_connections 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fecc8079270 0x7fecc807b730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece815fe20 0x7fece80c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fece815f9e0 0x7fece8155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 --2- 172.21.15.31:0/3993688594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece815bd70 0x7fece80c7a30 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 >> 172.21.15.31:0/3993688594 conn(0x7fece80cc910 msgr2=0x7fece80acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 shutdown_connections 2024-09-13T08:46:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:46.510+0000 7fecefe00640 1 -- 172.21.15.31:0/3993688594 wait complete. 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T08:46:46.527 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T08:46:46.528 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T08:46:46.528 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:46.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:46.528 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-13T08:46:46.569 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42130sec preferred_lft 42130sec 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T08:46:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::389c:7dff:fe43:49d7/64 scope link 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether fa:4b:19:95:91:a1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::f84b:19ff:fe95:91a1/64 scope link 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 1a:a1:15:5b:8a:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::18a1:15ff:fe5b:8ab7/64 scope link 2024-09-13T08:46:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-13T08:46:46.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:46.572 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T08:46:46.667 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-13T08:46:46.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:46.670 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-13T08:46:46.708 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-13T08:46:46.708 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T08:46:46.708 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T08:46:46.708 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T08:46:46.708 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T08:46:46.708 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-13T08:46:46.709 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-13T08:46:46.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T08:46:46.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T08:46:46.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T08:46:46.709 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-13T08:46:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-13T08:46:46.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:46.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:46.908 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:46:46.943 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:46:46.943 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:46:46.944 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:46.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:46.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-13T08:46:47.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:47.040 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:47.075 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T08:46:47.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T08:46:47.075 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:47.149 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T08:46:47.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:47.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:47.278 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:47.278 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:47.305 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T08:46:47.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T08:46:47.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.45:3300,172.21.15.45:3301" 2024-09-13T08:46:47.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 1@0251b593-a02a-47fa-b938-c065b7284822.cephfs=/ 2024-09-13T08:46:47.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.31:3300/172.21.15.45:3300/172.21.15.45:3301" will pass to kernel 2024-09-13T08:46:47.307 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T08:46:47.307 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-09-13T08:46:47.307 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T08:46:47.307 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T08:46:47.307 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T08:46:47.310 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T08:46:47.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:47.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:47.365 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T08:46:47.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:47.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-13T08:46:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:47.457 INFO:teuthology.orchestra.run.smithi045.stdout:client8704 2024-09-13T08:46:47.457 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:47.458 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:47.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-13T08:46:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 -- 172.21.15.31:0/1168134036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d1290 msgr2=0x7fa7bc0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 --2- 172.21.15.31:0/1168134036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc0cf900 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4005fd0 tx=0x7fa7b402a8b0 comp rx=0 tx=0).stop 2024-09-13T08:46:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 -- 172.21.15.31:0/1168134036 shutdown_connections 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 --2- 172.21.15.31:0/1168134036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc0cf900 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 -- 172.21.15.31:0/1168134036 >> 172.21.15.31:0/1168134036 conn(0x7fa7bc0d3d90 msgr2=0x7fa7bc0d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 -- 172.21.15.31:0/1168134036 shutdown_connections 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.668+0000 7fa7c1400640 1 -- 172.21.15.31:0/1168134036 wait complete. 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.670+0000 7fa7c1400640 1 Processor -- start 2024-09-13T08:46:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.670+0000 7fa7c1400640 1 -- start start 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7bc0b2f20 0x7fa7bc1f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b3b80 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b4ce0 con 0x7fa7bc0d1290 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c1400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b3d00 con 0x7fa7bc0b2f20 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35136/0 (socket says 172.21.15.31:35136) 2024-09-13T08:46:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 -- 172.21.15.31:0/2373254221 learned_addr learned my addr 172.21.15.31:0/2373254221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bbe00640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7bc0b2f20 0x7fa7bc1f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7c0a00640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 -- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7bc0b2f20 msgr2=0x7fa7bc1f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7bc0b2f20 0x7fa7bc1f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 -- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0d1290 msgr2=0x7fa7bc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.671+0000 7fa7bb400640 1 -- 172.21.15.31:0/2373254221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc15bdf0 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7bb400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c400 tx=0x7fa7a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8017070 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7a80035f0 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8003a40 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc1f5690 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.672+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc15bb10 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.673+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7bc0d32e0 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.675+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fa7a801a070 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.675+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 0x7fa79c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.676+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7fa7a80987b0 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.676+0000 7fa7bbe00640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 0x7fa79c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.676+0000 7fa7bbe00640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 0x7fa79c07b730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac000f60 tx=0x7fa7ac009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.680+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa7bc0d32e0 con 0x7fa7bc0d07a0 2024-09-13T08:46:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:47.913+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fa7bc0cf900 con 0x7fa7bc0d07a0 2024-09-13T08:46:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.459+0000 7fa7b3e00640 1 -- 172.21.15.31:0/2373254221 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v119) ==== 136+0+0 (secure 0 0 0) 0x7fa7bc15bdf0 con 0x7fa7bc0d07a0 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 msgr2=0x7fa79c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 0x7fa79c07b730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac000f60 tx=0x7fa7ac009040 comp rx=0 tx=0).stop 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 msgr2=0x7fa7bc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c400 tx=0x7fa7a800c8c0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 shutdown_connections 2024-09-13T08:46:48.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa79c079270 0x7fa79c07b730 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0d1290 0x7fa7bc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7bc0d07a0 0x7fa7bc06d540 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 --2- 172.21.15.31:0/2373254221 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa7bc0b2f20 0x7fa7bc1f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 >> 172.21.15.31:0/2373254221 conn(0x7fa7bc0d3d90 msgr2=0x7fa7bc0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 shutdown_connections 2024-09-13T08:46:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.462+0000 7fa7c1400640 1 -- 172.21.15.31:0/2373254221 wait complete. 2024-09-13T08:46:48.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 -- 172.21.15.31:0/2500108429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca015e930 msgr2=0x7f9ca015ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 --2- 172.21.15.31:0/2500108429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca015e930 0x7f9ca015ed70 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f9c88005fd0 tx=0x7f9c8802a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 -- 172.21.15.31:0/2500108429 shutdown_connections 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 --2- 172.21.15.31:0/2500108429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca015e930 0x7f9ca015ed70 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 --2- 172.21.15.31:0/2500108429 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca01446e0 0x7f9ca015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 -- 172.21.15.31:0/2500108429 >> 172.21.15.31:0/2500108429 conn(0x7f9ca00b18f0 msgr2=0x7f9ca00b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.642+0000 7f9ca5400640 1 -- 172.21.15.31:0/2500108429 shutdown_connections 2024-09-13T08:46:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.643+0000 7f9ca5400640 1 -- 172.21.15.31:0/2500108429 wait complete. 2024-09-13T08:46:48.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.644+0000 7f9ca5400640 1 Processor -- start 2024-09-13T08:46:48.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.644+0000 7f9ca5400640 1 -- start start 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 0x7f9ca0073780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca015e930 0x7f9ca0073cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00cf2c0 con 0x7f9ca00cec90 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00cf140 con 0x7f9ca01446e0 2024-09-13T08:46:48.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9ca5400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca00cf440 con 0x7f9ca015e930 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9c9e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9c9e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35160/0 (socket says 172.21.15.31:35160) 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9c9e800640 1 -- 172.21.15.31:0/1514845302 learned_addr learned my addr 172.21.15.31:0/1514845302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.645+0000 7f9c95e00640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 0x7f9ca0073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9f200640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca015e930 0x7f9ca0073cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 -- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca015e930 msgr2=0x7f9ca0073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca015e930 0x7f9ca0073cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 -- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 msgr2=0x7f9ca0073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 0x7f9ca0073780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 -- 172.21.15.31:0/1514845302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca00afd80 con 0x7f9ca00cec90 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c95e00640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 0x7f9ca0073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:48.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c9e800640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000d7a0 tx=0x7f9c9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90017070 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c90009d20 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90005280 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca00b05e0 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.646+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca00b0060 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.648+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca015bdc0 con 0x7f9ca00cec90 2024-09-13T08:46:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.649+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f9c90005420 con 0x7f9ca00cec90 2024-09-13T08:46:48.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.650+0000 7f9c97400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 0x7f9c7c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:48.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.650+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f9c90099610 con 0x7f9ca00cec90 2024-09-13T08:46:48.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.650+0000 7f9c95e00640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 0x7f9c7c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:48.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.651+0000 7f9c95e00640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 0x7f9c7c07b730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033000 tx=0x7f9c88004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:48.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.655+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9c90063510 con 0x7f9ca00cec90 2024-09-13T08:46:48.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.877+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f9ca009e0b0 con 0x7f9c7c079270 2024-09-13T08:46:48.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.878+0000 7f9c97400640 1 -- 172.21.15.31:0/1514845302 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f9ca009e0b0 con 0x7f9c7c079270 2024-09-13T08:46:48.879 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-13T08:46:48.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-13T08:46:48.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephfs" 2024-09-13T08:46:48.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-13T08:46:48.880 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-13T08:46:48.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 msgr2=0x7f9c7c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 0x7f9c7c07b730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033000 tx=0x7f9c88004730 comp rx=0 tx=0).stop 2024-09-13T08:46:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 msgr2=0x7f9ca013b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000d7a0 tx=0x7f9c9000dc60 comp rx=0 tx=0).stop 2024-09-13T08:46:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 shutdown_connections 2024-09-13T08:46:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c7c079270 0x7f9c7c07b730 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9ca015e930 0x7f9ca0073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca01446e0 0x7f9ca0073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 --2- 172.21.15.31:0/1514845302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ca00cec90 0x7f9ca013b870 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.880+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 >> 172.21.15.31:0/1514845302 conn(0x7f9ca00b18f0 msgr2=0x7f9ca00b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.881+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 shutdown_connections 2024-09-13T08:46:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:48.881+0000 7f9ca5400640 1 -- 172.21.15.31:0/1514845302 wait complete. 2024-09-13T08:46:48.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T08:46:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 -- 172.21.15.31:0/4029834960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 msgr2=0x7f8a0c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 --2- 172.21.15.31:0/4029834960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c06fc80 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0d0a70 tx=0x7f89f802a4d0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 -- 172.21.15.31:0/4029834960 shutdown_connections 2024-09-13T08:46:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 --2- 172.21.15.31:0/4029834960 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c0cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 --2- 172.21.15.31:0/4029834960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c06fc80 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 --2- 172.21.15.31:0/4029834960 >> v2:172.21.15.45:3300/0 conn(0x7f8a0c15bc00 0x7f8a0c073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 -- 172.21.15.31:0/4029834960 >> 172.21.15.31:0/4029834960 conn(0x7f8a0c0b1520 msgr2=0x7f8a0c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 -- 172.21.15.31:0/4029834960 shutdown_connections 2024-09-13T08:46:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.062+0000 7f8a12e00640 1 -- 172.21.15.31:0/4029834960 wait complete. 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.064+0000 7f8a12e00640 1 Processor -- start 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.064+0000 7f8a12e00640 1 -- start start 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.064+0000 7f8a12e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 0x7f8a0c0c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c0caad0 con 0x7f8a0c0b2820 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c0ca950 con 0x7f8a0c15bc00 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c0cac50 con 0x7f8a0c0cc3e0 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35184/0 (socket says 172.21.15.31:35184) 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 -- 172.21.15.31:0/12326220 learned_addr learned my addr 172.21.15.31:0/12326220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11000640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 -- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 msgr2=0x7f8a0c1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12400640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 0x7f8a0c0c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c1f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 -- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 msgr2=0x7f8a0c0c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 0x7f8a0c0c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 -- 172.21.15.31:0/12326220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c15d760 con 0x7f8a0c0b2820 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11000640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a12400640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 0x7f8a0c0c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.065+0000 7f8a11a00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000c970 tx=0x7f8a0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.066+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00007bb0 con 0x7f8a0c0b2820 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.066+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a00007e40 con 0x7f8a0c0b2820 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.066+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00005280 con 0x7f8a0c0b2820 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.066+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c15da40 con 0x7f8a0c0b2820 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.066+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c0c37e0 con 0x7f8a0c0b2820 2024-09-13T08:46:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.067+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0c0cc830 con 0x7f8a0c0b2820 2024-09-13T08:46:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.068+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f8a00005420 con 0x7f8a0c0b2820 2024-09-13T08:46:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.069+0000 7f8a09600640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 0x7f89ec07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.069+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f8a00099930 con 0x7f8a0c0b2820 2024-09-13T08:46:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.070+0000 7f8a11000640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 0x7f89ec07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.070+0000 7f8a11000640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 0x7f89ec07b730 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f89f8033000 tx=0x7f89f8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.074+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f8a00014030 con 0x7f8a0c0b2820 2024-09-13T08:46:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.268+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8a0c0af050 con 0x7f8a0c0b2820 2024-09-13T08:46:49.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.269+0000 7f8a09600640 1 -- 172.21.15.31:0/12326220 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f8a00063890 con 0x7f8a0c0b2820 2024-09-13T08:46:49.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 msgr2=0x7f89ec07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 0x7f89ec07b730 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f89f8033000 tx=0x7f89f8004730 comp rx=0 tx=0).stop 2024-09-13T08:46:49.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 msgr2=0x7f8a0c15d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000c970 tx=0x7f8a0000ce30 comp rx=0 tx=0).stop 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 shutdown_connections 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f89ec079270 0x7f89ec07b730 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a0c15bc00 0x7f8a0c0c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f8a0c0cc3e0 0x7f8a0c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 --2- 172.21.15.31:0/12326220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a0c0b2820 0x7f8a0c15d060 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 >> 172.21.15.31:0/12326220 conn(0x7f8a0c0b1520 msgr2=0x7f8a0c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 shutdown_connections 2024-09-13T08:46:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.271+0000 7f8a12e00640 1 -- 172.21.15.31:0/12326220 wait complete. 2024-09-13T08:46:49.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:49.288 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp67500/subvol63210) 2024-09-13T08:46:49.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:49.351 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-13T08:46:49.387 INFO:teuthology.orchestra.run.smithi031.stdout:cephfs_data 2024-09-13T08:46:49.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:49.388 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp67500/subvol63210) 2024-09-13T08:46:49.489 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:49.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp67500/subvol63210/.meta 2024-09-13T08:46:49.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol63210 subvol_grp67500 2024-09-13T08:46:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 -- 172.21.15.31:0/2125035875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 msgr2=0x7f9c840cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/2125035875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840cb740 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c005fd0 tx=0x7f9c6c02a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 -- 172.21.15.31:0/2125035875 shutdown_connections 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/2125035875 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c840cbc80 0x7f9c840b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/2125035875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840cb740 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 -- 172.21.15.31:0/2125035875 >> 172.21.15.31:0/2125035875 conn(0x7f9c840cfbb0 msgr2=0x7f9c840cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 -- 172.21.15.31:0/2125035875 shutdown_connections 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.753+0000 7f9c8aa00640 1 -- 172.21.15.31:0/2125035875 wait complete. 2024-09-13T08:46:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.755+0000 7f9c8aa00640 1 Processor -- start 2024-09-13T08:46:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.755+0000 7f9c8aa00640 1 -- start start 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c8415bfc0 0x7f9c841e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8406e740 con 0x7f9c840cbc80 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8406e5c0 con 0x7f9c84072560 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c8aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9c840b2f20 con 0x7f9c8415bfc0 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35190/0 (socket says 172.21.15.31:35190) 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c7b400640 1 -- 172.21.15.31:0/1146741662 learned_addr learned my addr 172.21.15.31:0/1146741662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c83e00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.756+0000 7f9c88a00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c8415bfc0 0x7f9c841e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 -- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c8415bfc0 msgr2=0x7f9c841e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c8415bfc0 0x7f9c841e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 -- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 msgr2=0x7f9c840c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 -- 172.21.15.31:0/1146741662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c840cf450 con 0x7f9c840cbc80 2024-09-13T08:46:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c7b400640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000c00 tx=0x7f9c6c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c03f070 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c6c005600 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c035030 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.757+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c841f8290 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.758+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c841f8850 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.759+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c840b3350 con 0x7f9c840cbc80 2024-09-13T08:46:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.760+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f9c6c042070 con 0x7f9c840cbc80 2024-09-13T08:46:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.761+0000 7f9c80c00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 0x7f9c6007b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.761+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f9c6c0b9eb0 con 0x7f9c840cbc80 2024-09-13T08:46:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.761+0000 7f9c83e00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 0x7f9c6007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.762+0000 7f9c83e00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 0x7f9c6007b730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9c74001720 tx=0x7f9c74009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.765+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9c840cf450 con 0x7f9c840cbc80 2024-09-13T08:46:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:49.988+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol63210", "group_name": "subvol_grp67500", "target": ["mon-mgr", ""]}) -- 0x7f9c840cb310 con 0x7f9c60079270 2024-09-13T08:46:49.996 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:49.995+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:49.996 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:49.995+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:49.996 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:49.995+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:49.996 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:49.995+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:49.996 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:49.995+0000 7fe16b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.030+0000 7f9c80c00640 1 -- 172.21.15.31:0/1146741662 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9c840cb310 con 0x7f9c60079270 2024-09-13T08:46:50.031 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp67500/subvol63210 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.032+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 msgr2=0x7f9c6007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.032+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 0x7f9c6007b730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9c74001720 tx=0x7f9c74009040 comp rx=0 tx=0).stop 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 msgr2=0x7f9c840cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000c00 tx=0x7f9c6c003820 comp rx=0 tx=0).stop 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 shutdown_connections 2024-09-13T08:46:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9c60079270 0x7f9c6007b730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9c8415bfc0 0x7f9c841e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c840cbc80 0x7f9c840cef10 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 --2- 172.21.15.31:0/1146741662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c84072560 0x7f9c840c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 >> 172.21.15.31:0/1146741662 conn(0x7f9c840cfbb0 msgr2=0x7f9c8415a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 shutdown_connections 2024-09-13T08:46:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.033+0000 7f9c8aa00640 1 -- 172.21.15.31:0/1146741662 wait complete. 2024-09-13T08:46:50.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol63210 subvol_grp67500 2024-09-13T08:46:50.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 -- 172.21.15.31:0/3092979138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 msgr2=0x7faddc0cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 --2- 172.21.15.31:0/3092979138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc0cdcc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009830 tx=0x7fadc402a480 comp rx=0 tx=0).stop 2024-09-13T08:46:50.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 -- 172.21.15.31:0/3092979138 shutdown_connections 2024-09-13T08:46:50.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 --2- 172.21.15.31:0/3092979138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc0cdcc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 --2- 172.21.15.31:0/3092979138 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 -- 172.21.15.31:0/3092979138 >> 172.21.15.31:0/3092979138 conn(0x7faddc0cfbb0 msgr2=0x7faddc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 -- 172.21.15.31:0/3092979138 shutdown_connections 2024-09-13T08:46:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.214+0000 7fade3c00640 1 -- 172.21.15.31:0/3092979138 wait complete. 2024-09-13T08:46:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.215+0000 7fade3c00640 1 Processor -- start 2024-09-13T08:46:50.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.215+0000 7fade3c00640 1 -- start start 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc1f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc1f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faddc0cb500 con 0x7faddc0cbc40 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faddc0cb380 con 0x7faddc0cc190 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.216+0000 7fade3c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7faddc0cb680 con 0x7faddc161480 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35224/0 (socket says 172.21.15.31:35224) 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 -- 172.21.15.31:0/3387385314 learned_addr learned my addr 172.21.15.31:0/3387385314 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fadd3e00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc1f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc1f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 -- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 msgr2=0x7faddc1f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc1f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 -- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 msgr2=0x7faddc1f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc1f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 -- 172.21.15.31:0/3387385314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc0b1e90 con 0x7faddc0cbc40 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fadd3e00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc1f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc1f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.217+0000 7fade1200640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e790 tx=0x7fadcc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.218+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00eea0 con 0x7faddc0cbc40 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.218+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadcc00cdb0 con 0x7faddc0cbc40 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.218+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc010690 con 0x7faddc0cbc40 2024-09-13T08:46:50.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.218+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc0b2100 con 0x7faddc0cbc40 2024-09-13T08:46:50.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.218+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc0c8930 con 0x7faddc0cbc40 2024-09-13T08:46:50.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.219+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faddc1625f0 con 0x7faddc0cbc40 2024-09-13T08:46:50.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.220+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7fadcc010830 con 0x7faddc0cbc40 2024-09-13T08:46:50.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.221+0000 7fadd9600640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 0x7fadb807b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.221+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7fadcc099da0 con 0x7faddc0cbc40 2024-09-13T08:46:50.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.221+0000 7fadd3e00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 0x7fadb807b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.222+0000 7fadd3e00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 0x7fadb807b760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fadc4034000 tx=0x7fadc4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:50.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.225+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fadcc063ca0 con 0x7faddc0cbc40 2024-09-13T08:46:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.453+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol63210", "group_name": "subvol_grp67500", "target": ["mon-mgr", ""]}) -- 0x7faddc0b5b30 con 0x7fadb80792a0 2024-09-13T08:46:50.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.457+0000 7fadd9600640 1 -- 172.21.15.31:0/3387385314 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faddc0b5b30 con 0x7fadb80792a0 2024-09-13T08:46:50.458 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/subvol_grp67500/subvol63210 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 msgr2=0x7fadb807b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 0x7fadb807b760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fadc4034000 tx=0x7fadc4004730 comp rx=0 tx=0).stop 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 msgr2=0x7faddc0b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e790 tx=0x7fadcc00ec50 comp rx=0 tx=0).stop 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 shutdown_connections 2024-09-13T08:46:50.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fadb80792a0 0x7fadb807b760 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7faddc161480 0x7faddc1f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faddc0cc190 0x7faddc1f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 --2- 172.21.15.31:0/3387385314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faddc0cbc40 0x7faddc0b2e10 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 >> 172.21.15.31:0/3387385314 conn(0x7faddc0cfbb0 msgr2=0x7faddc0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 shutdown_connections 2024-09-13T08:46:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.460+0000 7fade3c00640 1 -- 172.21.15.31:0/3387385314 wait complete. 2024-09-13T08:46:50.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:50.477 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/5d87c5dfc3a296b5f8670ab3f257d4cf.meta) 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout:[GLOBAL] 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout:version = 1 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout:type = subvolume 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout:path = /volumes/subvol_grp67500/subvol63210 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout:state = complete 2024-09-13T08:46:50.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:50.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol63210 subvol_grp67500 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1085200939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 msgr2=0x7f7cbc072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1085200939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc072560 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0cfab0 tx=0x7f7cac02a560 comp rx=0 tx=0).stop 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1085200939 shutdown_connections 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1085200939 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc06fec0 0x7f7cbc0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1085200939 >> v2:172.21.15.45:3300/0 conn(0x7f7cbc06fa60 0x7f7cbc0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1085200939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc072560 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1085200939 >> 172.21.15.31:0/1085200939 conn(0x7f7cbc0cfbb0 msgr2=0x7f7cbc0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1085200939 shutdown_connections 2024-09-13T08:46:50.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.706+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1085200939 wait complete. 2024-09-13T08:46:50.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.708+0000 7f7cc1e00640 1 Processor -- start 2024-09-13T08:46:50.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.708+0000 7f7cc1e00640 1 -- start start 2024-09-13T08:46:50.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 0x7f7cbc0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 0x7f7cbc1e8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b2b60 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b29e0 con 0x7f7cbc06fa60 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cc1e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b2ce0 con 0x7f7cbc1483c0 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cb2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cb2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35240/0 (socket says 172.21.15.31:35240) 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.709+0000 7f7cb2000640 1 -- 172.21.15.31:0/1247630222 learned_addr learned my addr 172.21.15.31:0/1247630222 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cbb400640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 0x7f7cbc1e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cbaa00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 0x7f7cbc0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 -- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 msgr2=0x7f7cbc1e8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 0x7f7cbc1e8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 -- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 msgr2=0x7f7cbc0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 0x7f7cbc0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 -- 172.21.15.31:0/1247630222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc0cf730 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cbaa00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 0x7f7cbc0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cbb400640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 0x7f7cbc1e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb2000640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400d770 tx=0x7f7ca400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4017070 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ca4009d20 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40106c0 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.710+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc1f86f0 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.711+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc1f8cb0 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.712+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc0b3200 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.713+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f7ca401a070 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.714+0000 7f7cb3e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 0x7f7c9807b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.715+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f7ca4099630 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.715+0000 7f7cbaa00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 0x7f7c9807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.715+0000 7f7cbaa00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 0x7f7c9807b730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7cac033000 tx=0x7f7cac004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.719+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7ca40634d0 con 0x7f7cbc15bfc0 2024-09-13T08:46:50.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.943+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol63210", "group_name": "subvol_grp67500", "target": ["mon-mgr", ""]}) -- 0x7f7cbc0b4610 con 0x7f7c98079270 2024-09-13T08:46:50.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.969+0000 7f7cb3e00640 1 -- 172.21.15.31:0/1247630222 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7cbc0b4610 con 0x7f7c98079270 2024-09-13T08:46:50.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 msgr2=0x7f7c9807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 0x7f7c9807b730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7cac033000 tx=0x7f7cac004730 comp rx=0 tx=0).stop 2024-09-13T08:46:50.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 msgr2=0x7f7cbc0c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:50.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400d770 tx=0x7f7ca400dc30 comp rx=0 tx=0).stop 2024-09-13T08:46:50.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 shutdown_connections 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7c98079270 0x7f7c9807b730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f7cbc1483c0 0x7f7cbc1e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cbc15bfc0 0x7f7cbc0c5e70 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 --2- 172.21.15.31:0/1247630222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cbc06fa60 0x7f7cbc0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 >> 172.21.15.31:0/1247630222 conn(0x7f7cbc0cfbb0 msgr2=0x7f7cbc074080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 shutdown_connections 2024-09-13T08:46:50.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:50.972+0000 7f7cc1e00640 1 -- 172.21.15.31:0/1247630222 wait complete. 2024-09-13T08:46:50.976 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.975+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.976 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.975+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.976 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.975+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.977 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.975+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.977 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.975+0000 7fe16e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.994 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-13T08:46:50.995 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.994+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.995 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.994+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.995 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.994+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.995 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.994+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:50.995 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-13T08:46:50.994+0000 7fe16da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T08:46:51.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-13T08:46:51.084 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-13T08:46:51.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-13T08:46:51.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp67500 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 -- 172.21.15.31:0/1696343099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 msgr2=0x7f158c0cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 --2- 172.21.15.31:0/1696343099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0cc830 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f1584005fd0 tx=0x7f158402a980 comp rx=0 tx=0).stop 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 -- 172.21.15.31:0/1696343099 shutdown_connections 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 --2- 172.21.15.31:0/1696343099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0cc830 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 --2- 172.21.15.31:0/1696343099 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c073ac0 0x7f158c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 -- 172.21.15.31:0/1696343099 >> 172.21.15.31:0/1696343099 conn(0x7f158c0b1520 msgr2=0x7f158c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:51.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 -- 172.21.15.31:0/1696343099 shutdown_connections 2024-09-13T08:46:51.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.256+0000 7f158b200640 1 -- 172.21.15.31:0/1696343099 wait complete. 2024-09-13T08:46:51.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.257+0000 7f158b200640 1 Processor -- start 2024-09-13T08:46:51.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 -- start start 2024-09-13T08:46:51.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f158c073ac0 0x7f158c156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:51.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:51.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c15bc00 0x7f158c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0cace0 con 0x7f158c0cc3e0 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.258+0000 7f158b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0cab60 con 0x7f158c073ac0 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f158b200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f158c0cae60 con 0x7f158c15bc00 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35318/0 (socket says 172.21.15.31:35318) 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 -- 172.21.15.31:0/3353419431 learned_addr learned my addr 172.21.15.31:0/3353419431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f158a800640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c15bc00 0x7f158c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589e00640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f158c073ac0 0x7f158c156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 -- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c15bc00 msgr2=0x7f158c0c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c15bc00 0x7f158c0c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 -- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f158c073ac0 msgr2=0x7f158c156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f158c073ac0 0x7f158c156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.259+0000 7f1589400640 1 -- 172.21.15.31:0/3353419431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c0c8d80 con 0x7f158c0cc3e0 2024-09-13T08:46:51.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f1589400640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f157800e9f0 tx=0x7f157800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:51.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157800cd80 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1578010430 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578010840 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158c156c50 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.260+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158c157210 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.261+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f158c0cb330 con 0x7f158c0cc3e0 2024-09-13T08:46:51.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.262+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f15780109e0 con 0x7f158c0cc3e0 2024-09-13T08:46:51.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.264+0000 7f1582000640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 0x7f156c07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:51.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.264+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f1578099f60 con 0x7f158c0cc3e0 2024-09-13T08:46:51.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.264+0000 7f1589e00640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 0x7f156c07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:51.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.265+0000 7f1589e00640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 0x7f156c07b730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f157c001720 tx=0x7f157c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:51.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.268+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f1578063870 con 0x7f158c0cc3e0 2024-09-13T08:46:51.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.491+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp67500", "target": ["mon-mgr", ""]}) -- 0x7f158c069700 con 0x7f156c079270 2024-09-13T08:46:51.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.495+0000 7f1582000640 1 -- 172.21.15.31:0/3353419431 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f158c069700 con 0x7f156c079270 2024-09-13T08:46:51.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 msgr2=0x7f156c07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:51.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 0x7f156c07b730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f157c001720 tx=0x7f157c009040 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 msgr2=0x7f158c0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f157800e9f0 tx=0x7f157800eeb0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 shutdown_connections 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f156c079270 0x7f156c07b730 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f158c15bc00 0x7f158c0c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f158c0cc3e0 0x7f158c0c6aa0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 --2- 172.21.15.31:0/3353419431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f158c073ac0 0x7f158c156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 >> 172.21.15.31:0/3353419431 conn(0x7f158c0b1520 msgr2=0x7f158c0cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 shutdown_connections 2024-09-13T08:46:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:51.497+0000 7f158b200640 1 -- 172.21.15.31:0/3353419431 wait complete. 2024-09-13T08:46:51.515 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T08:46:51.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:51.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T08:46:51.696 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T08:46:51.697 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:51.697 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:51.697 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:51.733 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:51.733 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:51.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:46:52.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 -- 172.21.15.31:0/3839234243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 msgr2=0x7f53fc15fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 --2- 172.21.15.31:0/3839234243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15fa50 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f53fc072340 tx=0x7f53f402aa10 comp rx=0 tx=0).stop 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 -- 172.21.15.31:0/3839234243 shutdown_connections 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 --2- 172.21.15.31:0/3839234243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15fa50 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 --2- 172.21.15.31:0/3839234243 >> v2:172.21.15.45:3300/0 conn(0x7f53fc0ced20 0x7f53fc0ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 --2- 172.21.15.31:0/3839234243 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 0x7f53fc0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 -- 172.21.15.31:0/3839234243 >> 172.21.15.31:0/3839234243 conn(0x7f53fc0b18f0 msgr2=0x7f53fc0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 -- 172.21.15.31:0/3839234243 shutdown_connections 2024-09-13T08:46:52.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.011+0000 7f5402400640 1 -- 172.21.15.31:0/3839234243 wait complete. 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.013+0000 7f5402400640 1 Processor -- start 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.013+0000 7f5402400640 1 -- start start 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53fc0ced20 0x7f53fc09b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 0x7f53fc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc160320 con 0x7f53fc0ce600 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1601a0 con 0x7f53fc0ced20 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f5402400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f53fc1604a0 con 0x7f53fc15be30 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35338/0 (socket says 172.21.15.31:35338) 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 -- 172.21.15.31:0/1830860701 learned_addr learned my addr 172.21.15.31:0/1830860701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53faa00640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53fc0ced20 0x7f53fc09b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fbe00640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 0x7f53fc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 -- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 msgr2=0x7f53fc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 0x7f53fc0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 -- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53fc0ced20 msgr2=0x7f53fc09b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53fc0ced20 0x7f53fc09b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.014+0000 7f53fb400640 1 -- 172.21.15.31:0/1830860701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53fc09b960 con 0x7f53fc0ce600 2024-09-13T08:46:52.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f53fb400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00b410 tx=0x7f53ec00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec015020 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53ec011940 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec011d90 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53fc1f8790 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.015+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53fc1f8d50 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.016+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53fc15fa50 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.017+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 102205+0+0 (secure 0 0 0) 0x7f53ec018020 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.019+0000 7f53f3400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 0x7f53dc07b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.019+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5753+0+0 (secure 0 0 0) 0x7f53ec099390 con 0x7f53fc0ce600 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.019+0000 7f53faa00640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 0x7f53dc07b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.020+0000 7f53faa00640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 0x7f53dc07b730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0cc2c0 tx=0x7f53e8008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.023+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f53ec01f020 con 0x7f53fc0ce600 2024-09-13T08:46:52.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.232+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f53fc1590b0 con 0x7f53fc0ce600 2024-09-13T08:46:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.233+0000 7f53f3400640 1 -- 172.21.15.31:0/1830860701 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v176) ==== 80+0+0 (secure 0 0 0) 0x7f53ec063230 con 0x7f53fc0ce600 2024-09-13T08:46:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:46:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 msgr2=0x7f53dc07b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 0x7f53dc07b730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0cc2c0 tx=0x7f53e8008040 comp rx=0 tx=0).stop 2024-09-13T08:46:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 msgr2=0x7f53fc15f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00b410 tx=0x7f53ec00b8d0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 shutdown_connections 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f53dc079270 0x7f53dc07b730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f53fc15be30 0x7f53fc0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53fc0ced20 0x7f53fc09b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 --2- 172.21.15.31:0/1830860701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53fc0ce600 0x7f53fc15f1b0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 >> 172.21.15.31:0/1830860701 conn(0x7f53fc0b18f0 msgr2=0x7f53fc0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 shutdown_connections 2024-09-13T08:46:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.235+0000 7f5402400640 1 -- 172.21.15.31:0/1830860701 wait complete. 2024-09-13T08:46:52.251 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T08:46:52.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:52.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T08:46:52.347 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:46:52.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:52.347 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T08:46:52.393 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:52.393 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:52.393 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:52.432 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:52.432 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:52.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T08:46:52.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 -- 172.21.15.31:0/508519289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29040051c0 msgr2=0x7f290400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 --2- 172.21.15.31:0/508519289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29040051c0 0x7f290400b9c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f291800c610 tx=0x7f2904005830 comp rx=0 tx=0).stop 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 -- 172.21.15.31:0/508519289 shutdown_connections 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 --2- 172.21.15.31:0/508519289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29040051c0 0x7f290400b9c0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 -- 172.21.15.31:0/508519289 >> 172.21.15.31:0/508519289 conn(0x7f29200b18f0 msgr2=0x7f29200b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 -- 172.21.15.31:0/508519289 shutdown_connections 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.678+0000 7f2925e00640 1 -- 172.21.15.31:0/508519289 wait complete. 2024-09-13T08:46:52.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.680+0000 7f2925e00640 1 Processor -- start 2024-09-13T08:46:52.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.680+0000 7f2925e00640 1 -- start start 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 0x7f292009bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 0x7f29201f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2920074040 con 0x7f29201f8680 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2920073ec0 con 0x7f29200c9e60 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2925e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f29200741c0 con 0x7f29040051c0 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35356/0 (socket says 172.21.15.31:35356) 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 -- 172.21.15.31:0/3642182888 learned_addr learned my addr 172.21.15.31:0/3642182888 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:52.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2916000640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 0x7f29201f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291ea00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 0x7f292009bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 -- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 msgr2=0x7f292009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 0x7f292009bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 -- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 msgr2=0x7f29201f8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 0x7f29201f8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291f400640 1 -- 172.21.15.31:0/3642182888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200c90a0 con 0x7f29201f8680 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f291ea00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 0x7f292009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.681+0000 7f2916000640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 0x7f29201f8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f291f400640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f291802b4d0 tx=0x7f291800c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918039070 con 0x7f29201f8680 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f291802bd60 con 0x7f29201f8680 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29200c9380 con 0x7f29201f8680 2024-09-13T08:46:52.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918006290 con 0x7f29201f8680 2024-09-13T08:46:52.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.682+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29200c9940 con 0x7f29201f8680 2024-09-13T08:46:52.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.683+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2920074720 con 0x7f29201f8680 2024-09-13T08:46:52.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.684+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f2918006430 con 0x7f29201f8680 2024-09-13T08:46:52.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.685+0000 7f2917e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 0x7f28fc07b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:52.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.685+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 5811+0+0 (secure 0 0 0) 0x7f29180b9180 con 0x7f29201f8680 2024-09-13T08:46:52.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.686+0000 7f291ea00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 0x7f28fc07b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:52.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.686+0000 7f291ea00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 0x7f28fc07b910 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f29100035f0 tx=0x7f2910006b70 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:52.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.690+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2918082fe0 con 0x7f29201f8680 2024-09-13T08:46:52.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.898+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f29200c9b10 con 0x7f29201f8680 2024-09-13T08:46:52.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.898+0000 7f2917e00640 1 -- 172.21.15.31:0/3642182888 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v177) ==== 80+0+0 (secure 0 0 0) 0x7f29180828e0 con 0x7f29201f8680 2024-09-13T08:46:52.899 INFO:teuthology.orchestra.run.smithi031.stderr:listed 0 entries 2024-09-13T08:46:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 msgr2=0x7f28fc07b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 0x7f28fc07b910 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f29100035f0 tx=0x7f2910006b70 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 msgr2=0x7f29200cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f291802b4d0 tx=0x7f291800c260 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 shutdown_connections 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f28fc079450 0x7f28fc07b910 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29201f8680 0x7f29200cab00 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29200c9e60 0x7f29201f8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 --2- 172.21.15.31:0/3642182888 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29040051c0 0x7f292009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 >> 172.21.15.31:0/3642182888 conn(0x7f29200b18f0 msgr2=0x7f29200aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:52.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 shutdown_connections 2024-09-13T08:46:52.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:52.901+0000 7f2925e00640 1 -- 172.21.15.31:0/3642182888 wait complete. 2024-09-13T08:46:52.917 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T08:46:52.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:52.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-13T08:46:52.987 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:46:52.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:46:52.987 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T08:46:53.025 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:53.026 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:46:53.026 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:53.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:46:53.062 INFO:tasks.cephfs.mount:Terminating background process 2024-09-13T08:46:53.062 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:46:53.062 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:46:53.102 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:46:53.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:53.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 -- 172.21.15.31:0/112455813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc072850 msgr2=0x7f20bc073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 --2- 172.21.15.31:0/112455813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc072850 0x7f20bc073fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f20a8005fd0 tx=0x7f20a802a440 comp rx=0 tx=0).stop 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 -- 172.21.15.31:0/112455813 shutdown_connections 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 --2- 172.21.15.31:0/112455813 >> v2:172.21.15.45:3300/0 conn(0x7f20bc074500 0x7f20bc0b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 --2- 172.21.15.31:0/112455813 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc072850 0x7f20bc073fc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 -- 172.21.15.31:0/112455813 >> 172.21.15.31:0/112455813 conn(0x7f20bc0cc910 msgr2=0x7f20bc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 -- 172.21.15.31:0/112455813 shutdown_connections 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.263+0000 7f20c4000640 1 -- 172.21.15.31:0/112455813 wait complete. 2024-09-13T08:46:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.265+0000 7f20c4000640 1 Processor -- start 2024-09-13T08:46:53.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.265+0000 7f20c4000640 1 -- start start 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc074500 0x7f20bc0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20bc152f10 0x7f20bc153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc15c490 con 0x7f20bc072850 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc0b1e50 con 0x7f20bc152f10 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c4000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f20bc0b1fd0 con 0x7f20bc074500 2024-09-13T08:46:53.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35360/0 (socket says 172.21.15.31:35360) 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c1600640 1 -- 172.21.15.31:0/2129660007 learned_addr learned my addr 172.21.15.31:0/2129660007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c2000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20bc152f10 0x7f20bc153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.266+0000 7f20c0c00640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc074500 0x7f20bc0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 -- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc074500 msgr2=0x7f20bc0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc074500 0x7f20bc0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 -- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20bc152f10 msgr2=0x7f20bc153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20bc152f10 0x7f20bc153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 -- 172.21.15.31:0/2129660007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc155640 con 0x7f20bc072850 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c1600640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f20b000c550 tx=0x7f20b000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b0017070 con 0x7f20bc072850 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20b00035f0 con 0x7f20bc072850 2024-09-13T08:46:53.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b0003a40 con 0x7f20bc072850 2024-09-13T08:46:53.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20bc155920 con 0x7f20bc072850 2024-09-13T08:46:53.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.267+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20bc155ee0 con 0x7f20bc072850 2024-09-13T08:46:53.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.268+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20bc073fc0 con 0x7f20bc072850 2024-09-13T08:46:53.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.269+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f20b001a070 con 0x7f20bc072850 2024-09-13T08:46:53.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.270+0000 7f20b9600640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 0x7f209c07b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.271+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 5811+0+0 (secure 0 0 0) 0x7f20b009b6f0 con 0x7f20bc072850 2024-09-13T08:46:53.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.271+0000 7f20c0c00640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 0x7f209c07b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.271+0000 7f20c0c00640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 0x7f209c07b910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f20a8033000 tx=0x7f20a8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:53.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.274+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f20b0001210 con 0x7f20bc072850 2024-09-13T08:46:53.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.516+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20bc0cce70 con 0x7f20bc072850 2024-09-13T08:46:53.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.517+0000 7f20b9600640 1 -- 172.21.15.31:0/2129660007 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5230 (secure 0 0 0) 0x7f20b00620d0 con 0x7f20bc072850 2024-09-13T08:46:53.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:53.519 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":119,"btime":"2024-09-13T08:46:48:452862+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:53.519 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 119 2024-09-13T08:46:53.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 msgr2=0x7f209c07b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 0x7f209c07b910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f20a8033000 tx=0x7f20a8004730 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 msgr2=0x7f20bc14e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f20b000c550 tx=0x7f20b000ca10 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 shutdown_connections 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f209c079450 0x7f209c07b910 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20bc152f10 0x7f20bc153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f20bc074500 0x7f20bc0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 --2- 172.21.15.31:0/2129660007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20bc072850 0x7f20bc14e5d0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 >> 172.21.15.31:0/2129660007 conn(0x7f20bc0cc910 msgr2=0x7f20bc0ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 shutdown_connections 2024-09-13T08:46:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.519+0000 7f20c4000640 1 -- 172.21.15.31:0/2129660007 wait complete. 2024-09-13T08:46:53.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 -- 172.21.15.31:0/1022932034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15bd70 msgr2=0x7efc0c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 --2- 172.21.15.31:0/1022932034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15bd70 0x7efc0c0cf850 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7efc0c0c6f10 tx=0x7efbfc02a530 comp rx=0 tx=0).stop 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 -- 172.21.15.31:0/1022932034 shutdown_connections 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 --2- 172.21.15.31:0/1022932034 >> v2:172.21.15.45:3300/0 conn(0x7efc0c15f9e0 0x7efc0c15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 --2- 172.21.15.31:0/1022932034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15bd70 0x7efc0c0cf850 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.694+0000 7efc10a00640 1 -- 172.21.15.31:0/1022932034 >> 172.21.15.31:0/1022932034 conn(0x7efc0c0cc910 msgr2=0x7efc0c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.695+0000 7efc10a00640 1 -- 172.21.15.31:0/1022932034 shutdown_connections 2024-09-13T08:46:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.695+0000 7efc10a00640 1 -- 172.21.15.31:0/1022932034 wait complete. 2024-09-13T08:46:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.696+0000 7efc10a00640 1 Processor -- start 2024-09-13T08:46:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.696+0000 7efc10a00640 1 -- start start 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc0c0ce3c0 0x7efc0c152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc0c15bd70 0x7efc0c0b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c0cb9d0 con 0x7efc0c15f9e0 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c0cb850 con 0x7efc0c0ce3c0 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc10a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7efc0c0cc360 con 0x7efc0c15bd70 2024-09-13T08:46:53.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.697+0000 7efc0a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc09e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc0c0ce3c0 0x7efc0c152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35378/0 (socket says 172.21.15.31:35378) 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 -- 172.21.15.31:0/1241540754 learned_addr learned my addr 172.21.15.31:0/1241540754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc01400640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc0c15bd70 0x7efc0c0b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 -- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc0c15bd70 msgr2=0x7efc0c0b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc0c15bd70 0x7efc0c0b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 -- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc0c0ce3c0 msgr2=0x7efc0c152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc0c0ce3c0 0x7efc0c152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 -- 172.21.15.31:0/1241540754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0c06fea0 con 0x7efc0c15f9e0 2024-09-13T08:46:53.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc0a800640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7efc0400cbc0 tx=0x7efc0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:53.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.698+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04016020 con 0x7efc0c15f9e0 2024-09-13T08:46:53.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.699+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc0400ae30 con 0x7efc0c15f9e0 2024-09-13T08:46:53.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.699+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04012660 con 0x7efc0c15f9e0 2024-09-13T08:46:53.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.699+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0c070180 con 0x7efc0c15f9e0 2024-09-13T08:46:53.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.699+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0c15e340 con 0x7efc0c15f9e0 2024-09-13T08:46:53.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.700+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc0c0cf850 con 0x7efc0c15f9e0 2024-09-13T08:46:53.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.701+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7efc04019020 con 0x7efc0c15f9e0 2024-09-13T08:46:53.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.702+0000 7efc02a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 0x7efbe807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:53.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.702+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7efc040997d0 con 0x7efc0c15f9e0 2024-09-13T08:46:53.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.702+0000 7efc09e00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 0x7efbe807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:53.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.703+0000 7efc09e00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 0x7efbe807b8e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efbfc034000 tx=0x7efbfc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:53.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.706+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7efc04020030 con 0x7efc0c15f9e0 2024-09-13T08:46:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.940+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc0c074200 con 0x7efc0c15f9e0 2024-09-13T08:46:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.941+0000 7efc02a00640 1 -- 172.21.15.31:0/1241540754 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5230 (secure 0 0 0) 0x7efc04062fe0 con 0x7efc0c15f9e0 2024-09-13T08:46:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:53.946 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":119,"btime":"2024-09-13T08:46:48:452862+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 119 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 msgr2=0x7efbe807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 0x7efbe807b8e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efbfc034000 tx=0x7efbfc004730 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 msgr2=0x7efc0c0b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7efc0400cbc0 tx=0x7efc0400a430 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 shutdown_connections 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7efbe8079420 0x7efbe807b8e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc0c15f9e0 0x7efc0c0b0580 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7efc0c15bd70 0x7efc0c0b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 --2- 172.21.15.31:0/1241540754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc0c0ce3c0 0x7efc0c152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 >> 172.21.15.31:0/1241540754 conn(0x7efc0c0cc910 msgr2=0x7efc0c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 shutdown_connections 2024-09-13T08:46:53.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:53.943+0000 7efc10a00640 1 -- 172.21.15.31:0/1241540754 wait complete. 2024-09-13T08:46:53.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.122+0000 7effa2000640 1 -- 172.21.15.31:0/2606282512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003a00 msgr2=0x7eff80003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.122+0000 7effa2000640 1 --2- 172.21.15.31:0/2606282512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80003a00 0x7eff80003e20 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009b50 tx=0x7eff80005830 comp rx=0 tx=0).stop 2024-09-13T08:46:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.123+0000 7eff9be00640 1 -- 172.21.15.31:0/2606282512 reap_dead start 2024-09-13T08:46:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.123+0000 7effa2000640 1 -- 172.21.15.31:0/2606282512 shutdown_connections 2024-09-13T08:46:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.123+0000 7effa2000640 1 -- 172.21.15.31:0/2606282512 >> 172.21.15.31:0/2606282512 conn(0x7eff9c0cc910 msgr2=0x7eff9c0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:54.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.123+0000 7effa2000640 1 -- 172.21.15.31:0/2606282512 shutdown_connections 2024-09-13T08:46:54.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.123+0000 7effa2000640 1 -- 172.21.15.31:0/2606282512 wait complete. 2024-09-13T08:46:54.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.124+0000 7effa2000640 1 Processor -- start 2024-09-13T08:46:54.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.124+0000 7effa2000640 1 -- start start 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 0x7eff9c1f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 0x7eff9c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c0b2020 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c0b21a0 con 0x7eff80003a00 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.125+0000 7effa2000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c0b2320 con 0x7eff9c09c0f0 2024-09-13T08:46:54.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35406/0 (socket says 172.21.15.31:35406) 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 -- 172.21.15.31:0/2408092384 learned_addr learned my addr 172.21.15.31:0/2408092384 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9b400640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 0x7eff9c1f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 -- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 msgr2=0x7eff9c06e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9be00640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 0x7eff9c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 0x7eff9c06e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 -- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 msgr2=0x7eff9c1f8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 0x7eff9c1f8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 -- 172.21.15.31:0/2408092384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c1f5060 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9b400640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 0x7eff9c1f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9be00640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 0x7eff9c06e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.126+0000 7eff9aa00640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7eff88011a50 tx=0x7eff88011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.127+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8800fd90 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.127+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff88012450 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.127+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88012840 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.127+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c1f5340 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.127+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c1f5900 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.128+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c073fc0 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.129+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7eff880129e0 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.130+0000 7eff93400640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 0x7eff7c07b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.130+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7eff8809a130 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.130+0000 7eff9b400640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 0x7eff7c07b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.131+0000 7eff9b400640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 0x7eff7c07b8e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7eff8c035000 tx=0x7eff8c02d040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.134+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7eff880641f0 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.330+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff9c0ae0f0 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.331+0000 7eff93400640 1 -- 172.21.15.31:0/2408092384 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) ==== 74+0+16791 (secure 0 0 0) 0x7eff88063af0 con 0x7eff9c1f4ae0 2024-09-13T08:46:54.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:54.333 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":178,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:53.482244+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:35.636566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:36.809469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"27.3","mappings":[{"from":7,"to":6}]},{"pgid":"27.27","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:54.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 msgr2=0x7eff7c07b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 0x7eff7c07b8e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7eff8c035000 tx=0x7eff8c02d040 comp rx=0 tx=0).stop 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 msgr2=0x7eff9c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7eff88011a50 tx=0x7eff88011f10 comp rx=0 tx=0).stop 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 shutdown_connections 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7eff7c079420 0x7eff7c07b8e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7eff9c09c0f0 0x7eff9c06e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9c1f4ae0 0x7eff9c15b070 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 --2- 172.21.15.31:0/2408092384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff80003a00 0x7eff9c1f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.333+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 >> 172.21.15.31:0/2408092384 conn(0x7eff9c0cc910 msgr2=0x7eff9c0af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:54.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.334+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 shutdown_connections 2024-09-13T08:46:54.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.334+0000 7effa2000640 1 -- 172.21.15.31:0/2408092384 wait complete. 2024-09-13T08:46:54.351 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:46:54.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 -- 172.21.15.31:0/4062366630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 msgr2=0x7fd3c815fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 --2- 172.21.15.31:0/4062366630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c815fe20 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005fd0 tx=0x7fd3b002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 -- 172.21.15.31:0/4062366630 shutdown_connections 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 --2- 172.21.15.31:0/4062366630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c815fe20 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 --2- 172.21.15.31:0/4062366630 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c815bd70 0x7fd3c80cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 -- 172.21.15.31:0/4062366630 >> 172.21.15.31:0/4062366630 conn(0x7fd3c80cc910 msgr2=0x7fd3c80cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 -- 172.21.15.31:0/4062366630 shutdown_connections 2024-09-13T08:46:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.515+0000 7fd3cf600640 1 -- 172.21.15.31:0/4062366630 wait complete. 2024-09-13T08:46:54.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.517+0000 7fd3cf600640 1 Processor -- start 2024-09-13T08:46:54.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.517+0000 7fd3cf600640 1 -- start start 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c80ce3c0 0x7fd3c81f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3c815bd70 0x7fd3c80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c80cb9d0 con 0x7fd3c815f9e0 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c80cb850 con 0x7fd3c815bd70 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cf600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c80cc360 con 0x7fd3c80ce3c0 2024-09-13T08:46:54.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35420/0 (socket says 172.21.15.31:35420) 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 -- 172.21.15.31:0/2246835006 learned_addr learned my addr 172.21.15.31:0/2246835006 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3bfe00640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3c815bd70 0x7fd3c80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3ccc00640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c80ce3c0 0x7fd3c81f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 -- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c80ce3c0 msgr2=0x7fd3c81f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c80ce3c0 0x7fd3c81f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 -- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3c815bd70 msgr2=0x7fd3c80b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3c815bd70 0x7fd3c80b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.518+0000 7fd3cd600640 1 -- 172.21.15.31:0/2246835006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c80b59c0 con 0x7fd3c815f9e0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3cd600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000c7d0 tx=0x7fd3c000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0003b70 con 0x7fd3c815f9e0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3c0003d10 con 0x7fd3c815f9e0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000aab0 con 0x7fd3c815f9e0 2024-09-13T08:46:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c80c8ac0 con 0x7fd3c815f9e0 2024-09-13T08:46:54.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.519+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c80c8770 con 0x7fd3c815f9e0 2024-09-13T08:46:54.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.520+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c80cf850 con 0x7fd3c815f9e0 2024-09-13T08:46:54.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.521+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fd3c000ac50 con 0x7fd3c815f9e0 2024-09-13T08:46:54.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.522+0000 7fd3c5600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 0x7fd3a407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.523+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7fd3c0099230 con 0x7fd3c815f9e0 2024-09-13T08:46:54.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.523+0000 7fd3ccc00640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 0x7fd3a407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.523+0000 7fd3ccc00640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 0x7fd3a407b8e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd3c809d850 tx=0x7fd3b800b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.527+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd3c0062a40 con 0x7fd3c815f9e0 2024-09-13T08:46:54.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.763+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3c80ccd90 con 0x7fd3c815f9e0 2024-09-13T08:46:54.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.763+0000 7fd3c5600640 1 -- 172.21.15.31:0/2246835006 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5230 (secure 0 0 0) 0x7fd3c0067470 con 0x7fd3c815f9e0 2024-09-13T08:46:54.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:54.766 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":119,"btime":"2024-09-13T08:46:48:452862+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:54.766 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 119 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 msgr2=0x7fd3a407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 0x7fd3a407b8e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd3c809d850 tx=0x7fd3b800b390 comp rx=0 tx=0).stop 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 msgr2=0x7fd3c80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000c7d0 tx=0x7fd3c000cc90 comp rx=0 tx=0).stop 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 shutdown_connections 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd3a4079420 0x7fd3a407b8e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3c815f9e0 0x7fd3c80b4f30 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3c815bd70 0x7fd3c80b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 --2- 172.21.15.31:0/2246835006 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd3c80ce3c0 0x7fd3c81f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 >> 172.21.15.31:0/2246835006 conn(0x7fd3c80cc910 msgr2=0x7fd3c80ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:54.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 shutdown_connections 2024-09-13T08:46:54.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.766+0000 7fd3cf600640 1 -- 172.21.15.31:0/2246835006 wait complete. 2024-09-13T08:46:54.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:46:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 -- 172.21.15.31:0/4280152220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100415bd70 msgr2=0x7f10040cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 --2- 172.21.15.31:0/4280152220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100415bd70 0x7f10040cf850 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f10040c6f10 tx=0x7f0ff802a540 comp rx=0 tx=0).stop 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 -- 172.21.15.31:0/4280152220 shutdown_connections 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 --2- 172.21.15.31:0/4280152220 >> v2:172.21.15.45:3300/0 conn(0x7f100415f9e0 0x7f100415fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 --2- 172.21.15.31:0/4280152220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f100415bd70 0x7f10040cf850 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 -- 172.21.15.31:0/4280152220 >> 172.21.15.31:0/4280152220 conn(0x7f10040cc910 msgr2=0x7f10040cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 -- 172.21.15.31:0/4280152220 shutdown_connections 2024-09-13T08:46:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.943+0000 7f100bc00640 1 -- 172.21.15.31:0/4280152220 wait complete. 2024-09-13T08:46:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.945+0000 7f100bc00640 1 Processor -- start 2024-09-13T08:46:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.945+0000 7f100bc00640 1 -- start start 2024-09-13T08:46:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 0x7f10040c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f100415f9e0 0x7f100406f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f10040cb9d0 con 0x7f10040ce3c0 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10040cb850 con 0x7f100415f9e0 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f100bc00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f10040cc360 con 0x7f100415bd70 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50020/0 (socket says 172.21.15.31:50020) 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 -- 172.21.15.31:0/2608044707 learned_addr learned my addr 172.21.15.31:0/2608044707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1003e00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 0x7f10040c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009c00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f100415f9e0 0x7f100406f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 -- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 msgr2=0x7f10040c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 0x7f10040c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 -- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f100415f9e0 msgr2=0x7f100406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f100415f9e0 0x7f100406f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1009200640 1 -- 172.21.15.31:0/2608044707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10041f8b40 con 0x7f10040ce3c0 2024-09-13T08:46:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.946+0000 7f1003e00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 0x7f10040c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f1009200640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009920 tx=0x7f0ff8032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8040070 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ff8005310 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100406e600 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80047a0 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.947+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100406ebc0 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.948+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10040cf850 con 0x7f10040ce3c0 2024-09-13T08:46:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.949+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f0ff8039040 con 0x7f10040ce3c0 2024-09-13T08:46:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.950+0000 7f1000c00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 0x7f0fe407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.950+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7f0ff80ba820 con 0x7f10040ce3c0 2024-09-13T08:46:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.951+0000 7f1003e00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 0x7f0fe407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.951+0000 7f1003e00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 0x7f0fe407b8e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f10040c6a40 tx=0x7f0ff0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:54.954+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f10041f8b40 con 0x7f10040ce3c0 2024-09-13T08:46:55.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.190+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10040cd070 con 0x7f10040ce3c0 2024-09-13T08:46:55.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.191+0000 7f1000c00640 1 -- 172.21.15.31:0/2608044707 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5230 (secure 0 0 0) 0x7f0ff8084600 con 0x7f10040ce3c0 2024-09-13T08:46:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:55.193 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":119,"btime":"2024-09-13T08:46:48:452862+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T08:46:37.846092+0000","modified":"2024-09-13T08:46:40.863828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9198":{"gid":9198,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":3,"addr":"172.21.15.31:6835/2620315837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2620315837},{"type":"v1","addr":"172.21.15.31:6835","nonce":2620315837}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9198,"qdb_cluster":[9198]},"id":13}]} 2024-09-13T08:46:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 119 2024-09-13T08:46:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.193+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 msgr2=0x7f0fe407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 0x7f0fe407b8e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f10040c6a40 tx=0x7f0ff0005f90 comp rx=0 tx=0).stop 2024-09-13T08:46:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 msgr2=0x7f10041f8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009920 tx=0x7f0ff8032c10 comp rx=0 tx=0).stop 2024-09-13T08:46:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 shutdown_connections 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fe4079420 0x7f0fe407b8e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f100415f9e0 0x7f100406f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f100415bd70 0x7f10040c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 --2- 172.21.15.31:0/2608044707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10040ce3c0 0x7f10041f8600 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 >> 172.21.15.31:0/2608044707 conn(0x7f10040cc910 msgr2=0x7f10040af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 shutdown_connections 2024-09-13T08:46:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.194+0000 7f100bc00640 1 -- 172.21.15.31:0/2608044707 wait complete. 2024-09-13T08:46:55.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 -- 172.21.15.31:0/3247746849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc15e440 msgr2=0x7f0fcc0b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/3247746849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc15e440 0x7f0fcc0b1b30 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0cab90 tx=0x7f0fbc02a4a0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 -- 172.21.15.31:0/3247746849 shutdown_connections 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/3247746849 >> v2:172.21.15.45:3300/0 conn(0x7f0fcc0b1600 0x7f0fcc0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/3247746849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc15e440 0x7f0fcc0b1b30 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 -- 172.21.15.31:0/3247746849 >> 172.21.15.31:0/3247746849 conn(0x7f0fcc074260 msgr2=0x7f0fcc074670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:55.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 -- 172.21.15.31:0/3247746849 shutdown_connections 2024-09-13T08:46:55.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.370+0000 7f0fd2a00640 1 -- 172.21.15.31:0/3247746849 wait complete. 2024-09-13T08:46:55.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.371+0000 7f0fd2a00640 1 Processor -- start 2024-09-13T08:46:55.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.372+0000 7f0fd2a00640 1 -- start start 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.372+0000 7f0fd2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 0x7f0fcc1f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd2a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 0x7f0fcc1f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd2a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc0ae4c0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd2a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc0ae340 con 0x7f0fcc15e440 2024-09-13T08:46:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd2a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f0fcc0ae640 con 0x7f0fcc15e820 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50032/0 (socket says 172.21.15.31:50032) 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 -- 172.21.15.31:0/385011741 learned_addr learned my addr 172.21.15.31:0/385011741 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd0a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 0x7f0fcc1f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 -- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 msgr2=0x7f0fcc1f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fc3e00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 0x7f0fcc1f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 0x7f0fcc1f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 -- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 msgr2=0x7f0fcc1f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 0x7f0fcc1f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fcbe00640 1 -- 172.21.15.31:0/385011741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc1f8ad0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fd0a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 0x7f0fcc1f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.373+0000 7f0fc3e00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 0x7f0fcc1f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fcbe00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009780 tx=0x7f0fbc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc040070 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fbc005310 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0047a0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fcc0c6110 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.374+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fcc0c66d0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.375+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fcc0ade00 con 0x7f0fcc0b1600 2024-09-13T08:46:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.376+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f0fbc039040 con 0x7f0fcc0b1600 2024-09-13T08:46:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.377+0000 7f0fc8c00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 0x7f0fa807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.377+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7f0fbc0ba8f0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.378+0000 7f0fc3e00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 0x7f0fa807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.378+0000 7f0fc3e00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 0x7f0fa807b8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc1e3cb0 tx=0x7f0fb4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:55.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.382+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0fbc0846f0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.579+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fcc0cb3f0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.580+0000 7f0fc8c00640 1 -- 172.21.15.31:0/385011741 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) ==== 74+0+16791 (secure 0 0 0) 0x7f0fbc083ff0 con 0x7f0fcc0b1600 2024-09-13T08:46:55.583 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:46:55.583 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":178,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:53.482244+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T08:46:35.636566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T08:46:36.809469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"27.3","mappings":[{"from":7,"to":6}]},{"pgid":"27.27","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 msgr2=0x7f0fa807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 0x7f0fa807b8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc1e3cb0 tx=0x7f0fb4009040 comp rx=0 tx=0).stop 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 msgr2=0x7f0fcc0d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009780 tx=0x7f0fbc032c10 comp rx=0 tx=0).stop 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 shutdown_connections 2024-09-13T08:46:55.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f0fa8079420 0x7f0fa807b8e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f0fcc15e820 0x7f0fcc1f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fcc15e440 0x7f0fcc1f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 --2- 172.21.15.31:0/385011741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc0b1600 0x7f0fcc0d0880 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 >> 172.21.15.31:0/385011741 conn(0x7f0fcc074260 msgr2=0x7f0fcc0b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 shutdown_connections 2024-09-13T08:46:55.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.583+0000 7f0fd2a00640 1 -- 172.21.15.31:0/385011741 wait complete. 2024-09-13T08:46:55.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 -- 172.21.15.31:0/665639925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd544073860 msgr2=0x7fd544073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 --2- 172.21.15.31:0/665639925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd544073860 0x7fd544073cf0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005fd0 tx=0x7fd53c02aaa0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 -- 172.21.15.31:0/665639925 shutdown_connections 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 --2- 172.21.15.31:0/665639925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd544073860 0x7fd544073cf0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 --2- 172.21.15.31:0/665639925 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 0x7fd5440ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 -- 172.21.15.31:0/665639925 >> 172.21.15.31:0/665639925 conn(0x7fd5440b18f0 msgr2=0x7fd5440b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 -- 172.21.15.31:0/665639925 shutdown_connections 2024-09-13T08:46:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.762+0000 7fd548c00640 1 -- 172.21.15.31:0/665639925 wait complete. 2024-09-13T08:46:55.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.764+0000 7fd548c00640 1 Processor -- start 2024-09-13T08:46:55.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.764+0000 7fd548c00640 1 -- start start 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd544073860 0x7fd5441f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 0x7fd5441f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd544074230 con 0x7fd54415be30 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5440ce9e0 con 0x7fd544073860 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd548c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd5440743b0 con 0x7fd5440ced20 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd542800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd542800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50052/0 (socket says 172.21.15.31:50052) 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.765+0000 7fd542800640 1 -- 172.21.15.31:0/199831736 learned_addr learned my addr 172.21.15.31:0/199831736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd539400640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 0x7fd5441f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd541e00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd544073860 0x7fd5441f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 -- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 msgr2=0x7fd5441f5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 0x7fd5441f5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 -- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd544073860 msgr2=0x7fd5441f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd544073860 0x7fd5441f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 -- 172.21.15.31:0/199831736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5441f6510 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd542800640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fd53c006010 tx=0x7fd53c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.766+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c046070 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.767+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd53c02e800 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.767+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0044b0 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.767+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5441f67f0 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.767+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441f3520 con 0x7fd54415be30 2024-09-13T08:46:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.768+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5440748a0 con 0x7fd54415be30 2024-09-13T08:46:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.769+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fd53c049070 con 0x7fd54415be30 2024-09-13T08:46:55.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.770+0000 7fd53aa00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 0x7fd52007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:55.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.770+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5811+0+0 (secure 0 0 0) 0x7fd53c0ba0d0 con 0x7fd54415be30 2024-09-13T08:46:55.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.770+0000 7fd541e00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 0x7fd52007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:55.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.771+0000 7fd541e00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 0x7fd52007b8e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd5441f5b90 tx=0x7fd534009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:55.774+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fd53c032030 con 0x7fd54415be30 2024-09-13T08:46:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.011+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd544069a50 con 0x7fd54415be30 2024-09-13T08:46:56.013 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-13T08:46:56.011+0000 7f8289e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T08:46:56.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.014+0000 7fd53aa00640 1 -- 172.21.15.31:0/199831736 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) ==== 161+0+0 (secure 0 0 0) 0x7fd53c083b40 con 0x7fd54415be30 2024-09-13T08:46:56.016 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T08:46:56.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.017+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 msgr2=0x7fd52007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:56.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.017+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 0x7fd52007b8e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd5441f5b90 tx=0x7fd534009040 comp rx=0 tx=0).stop 2024-09-13T08:46:56.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.017+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 msgr2=0x7fd544154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:56.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.017+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fd53c006010 tx=0x7fd53c003f10 comp rx=0 tx=0).stop 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 shutdown_connections 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd520079420 0x7fd52007b8e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54415be30 0x7fd544154520 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd5440ced20 0x7fd5441f5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 --2- 172.21.15.31:0/199831736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd544073860 0x7fd5441f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 >> 172.21.15.31:0/199831736 conn(0x7fd5440b18f0 msgr2=0x7fd5441582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 shutdown_connections 2024-09-13T08:46:56.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.018+0000 7fd548c00640 1 -- 172.21.15.31:0/199831736 wait complete. 2024-09-13T08:46:56.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T08:46:56.088 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-13T08:46:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.204+0000 7f4fd9600640 1 -- 172.21.15.31:0/1134815013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 msgr2=0x7f4fb80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.204+0000 7f4fd9600640 1 --2- 172.21.15.31:0/1134815013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fb80051c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f4fd406e280 tx=0x7f4fb8005830 comp rx=0 tx=0).stop 2024-09-13T08:46:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.204+0000 7f4fd3400640 1 -- 172.21.15.31:0/1134815013 reap_dead start 2024-09-13T08:46:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.205+0000 7f4fd9600640 1 -- 172.21.15.31:0/1134815013 shutdown_connections 2024-09-13T08:46:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.205+0000 7f4fd9600640 1 -- 172.21.15.31:0/1134815013 >> 172.21.15.31:0/1134815013 conn(0x7f4fd40cfb70 msgr2=0x7f4fd40cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.205+0000 7f4fd9600640 1 -- 172.21.15.31:0/1134815013 shutdown_connections 2024-09-13T08:46:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.205+0000 7f4fd9600640 1 -- 172.21.15.31:0/1134815013 wait complete. 2024-09-13T08:46:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.206+0000 7f4fd9600640 1 Processor -- start 2024-09-13T08:46:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.206+0000 7f4fd9600640 1 -- start start 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 0x7f4fd41509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 0x7f4fd40b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd40729a0 con 0x7f4fb8004d40 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd4072820 con 0x7f4fd40af5d0 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.207+0000 7f4fd9600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd406f690 con 0x7f4fd41f2e60 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50080/0 (socket says 172.21.15.31:50080) 2024-09-13T08:46:56.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 -- 172.21.15.31:0/3809680023 learned_addr learned my addr 172.21.15.31:0/3809680023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:56.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 -- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 msgr2=0x7f4fd41509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:56.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd3400640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 0x7f4fd40b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:56.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fca000640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 0x7f4fd41509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:56.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 0x7f4fd41509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 -- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 msgr2=0x7f4fd40b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 0x7f4fd40b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 -- 172.21.15.31:0/3809680023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd40c3b50 con 0x7f4fb8004d40 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fca000640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 0x7f4fd41509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd3400640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 0x7f4fd40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fd2a00640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40066e0 tx=0x7f4fc40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:56.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.208+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4011700 con 0x7f4fb8004d40 2024-09-13T08:46:56.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.209+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fc4016020 con 0x7f4fb8004d40 2024-09-13T08:46:56.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.209+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4011b60 con 0x7f4fb8004d40 2024-09-13T08:46:56.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.209+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd40c3e30 con 0x7f4fb8004d40 2024-09-13T08:46:56.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.209+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd40c43f0 con 0x7f4fb8004d40 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.210+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd40c45f0 con 0x7f4fb8004d40 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.211+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f4fc4011d00 con 0x7f4fb8004d40 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.212+0000 7f4fcb400640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 0x7f4fb007b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.212+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 5897+0+0 (secure 0 0 0) 0x7f4fc4099e10 con 0x7f4fb8004d40 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.212+0000 7f4fca000640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 0x7f4fb007b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:56.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.213+0000 7f4fca000640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 0x7f4fb007b910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:56.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.216+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f4fd40c45f0 con 0x7f4fb8004d40 2024-09-13T08:46:56.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:56.452+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4fd406fbf0 con 0x7f4fb8004d40 2024-09-13T08:46:57.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.016+0000 7f4fcb400640 1 -- 172.21.15.31:0/3809680023 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v121) ==== 104+0+0 (secure 0 0 0) 0x7f4fc4020020 con 0x7f4fb8004d40 2024-09-13T08:46:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 msgr2=0x7f4fb007b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 0x7f4fb007b910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc006b10 comp rx=0 tx=0).stop 2024-09-13T08:46:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 msgr2=0x7f4fd41511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40066e0 tx=0x7f4fc40039c0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 shutdown_connections 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f4fb0079450 0x7f4fb007b910 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4fd40af5d0 0x7f4fd40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f4fd41f2e60 0x7f4fd41509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 --2- 172.21.15.31:0/3809680023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fb8004d40 0x7f4fd41511c0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 >> 172.21.15.31:0/3809680023 conn(0x7f4fd40cfb70 msgr2=0x7f4fd40cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.019+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 shutdown_connections 2024-09-13T08:46:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.020+0000 7f4fd9600640 1 -- 172.21.15.31:0/3809680023 wait complete. 2024-09-13T08:46:57.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 -- 172.21.15.31:0/3851760009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fec0 msgr2=0x7fc5740cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 --2- 172.21.15.31:0/3851760009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fec0 0x7fc5740cbc80 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fc5740694f0 tx=0x7fc56c02aa30 comp rx=0 tx=0).stop 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 -- 172.21.15.31:0/3851760009 shutdown_connections 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 --2- 172.21.15.31:0/3851760009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fec0 0x7fc5740cbc80 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 --2- 172.21.15.31:0/3851760009 >> v2:172.21.15.45:3300/0 conn(0x7fc57406fa60 0x7fc5740cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 --2- 172.21.15.31:0/3851760009 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc57415bfc0 0x7fc574072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 -- 172.21.15.31:0/3851760009 >> 172.21.15.31:0/3851760009 conn(0x7fc5740cfbb0 msgr2=0x7fc5740cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:57.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.206+0000 7fc57b400640 1 -- 172.21.15.31:0/3851760009 shutdown_connections 2024-09-13T08:46:57.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.207+0000 7fc57b400640 1 -- 172.21.15.31:0/3851760009 wait complete. 2024-09-13T08:46:57.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.208+0000 7fc57b400640 1 Processor -- start 2024-09-13T08:46:57.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.208+0000 7fc57b400640 1 -- start start 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57415bfc0 0x7fc57414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5740cf3e0 0x7fc574157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5740b2b60 con 0x7fc57406fa60 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5740b29e0 con 0x7fc57415bfc0 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc57b400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc5740b2ce0 con 0x7fc5740cf3e0 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc578a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc578a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50104/0 (socket says 172.21.15.31:50104) 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.209+0000 7fc578a00640 1 -- 172.21.15.31:0/1231681747 learned_addr learned my addr 172.21.15.31:0/1231681747 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc579400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5740cf3e0 0x7fc574157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:57.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc573e00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57415bfc0 0x7fc57414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 -- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5740cf3e0 msgr2=0x7fc574157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5740cf3e0 0x7fc574157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 -- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57415bfc0 msgr2=0x7fc57414d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57415bfc0 0x7fc57414d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 -- 172.21.15.31:0/1231681747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57414d980 con 0x7fc57406fa60 2024-09-13T08:46:57.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc578a00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fc56800d740 tx=0x7fc56800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568017070 con 0x7fc57406fa60 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc568009d20 con 0x7fc57406fa60 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5740c6e90 con 0x7fc57406fa60 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.210+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5680106c0 con 0x7fc57406fa60 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.211+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5740c7450 con 0x7fc57406fa60 2024-09-13T08:46:57.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.211+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5740b31c0 con 0x7fc57406fa60 2024-09-13T08:46:57.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.213+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fc56801a070 con 0x7fc57406fa60 2024-09-13T08:46:57.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.214+0000 7fc570c00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 0x7fc55407b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:57.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.214+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 5897+0+0 (secure 0 0 0) 0x7fc5680995a0 con 0x7fc57406fa60 2024-09-13T08:46:57.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.214+0000 7fc573e00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 0x7fc55407b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:57.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.215+0000 7fc573e00640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 0x7fc55407b910 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc5741e94f0 tx=0x7fc560005e50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:57.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.218+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc568062db0 con 0x7fc57406fa60 2024-09-13T08:46:57.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:57.414+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc5740cb310 con 0x7fc57406fa60 2024-09-13T08:46:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.021+0000 7fc570c00640 1 -- 172.21.15.31:0/1231681747 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v180) ==== 181+0+0 (secure 0 0 0) 0x7fc5680677e0 con 0x7fc57406fa60 2024-09-13T08:46:58.024 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-09-13T08:46:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 msgr2=0x7fc55407b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 0x7fc55407b910 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc5741e94f0 tx=0x7fc560005e50 comp rx=0 tx=0).stop 2024-09-13T08:46:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 msgr2=0x7fc5740c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fc56800d740 tx=0x7fc56800dc00 comp rx=0 tx=0).stop 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 shutdown_connections 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc554079450 0x7fc55407b910 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc5740cf3e0 0x7fc574157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc57415bfc0 0x7fc57414d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 --2- 172.21.15.31:0/1231681747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc57406fa60 0x7fc5740c7c50 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 >> 172.21.15.31:0/1231681747 conn(0x7fc5740cfbb0 msgr2=0x7fc574151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 shutdown_connections 2024-09-13T08:46:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.026+0000 7fc57b400640 1 -- 172.21.15.31:0/1231681747 wait complete. 2024-09-13T08:46:58.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T08:46:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 -- 172.21.15.31:0/2041860102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 msgr2=0x7f9e580ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 --2- 172.21.15.31:0/2041860102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580ce660 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009830 tx=0x7f9e4002a510 comp rx=0 tx=0).stop 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 -- 172.21.15.31:0/2041860102 shutdown_connections 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 --2- 172.21.15.31:0/2041860102 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 0x7f9e5806f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 --2- 172.21.15.31:0/2041860102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580ce660 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 -- 172.21.15.31:0/2041860102 >> 172.21.15.31:0/2041860102 conn(0x7f9e580cfbb0 msgr2=0x7f9e580cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 -- 172.21.15.31:0/2041860102 shutdown_connections 2024-09-13T08:46:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.214+0000 7f9e5d200640 1 -- 172.21.15.31:0/2041860102 wait complete. 2024-09-13T08:46:58.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.216+0000 7f9e5d200640 1 Processor -- start 2024-09-13T08:46:58.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.216+0000 7f9e5d200640 1 -- start start 2024-09-13T08:46:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e58070650 0x7f9e5815bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 0x7f9e5815c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5806ff40 con 0x7f9e5815d5e0 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5806fdc0 con 0x7f9e58070650 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e5d200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9e580700c0 con 0x7f9e580cd780 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e56800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e56800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50122/0 (socket says 172.21.15.31:50122) 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e56800640 1 -- 172.21.15.31:0/4284872841 learned_addr learned my addr 172.21.15.31:0/4284872841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e4d400640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 0x7f9e5815c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.217+0000 7f9e55e00640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e58070650 0x7f9e5815bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 -- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 msgr2=0x7f9e5815c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 0x7f9e5815c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 -- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e58070650 msgr2=0x7f9e5815bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e58070650 0x7f9e5815bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 -- 172.21.15.31:0/4284872841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5815c6b0 con 0x7f9e5815d5e0 2024-09-13T08:46:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e56800640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000dda0 tx=0x7f9e5000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50004d20 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e5000be40 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e580b39a0 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.218+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000e8a0 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.219+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5815b390 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.220+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e580b3b70 con 0x7f9e5815d5e0 2024-09-13T08:46:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.221+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f9e5000ea40 con 0x7f9e5815d5e0 2024-09-13T08:46:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.222+0000 7f9e4ea00640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 0x7f9e3407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.222+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 5459+0+0 (secure 0 0 0) 0x7f9e50099de0 con 0x7f9e5815d5e0 2024-09-13T08:46:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.222+0000 7f9e55e00640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 0x7f9e3407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.222+0000 7f9e55e00640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 0x7f9e3407b8e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007fd0 tx=0x7f9e48009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:58.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.226+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f9e5815c6b0 con 0x7f9e5815d5e0 2024-09-13T08:46:58.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:58.423+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9e5806a180 con 0x7f9e5815d5e0 2024-09-13T08:46:59.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.023+0000 7f9e4ea00640 1 -- 172.21.15.31:0/4284872841 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v181) ==== 169+0+0 (secure 0 0 0) 0x7f9e5806a180 con 0x7f9e5815d5e0 2024-09-13T08:46:59.025 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.027+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 msgr2=0x7f9e3407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.027+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 0x7f9e3407b8e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007fd0 tx=0x7f9e48009040 comp rx=0 tx=0).stop 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.027+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 msgr2=0x7f9e580b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.027+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000dda0 tx=0x7f9e5000b730 comp rx=0 tx=0).stop 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 shutdown_connections 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e34079420 0x7f9e3407b8e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e5815d5e0 0x7f9e580b3e80 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e580cd780 0x7f9e5815c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 --2- 172.21.15.31:0/4284872841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e58070650 0x7f9e5815bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.028+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 >> 172.21.15.31:0/4284872841 conn(0x7f9e580cfbb0 msgr2=0x7f9e58158610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:59.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.029+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 shutdown_connections 2024-09-13T08:46:59.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.029+0000 7f9e5d200640 1 -- 172.21.15.31:0/4284872841 wait complete. 2024-09-13T08:46:59.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-13T08:46:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 -- 172.21.15.31:0/850660751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 msgr2=0x7ff0200cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 --2- 172.21.15.31:0/850660751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200cdb80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff014009830 tx=0x7ff01402a510 comp rx=0 tx=0).stop 2024-09-13T08:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 -- 172.21.15.31:0/850660751 shutdown_connections 2024-09-13T08:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 --2- 172.21.15.31:0/850660751 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff020073860 0x7ff020073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 --2- 172.21.15.31:0/850660751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200cdb80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 -- 172.21.15.31:0/850660751 >> 172.21.15.31:0/850660751 conn(0x7ff0200b18f0 msgr2=0x7ff0200b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:59.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 -- 172.21.15.31:0/850660751 shutdown_connections 2024-09-13T08:46:59.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.215+0000 7ff027a00640 1 -- 172.21.15.31:0/850660751 wait complete. 2024-09-13T08:46:59.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.217+0000 7ff027a00640 1 Processor -- start 2024-09-13T08:46:59.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.217+0000 7ff027a00640 1 -- start start 2024-09-13T08:46:59.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 0x7ff020156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0200733e0 con 0x7ff020073860 2024-09-13T08:46:59.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0200ce9e0 con 0x7ff02015be30 2024-09-13T08:46:59.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff027a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff020074230 con 0x7ff0200ced20 2024-09-13T08:46:59.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff025000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff025000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50138/0 (socket says 172.21.15.31:50138) 2024-09-13T08:46:59.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff025000640 1 -- 172.21.15.31:0/557378143 learned_addr learned my addr 172.21.15.31:0/557378143 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:59.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff01fe00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 0x7ff020156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.218+0000 7ff025a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 -- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 msgr2=0x7ff020156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 0x7ff020156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 -- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 msgr2=0x7ff0200c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 -- 172.21.15.31:0/557378143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0201f51c0 con 0x7ff020073860 2024-09-13T08:46:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff01fe00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 0x7ff020156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:46:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:46:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff025000640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff014009950 tx=0x7ff014033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014031d60 con 0x7ff020073860 2024-09-13T08:46:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff014033e10 con 0x7ff020073860 2024-09-13T08:46:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140332b0 con 0x7ff020073860 2024-09-13T08:46:59.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.219+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0201f54a0 con 0x7ff020073860 2024-09-13T08:46:59.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.220+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0201f5a60 con 0x7ff020073860 2024-09-13T08:46:59.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.221+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff020074720 con 0x7ff020073860 2024-09-13T08:46:59.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.222+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7ff014033450 con 0x7ff020073860 2024-09-13T08:46:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.223+0000 7ff01cc00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 0x7ff00007b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.223+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7ff0140ba530 con 0x7ff020073860 2024-09-13T08:46:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.223+0000 7ff01fe00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 0x7ff00007b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.224+0000 7ff01fe00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 0x7ff00007b910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff0201f50f0 tx=0x7ff00c007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.227+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff01404a030 con 0x7ff020073860 2024-09-13T08:46:59.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.422+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7ff0200b25f0 con 0x7ff020073860 2024-09-13T08:46:59.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.426+0000 7ff01cc00640 1 -- 172.21.15.31:0/557378143 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v49) ==== 71+0+0 (secure 0 0 0) 0x7ff0140be020 con 0x7ff020073860 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.428+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 msgr2=0x7ff00007b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.428+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 0x7ff00007b910 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff0201f50f0 tx=0x7ff00c007e90 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.428+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 msgr2=0x7ff0201f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.428+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff014009950 tx=0x7ff014033c20 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 shutdown_connections 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff000079450 0x7ff00007b910 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02015be30 0x7ff0200c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff0200ced20 0x7ff020156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 --2- 172.21.15.31:0/557378143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020073860 0x7ff0201f4bb0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 >> 172.21.15.31:0/557378143 conn(0x7ff0200b18f0 msgr2=0x7ff0200afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:59.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 shutdown_connections 2024-09-13T08:46:59.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.429+0000 7ff027a00640 1 -- 172.21.15.31:0/557378143 wait complete. 2024-09-13T08:46:59.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.610+0000 7f13c5e00640 1 -- 172.21.15.31:0/1981426227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4003c10 msgr2=0x7f13a4004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.610+0000 7f13c5e00640 1 --2- 172.21.15.31:0/1981426227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4003c10 0x7f13a4004010 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c3c0 tx=0x7f13a4005830 comp rx=0 tx=0).stop 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.610+0000 7f13bf400640 1 -- 172.21.15.31:0/1981426227 reap_dead start 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.610+0000 7f13c5e00640 1 -- 172.21.15.31:0/1981426227 shutdown_connections 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.611+0000 7f13c5e00640 1 -- 172.21.15.31:0/1981426227 >> 172.21.15.31:0/1981426227 conn(0x7f13c00cc910 msgr2=0x7f13c00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.611+0000 7f13c5e00640 1 -- 172.21.15.31:0/1981426227 shutdown_connections 2024-09-13T08:46:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.611+0000 7f13c5e00640 1 -- 172.21.15.31:0/1981426227 wait complete. 2024-09-13T08:46:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.612+0000 7f13c5e00640 1 Processor -- start 2024-09-13T08:46:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.612+0000 7f13c5e00640 1 -- start start 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13a4003c10 0x7f13c01508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 0x7f13c015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00cc6c0 con 0x7f13c015c3a0 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00cc540 con 0x7f13c0151220 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.613+0000 7f13c5e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f13c0073780 con 0x7f13a4003c10 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50156/0 (socket says 172.21.15.31:50156) 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 -- 172.21.15.31:0/840112953 learned_addr learned my addr 172.21.15.31:0/840112953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bea00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13a4003c10 0x7f13c01508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13b6000640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 0x7f13c015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 -- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13a4003c10 msgr2=0x7f13c01508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13a4003c10 0x7f13c01508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 -- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 msgr2=0x7f13c015a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 0x7f13c015a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 -- 172.21.15.31:0/840112953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c015b050 con 0x7f13c015c3a0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13b6000640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 0x7f13c015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13bf400640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af880 tx=0x7f13b8010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8003970 con 0x7f13c015c3a0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13b8015070 con 0x7f13c015c3a0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.614+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c015b330 con 0x7f13c015c3a0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.615+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8004030 con 0x7f13c015c3a0 2024-09-13T08:46:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.615+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c015b8f0 con 0x7f13c015c3a0 2024-09-13T08:46:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.616+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c00b1710 con 0x7f13c015c3a0 2024-09-13T08:46:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.617+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f13b80041d0 con 0x7f13c015c3a0 2024-09-13T08:46:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.618+0000 7f13b7e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 0x7f139c07b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:46:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.618+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f13b8099e10 con 0x7f13c015c3a0 2024-09-13T08:46:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.618+0000 7f13bea00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 0x7f139c07b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:46:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.619+0000 7f13bea00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 0x7f139c07b910 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c3c0 tx=0x7f13b002d070 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:46:59.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.622+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f13c00b1710 con 0x7f13c015c3a0 2024-09-13T08:46:59.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:46:59.889+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f13c0073ce0 con 0x7f13c015c3a0 2024-09-13T08:47:00.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.025+0000 7f13b7e00640 1 -- 172.21.15.31:0/840112953 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v546) ==== 163+0+0 (secure 0 0 0) 0x7f13b8063ed0 con 0x7f13c015c3a0 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 msgr2=0x7f139c07b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 0x7f139c07b910 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c3c0 tx=0x7f13b002d070 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 msgr2=0x7f13c015c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af880 tx=0x7f13b8010f40 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 shutdown_connections 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f139c079450 0x7f139c07b910 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13c015c3a0 0x7f13c015c850 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13c0151220 0x7f13c015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 --2- 172.21.15.31:0/840112953 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13a4003c10 0x7f13c01508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.028+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 >> 172.21.15.31:0/840112953 conn(0x7f13c00cc910 msgr2=0x7f13c01581a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.029+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 shutdown_connections 2024-09-13T08:47:00.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:00.029+0000 7f13c5e00640 1 -- 172.21.15.31:0/840112953 wait complete. 2024-09-13T08:47:00.047 INFO:tasks.cephfs_test_runner:Validate unparseable text .meta file on legacy subvol root doesn't break the system ... ok 2024-09-13T08:47:00.048 INFO:tasks.cephfs_test_runner: 2024-09-13T08:47:00.048 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-13T08:47:00.048 INFO:tasks.cephfs_test_runner:Ran 11 tests in 559.380s 2024-09-13T08:47:00.048 INFO:tasks.cephfs_test_runner: 2024-09-13T08:47:00.048 INFO:tasks.cephfs_test_runner:OK 2024-09-13T08:47:00.048 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-13T08:47:00.059 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-13T08:47:00.095 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-13T08:47:00.096 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:47:00.096 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:47:00.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:47:00.135 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-13T08:47:00.135 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:47:00.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T08:47:00.211 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:47:00.211 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:47:00.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:47:00.254 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-13T08:47:00.254 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T08:47:00.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T08:47:00.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:00.333 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:47:00.369 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T08:47:00.369 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T08:47:00.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:00.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T08:47:00.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:00.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T08:47:00.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:00.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-13T08:47:01.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.040 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-13T08:47:01.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-13T08:47:01.177 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-09-13T08:47:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:01.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.180 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:47:01.220 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T08:47:01.220 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T08:47:01.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T08:47:01.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T08:47:01.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-13T08:47:01.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.881 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-13T08:47:01.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T08:47:01.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-13T08:47:02.017 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-09-13T08:47:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:02.019 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T08:47:02.046 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T08:47:02.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:47:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 -- 172.21.15.31:0/133858829 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80ceab0 msgr2=0x7f2ba80cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 --2- 172.21.15.31:0/133858829 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80ceab0 0x7f2ba80cef10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0005fd0 tx=0x7f2ba002a820 comp rx=0 tx=0).stop 2024-09-13T08:47:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 -- 172.21.15.31:0/133858829 shutdown_connections 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 --2- 172.21.15.31:0/133858829 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80ceab0 0x7f2ba80cef10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 --2- 172.21.15.31:0/133858829 >> v2:172.21.15.45:3300/0 conn(0x7f2ba815e800 0x7f2ba815ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 -- 172.21.15.31:0/133858829 >> 172.21.15.31:0/133858829 conn(0x7f2ba806cb00 msgr2=0x7f2ba806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 -- 172.21.15.31:0/133858829 shutdown_connections 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.211+0000 7f2bade00640 1 -- 172.21.15.31:0/133858829 wait complete. 2024-09-13T08:47:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.212+0000 7f2bade00640 1 Processor -- start 2024-09-13T08:47:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.213+0000 7f2bade00640 1 -- start start 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.213+0000 7f2bade00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80cb9e0 0x7f2ba80c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bade00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 0x7f2ba8148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bade00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bade00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8073070 con 0x7f2ba815e800 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bade00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8072ef0 con 0x7f2ba80ceab0 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bade00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba80731f0 con 0x7f2ba80cb9e0 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50164/0 (socket says 172.21.15.31:50164) 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 -- 172.21.15.31:0/3945288778 learned_addr learned my addr 172.21.15.31:0/3945288778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2baca00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80cb9e0 0x7f2ba80c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2ba7e00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 0x7f2ba8148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 -- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80cb9e0 msgr2=0x7f2ba80c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80cb9e0 0x7f2ba80c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 -- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 msgr2=0x7f2ba8148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 0x7f2ba8148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2bad400640 1 -- 172.21.15.31:0/3945288778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba80c8890 con 0x7f2ba815e800 2024-09-13T08:47:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.214+0000 7f2ba7e00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 0x7f2ba8148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2bad400640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0035040 tx=0x7f2ba0032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0037040 con 0x7f2ba815e800 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2bade00640 1 -- 172.21.15.31:0/3945288778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba80c8b70 con 0x7f2ba815e800 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ba0042070 con 0x7f2ba815e800 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0003e60 con 0x7f2ba815e800 2024-09-13T08:47:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.215+0000 7f2bade00640 1 -- 172.21.15.31:0/3945288778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba80c9130 con 0x7f2ba815e800 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.216+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7c005290 con 0x7f2ba815e800 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.217+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f2ba0004240 con 0x7f2ba815e800 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.218+0000 7f2ba4c00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 0x7f2b8807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.218+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f2ba00ba210 con 0x7f2ba815e800 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.219+0000 7f2baca00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 0x7f2b8807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.219+0000 7f2baca00640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 0x7f2b8807b8e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0013f0 tx=0x7f2b9c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.222+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2ba0083ff0 con 0x7f2ba815e800 2024-09-13T08:47:02.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.413+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b7c000bc0 con 0x7f2b88079420 2024-09-13T08:47:02.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.414+0000 7f2ba4c00640 1 -- 172.21.15.31:0/3945288778 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13513 (secure 0 0 0) 0x7f2b7c000bc0 con 0x7f2b88079420 2024-09-13T08:47:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:02.417 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:47:02.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 msgr2=0x7f2b8807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 0x7f2b8807b8e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0013f0 tx=0x7f2b9c009040 comp rx=0 tx=0).stop 2024-09-13T08:47:02.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 msgr2=0x7f2ba81f3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0035040 tx=0x7f2ba0032c60 comp rx=0 tx=0).stop 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 shutdown_connections 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2b88079420 0x7f2b8807b8e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba815e800 0x7f2ba81f3c80 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ba80ceab0 0x7f2ba8148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 --2- 172.21.15.31:0/3945288778 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2ba80cb9e0 0x7f2ba80c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 >> 172.21.15.31:0/3945288778 conn(0x7f2ba806cb00 msgr2=0x7f2ba806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 shutdown_connections 2024-09-13T08:47:02.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.416+0000 7f2b9a000640 1 -- 172.21.15.31:0/3945288778 wait complete. 2024-09-13T08:47:02.434 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-13T08:47:01.021924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289160,"kb_used_data":18844,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754685560,"statfs":{"total":773094113280,"available":772798013440,"internally_reserved":0,"allocated":19296256,"data_stored":9257767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-51301,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-68,"num_write_kb":-74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001456"},"pg_stats":[{"pgid":"1.0","version":"78'79","reported_seq":425,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T08:46:56.018844+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:56.018844+0000","last_peered":"2024-09-13T08:46:56.018844+0000","last_clean":"2024-09-13T08:46:56.018844+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:56.018844+0000","last_undegraded":"2024-09-13T08:46:56.018844+0000","last_fullsized":"2024-09-13T08:46:56.018844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509563,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2252800,"data_stored":999576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509563,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37540,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334300,"statfs":{"total":96636764160,"available":96598323200,"internally_reserved":0,"allocated":2240512,"data_stored":993441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509563,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38128,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333712,"statfs":{"total":96636764160,"available":96597721088,"internally_reserved":0,"allocated":2846720,"data_stored":1584467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542266,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41708,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330132,"statfs":{"total":96636764160,"available":96594055168,"internally_reserved":0,"allocated":2252800,"data_stored":999644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509563,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2301952,"data_stored":1048027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509563,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2310144,"data_stored":1048131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509564,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33968,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337872,"statfs":{"total":96636764160,"available":96601980928,"internally_reserved":0,"allocated":2846720,"data_stored":1590225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509562,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2244608,"data_stored":994256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:47:02.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.594+0000 7f0355200640 1 -- 172.21.15.31:0/650455004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 msgr2=0x7f03500ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.594+0000 7f0355200640 1 --2- 172.21.15.31:0/650455004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f03500ce3c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f03500cfe60 tx=0x7f033802a5a0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.594+0000 7f0355200640 1 -- 172.21.15.31:0/650455004 shutdown_connections 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.595+0000 7f0355200640 1 --2- 172.21.15.31:0/650455004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f03500ce3c0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.595+0000 7f0355200640 1 -- 172.21.15.31:0/650455004 >> 172.21.15.31:0/650455004 conn(0x7f03500cc910 msgr2=0x7f03500cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.595+0000 7f0355200640 1 -- 172.21.15.31:0/650455004 shutdown_connections 2024-09-13T08:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.595+0000 7f0355200640 1 -- 172.21.15.31:0/650455004 wait complete. 2024-09-13T08:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.596+0000 7f0355200640 1 Processor -- start 2024-09-13T08:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.596+0000 7f0355200640 1 -- start start 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03500ce900 0x7f03500b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f035015bd70 0x7f03500b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03501600f0 con 0x7f03500cf850 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f03500cb850 con 0x7f03500ce900 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.597+0000 7f0355200640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f03500b3770 con 0x7f035015bd70 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50188/0 (socket says 172.21.15.31:50188) 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 -- 172.21.15.31:0/3611126629 learned_addr learned my addr 172.21.15.31:0/3611126629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f034de00640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03500ce900 0x7f03500b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f034e800640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f035015bd70 0x7f03500b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 -- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f035015bd70 msgr2=0x7f03500b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f035015bd70 0x7f03500b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 -- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03500ce900 msgr2=0x7f03500b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03500ce900 0x7f03500b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 -- 172.21.15.31:0/3611126629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03500b3a50 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.598+0000 7f0345400640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f0338009780 tx=0x7f0338032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.599+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338040070 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.599+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0338005310 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.599+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338003800 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.599+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03500b3d30 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.599+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03500b42f0 con 0x7f03500cf850 2024-09-13T08:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.600+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03500cc3b0 con 0x7f03500cf850 2024-09-13T08:47:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.601+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f0338039040 con 0x7f03500cf850 2024-09-13T08:47:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.602+0000 7f0346a00640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 0x7f032c07b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.602+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f03380ba660 con 0x7f03500cf850 2024-09-13T08:47:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.602+0000 7f034de00640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 0x7f032c07b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.603+0000 7f034de00640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 0x7f032c07b8e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0340001720 tx=0x7f0340009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.606+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f0338083e70 con 0x7f03500cf850 2024-09-13T08:47:02.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.798+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0350072aa0 con 0x7f032c079420 2024-09-13T08:47:02.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.798+0000 7f0346a00640 1 -- 172.21.15.31:0/3611126629 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13513 (secure 0 0 0) 0x7f0350072aa0 con 0x7f032c079420 2024-09-13T08:47:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:02.801 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 msgr2=0x7f032c07b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 0x7f032c07b8e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0340001720 tx=0x7f0340009040 comp rx=0 tx=0).stop 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 msgr2=0x7f035014d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f0338009780 tx=0x7f0338032d50 comp rx=0 tx=0).stop 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 shutdown_connections 2024-09-13T08:47:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f032c079420 0x7f032c07b8e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f035015bd70 0x7f03500b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03500cf850 0x7f035014d390 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 --2- 172.21.15.31:0/3611126629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03500ce900 0x7f03500b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 >> 172.21.15.31:0/3611126629 conn(0x7f03500cc910 msgr2=0x7f03500c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 shutdown_connections 2024-09-13T08:47:02.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.801+0000 7f0355200640 1 -- 172.21.15.31:0/3611126629 wait complete. 2024-09-13T08:47:02.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-13T08:47:01.021924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289160,"kb_used_data":18844,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754685560,"statfs":{"total":773094113280,"available":772798013440,"internally_reserved":0,"allocated":19296256,"data_stored":9257767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-51301,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-68,"num_write_kb":-74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001456"},"pg_stats":[{"pgid":"1.0","version":"78'79","reported_seq":425,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T08:46:56.018844+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:46:56.018844+0000","last_peered":"2024-09-13T08:46:56.018844+0000","last_clean":"2024-09-13T08:46:56.018844+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:46:56.018844+0000","last_undegraded":"2024-09-13T08:46:56.018844+0000","last_fullsized":"2024-09-13T08:46:56.018844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509563,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2252800,"data_stored":999576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509563,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37540,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334300,"statfs":{"total":96636764160,"available":96598323200,"internally_reserved":0,"allocated":2240512,"data_stored":993441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509563,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38128,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333712,"statfs":{"total":96636764160,"available":96597721088,"internally_reserved":0,"allocated":2846720,"data_stored":1584467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542266,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41708,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330132,"statfs":{"total":96636764160,"available":96594055168,"internally_reserved":0,"allocated":2252800,"data_stored":999644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509563,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2301952,"data_stored":1048027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509563,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2310144,"data_stored":1048131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509564,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33968,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337872,"statfs":{"total":96636764160,"available":96601980928,"internally_reserved":0,"allocated":2846720,"data_stored":1590225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509562,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2244608,"data_stored":994256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:47:02.819 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T08:47:02.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:47:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.978+0000 7ff8d4800640 1 -- 172.21.15.31:0/530848163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15bd70 msgr2=0x7ff8cc0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.978+0000 7ff8d4800640 1 --2- 172.21.15.31:0/530848163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15bd70 0x7ff8cc0cf850 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0cc8c0 tx=0x7ff8c002a560 comp rx=0 tx=0).stop 2024-09-13T08:47:02.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 -- 172.21.15.31:0/530848163 shutdown_connections 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 --2- 172.21.15.31:0/530848163 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15f9e0 0x7ff8cc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 --2- 172.21.15.31:0/530848163 >> v2:172.21.15.45:3300/0 conn(0x7ff8cc0ce3c0 0x7ff8cc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 --2- 172.21.15.31:0/530848163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15bd70 0x7ff8cc0cf850 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 -- 172.21.15.31:0/530848163 >> 172.21.15.31:0/530848163 conn(0x7ff8cc0cc910 msgr2=0x7ff8cc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 -- 172.21.15.31:0/530848163 shutdown_connections 2024-09-13T08:47:02.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.979+0000 7ff8d4800640 1 -- 172.21.15.31:0/530848163 wait complete. 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.980+0000 7ff8d4800640 1 Processor -- start 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.980+0000 7ff8d4800640 1 -- start start 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0ce3c0 0x7ff8cc1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 0x7ff8cc09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc0cb9d0 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc0cb850 con 0x7ff8cc0ce3c0 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.981+0000 7ff8d4800640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff8cc0cbc50 con 0x7ff8cc15bd70 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50204/0 (socket says 172.21.15.31:50204) 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 -- 172.21.15.31:0/3925801082 learned_addr learned my addr 172.21.15.31:0/3925801082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d1400640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 0x7ff8cc09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d1e00640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0ce3c0 0x7ff8cc1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 -- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 msgr2=0x7ff8cc09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 0x7ff8cc09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 -- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0ce3c0 msgr2=0x7ff8cc1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0ce3c0 0x7ff8cc1f45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 -- 172.21.15.31:0/3925801082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc0c6a20 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d1400640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 0x7ff8cc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:47:02.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.982+0000 7ff8d2800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800cce0 tx=0x7ff8c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.983+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c800f450 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.983+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8c800f700 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.983+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8cc1f4cd0 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.983+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c800fb50 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.983+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8cc1f5290 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.984+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8cc0cf850 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.985+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7ff8c800fcf0 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.986+0000 7ff8c6000640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 0x7ff8b007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.986+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7ff8c8099e50 con 0x7ff8cc15f9e0 2024-09-13T08:47:02.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.986+0000 7ff8d1e00640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 0x7ff8b007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:02.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.987+0000 7ff8d1e00640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 0x7ff8b007b8e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0033000 tx=0x7ff8c0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:02.990+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff8c8014030 con 0x7ff8cc15f9e0 2024-09-13T08:47:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.188+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff8cc15ffd0 con 0x7ff8b0079420 2024-09-13T08:47:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.188+0000 7ff8c6000640 1 -- 172.21.15.31:0/3925801082 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13488 (secure 0 0 0) 0x7ff8cc15ffd0 con 0x7ff8b0079420 2024-09-13T08:47:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 msgr2=0x7ff8b007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 0x7ff8b007b8e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0033000 tx=0x7ff8c0004730 comp rx=0 tx=0).stop 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 msgr2=0x7ff8cc1ee1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800cce0 tx=0x7ff8c800a620 comp rx=0 tx=0).stop 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 shutdown_connections 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff8b0079420 0x7ff8b007b8e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8cc15f9e0 0x7ff8cc1ee1a0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff8cc15bd70 0x7ff8cc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 --2- 172.21.15.31:0/3925801082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0ce3c0 0x7ff8cc1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 >> 172.21.15.31:0/3925801082 conn(0x7ff8cc0cc910 msgr2=0x7ff8cc0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 shutdown_connections 2024-09-13T08:47:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.191+0000 7ff8d4800640 1 -- 172.21.15.31:0/3925801082 wait complete. 2024-09-13T08:47:03.209 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-13T08:47:03.022254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":289304,"kb_used_data":18984,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754685416,"statfs":{"total":773094113280,"available":772797865984,"internally_reserved":0,"allocated":19439616,"data_stored":9271152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-51301,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-68,"num_write_kb":-74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.005489"},"pg_stats":[{"pgid":"1.0","version":"78'79","reported_seq":429,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-13T08:47:00.104888+0000","last_change":"2024-09-13T08:37:01.427882+0000","last_active":"2024-09-13T08:47:00.104888+0000","last_peered":"2024-09-13T08:47:00.104888+0000","last_clean":"2024-09-13T08:47:00.104888+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:47:00.104888+0000","last_undegraded":"2024-09-13T08:47:00.104888+0000","last_fullsized":"2024-09-13T08:47:00.104888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T08:36:59.880455+0000","last_clean_scrub_stamp":"2024-09-13T08:36:59.880455+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:00:04.973526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37580,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334260,"statfs":{"total":96636764160,"available":96598282240,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37576,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334264,"statfs":{"total":96636764160,"available":96598286336,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38156,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333684,"statfs":{"total":96636764160,"available":96597692416,"internally_reserved":0,"allocated":2875392,"data_stored":1601670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542267,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41740,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330100,"statfs":{"total":96636764160,"available":96594022400,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509564,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33420,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338420,"statfs":{"total":96636764160,"available":96602542080,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509565,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34000,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337840,"statfs":{"total":96636764160,"available":96601948160,"internally_reserved":0,"allocated":2875392,"data_stored":1601670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509563,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:47:03.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T08:47:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 -- 172.21.15.31:0/3505716099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e80051c0 msgr2=0x7f34e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 --2- 172.21.15.31:0/3505716099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e80051c0 0x7f34e800b9c0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f34fc005fd0 tx=0x7f34e8005830 comp rx=0 tx=0).stop 2024-09-13T08:47:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 -- 172.21.15.31:0/3505716099 shutdown_connections 2024-09-13T08:47:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 --2- 172.21.15.31:0/3505716099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e80051c0 0x7f34e800b9c0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 -- 172.21.15.31:0/3505716099 >> 172.21.15.31:0/3505716099 conn(0x7f350406cb00 msgr2=0x7f350406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 -- 172.21.15.31:0/3505716099 shutdown_connections 2024-09-13T08:47:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.369+0000 7f3508c00640 1 -- 172.21.15.31:0/3505716099 wait complete. 2024-09-13T08:47:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.371+0000 7f3508c00640 1 Processor -- start 2024-09-13T08:47:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.371+0000 7f3508c00640 1 -- start start 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80051c0 0x7f3504151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35040b2970 0x7f35040b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35040ca7d0 con 0x7f35040b0df0 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f35040ca650 con 0x7f34e80051c0 2024-09-13T08:47:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3508c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f35040ca950 con 0x7f35040b2970 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f34faa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f34faa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50226/0 (socket says 172.21.15.31:50226) 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f34faa00640 1 -- 172.21.15.31:0/949072114 learned_addr learned my addr 172.21.15.31:0/949072114 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3503400640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80051c0 0x7f3504151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.372+0000 7f3503e00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35040b2970 0x7f35040b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 -- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35040b2970 msgr2=0x7f35040b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35040b2970 0x7f35040b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 -- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80051c0 msgr2=0x7f3504151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80051c0 0x7f3504151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 -- 172.21.15.31:0/949072114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3504159470 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34faa00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00e570 tx=0x7f34ec00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec015070 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34ec010980 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec010d70 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3504159750 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.373+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3504159d10 con 0x7f35040b0df0 2024-09-13T08:47:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.375+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350415e2d0 con 0x7f35040b0df0 2024-09-13T08:47:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.376+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f34ec018070 con 0x7f35040b0df0 2024-09-13T08:47:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.377+0000 7f34fbe00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 0x7f34e007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.377+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f34ec099840 con 0x7f35040b0df0 2024-09-13T08:47:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.377+0000 7f3503400640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 0x7f34e007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.377+0000 7f3503400640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 0x7f34e007b8e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000c00 tx=0x7f34f4006b70 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.381+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f34ec063010 con 0x7f35040b0df0 2024-09-13T08:47:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.574+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3504153cb0 con 0x7f35040b0df0 2024-09-13T08:47:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.575+0000 7f34fbe00640 1 -- 172.21.15.31:0/949072114 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) ==== 74+0+13061 (secure 0 0 0) 0x7f34ec067a40 con 0x7f35040b0df0 2024-09-13T08:47:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:03.577 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":181,"fsid":"0251b593-a02a-47fa-b938-c065b7284822","created":"2024-09-13T08:36:43.729442+0000","modified":"2024-09-13T08:46:59.023219+0000","last_up_change":"2024-09-13T08:36:57.874656+0000","last_in_change":"2024-09-13T08:36:47.976057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T08:36:59.061847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7903a6a-0c16-44bc-ad3f-35e6d004dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6809","nonce":1940612994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6811","nonce":1940612994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6815","nonce":1940612994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1940612994},{"type":"v1","addr":"172.21.15.31:6813","nonce":1940612994}]},"public_addr":"172.21.15.31:6809/1940612994","cluster_addr":"172.21.15.31:6811/1940612994","heartbeat_back_addr":"172.21.15.31:6815/1940612994","heartbeat_front_addr":"172.21.15.31:6813/1940612994","state":["exists","up"]},{"osd":1,"uuid":"15a51edd-43e9-4b8c-9d97-70734e704624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3396863731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6827","nonce":3396863731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6831","nonce":3396863731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3396863731},{"type":"v1","addr":"172.21.15.31:6829","nonce":3396863731}]},"public_addr":"172.21.15.31:6825/3396863731","cluster_addr":"172.21.15.31:6827/3396863731","heartbeat_back_addr":"172.21.15.31:6831/3396863731","heartbeat_front_addr":"172.21.15.31:6829/3396863731","state":["exists","up"]},{"osd":2,"uuid":"1e2d33eb-4d88-4f66-9a77-a7fd6ea618fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6817","nonce":657376189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6819","nonce":657376189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6823","nonce":657376189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":657376189},{"type":"v1","addr":"172.21.15.31:6821","nonce":657376189}]},"public_addr":"172.21.15.31:6817/657376189","cluster_addr":"172.21.15.31:6819/657376189","heartbeat_back_addr":"172.21.15.31:6823/657376189","heartbeat_front_addr":"172.21.15.31:6821/657376189","state":["exists","up"]},{"osd":3,"uuid":"022dffd3-c8d6-406e-a985-d47b5c8e8a91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6801","nonce":2845481020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6803","nonce":2845481020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6807","nonce":2845481020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2845481020},{"type":"v1","addr":"172.21.15.31:6805","nonce":2845481020}]},"public_addr":"172.21.15.31:6801/2845481020","cluster_addr":"172.21.15.31:6803/2845481020","heartbeat_back_addr":"172.21.15.31:6807/2845481020","heartbeat_front_addr":"172.21.15.31:6805/2845481020","state":["exists","up"]},{"osd":4,"uuid":"dc59624a-a6b1-451d-b54c-a2b12eaed3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6803","nonce":1087405525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6805","nonce":1087405525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6809","nonce":1087405525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1087405525},{"type":"v1","addr":"172.21.15.45:6807","nonce":1087405525}]},"public_addr":"172.21.15.45:6803/1087405525","cluster_addr":"172.21.15.45:6805/1087405525","heartbeat_back_addr":"172.21.15.45:6809/1087405525","heartbeat_front_addr":"172.21.15.45:6807/1087405525","state":["exists","up"]},{"osd":5,"uuid":"a4268a77-31dd-4270-958a-14dc24be7bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6819","nonce":2092128205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6821","nonce":2092128205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6825","nonce":2092128205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2092128205},{"type":"v1","addr":"172.21.15.45:6823","nonce":2092128205}]},"public_addr":"172.21.15.45:6819/2092128205","cluster_addr":"172.21.15.45:6821/2092128205","heartbeat_back_addr":"172.21.15.45:6825/2092128205","heartbeat_front_addr":"172.21.15.45:6823/2092128205","state":["exists","up"]},{"osd":6,"uuid":"eb828891-dcac-4b30-9dfe-139f49927ca2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6811","nonce":636367962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6813","nonce":636367962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6817","nonce":636367962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":636367962},{"type":"v1","addr":"172.21.15.45:6815","nonce":636367962}]},"public_addr":"172.21.15.45:6811/636367962","cluster_addr":"172.21.15.45:6813/636367962","heartbeat_back_addr":"172.21.15.45:6817/636367962","heartbeat_front_addr":"172.21.15.45:6815/636367962","state":["exists","up"]},{"osd":7,"uuid":"ee3ea50e-f92c-4079-bbbc-68ed25d6cb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6827","nonce":1264531367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6829","nonce":1264531367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6833","nonce":1264531367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1264531367},{"type":"v1","addr":"172.21.15.45:6831","nonce":1264531367}]},"public_addr":"172.21.15.45:6827/1264531367","cluster_addr":"172.21.15.45:6829/1264531367","heartbeat_back_addr":"172.21.15.45:6833/1264531367","heartbeat_front_addr":"172.21.15.45:6831/1264531367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.053806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:55.037293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.503230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.789255+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.345359+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.827984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.508158+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T08:36:54.890214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6835/2620315837":"2024-09-14T08:46:56.013242+0000","172.21.15.31:6833/2620315837":"2024-09-14T08:46:56.013242+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 msgr2=0x7f34e007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 0x7f34e007b8e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000c00 tx=0x7f34f4006b70 comp rx=0 tx=0).stop 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 msgr2=0x7f35040b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00e570 tx=0x7f34ec00ea30 comp rx=0 tx=0).stop 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 shutdown_connections 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f34e0079420 0x7f34e007b8e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f35040b2970 0x7f35040b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35040b0df0 0x7f35040b5320 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 --2- 172.21.15.31:0/949072114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e80051c0 0x7f3504151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 >> 172.21.15.31:0/949072114 conn(0x7f350406cb00 msgr2=0x7f3504152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 shutdown_connections 2024-09-13T08:47:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:03.578+0000 7f3508c00640 1 -- 172.21.15.31:0/949072114 wait complete. 2024-09-13T08:47:04.597 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T08:47:04.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 -- 172.21.15.31:0/829180623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe19415bfc0 msgr2=0x7fe194072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 --2- 172.21.15.31:0/829180623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe19415bfc0 0x7fe194072560 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe184009830 tx=0x7fe18402a5b0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 -- 172.21.15.31:0/829180623 shutdown_connections 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 --2- 172.21.15.31:0/829180623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe19415bfc0 0x7fe194072560 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 -- 172.21.15.31:0/829180623 >> 172.21.15.31:0/829180623 conn(0x7fe1940cfbb0 msgr2=0x7fe1940cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 -- 172.21.15.31:0/829180623 shutdown_connections 2024-09-13T08:47:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.770+0000 7fe19aa00640 1 -- 172.21.15.31:0/829180623 wait complete. 2024-09-13T08:47:04.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.772+0000 7fe19aa00640 1 Processor -- start 2024-09-13T08:47:04.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.772+0000 7fe19aa00640 1 -- start start 2024-09-13T08:47:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe19406fa60 0x7fe1940c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe19415bfc0 0x7fe1940c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 0x7fe1941f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940cbfd0 con 0x7fe1940b4510 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940cbe50 con 0x7fe19406fa60 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe19aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fe1940b30f0 con 0x7fe19415bfc0 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe198a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 0x7fe1941f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe193e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe19406fa60 0x7fe1940c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe198a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 0x7fe1941f4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50244/0 (socket says 172.21.15.31:50244) 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe198a00640 1 -- 172.21.15.31:0/46743780 learned_addr learned my addr 172.21.15.31:0/46743780 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.773+0000 7fe18b400640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe19415bfc0 0x7fe1940c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 -- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe19415bfc0 msgr2=0x7fe1940c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fe19415bfc0 0x7fe1940c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 -- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe19406fa60 msgr2=0x7fe1940c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe19406fa60 0x7fe1940c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1940c73f0 con 0x7fe1940b4510 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe193e00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe19406fa60 0x7fe1940c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe198a00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 0x7fe1941f4790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c7a0 tx=0x7fe18c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c003b70 con 0x7fe1940b4510 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.774+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe18c003d10 con 0x7fe1940b4510 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.775+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c00aa50 con 0x7fe1940b4510 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.775+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1940c8b00 con 0x7fe1940b4510 2024-09-13T08:47:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.775+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1940c90c0 con 0x7fe1940b4510 2024-09-13T08:47:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.776+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe194072560 con 0x7fe1940b4510 2024-09-13T08:47:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.777+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fe18c00abf0 con 0x7fe1940b4510 2024-09-13T08:47:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe190c00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe170079420 0x7fe17007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7fe18c099240 con 0x7fe1940b4510 2024-09-13T08:47:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe193e00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe170079420 0x7fe17007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe190c00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7fe17007f310 0x7fe170081730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe170081de0 con 0x7fe17007f310 2024-09-13T08:47:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7fe18c0a02a0 con 0x7fe1940b4510 2024-09-13T08:47:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.778+0000 7fe18b400640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7fe17007f310 0x7fe170081730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.779+0000 7fe193e00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe170079420 0x7fe17007b8e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe184034000 tx=0x7fe184004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.779+0000 7fe18b400640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7fe17007f310 0x7fe170081730 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.780+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== osd.0 v2:172.21.15.31:6808/1940612994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe170081de0 con 0x7fe17007f310 2024-09-13T08:47:04.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.793+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 --> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe19415b020 con 0x7fe17007f310 2024-09-13T08:47:04.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe190c00640 1 -- 172.21.15.31:0/46743780 <== osd.0 v2:172.21.15.31:6808/1940612994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe19415b020 con 0x7fe17007f310 2024-09-13T08:47:04.796 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:04.796 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:04.796 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7fe17007f310 msgr2=0x7fe170081730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6808/1940612994,v1:172.21.15.31:6809/1940612994] conn(0x7fe17007f310 0x7fe170081730 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe170079420 msgr2=0x7fe17007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fe170079420 0x7fe17007b8e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe184034000 tx=0x7fe184004730 comp rx=0 tx=0).stop 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 msgr2=0x7fe1941f4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 --2- 172.21.15.31:0/46743780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1940b4510 0x7fe1941f4790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c7a0 tx=0x7fe18c00cc60 comp rx=0 tx=0).stop 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe198a00640 1 -- 172.21.15.31:0/46743780 reap_dead start 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 shutdown_connections 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 >> 172.21.15.31:0/46743780 conn(0x7fe1940cfbb0 msgr2=0x7fe194153210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 shutdown_connections 2024-09-13T08:47:04.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.794+0000 7fe19aa00640 1 -- 172.21.15.31:0/46743780 wait complete. 2024-09-13T08:47:04.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 -- 172.21.15.31:0/663387252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff815be30 msgr2=0x7f6ff80cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 --2- 172.21.15.31:0/663387252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff815be30 0x7f6ff80cdb80 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8005fd0 tx=0x7f6fe802a530 comp rx=0 tx=0).stop 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 -- 172.21.15.31:0/663387252 shutdown_connections 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 --2- 172.21.15.31:0/663387252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff8073860 0x7f6ff8073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 --2- 172.21.15.31:0/663387252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff815be30 0x7f6ff80cdb80 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 -- 172.21.15.31:0/663387252 >> 172.21.15.31:0/663387252 conn(0x7f6ff80b18f0 msgr2=0x7f6ff80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 -- 172.21.15.31:0/663387252 shutdown_connections 2024-09-13T08:47:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.971+0000 7f6fff600640 1 -- 172.21.15.31:0/663387252 wait complete. 2024-09-13T08:47:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.973+0000 7f6fff600640 1 Processor -- start 2024-09-13T08:47:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.973+0000 7f6fff600640 1 -- start start 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 0x7f6ff814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff815be30 0x7f6ff80b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff80733e0 con 0x7f6ff80ced20 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff80ce9e0 con 0x7f6ff8073860 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fff600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff8074230 con 0x7f6ff815be30 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fefe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fefe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53688/0 (socket says 172.21.15.31:53688) 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6fefe00640 1 -- 172.21.15.31:0/3814633321 learned_addr learned my addr 172.21.15.31:0/3814633321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6ffcc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 0x7f6ff814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.974+0000 7f6ffd600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff815be30 0x7f6ff80b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 -- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff815be30 msgr2=0x7f6ff80b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff815be30 0x7f6ff80b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 -- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 msgr2=0x7f6ff80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 -- 172.21.15.31:0/3814633321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff81f5510 con 0x7f6ff8073860 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6fefe00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T08:47:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.975+0000 7f6ffcc00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 0x7f6ff814a840 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80b25a0 tx=0x7f6fe80039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.976+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe803e030 con 0x7f6ff8073860 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.976+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fe80047a0 con 0x7f6ff8073860 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.976+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8035040 con 0x7f6ff8073860 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.976+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff81f57f0 con 0x7f6ff8073860 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.976+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff81f5db0 con 0x7f6ff8073860 2024-09-13T08:47:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.977+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff81f5ff0 con 0x7f6ff8073860 2024-09-13T08:47:04.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.978+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f6fe8041030 con 0x7f6ff8073860 2024-09-13T08:47:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.979+0000 7f6ff5600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 0x7f6fd407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.979+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f6fe80b9bf0 con 0x7f6ff8073860 2024-09-13T08:47:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.979+0000 7f6fefe00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 0x7f6fd407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.980+0000 7f6fefe00640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 0x7f6fd407b8e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6ff81f61f0 tx=0x7f6fe0007ef0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:04.985+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f6ff81f5ff0 con 0x7f6ff8073860 2024-09-13T08:47:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.185+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f6ff8155150 con 0x7f6fd4079420 2024-09-13T08:47:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.185+0000 7f6ff5600640 1 -- 172.21.15.31:0/3814633321 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6ff8155150 con 0x7f6fd4079420 2024-09-13T08:47:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T08:47:05.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 msgr2=0x7f6fd407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 0x7f6fd407b8e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6ff81f61f0 tx=0x7f6fe0007ef0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 msgr2=0x7f6ff814a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 0x7f6ff814a840 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80b25a0 tx=0x7f6fe80039e0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 shutdown_connections 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fd4079420 0x7f6fd407b8e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6ff815be30 0x7f6ff80b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ff80ced20 0x7f6ff80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 --2- 172.21.15.31:0/3814633321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff8073860 0x7f6ff814a840 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 >> 172.21.15.31:0/3814633321 conn(0x7f6ff80b18f0 msgr2=0x7f6ff80ad400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 shutdown_connections 2024-09-13T08:47:05.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.188+0000 7f6fff600640 1 -- 172.21.15.31:0/3814633321 wait complete. 2024-09-13T08:47:05.207 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T08:47:05.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:05.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 -- 172.21.15.31:0/3930914230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c06fa60 msgr2=0x7f6a0c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 --2- 172.21.15.31:0/3930914230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c06fa60 0x7f6a0c0cb740 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c069dd0 tx=0x7f69f802a540 comp rx=0 tx=0).stop 2024-09-13T08:47:05.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 -- 172.21.15.31:0/3930914230 shutdown_connections 2024-09-13T08:47:05.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 --2- 172.21.15.31:0/3930914230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c06fa60 0x7f6a0c0cb740 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 --2- 172.21.15.31:0/3930914230 >> v2:172.21.15.45:3300/0 conn(0x7f6a0c15bfc0 0x7f6a0c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 -- 172.21.15.31:0/3930914230 >> 172.21.15.31:0/3930914230 conn(0x7f6a0c0cfbb0 msgr2=0x7f6a0c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 -- 172.21.15.31:0/3930914230 shutdown_connections 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.371+0000 7f6a14400640 1 -- 172.21.15.31:0/3930914230 wait complete. 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.372+0000 7f6a14400640 1 Processor -- start 2024-09-13T08:47:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.373+0000 7f6a14400640 1 -- start start 2024-09-13T08:47:05.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a0c06fa60 0x7f6a0c1f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 0x7f6a0c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a0c155af0 0x7f6a0c149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c0b3270 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c0b30f0 con 0x7f6a0c06fa60 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a14400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6a0c0b33f0 con 0x7f6a0c155af0 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 0x7f6a0c152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 0x7f6a0c152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53714/0 (socket says 172.21.15.31:53714) 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 -- 172.21.15.31:0/2128056860 learned_addr learned my addr 172.21.15.31:0/2128056860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11a00640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a0c06fa60 0x7f6a0c1f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a12400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a0c155af0 0x7f6a0c149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 -- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a0c155af0 msgr2=0x7f6a0c149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6a0c155af0 0x7f6a0c149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 -- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a0c06fa60 msgr2=0x7f6a0c1f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a0c06fa60 0x7f6a0c1f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.374+0000 7f6a11000640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c1563b0 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a11000640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 0x7f6a0c152d40 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009920 tx=0x7f69f8032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f8040070 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0c156690 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69f8032e70 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f80055b0 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.375+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0c1490f0 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.376+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a0c153280 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.378+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f69f8039040 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.378+0000 7f6a09600640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69ec079450 0x7f69ec07b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.378+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f69f80baad0 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a11a00640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69ec079450 0x7f69ec07b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a09600640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f69ec07f340 0x7f69ec081760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69f8001d30 con 0x7f69ec07f340 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f69f80bd7e0 con 0x7f6a0c15bfc0 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a12400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f69ec07f340 0x7f69ec081760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.379+0000 7f6a11a00640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69ec079450 0x7f69ec07b910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000bb50 tx=0x7f6a0000b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.380+0000 7f6a12400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f69ec07f340 0x7f69ec081760 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.380+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== osd.1 v2:172.21.15.31:6824/3396863731 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69f8001d30 con 0x7f69ec07f340 2024-09-13T08:47:05.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 --> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6a0c148ee0 con 0x7f69ec07f340 2024-09-13T08:47:05.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a09600640 1 -- 172.21.15.31:0/2128056860 <== osd.1 v2:172.21.15.31:6824/3396863731 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6a0c148ee0 con 0x7f69ec07f340 2024-09-13T08:47:05.396 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:05.396 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:05.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f69ec07f340 msgr2=0x7f69ec081760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6824/3396863731,v1:172.21.15.31:6825/3396863731] conn(0x7f69ec07f340 0x7f69ec081760 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69ec079450 msgr2=0x7f69ec07b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f69ec079450 0x7f69ec07b910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000bb50 tx=0x7f6a0000b2e0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 msgr2=0x7f6a0c152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a14400640 1 --2- 172.21.15.31:0/2128056860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a0c15bfc0 0x7f6a0c152d40 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009920 tx=0x7f69f8032c80 comp rx=0 tx=0).stop 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.394+0000 7f6a12400640 1 -- 172.21.15.31:0/2128056860 reap_dead start 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.395+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 shutdown_connections 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.395+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 >> 172.21.15.31:0/2128056860 conn(0x7f6a0c0cfbb0 msgr2=0x7f6a0c0b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.395+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 shutdown_connections 2024-09-13T08:47:05.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.395+0000 7f6a14400640 1 -- 172.21.15.31:0/2128056860 wait complete. 2024-09-13T08:47:05.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.571+0000 7fc307e00640 1 -- 172.21.15.31:0/4082751518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2e8003a00 msgr2=0x7fc2e8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.571+0000 7fc307e00640 1 --2- 172.21.15.31:0/4082751518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2e8003a00 0x7fc2e8003e20 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400c290 tx=0x7fc2e8005830 comp rx=0 tx=0).stop 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.571+0000 7fc305e00640 1 -- 172.21.15.31:0/4082751518 reap_dead start 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.572+0000 7fc307e00640 1 -- 172.21.15.31:0/4082751518 shutdown_connections 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.572+0000 7fc307e00640 1 -- 172.21.15.31:0/4082751518 >> 172.21.15.31:0/4082751518 conn(0x7fc3000b18f0 msgr2=0x7fc3000b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.572+0000 7fc307e00640 1 -- 172.21.15.31:0/4082751518 shutdown_connections 2024-09-13T08:47:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.572+0000 7fc307e00640 1 -- 172.21.15.31:0/4082751518 wait complete. 2024-09-13T08:47:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.573+0000 7fc307e00640 1 Processor -- start 2024-09-13T08:47:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.573+0000 7fc307e00640 1 -- start start 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8003a00 0x7fc300157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30014db00 0x7fc30014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc30006f710 con 0x7fc3000add90 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc30006f590 con 0x7fc30014db00 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.574+0000 7fc307e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fc3000b3ef0 con 0x7fc2e8003a00 2024-09-13T08:47:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53734/0 (socket says 172.21.15.31:53734) 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 -- 172.21.15.31:0/2385781000 learned_addr learned my addr 172.21.15.31:0/2385781000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc305400640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8003a00 0x7fc300157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc305e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30014db00 0x7fc30014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 -- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8003a00 msgr2=0x7fc300157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8003a00 0x7fc300157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 -- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30014db00 msgr2=0x7fc30014df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30014db00 0x7fc30014df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 -- 172.21.15.31:0/2385781000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3001f4bc0 con 0x7fc3000add90 2024-09-13T08:47:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.575+0000 7fc304a00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec011ae0 tx=0x7fc2ec009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.576+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec01e070 con 0x7fc3000add90 2024-09-13T08:47:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.576+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2ec004460 con 0x7fc3000add90 2024-09-13T08:47:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.576+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00fc60 con 0x7fc3000add90 2024-09-13T08:47:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.576+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3001f4ea0 con 0x7fc3000add90 2024-09-13T08:47:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.576+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3001f5460 con 0x7fc3000add90 2024-09-13T08:47:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.577+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000b39b0 con 0x7fc3000add90 2024-09-13T08:47:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.578+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fc2ec021070 con 0x7fc3000add90 2024-09-13T08:47:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.579+0000 7fc2fd600640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 0x7fc2e007b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.580+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7fc2ec099f90 con 0x7fc3000add90 2024-09-13T08:47:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.580+0000 7fc305400640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 0x7fc2e007b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.580+0000 7fc305400640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 0x7fc2e007b910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4033000 tx=0x7fc2f402d000 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.584+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fc2ec063860 con 0x7fc3000add90 2024-09-13T08:47:05.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.777+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fc3000cd860 con 0x7fc2e0079450 2024-09-13T08:47:05.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.778+0000 7fc2fd600640 1 -- 172.21.15.31:0/2385781000 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc3000cd860 con 0x7fc2e0079450 2024-09-13T08:47:05.780 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T08:47:05.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 msgr2=0x7fc2e007b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 0x7fc2e007b910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4033000 tx=0x7fc2f402d000 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 msgr2=0x7fc3000ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec011ae0 tx=0x7fc2ec009e90 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 shutdown_connections 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fc2e0079450 0x7fc2e007b910 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30014db00 0x7fc30014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3000add90 0x7fc3000ae210 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 --2- 172.21.15.31:0/2385781000 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fc2e8003a00 0x7fc300157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 >> 172.21.15.31:0/2385781000 conn(0x7fc3000b18f0 msgr2=0x7fc3001611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.780+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 shutdown_connections 2024-09-13T08:47:05.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.781+0000 7fc307e00640 1 -- 172.21.15.31:0/2385781000 wait complete. 2024-09-13T08:47:05.799 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T08:47:05.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:05.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 -- 172.21.15.31:0/831749640 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 msgr2=0x7f3994073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 --2- 172.21.15.31:0/831749640 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 0x7f3994073ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3988009830 tx=0x7f398802a480 comp rx=0 tx=0).stop 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 -- 172.21.15.31:0/831749640 shutdown_connections 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 --2- 172.21.15.31:0/831749640 >> v2:172.21.15.45:3300/0 conn(0x7f39940cc3e0 0x7f39940cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 --2- 172.21.15.31:0/831749640 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 0x7f3994073ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 -- 172.21.15.31:0/831749640 >> 172.21.15.31:0/831749640 conn(0x7f39940b1520 msgr2=0x7f39940b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 -- 172.21.15.31:0/831749640 shutdown_connections 2024-09-13T08:47:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.962+0000 7f399aa00640 1 -- 172.21.15.31:0/831749640 wait complete. 2024-09-13T08:47:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.964+0000 7f399aa00640 1 Processor -- start 2024-09-13T08:47:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.964+0000 7f399aa00640 1 -- start start 2024-09-13T08:47:05.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 0x7f39941f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39940cc3e0 0x7f39940cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 0x7f39940c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39940cace0 con 0x7f39940b2820 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39940cab60 con 0x7f39940cc3e0 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f399aa00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39940cae60 con 0x7f399415bc00 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f3999600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 0x7f39941f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f3999600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 0x7f39941f4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53746/0 (socket says 172.21.15.31:53746) 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.965+0000 7f3999600640 1 -- 172.21.15.31:0/4165637859 learned_addr learned my addr 172.21.15.31:0/4165637859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 -- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 msgr2=0x7f39940c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f399415bc00 0x7f39940c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3998c00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39940cc3e0 0x7f39940cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 -- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39940cc3e0 msgr2=0x7f39940cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39940cc3e0 0x7f39940cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940c7ee0 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3998c00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39940cc3e0 0x7f39940cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3999600640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 0x7f39941f4c10 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3988009950 tx=0x7f3988033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988005c90 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3988031e50 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39941f5440 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988005260 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.966+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39941f5a00 con 0x7f39940b2820 2024-09-13T08:47:05.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.967+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3994050ee0 con 0x7f39940b2820 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.969+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f3988005400 con 0x7f39940b2820 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3991600640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3974079450 0x7f397407b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f39880baa90 con 0x7f39940b2820 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3998c00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3974079450 0x7f397407b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3991600640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f397407f340 0x7f3974081760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3988045e50 con 0x7f397407f340 2024-09-13T08:47:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f39880bd390 con 0x7f39940b2820 2024-09-13T08:47:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.970+0000 7f399a000640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f397407f340 0x7f3974081760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.971+0000 7f3998c00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3974079450 0x7f397407b910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39800013f0 tx=0x7f3980009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.971+0000 7f399a000640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f397407f340 0x7f3974081760 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.972+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== osd.2 v2:172.21.15.31:6816/657376189 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3988045e50 con 0x7f397407f340 2024-09-13T08:47:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.985+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 --> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3994076bd0 con 0x7f397407f340 2024-09-13T08:47:05.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.985+0000 7f3991600640 1 -- 172.21.15.31:0/4165637859 <== osd.2 v2:172.21.15.31:6816/657376189 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3994076bd0 con 0x7f397407f340 2024-09-13T08:47:05.988 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:05.988 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f397407f340 msgr2=0x7f3974081760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6816/657376189,v1:172.21.15.31:6817/657376189] conn(0x7f397407f340 0x7f3974081760 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3974079450 msgr2=0x7f397407b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3974079450 0x7f397407b910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39800013f0 tx=0x7f3980009040 comp rx=0 tx=0).stop 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 msgr2=0x7f39941f4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 --2- 172.21.15.31:0/4165637859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39940b2820 0x7f39941f4c10 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3988009950 tx=0x7f3988033c20 comp rx=0 tx=0).stop 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399a000640 1 -- 172.21.15.31:0/4165637859 reap_dead start 2024-09-13T08:47:05.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 shutdown_connections 2024-09-13T08:47:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 >> 172.21.15.31:0/4165637859 conn(0x7f39940b1520 msgr2=0x7f39940b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 shutdown_connections 2024-09-13T08:47:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:05.986+0000 7f399aa00640 1 -- 172.21.15.31:0/4165637859 wait complete. 2024-09-13T08:47:06.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 -- 172.21.15.31:0/1677876747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75700041f0 msgr2=0x7f7570004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 --2- 172.21.15.31:0/1677876747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75700041f0 0x7f7570004800 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7574006f00 tx=0x7f7570005830 comp rx=0 tx=0).stop 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f758a800640 1 -- 172.21.15.31:0/1677876747 reap_dead start 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 -- 172.21.15.31:0/1677876747 shutdown_connections 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 -- 172.21.15.31:0/1677876747 >> 172.21.15.31:0/1677876747 conn(0x7f758c0b18f0 msgr2=0x7f758c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 -- 172.21.15.31:0/1677876747 shutdown_connections 2024-09-13T08:47:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.165+0000 7f7591000640 1 -- 172.21.15.31:0/1677876747 wait complete. 2024-09-13T08:47:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.167+0000 7f7591000640 1 Processor -- start 2024-09-13T08:47:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.167+0000 7f7591000640 1 -- start start 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 0x7f758c1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f758c15d320 0x7f758c155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f758c0749f0 con 0x7f758c155a60 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f758c074870 con 0x7f75700041f0 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7591000640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f758c1651b0 con 0x7f758c15d320 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f758a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7589e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 0x7f758c1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7589e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 0x7f758c1f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.31:48982/0 (socket says 172.21.15.31:48982) 2024-09-13T08:47:06.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f758a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53760/0 (socket says 172.21.15.31:53760) 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7589e00640 1 -- 172.21.15.31:0/3765984712 learned_addr learned my addr 172.21.15.31:0/3765984712 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.168+0000 7f7581400640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f758c15d320 0x7f758c155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 -- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f758c15d320 msgr2=0x7f758c155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f758c15d320 0x7f758c155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 -- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 msgr2=0x7f758c1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 0x7f758c1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 -- 172.21.15.31:0/3765984712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c15c860 con 0x7f758c155a60 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f758a800640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f758c0cf770 tx=0x7f758400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584004d20 con 0x7f758c155a60 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7584003970 con 0x7f758c155a60 2024-09-13T08:47:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758c1f71b0 con 0x7f758c155a60 2024-09-13T08:47:06.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584003da0 con 0x7f758c155a60 2024-09-13T08:47:06.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.169+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758c15ce00 con 0x7f758c155a60 2024-09-13T08:47:06.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.170+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758c0cdb80 con 0x7f758c155a60 2024-09-13T08:47:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.172+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f758400e5c0 con 0x7f758c155a60 2024-09-13T08:47:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.172+0000 7f7582a00640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 0x7f756807b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.173+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f758409a180 con 0x7f758c155a60 2024-09-13T08:47:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.173+0000 7f7589e00640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 0x7f756807b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.173+0000 7f7589e00640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 0x7f756807b910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f757c0036b0 tx=0x7f757c006b10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.177+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f7584063990 con 0x7f758c155a60 2024-09-13T08:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.376+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f758c0b25f0 con 0x7f7568079450 2024-09-13T08:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.376+0000 7f7582a00640 1 -- 172.21.15.31:0/3765984712 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f758c0b25f0 con 0x7f7568079450 2024-09-13T08:47:06.378 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 msgr2=0x7f756807b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 0x7f756807b910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f757c0036b0 tx=0x7f757c006b10 comp rx=0 tx=0).stop 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 msgr2=0x7f758c1f6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f758c0cf770 tx=0x7f758400f790 comp rx=0 tx=0).stop 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 shutdown_connections 2024-09-13T08:47:06.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f7568079450 0x7f756807b910 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f758c155a60 0x7f758c1f6480 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f758c15d320 0x7f758c155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 --2- 172.21.15.31:0/3765984712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75700041f0 0x7f758c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 >> 172.21.15.31:0/3765984712 conn(0x7f758c0b18f0 msgr2=0x7f758c0b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 shutdown_connections 2024-09-13T08:47:06.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.379+0000 7f7591000640 1 -- 172.21.15.31:0/3765984712 wait complete. 2024-09-13T08:47:06.397 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T08:47:06.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:06.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.559+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2620291232 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00aef50 msgr2=0x7f3ed006f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.559+0000 7f3ed6c00640 1 --2- 172.21.15.31:0/2620291232 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00aef50 0x7f3ed006f710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009860 tx=0x7f3ec402a530 comp rx=0 tx=0).stop 2024-09-13T08:47:06.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.560+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2620291232 shutdown_connections 2024-09-13T08:47:06.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.560+0000 7f3ed6c00640 1 --2- 172.21.15.31:0/2620291232 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00aef50 0x7f3ed006f710 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.560+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2620291232 >> 172.21.15.31:0/2620291232 conn(0x7f3ed00b1520 msgr2=0x7f3ed00b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.560+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2620291232 shutdown_connections 2024-09-13T08:47:06.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.560+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2620291232 wait complete. 2024-09-13T08:47:06.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.561+0000 7f3ed6c00640 1 Processor -- start 2024-09-13T08:47:06.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.561+0000 7f3ed6c00640 1 -- start start 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 0x7f3ed01e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00ca730 0x7f3ed01f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ed00cf440 0x7f3ed01e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed015ddf0 con 0x7f3ed00aef50 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed006fde0 con 0x7f3ed00cf440 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.562+0000 7f3ed6c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed0072b10 con 0x7f3ed00ca730 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 0x7f3ed01e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 0x7f3ed01e8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53770/0 (socket says 172.21.15.31:53770) 2024-09-13T08:47:06.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 -- 172.21.15.31:0/2863378664 learned_addr learned my addr 172.21.15.31:0/2863378664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed4e00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00ca730 0x7f3ed01f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed6200640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ed00cf440 0x7f3ed01e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 -- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00ca730 msgr2=0x7f3ed01f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3ed00ca730 0x7f3ed01f5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 -- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ed00cf440 msgr2=0x7f3ed01e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ed00cf440 0x7f3ed01e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed01e7cc0 con 0x7f3ed00aef50 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.563+0000 7f3ed5800640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 0x7f3ed01e8d60 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3ec402ab20 tx=0x7f3ec4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.564+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4005c90 con 0x7f3ed00aef50 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.564+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ec4005020 con 0x7f3ed00aef50 2024-09-13T08:47:06.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.564+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4005410 con 0x7f3ed00aef50 2024-09-13T08:47:06.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.564+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01e7fa0 con 0x7f3ed00aef50 2024-09-13T08:47:06.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.564+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01e8560 con 0x7f3ed00aef50 2024-09-13T08:47:06.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.565+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ed006f710 con 0x7f3ed00aef50 2024-09-13T08:47:06.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.566+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f3ec4039040 con 0x7f3ed00aef50 2024-09-13T08:47:06.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ecd600640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3eb0079420 0x7f3eb007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f3ec40ba4a0 con 0x7f3ed00aef50 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ed4e00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3eb0079420 0x7f3eb007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ecd600640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f3eb007f310 0x7f3eb0081730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ec4031400 con 0x7f3eb007f310 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.567+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f3ec40c23b0 con 0x7f3ed00aef50 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.568+0000 7f3ed6200640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f3eb007f310 0x7f3eb0081730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.568+0000 7f3ed4e00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3eb0079420 0x7f3eb007b8e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc001860 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.568+0000 7f3ed6200640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f3eb007f310 0x7f3eb0081730 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.569+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== osd.3 v2:172.21.15.31:6800/2845481020 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ec4031400 con 0x7f3eb007f310 2024-09-13T08:47:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.582+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 --> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3ed00ce2a0 con 0x7f3eb007f310 2024-09-13T08:47:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.582+0000 7f3ecd600640 1 -- 172.21.15.31:0/2863378664 <== osd.3 v2:172.21.15.31:6800/2845481020 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3ed00ce2a0 con 0x7f3eb007f310 2024-09-13T08:47:06.585 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:06.585 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f3eb007f310 msgr2=0x7f3eb0081730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6800/2845481020,v1:172.21.15.31:6801/2845481020] conn(0x7f3eb007f310 0x7f3eb0081730 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3eb0079420 msgr2=0x7f3eb007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3eb0079420 0x7f3eb007b8e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc001860 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 msgr2=0x7f3ed01e8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 --2- 172.21.15.31:0/2863378664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed00aef50 0x7f3ed01e8d60 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3ec402ab20 tx=0x7f3ec4003c10 comp rx=0 tx=0).stop 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6200640 1 -- 172.21.15.31:0/2863378664 reap_dead start 2024-09-13T08:47:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 shutdown_connections 2024-09-13T08:47:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 >> 172.21.15.31:0/2863378664 conn(0x7f3ed00b1520 msgr2=0x7f3ed015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 shutdown_connections 2024-09-13T08:47:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.583+0000 7f3ed6c00640 1 -- 172.21.15.31:0/2863378664 wait complete. 2024-09-13T08:47:06.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T08:47:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 -- 172.21.15.31:0/4274718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 msgr2=0x7f39700b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 --2- 172.21.15.31:0/4274718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700b39b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3968005fd0 tx=0x7f396802a8a0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 -- 172.21.15.31:0/4274718606 shutdown_connections 2024-09-13T08:47:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 --2- 172.21.15.31:0/4274718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700b39b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 --2- 172.21.15.31:0/4274718606 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f3970158700 0x7f397006e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 -- 172.21.15.31:0/4274718606 >> 172.21.15.31:0/4274718606 conn(0x7f39700b18f0 msgr2=0x7f39700b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 -- 172.21.15.31:0/4274718606 shutdown_connections 2024-09-13T08:47:06.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.760+0000 7f3978400640 1 -- 172.21.15.31:0/4274718606 wait complete. 2024-09-13T08:47:06.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.761+0000 7f3978400640 1 Processor -- start 2024-09-13T08:47:06.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.762+0000 7f3978400640 1 -- start start 2024-09-13T08:47:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.762+0000 7f3978400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f397006eba0 0x7f39701f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.762+0000 7f3978400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.762+0000 7f3978400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3978400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39700b4280 con 0x7f3970158700 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3978400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39700b4100 con 0x7f397006fa90 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3978400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f39700b4400 con 0x7f397006eba0 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53776/0 (socket says 172.21.15.31:53776) 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 -- 172.21.15.31:0/1792541860 learned_addr learned my addr 172.21.15.31:0/1792541860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3975a00640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f397006eba0 0x7f39701f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3975000640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 -- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f397006eba0 msgr2=0x7f39701f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f397006eba0 0x7f39701f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 -- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 msgr2=0x7f39700ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700ae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.763+0000 7f3976400640 1 -- 172.21.15.31:0/1792541860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39700ad220 con 0x7f3970158700 2024-09-13T08:47:06.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f3976400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3968009cd0 tx=0x7f3968032db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968031430 con 0x7f3970158700 2024-09-13T08:47:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3968004450 con 0x7f3970158700 2024-09-13T08:47:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968003e60 con 0x7f3970158700 2024-09-13T08:47:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39700ad500 con 0x7f3970158700 2024-09-13T08:47:06.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.764+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39700adac0 con 0x7f3970158700 2024-09-13T08:47:06.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.765+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39700b4920 con 0x7f3970158700 2024-09-13T08:47:06.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.766+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f3968043020 con 0x7f3970158700 2024-09-13T08:47:06.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.767+0000 7f396d600640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 0x7f395007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:06.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.767+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f396802e020 con 0x7f3970158700 2024-09-13T08:47:06.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.768+0000 7f3975a00640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 0x7f395007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:06.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.768+0000 7f3975a00640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 0x7f395007b8e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3970072ac0 tx=0x7f3964005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:06.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.772+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f3968084090 con 0x7f3970158700 2024-09-13T08:47:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.967+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f397015fa90 con 0x7f3950079420 2024-09-13T08:47:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.967+0000 7f396d600640 1 -- 172.21.15.31:0/1792541860 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f397015fa90 con 0x7f3950079420 2024-09-13T08:47:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T08:47:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 msgr2=0x7f395007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 0x7f395007b8e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3970072ac0 tx=0x7f3964005f90 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 msgr2=0x7f3970074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3968009cd0 tx=0x7f3968032db0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 shutdown_connections 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f3950079420 0x7f395007b8e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3970158700 0x7f3970074500 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f397006fa90 0x7f39700ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 --2- 172.21.15.31:0/1792541860 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f397006eba0 0x7f39701f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 >> 172.21.15.31:0/1792541860 conn(0x7f39700b18f0 msgr2=0x7f3970162400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 shutdown_connections 2024-09-13T08:47:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:06.970+0000 7f3978400640 1 -- 172.21.15.31:0/1792541860 wait complete. 2024-09-13T08:47:06.988 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T08:47:06.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:07.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 -- 172.21.15.31:0/1267156486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00cabb0 msgr2=0x7f6fe0165320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 --2- 172.21.15.31:0/1267156486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00cabb0 0x7f6fe0165320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8005fd0 tx=0x7f6fd802bc80 comp rx=0 tx=0).stop 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 -- 172.21.15.31:0/1267156486 shutdown_connections 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 --2- 172.21.15.31:0/1267156486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe00cabb0 0x7f6fe0165320 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 -- 172.21.15.31:0/1267156486 >> 172.21.15.31:0/1267156486 conn(0x7f6fe00cfbb0 msgr2=0x7f6fe00cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 -- 172.21.15.31:0/1267156486 shutdown_connections 2024-09-13T08:47:07.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.152+0000 7f6fe7e00640 1 -- 172.21.15.31:0/1267156486 wait complete. 2024-09-13T08:47:07.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.154+0000 7f6fe7e00640 1 Processor -- start 2024-09-13T08:47:07.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.154+0000 7f6fe7e00640 1 -- start start 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 0x7f6fe0159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00cabb0 0x7f6fe0072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe015ad00 0x7f6fe00733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe006e950 con 0x7f6fe006ee50 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe006e7d0 con 0x7f6fe00cabb0 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe7e00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe006ead0 con 0x7f6fe015ad00 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 0x7f6fe0159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 0x7f6fe0159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53800/0 (socket says 172.21.15.31:53800) 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe5400640 1 -- 172.21.15.31:0/3454743167 learned_addr learned my addr 172.21.15.31:0/3454743167 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe4a00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00cabb0 0x7f6fe0072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.155+0000 7f6fe5e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe015ad00 0x7f6fe00733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 -- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe015ad00 msgr2=0x7f6fe00733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f6fe015ad00 0x7f6fe00733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 -- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00cabb0 msgr2=0x7f6fe0072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00cabb0 0x7f6fe0072e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe00b1860 con 0x7f6fe006ee50 2024-09-13T08:47:07.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe5400640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 0x7f6fe0159aa0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c400 tx=0x7f6fd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011790 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe014b6f0 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fd4017070 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.156+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011cc0 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.157+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe01f8440 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.157+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6fe01f8b00 con 0x7f6fe006ee50 2024-09-13T08:47:07.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.159+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f6fd401c460 con 0x7f6fe006ee50 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fdd600640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fc0079420 0x7f6fc007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f6fd40992a0 con 0x7f6fe006ee50 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fe4a00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fc0079420 0x7f6fc007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fdd600640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f6fc007f310 0x7f6fc0081730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fd4009e50 con 0x7f6fc007f310 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.160+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f6fd40a2050 con 0x7f6fe006ee50 2024-09-13T08:47:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.161+0000 7f6fe4a00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fc0079420 0x7f6fc007b8e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6fe01e7d70 tx=0x7f6fcc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.161+0000 7f6fe5e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f6fc007f310 0x7f6fc0081730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.161+0000 7f6fe5e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f6fc007f310 0x7f6fc0081730 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.162+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== osd.4 v2:172.21.15.45:6802/1087405525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6fd4009e50 con 0x7f6fc007f310 2024-09-13T08:47:07.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.175+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 --> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6fe00ca4a0 con 0x7f6fc007f310 2024-09-13T08:47:07.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.176+0000 7f6fdd600640 1 -- 172.21.15.31:0/3454743167 <== osd.4 v2:172.21.15.45:6802/1087405525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6fe00ca4a0 con 0x7f6fc007f310 2024-09-13T08:47:07.178 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:07.178 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:07.179 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:07.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f6fc007f310 msgr2=0x7f6fc0081730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.45:6802/1087405525,v1:172.21.15.45:6803/1087405525] conn(0x7f6fc007f310 0x7f6fc0081730 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fc0079420 msgr2=0x7f6fc007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f6fc0079420 0x7f6fc007b8e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6fe01e7d70 tx=0x7f6fcc009040 comp rx=0 tx=0).stop 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 msgr2=0x7f6fe0159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 --2- 172.21.15.31:0/3454743167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fe006ee50 0x7f6fe0159aa0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c400 tx=0x7f6fd400c8c0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe5e00640 1 -- 172.21.15.31:0/3454743167 reap_dead start 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 shutdown_connections 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 >> 172.21.15.31:0/3454743167 conn(0x7f6fe00cfbb0 msgr2=0x7f6fe00cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 shutdown_connections 2024-09-13T08:47:07.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.177+0000 7f6fe7e00640 1 -- 172.21.15.31:0/3454743167 wait complete. 2024-09-13T08:47:07.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T08:47:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 -- 172.21.15.31:0/2664800428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa264073860 msgr2=0x7fa264073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 --2- 172.21.15.31:0/2664800428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa264073860 0x7fa264073cb0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fa26406e610 tx=0x7fa25c02aac0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 -- 172.21.15.31:0/2664800428 shutdown_connections 2024-09-13T08:47:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 --2- 172.21.15.31:0/2664800428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa264073860 0x7fa264073cb0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 --2- 172.21.15.31:0/2664800428 >> v2:172.21.15.45:3300/0 conn(0x7fa26415be30 0x7fa2640cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 -- 172.21.15.31:0/2664800428 >> 172.21.15.31:0/2664800428 conn(0x7fa2640b18f0 msgr2=0x7fa2640b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 -- 172.21.15.31:0/2664800428 shutdown_connections 2024-09-13T08:47:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.354+0000 7fa26b600640 1 -- 172.21.15.31:0/2664800428 wait complete. 2024-09-13T08:47:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.356+0000 7fa26b600640 1 Processor -- start 2024-09-13T08:47:07.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.356+0000 7fa26b600640 1 -- start start 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa264073860 0x7fa264153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa26415be30 0x7fa2641562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa2640733e0 con 0x7fa2640ced20 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa2640ce9e0 con 0x7fa264073860 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa26b600640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fa2640741f0 con 0x7fa26415be30 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa25be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa25be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53818/0 (socket says 172.21.15.31:53818) 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa25be00640 1 -- 172.21.15.31:0/3019207252 learned_addr learned my addr 172.21.15.31:0/3019207252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa269600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa26415be30 0x7fa2641562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.357+0000 7fa268c00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa264073860 0x7fa264153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 -- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa26415be30 msgr2=0x7fa2641562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa26415be30 0x7fa2641562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 -- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa264073860 msgr2=0x7fa264153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa264073860 0x7fa264153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 -- 172.21.15.31:0/3019207252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26415b640 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa25be00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00c450 tx=0x7fa24c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0052d0 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa24c005470 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c012870 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.358+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa264159fb0 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.359+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26415a570 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.360+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa264073cb0 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.361+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fa24c0035f0 con 0x7fa2640ced20 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.362+0000 7fa261600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 0x7fa24007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.362+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7fa24c098ef0 con 0x7fa2640ced20 2024-09-13T08:47:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.362+0000 7fa268c00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 0x7fa24007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.363+0000 7fa268c00640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 0x7fa24007b8e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa26406ad00 tx=0x7fa254005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.366+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7fa24c062760 con 0x7fa2640ced20 2024-09-13T08:47:07.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.562+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fa264157930 con 0x7fa240079420 2024-09-13T08:47:07.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.563+0000 7fa261600640 1 -- 172.21.15.31:0/3019207252 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa264157930 con 0x7fa240079420 2024-09-13T08:47:07.565 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T08:47:07.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 msgr2=0x7fa24007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 0x7fa24007b8e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa26406ad00 tx=0x7fa254005f90 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 msgr2=0x7fa2641540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00c450 tx=0x7fa24c00c910 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 shutdown_connections 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fa240079420 0x7fa24007b8e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fa26415be30 0x7fa2641562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2640ced20 0x7fa2641540d0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 --2- 172.21.15.31:0/3019207252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa264073860 0x7fa264153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 >> 172.21.15.31:0/3019207252 conn(0x7fa2640b18f0 msgr2=0x7fa26406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.565+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 shutdown_connections 2024-09-13T08:47:07.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.566+0000 7fa26b600640 1 -- 172.21.15.31:0/3019207252 wait complete. 2024-09-13T08:47:07.583 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T08:47:07.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:07.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:07.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:07.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 -- 172.21.15.31:0/4148109028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fec0 msgr2=0x7f9e480b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/4148109028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fec0 0x7f9e480b29e0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9e40005fd0 tx=0x7f9e4002a8b0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 -- 172.21.15.31:0/4148109028 shutdown_connections 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/4148109028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fec0 0x7f9e480b29e0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/4148109028 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e4815bfc0 0x7f9e48072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 -- 172.21.15.31:0/4148109028 >> 172.21.15.31:0/4148109028 conn(0x7f9e480cfbb0 msgr2=0x7f9e480cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 -- 172.21.15.31:0/4148109028 shutdown_connections 2024-09-13T08:47:07.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.747+0000 7f9e4ee00640 1 -- 172.21.15.31:0/4148109028 wait complete. 2024-09-13T08:47:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.749+0000 7f9e4ee00640 1 Processor -- start 2024-09-13T08:47:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.749+0000 7f9e4ee00640 1 -- start start 2024-09-13T08:47:07.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 0x7f9e481f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e4815bfc0 0x7f9e480c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e480c8dc0 0x7f9e480b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e480cbfd0 con 0x7f9e4806fa60 2024-09-13T08:47:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e480cbe50 con 0x7f9e4815bfc0 2024-09-13T08:47:07.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f9e480b2f20 con 0x7f9e480c8dc0 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e47e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 0x7f9e481f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e47e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 0x7f9e481f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53844/0 (socket says 172.21.15.31:53844) 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e47e00640 1 -- 172.21.15.31:0/1734657200 learned_addr learned my addr 172.21.15.31:0/1734657200 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e4ce00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e480c8dc0 0x7f9e480b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.750+0000 7f9e47400640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e4815bfc0 0x7f9e480c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 -- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e480c8dc0 msgr2=0x7f9e480b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f9e480c8dc0 0x7f9e480b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 -- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e4815bfc0 msgr2=0x7f9e480c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e4815bfc0 0x7f9e480c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e480c9330 con 0x7f9e4806fa60 2024-09-13T08:47:07.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e47e00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 0x7f9e481f59d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e7c0 tx=0x7f9e3800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3800ccc0 con 0x7f9e4806fa60 2024-09-13T08:47:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e38010430 con 0x7f9e4806fa60 2024-09-13T08:47:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e480c9610 con 0x7f9e4806fa60 2024-09-13T08:47:07.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38010860 con 0x7f9e4806fa60 2024-09-13T08:47:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.751+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e480c9bd0 con 0x7f9e4806fa60 2024-09-13T08:47:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.752+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e480b3370 con 0x7f9e4806fa60 2024-09-13T08:47:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.754+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f9e38010a00 con 0x7f9e4806fa60 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.754+0000 7f9e3fe00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e28079450 0x7f9e2807b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f9e38099bd0 con 0x7f9e4806fa60 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e47400640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e28079450 0x7f9e2807b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e3fe00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7f9e2807f340 0x7f9e28081760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e38012e50 con 0x7f9e2807f340 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f9e380a3050 con 0x7f9e4806fa60 2024-09-13T08:47:07.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e4ce00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7f9e2807f340 0x7f9e28081760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.755+0000 7f9e47400640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e28079450 0x7f9e2807b910 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9e48156d60 tx=0x7f9e34007e90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.756+0000 7f9e4ce00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7f9e2807f340 0x7f9e28081760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:07.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.757+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== osd.5 v2:172.21.15.45:6818/2092128205 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e38012e50 con 0x7f9e2807f340 2024-09-13T08:47:07.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.770+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 --> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9e480cb310 con 0x7f9e2807f340 2024-09-13T08:47:07.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e3fe00640 1 -- 172.21.15.31:0/1734657200 <== osd.5 v2:172.21.15.45:6818/2092128205 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9e480cb310 con 0x7f9e2807f340 2024-09-13T08:47:07.773 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:07.773 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:07.773 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7f9e2807f340 msgr2=0x7f9e28081760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.45:6818/2092128205,v1:172.21.15.45:6819/2092128205] conn(0x7f9e2807f340 0x7f9e28081760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e28079450 msgr2=0x7f9e2807b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9e28079450 0x7f9e2807b910 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9e48156d60 tx=0x7f9e34007e90 comp rx=0 tx=0).stop 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 msgr2=0x7f9e481f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 --2- 172.21.15.31:0/1734657200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e4806fa60 0x7f9e481f59d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e7c0 tx=0x7f9e3800ec80 comp rx=0 tx=0).stop 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ce00640 1 -- 172.21.15.31:0/1734657200 reap_dead start 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 shutdown_connections 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 >> 172.21.15.31:0/1734657200 conn(0x7f9e480cfbb0 msgr2=0x7f9e480ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 shutdown_connections 2024-09-13T08:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.772+0000 7f9e4ee00640 1 -- 172.21.15.31:0/1734657200 wait complete. 2024-09-13T08:47:07.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 -- 172.21.15.31:0/3729877671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c0ced20 msgr2=0x7f2c0c0ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 --2- 172.21.15.31:0/3729877671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0ce0c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c06e610 tx=0x7f2bf402a530 comp rx=0 tx=0).stop 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 -- 172.21.15.31:0/3729877671 shutdown_connections 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 --2- 172.21.15.31:0/3729877671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0ce0c0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 --2- 172.21.15.31:0/3729877671 >> v2:172.21.15.45:3300/0 conn(0x7f2c0c15be30 0x7f2c0c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 -- 172.21.15.31:0/3729877671 >> 172.21.15.31:0/3729877671 conn(0x7f2c0c0b18f0 msgr2=0x7f2c0c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 -- 172.21.15.31:0/3729877671 shutdown_connections 2024-09-13T08:47:07.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.992+0000 7f2c11c00640 1 -- 172.21.15.31:0/3729877671 wait complete. 2024-09-13T08:47:07.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.994+0000 7f2c11c00640 1 Processor -- start 2024-09-13T08:47:07.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.994+0000 7f2c11c00640 1 -- start start 2024-09-13T08:47:07.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c0c0ce600 0x7f2c0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0749f0 con 0x7f2c0c15be30 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c074870 con 0x7f2c0c0ced20 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c11c00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c160160 con 0x7f2c0c0ce600 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c0b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c0b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53856/0 (socket says 172.21.15.31:53856) 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c0b400640 1 -- 172.21.15.31:0/2216968926 learned_addr learned my addr 172.21.15.31:0/2216968926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:07.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c02000640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.995+0000 7f2c0aa00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c0c0ce600 0x7f2c0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 -- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c0c0ce600 msgr2=0x7f2c0c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c0c0ce600 0x7f2c0c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 -- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c0c0ced20 msgr2=0x7f2c0c0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 -- 172.21.15.31:0/2216968926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c1f8220 con 0x7f2c0c15be30 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c0b400640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400cbc0 tx=0x7f2c0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04016020 con 0x7f2c0c15be30 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c0400ae30 con 0x7f2c0c15be30 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04012660 con 0x7f2c0c15be30 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c0c1f8500 con 0x7f2c0c15be30 2024-09-13T08:47:08.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.996+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c0c1f8ac0 con 0x7f2c0c15be30 2024-09-13T08:47:08.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.998+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c0c1f4c80 con 0x7f2c0c15be30 2024-09-13T08:47:08.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:07.999+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f2c04019020 con 0x7f2c0c15be30 2024-09-13T08:47:08.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.000+0000 7f2c03400640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 0x7f2be807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.000+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f2c040997d0 con 0x7f2c0c15be30 2024-09-13T08:47:08.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.001+0000 7f2c0aa00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 0x7f2be807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.001+0000 7f2c0aa00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 0x7f2be807b8e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cc310 tx=0x7f2bfc00b2e0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.005+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2c0c1f4c80 con 0x7f2c0c15be30 2024-09-13T08:47:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.202+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2c0c1605c0 con 0x7f2be8079420 2024-09-13T08:47:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.202+0000 7f2c03400640 1 -- 172.21.15.31:0/2216968926 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2c0c1605c0 con 0x7f2be8079420 2024-09-13T08:47:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.204+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 msgr2=0x7f2be807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.204+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 0x7f2be807b8e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cc310 tx=0x7f2bfc00b2e0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 msgr2=0x7f2c0c1eec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400cbc0 tx=0x7f2c0400a430 comp rx=0 tx=0).stop 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 shutdown_connections 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2be8079420 0x7f2be807b8e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c0c15be30 0x7f2c0c1eec40 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c0c0ced20 0x7f2c0c0b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 --2- 172.21.15.31:0/2216968926 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f2c0c0ce600 0x7f2c0c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 >> 172.21.15.31:0/2216968926 conn(0x7f2c0c0b18f0 msgr2=0x7f2c0c0cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 shutdown_connections 2024-09-13T08:47:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.205+0000 7f2c11c00640 1 -- 172.21.15.31:0/2216968926 wait complete. 2024-09-13T08:47:08.225 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T08:47:08.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 -- 172.21.15.31:0/2750471216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 msgr2=0x7f138806f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 --2- 172.21.15.31:0/2750471216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f138806f530 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f138806edb0 tx=0x7f137c02a560 comp rx=0 tx=0).stop 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 -- 172.21.15.31:0/2750471216 shutdown_connections 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 --2- 172.21.15.31:0/2750471216 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13880710c0 0x7f138806cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 --2- 172.21.15.31:0/2750471216 >> v2:172.21.15.45:3300/0 conn(0x7f138806d040 0x7f13880738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 --2- 172.21.15.31:0/2750471216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f138806f530 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 -- 172.21.15.31:0/2750471216 >> 172.21.15.31:0/2750471216 conn(0x7f13880d0ae0 msgr2=0x7f13880d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 -- 172.21.15.31:0/2750471216 shutdown_connections 2024-09-13T08:47:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.389+0000 7f138ee00640 1 -- 172.21.15.31:0/2750471216 wait complete. 2024-09-13T08:47:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.391+0000 7f138ee00640 1 Processor -- start 2024-09-13T08:47:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.391+0000 7f138ee00640 1 -- start start 2024-09-13T08:47:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.391+0000 7f138ee00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f138806d040 0x7f13880c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138ee00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13880710c0 0x7f1388150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f13880b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1388073f70 con 0x7f138815bbe0 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138ee00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1388073df0 con 0x7f138806d040 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138ee00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f13880740f0 con 0x7f13880710c0 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f13880b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f13880b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53876/0 (socket says 172.21.15.31:53876) 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138d000640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13880710c0 0x7f1388150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 -- 172.21.15.31:0/567278594 learned_addr learned my addr 172.21.15.31:0/567278594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138da00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f138806d040 0x7f13880c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 -- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13880710c0 msgr2=0x7f1388150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f13880710c0 0x7f1388150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 -- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f138806d040 msgr2=0x7f13880c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f138806d040 0x7f13880c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.392+0000 7f138e400640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13881f8320 con 0x7f138815bbe0 2024-09-13T08:47:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f138e400640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f13880b5bd0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f138000eaa0 tx=0x7f138000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380004b20 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1380004db0 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380005280 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13881f5650 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.393+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13880c9450 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.394+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f138806f890 con 0x7f138815bbe0 2024-09-13T08:47:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.395+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f1380005420 con 0x7f138815bbe0 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.396+0000 7f1385600640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1368079420 0x7f136807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.396+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f138009a700 con 0x7f138815bbe0 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.396+0000 7f138da00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1368079420 0x7f136807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.396+0000 7f1385600640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f136807f310 0x7f1368081730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.397+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1380010e50 con 0x7f136807f310 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.397+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f138009d380 con 0x7f138815bbe0 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.397+0000 7f138d000640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f136807f310 0x7f1368081730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.397+0000 7f138da00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1368079420 0x7f136807b8e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f137c033000 tx=0x7f137c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.398+0000 7f138d000640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f136807f310 0x7f1368081730 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.398+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== osd.6 v2:172.21.15.45:6810/636367962 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1380010e50 con 0x7f136807f310 2024-09-13T08:47:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.412+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 --> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f138806cb00 con 0x7f136807f310 2024-09-13T08:47:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.412+0000 7f1385600640 1 -- 172.21.15.31:0/567278594 <== osd.6 v2:172.21.15.45:6810/636367962 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f138806cb00 con 0x7f136807f310 2024-09-13T08:47:08.415 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:08.415 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:08.415 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.412+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f136807f310 msgr2=0x7f1368081730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.412+0000 7f138ee00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.45:6810/636367962,v1:172.21.15.45:6811/636367962] conn(0x7f136807f310 0x7f1368081730 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1368079420 msgr2=0x7f136807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f1368079420 0x7f136807b8e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f137c033000 tx=0x7f137c004730 comp rx=0 tx=0).stop 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 msgr2=0x7f13880b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 --2- 172.21.15.31:0/567278594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f138815bbe0 0x7f13880b5bd0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f138000eaa0 tx=0x7f138000ef60 comp rx=0 tx=0).stop 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138e400640 1 -- 172.21.15.31:0/567278594 reap_dead start 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 shutdown_connections 2024-09-13T08:47:08.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 >> 172.21.15.31:0/567278594 conn(0x7f13880d0ae0 msgr2=0x7f138815b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 shutdown_connections 2024-09-13T08:47:08.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.413+0000 7f138ee00640 1 -- 172.21.15.31:0/567278594 wait complete. 2024-09-13T08:47:08.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 -- 172.21.15.31:0/3680746436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 msgr2=0x7ff3b80ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3680746436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b80ce290 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0005fd0 tx=0x7ff3a002a5d0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 -- 172.21.15.31:0/3680746436 shutdown_connections 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3680746436 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 0x7ff3b8073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3680746436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b80ce290 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 -- 172.21.15.31:0/3680746436 >> 172.21.15.31:0/3680746436 conn(0x7ff3b80b18f0 msgr2=0x7ff3b80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 -- 172.21.15.31:0/3680746436 shutdown_connections 2024-09-13T08:47:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.588+0000 7ff3bf400640 1 -- 172.21.15.31:0/3680746436 wait complete. 2024-09-13T08:47:08.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.590+0000 7ff3bf400640 1 Processor -- start 2024-09-13T08:47:08.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.590+0000 7ff3bf400640 1 -- start start 2024-09-13T08:47:08.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 0x7ff3b8151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b815be30 0x7ff3b80ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80733e0 con 0x7ff3b80ced20 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80ce9e0 con 0x7ff3b815be30 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bf400640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b8074230 con 0x7ff3b8073860 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53908/0 (socket says 172.21.15.31:53908) 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3afe00640 1 -- 172.21.15.31:0/3537244176 learned_addr learned my addr 172.21.15.31:0/3537244176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bca00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 0x7ff3b8151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.591+0000 7ff3bd400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b815be30 0x7ff3b80ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 -- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 msgr2=0x7ff3b8151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 0x7ff3b8151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 -- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b815be30 msgr2=0x7ff3b80ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b815be30 0x7ff3b80ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 -- 172.21.15.31:0/3537244176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b8150570 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3afe00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009850 tx=0x7ff3a00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a003f070 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3a0003ce0 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b814f5a0 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.592+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0035030 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.593+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b814fb60 con 0x7ff3b80ced20 2024-09-13T08:47:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.594+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b80b25f0 con 0x7ff3b80ced20 2024-09-13T08:47:08.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.595+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7ff3a0042070 con 0x7ff3b80ced20 2024-09-13T08:47:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.596+0000 7ff3b5600640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 0x7ff39407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.596+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7ff3a00ba160 con 0x7ff3b80ced20 2024-09-13T08:47:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.596+0000 7ff3bca00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 0x7ff39407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.597+0000 7ff3bca00640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 0x7ff39407b8e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001720 tx=0x7ff3a8009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.600+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7ff3a00838f0 con 0x7ff3b80ced20 2024-09-13T08:47:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.797+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7ff3b80b16b0 con 0x7ff394079420 2024-09-13T08:47:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.798+0000 7ff3b5600640 1 -- 172.21.15.31:0/3537244176 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff3b80b16b0 con 0x7ff394079420 2024-09-13T08:47:08.800 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T08:47:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 msgr2=0x7ff39407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 0x7ff39407b8e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001720 tx=0x7ff3a8009040 comp rx=0 tx=0).stop 2024-09-13T08:47:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 msgr2=0x7ff3b814e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009850 tx=0x7ff3a00035f0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 shutdown_connections 2024-09-13T08:47:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7ff394079420 0x7ff39407b8e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b815be30 0x7ff3b80ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3b80ced20 0x7ff3b814e100 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 --2- 172.21.15.31:0/3537244176 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7ff3b8073860 0x7ff3b8151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.800+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 >> 172.21.15.31:0/3537244176 conn(0x7ff3b80b18f0 msgr2=0x7ff3b80cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.801+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 shutdown_connections 2024-09-13T08:47:08.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.801+0000 7ff3bf400640 1 -- 172.21.15.31:0/3537244176 wait complete. 2024-09-13T08:47:08.818 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T08:47:08.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T08:47:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 -- 172.21.15.31:0/2122323109 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c1618d0 msgr2=0x7fd98c0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 --2- 172.21.15.31:0/2122323109 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c1618d0 0x7fd98c0ceb90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd974005fd0 tx=0x7fd97402a540 comp rx=0 tx=0).stop 2024-09-13T08:47:08.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 -- 172.21.15.31:0/2122323109 shutdown_connections 2024-09-13T08:47:08.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 --2- 172.21.15.31:0/2122323109 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c1618d0 0x7fd98c0ceb90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 --2- 172.21.15.31:0/2122323109 >> v2:172.21.15.45:3300/0 conn(0x7fd98c1614f0 0x7fd98c0ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 -- 172.21.15.31:0/2122323109 >> 172.21.15.31:0/2122323109 conn(0x7fd98c0d1dd0 msgr2=0x7fd98c0d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:08.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 -- 172.21.15.31:0/2122323109 shutdown_connections 2024-09-13T08:47:08.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.981+0000 7fd992a00640 1 -- 172.21.15.31:0/2122323109 wait complete. 2024-09-13T08:47:08.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.983+0000 7fd992a00640 1 Processor -- start 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.983+0000 7fd992a00640 1 -- start start 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 0x7fd98c1f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c1614f0 0x7fd98c1f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c1618d0 0x7fd98c0b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0d1340 con 0x7fd98c1614f0 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0d11c0 con 0x7fd98c1618d0 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd992a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c0d14c0 con 0x7fd98c0cc870 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd98be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 0x7fd98c1f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd98be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 0x7fd98c1f44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3301/0 says I am v2:172.21.15.31:50698/0 (socket says 172.21.15.31:50698) 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd98be00640 1 -- 172.21.15.31:0/571769844 learned_addr learned my addr 172.21.15.31:0/571769844 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd983400640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c1614f0 0x7fd98c1f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.984+0000 7fd990a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c1618d0 0x7fd98c0b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 -- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c1618d0 msgr2=0x7fd98c0b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98c1618d0 0x7fd98c0b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 -- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c1614f0 msgr2=0x7fd98c1f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd98c1614f0 0x7fd98c1f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0c8fd0 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd98be00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 0x7fd98c1f44a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00ea20 tx=0x7fd97c00eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00ce40 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd97c004110 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c0c92b0 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.985+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c005030 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.986+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c0c9d90 con 0x7fd98c0cc870 2024-09-13T08:47:08.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.986+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd98c0b7670 con 0x7fd98c0cc870 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.988+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7fd97c0051d0 con 0x7fd98c0cc870 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd988c00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd968079450 0x7fd96807b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7fd97c09a390 con 0x7fd98c0cc870 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd983400640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd968079450 0x7fd96807b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd988c00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7fd96807f340 0x7fd968081760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd968081e10 con 0x7fd96807f340 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.989+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== mon.2 v2:172.21.15.45:3301/0 6 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7fd97c09cff0 con 0x7fd98c0cc870 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.990+0000 7fd990a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7fd96807f340 0x7fd968081760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.990+0000 7fd983400640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd968079450 0x7fd96807b910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.990+0000 7fd990a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7fd96807f340 0x7fd968081760 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:08.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:08.991+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== osd.7 v2:172.21.15.45:6826/1264531367 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd968081e10 con 0x7fd96807f340 2024-09-13T08:47:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.005+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 --> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd98c0cccc0 con 0x7fd96807f340 2024-09-13T08:47:09.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.005+0000 7fd988c00640 1 -- 172.21.15.31:0/571769844 <== osd.7 v2:172.21.15.45:6826/1264531367 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd98c0cccc0 con 0x7fd96807f340 2024-09-13T08:47:09.007 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-13T08:47:09.007 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T08:47:09.007 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-13T08:47:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7fd96807f340 msgr2=0x7fd968081760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:6826/1264531367,v1:172.21.15.45:6827/1264531367] conn(0x7fd96807f340 0x7fd968081760 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd968079450 msgr2=0x7fd96807b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7fd968079450 0x7fd96807b910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974004730 comp rx=0 tx=0).stop 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 msgr2=0x7fd98c1f44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 --2- 172.21.15.31:0/571769844 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7fd98c0cc870 0x7fd98c1f44a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00ea20 tx=0x7fd97c00eee0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd990a00640 1 -- 172.21.15.31:0/571769844 reap_dead start 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 shutdown_connections 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 >> 172.21.15.31:0/571769844 conn(0x7fd98c0d1dd0 msgr2=0x7fd98c0cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 shutdown_connections 2024-09-13T08:47:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.006+0000 7fd992a00640 1 -- 172.21.15.31:0/571769844 wait complete. 2024-09-13T08:47:09.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T08:47:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 -- 172.21.15.31:0/415460851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 msgr2=0x7f831c0cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 --2- 172.21.15.31:0/415460851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c0cee40 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f831c0743f0 tx=0x7f830c02a580 comp rx=0 tx=0).stop 2024-09-13T08:47:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 -- 172.21.15.31:0/415460851 shutdown_connections 2024-09-13T08:47:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 --2- 172.21.15.31:0/415460851 >> v2:172.21.15.45:3300/0 conn(0x7f831c15f820 0x7f831c15fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 --2- 172.21.15.31:0/415460851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c0cee40 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 -- 172.21.15.31:0/415460851 >> 172.21.15.31:0/415460851 conn(0x7f831c0cc7a0 msgr2=0x7f831c0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 -- 172.21.15.31:0/415460851 shutdown_connections 2024-09-13T08:47:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.184+0000 7f831be00640 1 -- 172.21.15.31:0/415460851 wait complete. 2024-09-13T08:47:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.185+0000 7f831be00640 1 Processor -- start 2024-09-13T08:47:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.186+0000 7f831be00640 1 -- start start 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.186+0000 7f831be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f831c06e670 0x7f831c1f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.186+0000 7f831be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831be00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f831c15f820 0x7f831c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f831c070040 con 0x7f831c15bb70 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f831c06fec0 con 0x7f831c06e670 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831be00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f831c0701c0 con 0x7f831c15f820 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831b400640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f831c15f820 0x7f831c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53944/0 (socket says 172.21.15.31:53944) 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 -- 172.21.15.31:0/3248082952 learned_addr learned my addr 172.21.15.31:0/3248082952 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f831aa00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f831c06e670 0x7f831c1f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 -- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f831c15f820 msgr2=0x7f831c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f831c15f820 0x7f831c15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 -- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f831c06e670 msgr2=0x7f831c1f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f831c06e670 0x7f831c1f89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.187+0000 7f8313e00640 1 -- 172.21.15.31:0/3248082952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831c15ea20 con 0x7f831c15bb70 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f8313e00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f830400c480 tx=0x7f830400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83040052d0 con 0x7f831c15bb70 2024-09-13T08:47:09.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8304005470 con 0x7f831c15bb70 2024-09-13T08:47:09.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304012870 con 0x7f831c15bb70 2024-09-13T08:47:09.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831c15ed00 con 0x7f831c15bb70 2024-09-13T08:47:09.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.188+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831c15f2c0 con 0x7f831c15bb70 2024-09-13T08:47:09.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.189+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f831c0c7ef0 con 0x7f831c15bb70 2024-09-13T08:47:09.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.190+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f83040035f0 con 0x7f831c15bb70 2024-09-13T08:47:09.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.191+0000 7f8312a00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 0x7f82f807b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.191+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f8304098fe0 con 0x7f831c15bb70 2024-09-13T08:47:09.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.191+0000 7f831aa00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 0x7f82f807b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.192+0000 7f831aa00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 0x7f82f807b8e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f830c034000 tx=0x7f830c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.195+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f83040627f0 con 0x7f831c15bb70 2024-09-13T08:47:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.393+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f831c070670 con 0x7f82f8079420 2024-09-13T08:47:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.393+0000 7f8312a00640 1 -- 172.21.15.31:0/3248082952 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f831c070670 con 0x7f82f8079420 2024-09-13T08:47:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T08:47:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 msgr2=0x7f82f807b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 0x7f82f807b8e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f830c034000 tx=0x7f830c004730 comp rx=0 tx=0).stop 2024-09-13T08:47:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 msgr2=0x7f831c148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f830400c480 tx=0x7f830400c940 comp rx=0 tx=0).stop 2024-09-13T08:47:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 shutdown_connections 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f82f8079420 0x7f82f807b8e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f831c15f820 0x7f831c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831c15bb70 0x7f831c148a20 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 --2- 172.21.15.31:0/3248082952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f831c06e670 0x7f831c1f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 >> 172.21.15.31:0/3248082952 conn(0x7f831c0cc7a0 msgr2=0x7f831c15a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 shutdown_connections 2024-09-13T08:47:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.396+0000 7f831be00640 1 -- 172.21.15.31:0/3248082952 wait complete. 2024-09-13T08:47:09.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T08:47:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.574+0000 7f61afe00640 1 -- 172.21.15.31:0/3638722328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00b52d0 msgr2=0x7f61b0152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.574+0000 7f61afe00640 1 --2- 172.21.15.31:0/3638722328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00b52d0 0x7f61b0152680 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f61b00b26c0 tx=0x7f61a802aac0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 -- 172.21.15.31:0/3638722328 shutdown_connections 2024-09-13T08:47:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 --2- 172.21.15.31:0/3638722328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00b52d0 0x7f61b0152680 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 --2- 172.21.15.31:0/3638722328 >> v2:172.21.15.45:3300/0 conn(0x7f61b00cc720 0x7f61b006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 -- 172.21.15.31:0/3638722328 >> 172.21.15.31:0/3638722328 conn(0x7f61b006cb00 msgr2=0x7f61b006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 -- 172.21.15.31:0/3638722328 shutdown_connections 2024-09-13T08:47:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.575+0000 7f61afe00640 1 -- 172.21.15.31:0/3638722328 wait complete. 2024-09-13T08:47:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.576+0000 7f61afe00640 1 Processor -- start 2024-09-13T08:47:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.576+0000 7f61afe00640 1 -- start start 2024-09-13T08:47:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f61b00b52d0 0x7f61b0150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61b00cc720 0x7f61b014cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0152bc0 con 0x7f61b00ccb00 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0070460 con 0x7f61b00cc720 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.577+0000 7f61afe00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f61b0152d40 con 0x7f61b00b52d0 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53966/0 (socket says 172.21.15.31:53966) 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 -- 172.21.15.31:0/4159710659 learned_addr learned my addr 172.21.15.31:0/4159710659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61a6000640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61b00cc720 0x7f61b014cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61aea00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f61b00b52d0 0x7f61b0150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 -- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f61b00b52d0 msgr2=0x7f61b0150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f61b00b52d0 0x7f61b0150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 -- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61b00cc720 msgr2=0x7f61b014cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61b00cc720 0x7f61b014cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 -- 172.21.15.31:0/4159710659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b014dd60 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.578+0000 7f61af400640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009d60 tx=0x7f61a8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.579+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8044070 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.579+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61a8004250 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.579+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8004680 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.579+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b00c4da0 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.579+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b00c5360 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.580+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b006aa20 con 0x7f61b00ccb00 2024-09-13T08:47:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.581+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f61a802c840 con 0x7f61b00ccb00 2024-09-13T08:47:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.582+0000 7f61a7400640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 0x7f618c07b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.582+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f61a80b9ad0 con 0x7f61b00ccb00 2024-09-13T08:47:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.582+0000 7f61aea00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 0x7f618c07b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.583+0000 7f61aea00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 0x7f618c07b8e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f61a0005d80 tx=0x7f61a000b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.586+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f61b006aa20 con 0x7f61b00ccb00 2024-09-13T08:47:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.776+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 --> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61b0153260 con 0x7f618c079420 2024-09-13T08:47:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.777+0000 7f61a7400640 1 -- 172.21.15.31:0/4159710659 <== mgr.4118 v2:172.21.15.31:6836/1500367474 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13380 (secure 0 0 0) 0x7f61b0153260 con 0x7f618c079420 2024-09-13T08:47:09.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-13T08:47:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 msgr2=0x7f618c07b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 0x7f618c07b8e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f61a0005d80 tx=0x7f61a000b390 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 msgr2=0x7f61b014d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009d60 tx=0x7f61a8003f10 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 shutdown_connections 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f618c079420 0x7f618c07b8e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61b00ccb00 0x7f61b014d0b0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61b00cc720 0x7f61b014cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 --2- 172.21.15.31:0/4159710659 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f61b00b52d0 0x7f61b0150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 >> 172.21.15.31:0/4159710659 conn(0x7f61b006cb00 msgr2=0x7f61b006f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 shutdown_connections 2024-09-13T08:47:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.779+0000 7f61afe00640 1 -- 172.21.15.31:0/4159710659 wait complete. 2024-09-13T08:47:09.797 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-13T08:47:09.023158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":289300,"kb_used_data":18984,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":754685420,"statfs":{"total":773094113280,"available":772797870080,"internally_reserved":0,"allocated":19439616,"data_stored":9271152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-4,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001613"},"pg_stats":[{"pgid":"1.0","version":"181'81","reported_seq":440,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-13T08:47:06.294757+0000","last_change":"2024-09-13T08:47:06.294699+0000","last_active":"2024-09-13T08:47:06.294757+0000","last_peered":"2024-09-13T08:47:06.294757+0000","last_clean":"2024-09-13T08:47:06.294757+0000","last_became_active":"2024-09-13T08:37:01.427513+0000","last_became_peered":"2024-09-13T08:37:01.427513+0000","last_unstale":"2024-09-13T08:47:06.294757+0000","last_undegraded":"2024-09-13T08:47:06.294757+0000","last_fullsized":"2024-09-13T08:47:06.294757+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'81","last_scrub_stamp":"2024-09-13T08:47:06.294617+0000","last_deep_scrub":"181'81","last_deep_scrub_stamp":"2024-09-13T08:47:06.294617+0000","last_clean_scrub_stamp":"2024-09-13T08:47:06.294617+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:19.326317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37580,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334260,"statfs":{"total":96636764160,"available":96598282240,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37576,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334264,"statfs":{"total":96636764160,"available":96598286336,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":2875392,"data_stored":1601670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41740,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330100,"statfs":{"total":96636764160,"available":96594022400,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33420,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338420,"statfs":{"total":96636764160,"available":96602542080,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34000,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337840,"statfs":{"total":96636764160,"available":96601948160,"internally_reserved":0,"allocated":2875392,"data_stored":1601670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T08:47:09.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.954+0000 7f299ca00640 1 -- 172.21.15.31:0/562152942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f299815bc00 msgr2=0x7f2998073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.954+0000 7f299ca00640 1 --2- 172.21.15.31:0/562152942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f299815bc00 0x7f2998073ac0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f2988009900 tx=0x7f298802a580 comp rx=0 tx=0).stop 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 -- 172.21.15.31:0/562152942 shutdown_connections 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 --2- 172.21.15.31:0/562152942 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f29980b2820 0x7f299806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 --2- 172.21.15.31:0/562152942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f299815bc00 0x7f2998073ac0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 -- 172.21.15.31:0/562152942 >> 172.21.15.31:0/562152942 conn(0x7f29980b1520 msgr2=0x7f29980b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 -- 172.21.15.31:0/562152942 shutdown_connections 2024-09-13T08:47:09.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.955+0000 7f299ca00640 1 -- 172.21.15.31:0/562152942 wait complete. 2024-09-13T08:47:09.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.956+0000 7f299ca00640 1 Processor -- start 2024-09-13T08:47:09.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.957+0000 7f299ca00640 1 -- start start 2024-09-13T08:47:09.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.957+0000 7f299ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 0x7f29980c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.957+0000 7f299ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.957+0000 7f299ca00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 0x7f29981f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f299ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29980ccbc0 con 0x7f29980cc3e0 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f299ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29980cca40 con 0x7f29980b2820 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f299ca00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f29980ccd40 con 0x7f299815bc00 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53976/0 (socket says 172.21.15.31:53976) 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 -- 172.21.15.31:0/3403312807 learned_addr learned my addr 172.21.15.31:0/3403312807 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f2996800640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 0x7f29980c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f2997200640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 0x7f29981f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 -- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 msgr2=0x7f29981f84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 0x7f29981f84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 -- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 msgr2=0x7f29980c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 0x7f29980c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 -- 172.21.15.31:0/3403312807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29980c6280 con 0x7f29980cc3e0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f2997200640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 0x7f29981f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f2996800640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 0x7f29980c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.958+0000 7f298fe00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f298000d770 tx=0x7f298000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.959+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980017070 con 0x7f29980cc3e0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.959+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2980009d20 con 0x7f29980cc3e0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.959+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980005280 con 0x7f29980cc3e0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.959+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29980b2f20 con 0x7f29980cc3e0 2024-09-13T08:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.959+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29980b34e0 con 0x7f29980cc3e0 2024-09-13T08:47:09.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.960+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29980cb370 con 0x7f29980cc3e0 2024-09-13T08:47:09.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.961+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f2980005420 con 0x7f29980cc3e0 2024-09-13T08:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.962+0000 7f298ea00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 0x7f297407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.962+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f2980099720 con 0x7f29980cc3e0 2024-09-13T08:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.962+0000 7f2996800640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 0x7f297407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.963+0000 7f2996800640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 0x7f297407b8e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2988033000 tx=0x7f2988004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:09.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:09.967+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f2980063580 con 0x7f29980cc3e0 2024-09-13T08:47:10.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.164+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f29980b36b0 con 0x7f29980cc3e0 2024-09-13T08:47:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.168+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v50)=0 v50) ==== 123+0+0 (secure 0 0 0) 0x7f2980062e80 con 0x7f29980cc3e0 2024-09-13T08:47:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.170+0000 7f298ea00640 1 -- 172.21.15.31:0/3403312807 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f2980062990 con 0x7f29980cc3e0 2024-09-13T08:47:10.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 msgr2=0x7f297407b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 0x7f297407b8e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2988033000 tx=0x7f2988004730 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 msgr2=0x7f29981f7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f298000d770 tx=0x7f298000dc30 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 shutdown_connections 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f2974079420 0x7f297407b8e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f299815bc00 0x7f29981f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29980cc3e0 0x7f29981f7fb0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 --2- 172.21.15.31:0/3403312807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29980b2820 0x7f29980c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 >> 172.21.15.31:0/3403312807 conn(0x7f29980b1520 msgr2=0x7f29980cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 shutdown_connections 2024-09-13T08:47:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.171+0000 7f299ca00640 1 -- 172.21.15.31:0/3403312807 wait complete. 2024-09-13T08:47:10.190 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T08:47:10.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T08:47:10.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 -- 172.21.15.31:0/2029595275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 msgr2=0x7f96a40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 --2- 172.21.15.31:0/2029595275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a40cf850 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f96a40ca470 tx=0x7f969402a560 comp rx=0 tx=0).stop 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 -- 172.21.15.31:0/2029595275 shutdown_connections 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 --2- 172.21.15.31:0/2029595275 >> v2:172.21.15.45:3300/0 conn(0x7f96a415fe20 0x7f96a40cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 --2- 172.21.15.31:0/2029595275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a40cf850 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 -- 172.21.15.31:0/2029595275 >> 172.21.15.31:0/2029595275 conn(0x7f96a40cc910 msgr2=0x7f96a40cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 -- 172.21.15.31:0/2029595275 shutdown_connections 2024-09-13T08:47:10.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.350+0000 7f96a9a00640 1 -- 172.21.15.31:0/2029595275 wait complete. 2024-09-13T08:47:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.352+0000 7f96a9a00640 1 Processor -- start 2024-09-13T08:47:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.352+0000 7f96a9a00640 1 -- start start 2024-09-13T08:47:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96a415f9e0 0x7f96a409c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 --2- >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96a415fe20 0x7f96a409d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96a40cec50 con 0x7f96a415bd70 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96a40cead0 con 0x7f96a415f9e0 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a9a00640 1 -- --> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] -- mon_getmap magic: 0 -- 0x7f96a40cb850 con 0x7f96a415fe20 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53980/0 (socket says 172.21.15.31:53980) 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a2a00640 1 -- 172.21.15.31:0/339861058 learned_addr learned my addr 172.21.15.31:0/339861058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f969a000640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96a415f9e0 0x7f96a409c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.353+0000 7f96a3400640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96a415fe20 0x7f96a409d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 -- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96a415fe20 msgr2=0x7f96a409d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96a415fe20 0x7f96a409d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 -- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96a415f9e0 msgr2=0x7f96a409c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96a415f9e0 0x7f96a409c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 -- 172.21.15.31:0/339861058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a409aea0 con 0x7f96a415bd70 2024-09-13T08:47:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a2a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f9694009850 tx=0x7f9694003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969403f070 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f9694004a40 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a40b3100 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.354+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694035030 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.355+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a41594d0 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.356+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96a40cc450 con 0x7f96a415bd70 2024-09-13T08:47:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.357+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 102358+0+0 (secure 0 0 0) 0x7f9694042070 con 0x7f96a415bd70 2024-09-13T08:47:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.358+0000 7f969b400640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 0x7f968007b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T08:47:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.358+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 4987+0+0 (secure 0 0 0) 0x7f96940ba4e0 con 0x7f96a415bd70 2024-09-13T08:47:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.358+0000 7f969a000640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 0x7f968007b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T08:47:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.359+0000 7f969a000640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 0x7f968007b8e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a409e090 tx=0x7f968c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T08:47:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.362+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196643 (secure 0 0 0) 0x7f96940c1070 con 0x7f96a415bd70 2024-09-13T08:47:10.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.594+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96a40cbdb0 con 0x7f96a415bd70 2024-09-13T08:47:10.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.594+0000 7f969b400640 1 -- 172.21.15.31:0/339861058 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3913 (secure 0 0 0) 0x7f9694083cf0 con 0x7f96a415bd70 2024-09-13T08:47:10.596 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-13T08:47:10.597 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":121,"btime":"2024-09-13T08:46:57:015813+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7640,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1301248350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1301248350},{"type":"v1","addr":"172.21.15.45:6835","nonce":1301248350}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":8584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6801/1635479656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1635479656},{"type":"v1","addr":"172.21.15.45:6801","nonce":1635479656}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114},{"gid":9195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/160961409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":160961409},{"type":"v1","addr":"172.21.15.31:6834","nonce":160961409}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":113},{"gid":9483,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1108415823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":1108415823},{"type":"v1","addr":"172.21.15.31:6835","nonce":1108415823}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[]} 2024-09-13T08:47:10.597 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 121 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 msgr2=0x7f968007b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 0x7f968007b8e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a409e090 tx=0x7f968c009040 comp rx=0 tx=0).stop 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 msgr2=0x7f96a41f88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f9694009850 tx=0x7f9694003c10 comp rx=0 tx=0).stop 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 shutdown_connections 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:6836/1500367474,v1:172.21.15.31:6837/1500367474] conn(0x7f9680079420 0x7f968007b8e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] conn(0x7f96a415fe20 0x7f96a409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96a415f9e0 0x7f96a409c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 --2- 172.21.15.31:0/339861058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96a415bd70 0x7f96a41f88f0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T08:47:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 >> 172.21.15.31:0/339861058 conn(0x7f96a40cc910 msgr2=0x7f96a40ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T08:47:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 shutdown_connections 2024-09-13T08:47:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-13T08:47:10.597+0000 7f96a9a00640 1 -- 172.21.15.31:0/339861058 wait complete. 2024-09-13T08:47:10.614 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-13T08:47:10.615 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T08:47:10.615 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T08:47:10.615 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.629 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T08:47:10.629 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T08:47:10.629 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.684 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T08:47:10.684 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T08:47:10.684 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.701 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T08:47:10.701 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T08:47:10.702 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.760 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T08:47:10.760 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T08:47:10.760 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T08:47:10.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.807 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T08:47:10.807 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T08:47:10.807 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.867 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T08:47:10.868 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T08:47:10.868 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.909 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T08:47:10.909 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T08:47:10.910 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:10.968 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T08:47:10.968 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T08:47:10.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.022 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T08:47:11.022 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T08:47:11.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.075 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T08:47:11.075 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T08:47:11.075 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.135 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T08:47:11.135 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T08:47:11.136 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.188 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T08:47:11.188 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T08:47:11.189 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T08:47:11.189 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.264 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T08:47:11.265 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T08:47:11.265 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.332 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T08:47:11.332 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T08:47:11.333 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T08:47:11.333 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.356 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T08:47:11.356 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T08:47:11.356 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.376 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T08:47:11.376 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T08:47:11.376 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T08:47:11.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-13T08:47:11.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-13T08:47:11.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-13T08:47:11.437 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T08:47:11.437 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T08:47:11.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-09-13T08:47:11.525 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:47:11.525 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T08:47:11.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:47:11.920 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T08:47:12.130 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:47:12.130 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T08:47:12.229 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:47:12.230 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T08:47:12.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:47:12.339 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T08:47:12.564 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:47:12.565 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T08:47:12.699 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:47:12.699 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T08:47:12.805 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:47:12.805 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T08:47:12.912 INFO:tasks.ceph:Archiving mon data... 2024-09-13T08:47:12.914 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/data/mon.a.tgz 2024-09-13T08:47:12.914 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-13T08:47:12.951 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.bSAy4eZSEF 2024-09-13T08:47:12.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bSAy4eZSEF 2024-09-13T08:47:13.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.bSAy4eZSEF 2024-09-13T08:47:13.361 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.bSAy4eZSEF is 1MB 2024-09-13T08:47:13.465 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.bSAy4eZSEF 2024-09-13T08:47:13.501 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/data/mon.b.tgz 2024-09-13T08:47:13.501 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-13T08:47:13.537 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.vdcnRViKBI 2024-09-13T08:47:13.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vdcnRViKBI 2024-09-13T08:47:13.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.vdcnRViKBI 2024-09-13T08:47:13.951 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.vdcnRViKBI is 1MB 2024-09-13T08:47:14.054 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.vdcnRViKBI 2024-09-13T08:47:14.089 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/data/mon.c.tgz 2024-09-13T08:47:14.089 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-13T08:47:14.163 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.9PSfgFXVZC 2024-09-13T08:47:14.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9PSfgFXVZC 2024-09-13T08:47:14.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.9PSfgFXVZC 2024-09-13T08:47:14.568 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.9PSfgFXVZC is 1MB 2024-09-13T08:47:14.681 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.9PSfgFXVZC 2024-09-13T08:47:14.716 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T08:47:14.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T08:47:14.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T08:47:14.854 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T08:47:14.889 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T08:47:14.890 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T08:47:14.891 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi031/crash 2024-09-13T08:47:14.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T08:47:14.952 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi045/crash 2024-09-13T08:47:14.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T08:47:15.007 INFO:tasks.ceph:Shutting down logrotate 2024-09-13T08:47:15.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:15.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:15.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:15.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T08:47:15.227 INFO:tasks.ceph:Compressing logs... 2024-09-13T08:47:15.228 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T08:47:15.232 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T08:47:15.282 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T08:47:15.283 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51863.log 2024-09-13T08:47:15.284 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82069.log 2024-09-13T08:47:15.285 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66774.log 2024-09-13T08:47:15.285 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51863.log.gz 2024-09-13T08:47:15.286 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54615.log 2024-09-13T08:47:15.286 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35743.log 2024-09-13T08:47:15.287 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.82069.log.gz 2024-09-13T08:47:15.287 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66774.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66774.log.gz 2024-09-13T08:47:15.287 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67381.log 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47133.log 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54615.log.gz 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66817.log 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67381.log.gz 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43063.log 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43606.log 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35743.log.gz 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.43063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43063.log.gz 2024-09-13T08:47:15.289 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35863.log 2024-09-13T08:47:15.291 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.43606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43606.log.gz 2024-09-13T08:47:15.291 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-13T08:47:15.291 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35863.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35863.log.gz 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35983.log 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44862.log 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47133.log.gz 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60867.log 2024-09-13T08:47:15.292 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66817.log.gz 2024-09-13T08:47:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.2.44862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.44862.log.gz 2024-09-13T08:47:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61422.log 2024-09-13T08:47:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53010.log 2024-09-13T08:47:15.293 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60867.log.gz 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61422.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61422.log.gz 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63606.log 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73817.log 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53010.log.gz 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69300.log 2024-09-13T08:47:15.295 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63606.log.gz 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46144.log 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35983.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35983.log.gz 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.46144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46144.log.gz 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.40428.log 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.40428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.40428.log.gz 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T08:47:15.297 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42529.log 2024-09-13T08:47:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35903.log 2024-09-13T08:47:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.42529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42529.log.gz 2024-09-13T08:47:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62615.log 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73817.log.gz 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69300.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69300.log.gz 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46426.log 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51821.log 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.62615.log.gz 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61380.log 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46426.log.gz 2024-09-13T08:47:15.301 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61296.log 2024-09-13T08:47:15.302 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51821.log.gz 2024-09-13T08:47:15.302 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61380.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61380.log.gz 2024-09-13T08:47:15.302 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57853.log 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35823.log 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35903.log.gz 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46642.log 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.40843.log 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.46642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46642.log.gz 2024-09-13T08:47:15.306 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-13T08:47:15.307 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.3.40843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.40843.log.gz 2024-09-13T08:47:15.307 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45155.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68659.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61296.log.gz 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36486.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57853.log.gz 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56322.log 2024-09-13T08:47:15.309 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68659.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36486.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.36486.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67158.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47175.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56322.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67158.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67158.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45224.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47175.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58965.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57955.log 2024-09-13T08:47:15.310 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45224.log.gz 2024-09-13T08:47:15.315 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.1.45155.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-13T08:47:15.315 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.1.45155.log.gz 2024-09-13T08:47:15.315 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T08:47:15.315 INFO:teuthology.orchestra.run.smithi045.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T08:47:15.315 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45652.log 2024-09-13T08:47:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81673.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58965.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47073.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57955.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76363.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81673.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71999.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.47073.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76363.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76363.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73592.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75892.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.71999.log.gz 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43438.log 2024-09-13T08:47:15.317 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73592.log.gz 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53476.log 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75892.log.gz 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43080.log 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43438.log.gz 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-client.1.45652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45652.log.gz 2024-09-13T08:47:15.318 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T08:47:15.319 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65919.log 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53476.log.gz 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81421.log 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43080.log.gz 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36169.log 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65919.log.gz 2024-09-13T08:47:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44404.log 2024-09-13T08:47:15.321 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81421.log.gz 2024-09-13T08:47:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57369.log 2024-09-13T08:47:15.321 INFO:teuthology.orchestra.run.smithi031.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.36169.log.gz 2024-09-13T08:47:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.44404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44404.log.gz 2024-09-13T08:47:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78706.log 2024-09-13T08:47:15.323 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50945.log 2024-09-13T08:47:15.323 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57369.log.gz 2024-09-13T08:47:15.324 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43877.log 2024-09-13T08:47:15.324 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78706.log.gz 2024-09-13T08:47:15.324 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44674.log 2024-09-13T08:47:15.325 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66522.log 2024-09-13T08:47:15.325 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50945.log.gz 2024-09-13T08:47:15.325 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42533.log 2024-09-13T08:47:15.326 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43877.log.gz 2024-09-13T08:47:15.326 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62963.log 2024-09-13T08:47:15.326 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66522.log.gz 2024-09-13T08:47:15.327 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35783.log 2024-09-13T08:47:15.328 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.44674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44674.log.gz 2024-09-13T08:47:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69570.log 2024-09-13T08:47:15.328 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42533.log.gz 2024-09-13T08:47:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61674.log 2024-09-13T08:47:15.329 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.62963.log.gz 2024-09-13T08:47:15.329 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53626.log 2024-09-13T08:47:15.329 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69570.log.gz 2024-09-13T08:47:15.330 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80707.log 2024-09-13T08:47:15.330 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61674.log.gz 2024-09-13T08:47:15.331 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73165.log 2024-09-13T08:47:15.331 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53626.log.gz 2024-09-13T08:47:15.332 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75243.log 2024-09-13T08:47:15.332 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80707.log.gz 2024-09-13T08:47:15.333 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69342.log 2024-09-13T08:47:15.333 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73165.log.gz 2024-09-13T08:47:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35783.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35783.log.gz 2024-09-13T08:47:15.334 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75243.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75243.log.gz 2024-09-13T08:47:15.334 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69258.log 2024-09-13T08:47:15.335 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57998.log 2024-09-13T08:47:15.335 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69342.log.gz 2024-09-13T08:47:15.335 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41438.log 2024-09-13T08:47:15.336 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82151.log 2024-09-13T08:47:15.336 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69258.log.gz 2024-09-13T08:47:15.336 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58337.log 2024-09-13T08:47:15.337 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57998.log.gz 2024-09-13T08:47:15.337 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82151.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82151.log.gz 2024-09-13T08:47:15.337 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58432.log 2024-09-13T08:47:15.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43067.log 2024-09-13T08:47:15.339 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41967.log 2024-09-13T08:47:15.339 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58337.log.gz 2024-09-13T08:47:15.340 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-09-13T08:47:15.340 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58432.log.gz 2024-09-13T08:47:15.340 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.41438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41438.log.gz 2024-09-13T08:47:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48624.log 2024-09-13T08:47:15.341 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43067.log.gz 2024-09-13T08:47:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47031.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-09-13T08:47:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53518.log 2024-09-13T08:47:15.342 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43304.log 2024-09-13T08:47:15.342 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48624.log.gz 2024-09-13T08:47:15.343 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44133.log 2024-09-13T08:47:15.343 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53434.log 2024-09-13T08:47:15.343 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53518.log.gz 2024-09-13T08:47:15.344 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.41967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41967.log.gz 2024-09-13T08:47:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76843.log 2024-09-13T08:47:15.345 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43304.log.gz 2024-09-13T08:47:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51397.log 2024-09-13T08:47:15.345 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53434.log.gz 2024-09-13T08:47:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79155.log 2024-09-13T08:47:15.347 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36023.log 2024-09-13T08:47:15.347 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.1.44133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44133.log.gz 2024-09-13T08:47:15.347 INFO:teuthology.orchestra.run.smithi031.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76843.log.gz 2024-09-13T08:47:15.347 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43354.log 2024-09-13T08:47:15.347 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51397.log.gz 2024-09-13T08:47:15.348 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67637.log 2024-09-13T08:47:15.348 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79155.log.gz 2024-09-13T08:47:15.349 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50052.log 2024-09-13T08:47:15.349 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43354.log.gz 2024-09-13T08:47:15.349 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57769.log 2024-09-13T08:47:15.349 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67637.log.gz 2024-09-13T08:47:15.350 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.36023.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36023.log.gz 2024-09-13T08:47:15.351 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.50052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50052.log.gz 2024-09-13T08:47:15.351 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80403.log 2024-09-13T08:47:15.351 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49504.log 2024-09-13T08:47:15.351 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57769.log.gz 2024-09-13T08:47:15.352 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61464.log 2024-09-13T08:47:15.352 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80403.log.gz 2024-09-13T08:47:15.353 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57811.log 2024-09-13T08:47:15.353 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49504.log.gz 2024-09-13T08:47:15.353 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75808.log 2024-09-13T08:47:15.354 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61464.log.gz 2024-09-13T08:47:15.354 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68847.log 2024-09-13T08:47:15.354 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57811.log.gz 2024-09-13T08:47:15.355 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36891.log 2024-09-13T08:47:15.355 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75808.log.gz 2024-09-13T08:47:15.356 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69697.log 2024-09-13T08:47:15.356 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68847.log.gz 2024-09-13T08:47:15.357 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75682.log 2024-09-13T08:47:15.357 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36891.log.gz 2024-09-13T08:47:15.358 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57895.log 2024-09-13T08:47:15.358 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69697.log.gz 2024-09-13T08:47:15.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2024-09-13T08:47:15.359 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75682.log.gz 2024-09-13T08:47:15.360 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36067.log 2024-09-13T08:47:15.360 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57895.log.gz 2024-09-13T08:47:15.361 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51779.log 2024-09-13T08:47:15.361 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2024-09-13T08:47:15.361 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36067.log: 77.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.36067.log.gz -5 --verbose 2024-09-13T08:47:15.361 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.41773.log 2024-09-13T08:47:15.362 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43082.log 2024-09-13T08:47:15.362 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51779.log.gz 2024-09-13T08:47:15.363 INFO:teuthology.orchestra.run.smithi045.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-13T08:47:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73123.log 2024-09-13T08:47:15.364 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41773.log.gz 2024-09-13T08:47:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56070.log 2024-09-13T08:47:15.364 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43082.log.gz 2024-09-13T08:47:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73123.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73123.log.gz 2024-09-13T08:47:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74195.log 2024-09-13T08:47:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58923.log 2024-09-13T08:47:15.366 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56070.log.gz 2024-09-13T08:47:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74764.log 2024-09-13T08:47:15.367 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74195.log.gz 2024-09-13T08:47:15.368 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80481.log 2024-09-13T08:47:15.368 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58923.log.gz 2024-09-13T08:47:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74764.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74764.log.gz 2024-09-13T08:47:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2024-09-13T08:47:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41097.log 2024-09-13T08:47:15.369 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80481.log.gz 2024-09-13T08:47:15.370 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77624.log 2024-09-13T08:47:15.370 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2024-09-13T08:47:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53182.log 2024-09-13T08:47:15.371 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2024-09-13T08:47:15.372 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-09-13T08:47:15.372 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.77624.log.gz 2024-09-13T08:47:15.373 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82027.log 2024-09-13T08:47:15.373 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53182.log.gz 2024-09-13T08:47:15.374 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56508.log 2024-09-13T08:47:15.374 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2024-09-13T08:47:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82027.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.82027.log.gz 2024-09-13T08:47:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T08:47:15.375 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75934.log 2024-09-13T08:47:15.376 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.56508.log.gz 2024-09-13T08:47:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61506.log 2024-09-13T08:47:15.377 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63921.log 2024-09-13T08:47:15.378 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75934.log.gz 2024-09-13T08:47:15.378 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61506.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61506.log.gz 2024-09-13T08:47:15.379 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59007.log 2024-09-13T08:47:15.380 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76029.log 2024-09-13T08:47:15.380 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63921.log.gz 2024-09-13T08:47:15.381 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59007.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59007.log.gz 2024-09-13T08:47:15.381 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65811.log 2024-09-13T08:47:15.382 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-09-13T08:47:15.382 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76029.log.gz 2024-09-13T08:47:15.383 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65811.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65811.log.gz 2024-09-13T08:47:15.383 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2024-09-13T08:47:15.384 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43426.log 2024-09-13T08:47:15.384 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-09-13T08:47:15.385 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50451.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2024-09-13T08:47:15.385 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73548.log 2024-09-13T08:47:15.386 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74045.log 2024-09-13T08:47:15.386 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43426.log.gz 2024-09-13T08:47:15.387 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73548.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73548.log.gz 2024-09-13T08:47:15.387 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-09-13T08:47:15.388 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77803.log 2024-09-13T08:47:15.388 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74045.log.gz 2024-09-13T08:47:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41631.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-09-13T08:47:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46552.log 2024-09-13T08:47:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66210.log 2024-09-13T08:47:15.390 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77803.log.gz 2024-09-13T08:47:15.391 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46552.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46552.log.gz 2024-09-13T08:47:15.391 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48437.log 2024-09-13T08:47:15.392 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74668.log 2024-09-13T08:47:15.392 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66210.log.gz 2024-09-13T08:47:15.393 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70727.log 2024-09-13T08:47:15.393 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.48437.log.gz 2024-09-13T08:47:15.394 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46468.log 2024-09-13T08:47:15.394 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.74668.log.gz 2024-09-13T08:47:15.395 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70727.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70727.log.gz 2024-09-13T08:47:15.395 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56280.log 2024-09-13T08:47:15.396 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52031.log 2024-09-13T08:47:15.397 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46468.log.gz 2024-09-13T08:47:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56280.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56280.log.gz 2024-09-13T08:47:15.398 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29863.log 2024-09-13T08:47:15.398 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75327.log 2024-09-13T08:47:15.399 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52031.log.gz 2024-09-13T08:47:15.399 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.29863.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29863.log.gz 2024-09-13T08:47:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65708.log 2024-09-13T08:47:15.401 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51989.log 2024-09-13T08:47:15.401 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75327.log.gz 2024-09-13T08:47:15.402 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65708.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65708.log.gz 2024-09-13T08:47:15.402 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78786.log 2024-09-13T08:47:15.403 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66954.log 2024-09-13T08:47:15.403 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51989.log.gz 2024-09-13T08:47:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78786.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78786.log.gz 2024-09-13T08:47:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70601.log 2024-09-13T08:47:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79600.log 2024-09-13T08:47:15.405 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66954.log.gz 2024-09-13T08:47:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36287.log 2024-09-13T08:47:15.406 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.70601.log.gz 2024-09-13T08:47:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71659.log 2024-09-13T08:47:15.407 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79600.log.gz 2024-09-13T08:47:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60615.log 2024-09-13T08:47:15.408 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36287.log.gz 2024-09-13T08:47:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71659.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71659.log.gz 2024-09-13T08:47:15.409 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43919.log 2024-09-13T08:47:15.410 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81463.log 87.9% 2024-09-13T08:47:15.410 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60615.log.gz 2024-09-13T08:47:15.411 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45970.log 2024-09-13T08:47:15.411 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43919.log.gz 2024-09-13T08:47:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81463.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81463.log.gz 2024-09-13T08:47:15.412 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65624.log 2024-09-13T08:47:15.413 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45970.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.45970.log.gz 2024-09-13T08:47:15.413 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66378.log 2024-09-13T08:47:15.414 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65624.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.65624.log.gz 2024-09-13T08:47:15.414 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54322.log 2024-09-13T08:47:15.415 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66378.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66378.log.gz 2024-09-13T08:47:15.415 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60021.log 2024-09-13T08:47:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54322.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54322.log.gz 2024-09-13T08:47:15.416 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50257.log 2024-09-13T08:47:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60021.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60021.log.gz 2024-09-13T08:47:15.417 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46384.log 2024-09-13T08:47:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59676.log 2024-09-13T08:47:15.418 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50257.log.gz 2024-09-13T08:47:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46384.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46384.log.gz 2024-09-13T08:47:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-09-13T08:47:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59676.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59676.log.gz 2024-09-13T08:47:15.419 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55238.log 2024-09-13T08:47:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42197.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-09-13T08:47:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69426.log 2024-09-13T08:47:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55238.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55238.log.gz 2024-09-13T08:47:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69655.log 2024-09-13T08:47:15.422 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61548.log 2024-09-13T08:47:15.422 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69426.log.gz 2024-09-13T08:47:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69655.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.69655.log.gz --verbose 2024-09-13T08:47:15.423 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.66110.log 2024-09-13T08:47:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61548.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61548.log.gz 2024-09-13T08:47:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2024-09-13T08:47:15.425 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66110.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66110.log.gz 2024-09-13T08:47:15.425 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71312.log 2024-09-13T08:47:15.426 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50409.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.75185.log -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2024-09-13T08:47:15.426 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.427 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71312.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71312.log.gz 2024-09-13T08:47:15.427 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47954.log 2024-09-13T08:47:15.427 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75185.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75185.log.gz 2024-09-13T08:47:15.428 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49714.log 2024-09-13T08:47:15.429 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47954.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47954.log.gz 2024-09-13T08:47:15.429 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68138.log 2024-09-13T08:47:15.430 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49714.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49714.log.gz 2024-09-13T08:47:15.430 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2024-09-13T08:47:15.431 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68138.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68138.log.gz 2024-09-13T08:47:15.431 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54019.log 2024-09-13T08:47:15.431 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46636.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gz 2024-09-13T08:47:15.431 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71700.log 2024-09-13T08:47:15.432 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54019.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.54019.log.gz 2024-09-13T08:47:15.432 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2024-09-13T08:47:15.433 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71700.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.71700.log.gz 2024-09-13T08:47:15.433 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71354.log 2024-09-13T08:47:15.434 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47676.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2024-09-13T08:47:15.434 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63564.log 2024-09-13T08:47:15.435 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71354.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.71354.log.gz 2024-09-13T08:47:15.435 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61590.log 2024-09-13T08:47:15.436 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63564.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63564.log.gz 2024-09-13T08:47:15.436 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63354.log 2024-09-13T08:47:15.437 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61590.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.61590.log.gz 2024-09-13T08:47:15.437 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49323.log 2024-09-13T08:47:15.438 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68096.log 2024-09-13T08:47:15.438 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63354.log.gz 2024-09-13T08:47:15.439 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49323.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49323.log.gz 2024-09-13T08:47:15.439 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60157.log 2024-09-13T08:47:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68096.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68096.log.gz 2024-09-13T08:47:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52727.log 2024-09-13T08:47:15.441 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60157.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60157.log.gzgzip 2024-09-13T08:47:15.441 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58654.log 2024-09-13T08:47:15.442 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52727.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52727.log.gz 2024-09-13T08:47:15.442 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46054.log 2024-09-13T08:47:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58654.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58654.log.gzgzip 2024-09-13T08:47:15.443 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67034.log 2024-09-13T08:47:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46054.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46054.log.gz 2024-09-13T08:47:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79281.log 2024-09-13T08:47:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67034.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67034.log.gz 2024-09-13T08:47:15.444 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60367.log 2024-09-13T08:47:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79281.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79281.log.gz 2024-09-13T08:47:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67511.log 2024-09-13T08:47:15.446 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60367.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60367.log.gz 2024-09-13T08:47:15.446 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55196.log 2024-09-13T08:47:15.447 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67511.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.0.68017.log 2024-09-13T08:47:15.447 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67511.log.gz 2024-09-13T08:47:15.448 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55196.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55196.log.gz 2024-09-13T08:47:15.448 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76489.log 2024-09-13T08:47:15.449 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.68017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.68017.log.gz 2024-09-13T08:47:15.449 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49261.log 2024-09-13T08:47:15.450 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76489.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76489.log.gz 2024-09-13T08:47:15.450 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61252.log 2024-09-13T08:47:15.451 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49261.log: gzip -5 --verbose 87.1% -- -- replaced with /var/log/ceph/ceph-client.admin.49261.log.gz 2024-09-13T08:47:15.451 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.61338.log 2024-09-13T08:47:15.452 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61252.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.61252.log.gz 2024-09-13T08:47:15.452 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55468.log 2024-09-13T08:47:15.452 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61338.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61338.log.gz -5 2024-09-13T08:47:15.453 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53266.log 2024-09-13T08:47:15.453 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55468.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55468.log.gz 2024-09-13T08:47:15.454 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78996.log 2024-09-13T08:47:15.454 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53266.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53266.log.gz 2024-09-13T08:47:15.455 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46510.log 2024-09-13T08:47:15.455 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78996.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.78996.log.gz 2024-09-13T08:47:15.455 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59049.log 2024-09-13T08:47:15.456 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46510.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46510.log.gz 2024-09-13T08:47:15.457 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47217.log 2024-09-13T08:47:15.457 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59049.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59049.log.gz 2024-09-13T08:47:15.457 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54116.log 2024-09-13T08:47:15.458 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62573.log 2024-09-13T08:47:15.459 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47217.log.gz 2024-09-13T08:47:15.459 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54116.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54116.log.gz 2024-09-13T08:47:15.459 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72768.log 2024-09-13T08:47:15.460 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62573.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62573.log.gz 2024-09-13T08:47:15.461 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58612.log 2024-09-13T08:47:15.461 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77302.log 2024-09-13T08:47:15.461 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72768.log.gz 2024-09-13T08:47:15.462 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58612.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.58612.log.gz 2024-09-13T08:47:15.462 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78952.log 2024-09-13T08:47:15.463 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77302.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.77302.log.gz 2024-09-13T08:47:15.463 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.71107.log 2024-09-13T08:47:15.464 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78952.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78952.log.gzgzip 2024-09-13T08:47:15.464 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81211.log 2024-09-13T08:47:15.464 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.71107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.71107.log.gz 2024-09-13T08:47:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78401.log 2024-09-13T08:47:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81211.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81211.log.gz 2024-09-13T08:47:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44558.log 2024-09-13T08:47:15.467 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78401.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.78401.log.gz 2024-09-13T08:47:15.467 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71741.log 2024-09-13T08:47:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44558.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44558.log.gz 2024-09-13T08:47:15.468 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52218.log 2024-09-13T08:47:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71741.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.71741.log.gz 2024-09-13T08:47:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78910.log 2024-09-13T08:47:15.470 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52218.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52218.log.gz 2024-09-13T08:47:15.470 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79365.log 2024-09-13T08:47:15.470 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78910.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78910.log.gz 2024-09-13T08:47:15.471 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56883.log 2024-09-13T08:47:15.471 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79365.log: gzip -5 87.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.79365.log.gz -- 2024-09-13T08:47:15.471 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.59592.log 2024-09-13T08:47:15.472 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56883.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56883.log.gz 2024-09-13T08:47:15.472 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69613.log 2024-09-13T08:47:15.473 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59592.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59592.log.gz 2024-09-13T08:47:15.473 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67553.log 2024-09-13T08:47:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69613.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69613.log.gz 2024-09-13T08:47:15.474 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54537.log 2024-09-13T08:47:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67553.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67553.log.gz 2024-09-13T08:47:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80308.log 2024-09-13T08:47:15.476 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54537.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54537.log.gz 2024-09-13T08:47:15.476 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68575.log 2024-09-13T08:47:15.477 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80308.log: gzip -5 --verbose 87.1% -- -- replaced with /var/log/ceph/ceph-client.admin.80308.log.gz /var/log/ceph/ceph-client.admin.42754.log 2024-09-13T08:47:15.477 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68575.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68575.log.gz 2024-09-13T08:47:15.478 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76405.log 2024-09-13T08:47:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42754.log: gzip -5 87.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42754.log.gz /var/log/ceph/ceph-client.admin.55717.log 2024-09-13T08:47:15.479 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76405.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76405.log.gz 2024-09-13T08:47:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73039.log 2024-09-13T08:47:15.481 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55717.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-09-13T08:47:15.481 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56154.log 2024-09-13T08:47:15.481 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73039.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73039.log.gzgzip 2024-09-13T08:47:15.481 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.47471.log 2024-09-13T08:47:15.482 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60825.log 2024-09-13T08:47:15.483 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56154.log.gz 2024-09-13T08:47:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.47471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47471.log.gz 2024-09-13T08:47:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49217.log 2024-09-13T08:47:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60825.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60825.log.gz 2024-09-13T08:47:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2024-09-13T08:47:15.485 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49217.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49217.log.gz 2024-09-13T08:47:15.485 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42407.log 2024-09-13T08:47:15.486 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50215.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gzgzip 2024-09-13T08:47:15.486 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60531.log 2024-09-13T08:47:15.487 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42407.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42407.log.gzgzip 2024-09-13T08:47:15.487 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55940.log 2024-09-13T08:47:15.488 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60531.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60531.log.gz -5 2024-09-13T08:47:15.488 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52302.log 2024-09-13T08:47:15.489 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55940.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55940.log.gz 2024-09-13T08:47:15.489 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54673.log 2024-09-13T08:47:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52302.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52302.log.gz 2024-09-13T08:47:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61758.log 2024-09-13T08:47:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54673.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54673.log.gz 2024-09-13T08:47:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62091.log 2024-09-13T08:47:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47283.log 2024-09-13T08:47:15.492 INFO:teuthology.orchestra.run.smithi031.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61758.log.gz 2024-09-13T08:47:15.493 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62091.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62091.log.gz 2024-09-13T08:47:15.493 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72642.log 2024-09-13T08:47:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47283.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47283.log.gz 2024-09-13T08:47:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71577.log 2024-09-13T08:47:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72642.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.72642.log.gz 2024-09-13T08:47:15.495 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78023.log 2024-09-13T08:47:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59513.log 2024-09-13T08:47:15.496 INFO:teuthology.orchestra.run.smithi031.stderr: 84.1% -- replaced with /var/log/ceph/ceph-client.admin.71577.log.gz 2024-09-13T08:47:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78023.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78023.log.gz 2024-09-13T08:47:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60487.log 2024-09-13T08:47:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.59513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59513.log.gz 2024-09-13T08:47:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79951.log 2024-09-13T08:47:15.498 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60487.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60487.log.gz 2024-09-13T08:47:15.499 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58839.log 2024-09-13T08:47:15.499 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79951.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79951.log.gz 2024-09-13T08:47:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81901.log 2024-09-13T08:47:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58839.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58839.log.gz 2024-09-13T08:47:15.500 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62217.log 2024-09-13T08:47:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81901.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.81901.log.gz 2024-09-13T08:47:15.501 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36690.log 2024-09-13T08:47:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63733.log 2024-09-13T08:47:15.502 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62217.log.gz 2024-09-13T08:47:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36690.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36690.log.gz 2024-09-13T08:47:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72913.log 2024-09-13T08:47:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63733.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.63733.log.gz 2024-09-13T08:47:15.504 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57632.log 2024-09-13T08:47:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72913.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72913.log.gz 2024-09-13T08:47:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76275.log 2024-09-13T08:47:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42756.log 2024-09-13T08:47:15.506 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57632.log.gz 2024-09-13T08:47:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76275.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76275.log.gz 2024-09-13T08:47:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49416.log 2024-09-13T08:47:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46012.log 2024-09-13T08:47:15.508 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42756.log.gz 2024-09-13T08:47:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49416.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49416.log.gz 2024-09-13T08:47:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78233.log 2024-09-13T08:47:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46012.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46012.log.gz 2024-09-13T08:47:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68222.log 2024-09-13T08:47:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78233.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78233.log.gz 2024-09-13T08:47:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79492.log 2024-09-13T08:47:15.511 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68222.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.63312.log -- replaced with /var/log/ceph/ceph-client.admin.68222.log.gz 2024-09-13T08:47:15.511 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79492.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79492.log.gz -5 2024-09-13T08:47:15.512 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49672.log 2024-09-13T08:47:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63312.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63312.log.gz 2024-09-13T08:47:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51131.log 2024-09-13T08:47:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49672.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.49672.log.gz 2024-09-13T08:47:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77981.log 2024-09-13T08:47:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57119.log 2024-09-13T08:47:15.515 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51131.log.gz 2024-09-13T08:47:15.516 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61207.log 2024-09-13T08:47:15.516 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77981.log.gz 2024-09-13T08:47:15.516 INFO:teuthology.orchestra.run.smithi045.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T08:47:15.517 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57119.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57119.log.gz 2024-09-13T08:47:15.517 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T08:47:15.518 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61207.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61207.log.gz 2024-09-13T08:47:15.518 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58740.log 2024-09-13T08:47:15.519 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55070.log 2024-09-13T08:47:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58740.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58740.log.gz 2024-09-13T08:47:15.520 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59175.log 2024-09-13T08:47:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55070.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55070.log.gz 2024-09-13T08:47:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54715.log 2024-09-13T08:47:15.522 INFO:teuthology.orchestra.run.smithi031.stderr: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz/var/log/ceph/ceph-client.admin.59175.log: 2024-09-13T08:47:15.522 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42071.log 2024-09-13T08:47:15.523 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.59175.log.gz 2024-09-13T08:47:15.523 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52134.log 2024-09-13T08:47:15.523 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54715.log.gz 2024-09-13T08:47:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42071.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42071.log.gz 2024-09-13T08:47:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74588.log 2024-09-13T08:47:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52134.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.52134.log.gz 2024-09-13T08:47:15.525 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2024-09-13T08:47:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74588.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74588.log.gz 2024-09-13T08:47:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71823.log 2024-09-13T08:47:15.527 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52811.log 87.2% 2024-09-13T08:47:15.527 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2024-09-13T08:47:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71823.log: gzip -5 --verbose -- 84.6% /var/log/ceph/ceph-client.admin.49374.log -- replaced with /var/log/ceph/ceph-client.admin.71823.log.gz 2024-09-13T08:47:15.528 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2024-09-13T08:47:15.529 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.52811.log.gz 2024-09-13T08:47:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49374.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49374.log.gzgzip 2024-09-13T08:47:15.530 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48792.log 2024-09-13T08:47:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50679.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2024-09-13T08:47:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79109.log 2024-09-13T08:47:15.532 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48792.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48792.log.gz 2024-09-13T08:47:15.532 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65666.log 2024-09-13T08:47:15.532 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79109.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79109.log.gz 2024-09-13T08:47:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62405.log 2024-09-13T08:47:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65666.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65666.log.gz 2024-09-13T08:47:15.534 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63144.log 2024-09-13T08:47:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62405.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62405.log.gz 2024-09-13T08:47:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76759.log 2024-09-13T08:47:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63144.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63144.log.gz 2024-09-13T08:47:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60099.log 2024-09-13T08:47:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48395.log 87.2% 2024-09-13T08:47:15.536 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76759.log.gz 2024-09-13T08:47:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60099.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60099.log.gz 2024-09-13T08:47:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81020.log 2024-09-13T08:47:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48395.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48395.log.gz 2024-09-13T08:47:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81715.log 2024-09-13T08:47:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81020.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.81020.log.gz 2024-09-13T08:47:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71228.log 2024-09-13T08:47:15.540 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81715.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81715.log.gz 2024-09-13T08:47:15.540 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60241.log 2024-09-13T08:47:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71228.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71228.log.gz 2024-09-13T08:47:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67339.log 2024-09-13T08:47:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60241.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60241.log.gz 2024-09-13T08:47:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36981.log 2024-09-13T08:47:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49630.log 2024-09-13T08:47:15.543 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67339.log.gz 2024-09-13T08:47:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62531.log 2024-09-13T08:47:15.544 INFO:teuthology.orchestra.run.smithi031.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.36981.log.gz 2024-09-13T08:47:15.545 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78611.log 2024-09-13T08:47:15.545 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49630.log.gz 2024-09-13T08:47:15.545 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62531.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.62531.log.gz 2024-09-13T08:47:15.546 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43422.log 2024-09-13T08:47:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78611.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78611.log.gz 2024-09-13T08:47:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53893.log 2024-09-13T08:47:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43422.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.43422.log.gz 2024-09-13T08:47:15.548 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67829.log 2024-09-13T08:47:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53893.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53893.log.gz 2024-09-13T08:47:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2024-09-13T08:47:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67829.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67829.log.gz 2024-09-13T08:47:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54841.log 2024-09-13T08:47:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60409.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2024-09-13T08:47:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52917.log 2024-09-13T08:47:15.551 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54841.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54841.log.gz 2024-09-13T08:47:15.551 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72726.log 2024-09-13T08:47:15.552 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52917.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.52917.log.gzgzip 2024-09-13T08:47:15.552 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68521.log 2024-09-13T08:47:15.553 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72726.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72726.log.gz 2024-09-13T08:47:15.553 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41859.log 2024-09-13T08:47:15.554 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77939.log 2024-09-13T08:47:15.554 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68521.log.gz 2024-09-13T08:47:15.555 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41859.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.41859.log.gz /var/log/ceph/ceph-client.admin.80936.log 2024-09-13T08:47:15.555 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.556 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77939.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77939.log.gz 2024-09-13T08:47:15.556 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77260.log 2024-09-13T08:47:15.557 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80936.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80936.log.gz 2024-09-13T08:47:15.557 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73775.log 2024-09-13T08:47:15.558 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77260.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77260.log.gz 2024-09-13T08:47:15.558 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59760.log 2024-09-13T08:47:15.558 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73775.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.73775.log.gz --verbose 2024-09-13T08:47:15.559 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.72828.log 2024-09-13T08:47:15.560 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78569.log 2024-09-13T08:47:15.560 INFO:teuthology.orchestra.run.smithi031.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.59760.log.gz 2024-09-13T08:47:15.560 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72828.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72828.log.gz 2024-09-13T08:47:15.560 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44057.log 2024-09-13T08:47:15.561 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78569.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78569.log.gz 2024-09-13T08:47:15.561 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74462.log 2024-09-13T08:47:15.562 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44057.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44057.log.gz 2024-09-13T08:47:15.562 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65750.log 2024-09-13T08:47:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74462.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74462.log.gz 2024-09-13T08:47:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57515.log 2024-09-13T08:47:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65750.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65750.log.gz 2024-09-13T08:47:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56574.log 2024-09-13T08:47:15.565 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57515.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.57515.log.gz 2024-09-13T08:47:15.565 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72684.log 2024-09-13T08:47:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56574.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56574.log.gz 2024-09-13T08:47:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56466.log 2024-09-13T08:47:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72684.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72684.log.gz 2024-09-13T08:47:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68889.log 2024-09-13T08:47:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72204.log 2024-09-13T08:47:15.568 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56466.log.gz 2024-09-13T08:47:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68889.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.68889.log.gz 2024-09-13T08:47:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56028.log 2024-09-13T08:47:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72204.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.72204.log.gz 2024-09-13T08:47:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56238.log 2024-09-13T08:47:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56028.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56028.log.gz 2024-09-13T08:47:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54280.log 2024-09-13T08:47:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56238.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56238.log.gz 2024-09-13T08:47:15.571 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74952.log 2024-09-13T08:47:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54280.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54280.log.gz 2024-09-13T08:47:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43348.log 2024-09-13T08:47:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74952.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74952.log.gz 2024-09-13T08:47:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70811.log 2024-09-13T08:47:15.574 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71905.log 2024-09-13T08:47:15.574 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43348.log.gz 2024-09-13T08:47:15.575 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71395.log 2024-09-13T08:47:15.575 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70811.log.gz 2024-09-13T08:47:15.576 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71905.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.71905.log.gz 2024-09-13T08:47:15.576 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61824.log 2024-09-13T08:47:15.577 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71395.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71395.log.gz 2024-09-13T08:47:15.577 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53560.log 2024-09-13T08:47:15.578 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61824.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.61824.log.gz 2024-09-13T08:47:15.578 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81985.log 2024-09-13T08:47:15.579 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61165.log 2024-09-13T08:47:15.579 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53560.log.gz 2024-09-13T08:47:15.580 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81985.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81985.log.gz 2024-09-13T08:47:15.580 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43093.log 2024-09-13T08:47:15.581 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61165.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61165.log.gzgzip 2024-09-13T08:47:15.581 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59217.log 2024-09-13T08:47:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60325.log 2024-09-13T08:47:15.582 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43093.log.gz 2024-09-13T08:47:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36499.log 2024-09-13T08:47:15.583 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59217.log.gz 2024-09-13T08:47:15.583 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60325.log: gzip -5 --verbose 86.9% -- -- replaced with /var/log/ceph/ceph-client.admin.60325.log.gz 2024-09-13T08:47:15.583 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.69823.log 2024-09-13T08:47:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36499.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36499.log.gz 2024-09-13T08:47:15.584 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50741.log 2024-09-13T08:47:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69823.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69823.log.gz 2024-09-13T08:47:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69949.log 2024-09-13T08:47:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74504.log 2024-09-13T08:47:15.586 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50741.log.gz 2024-09-13T08:47:15.587 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75850.log 87.2% 2024-09-13T08:47:15.587 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69949.log.gz 2024-09-13T08:47:15.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74504.log: gzip -5 87.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.74504.log.gz -- 2024-09-13T08:47:15.588 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.80108.log 2024-09-13T08:47:15.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75850.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75850.log.gz 2024-09-13T08:47:15.589 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66606.log 2024-09-13T08:47:15.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80108.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.80108.log.gz 2024-09-13T08:47:15.590 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81943.log 2024-09-13T08:47:15.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66606.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66606.log.gz 2024-09-13T08:47:15.591 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71186.log 2024-09-13T08:47:15.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81943.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81943.log.gz 2024-09-13T08:47:15.592 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59091.log 2024-09-13T08:47:15.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71186.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71186.log.gz 2024-09-13T08:47:15.593 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-09-13T08:47:15.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59091.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59091.log.gz 2024-09-13T08:47:15.593 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36479.log 2024-09-13T08:47:15.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45718.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-09-13T08:47:15.594 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76591.log 2024-09-13T08:47:15.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36479.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36479.log.gz 2024-09-13T08:47:15.595 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69009.log 2024-09-13T08:47:15.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76591.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76591.log.gzgzip 2024-09-13T08:47:15.596 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2024-09-13T08:47:15.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65521.log 87.4% 2024-09-13T08:47:15.597 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69009.log.gz 2024-09-13T08:47:15.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-09-13T08:47:15.598 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2024-09-13T08:47:15.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65521.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65521.log.gz 2024-09-13T08:47:15.599 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80665.log 2024-09-13T08:47:15.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36236.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-09-13T08:47:15.600 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45140.log 2024-09-13T08:47:15.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80665.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.80665.log.gz 2024-09-13T08:47:15.601 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48127.log 2024-09-13T08:47:15.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45140.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45140.log.gz 2024-09-13T08:47:15.601 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77176.log 2024-09-13T08:47:15.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48127.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.48127.log.gz 2024-09-13T08:47:15.602 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50513.log 2024-09-13T08:47:15.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77176.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77176.log.gz 2024-09-13T08:47:15.603 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67595.log 2024-09-13T08:47:15.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50513.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.50513.log.gz 2024-09-13T08:47:15.604 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67679.log 2024-09-13T08:47:15.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67595.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67595.log.gz 2024-09-13T08:47:15.605 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-13T08:47:15.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64067.log 2024-09-13T08:47:15.606 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67679.log.gz 2024-09-13T08:47:15.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63690.log 2024-09-13T08:47:15.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64067.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64067.log.gz 2024-09-13T08:47:15.608 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50821.log 2024-09-13T08:47:15.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63690.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63690.log.gz 2024-09-13T08:47:15.610 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70290.log 2024-09-13T08:47:15.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49051.log 2024-09-13T08:47:15.611 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50821.log.gz 2024-09-13T08:47:15.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70290.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70290.log.gz 2024-09-13T08:47:15.612 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52176.log 2024-09-13T08:47:15.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49051.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49051.log.gz 2024-09-13T08:47:15.613 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66252.log 2024-09-13T08:47:15.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81337.log 2024-09-13T08:47:15.614 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52176.log.gz 2024-09-13T08:47:15.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66252.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66252.log.gz 2024-09-13T08:47:15.615 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49756.log 2024-09-13T08:47:15.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81337.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81337.log.gz 2024-09-13T08:47:15.616 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76549.log 2024-09-13T08:47:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73506.log 2024-09-13T08:47:15.617 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49756.log.gz 2024-09-13T08:47:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42281.log 2024-09-13T08:47:15.619 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76549.log.gz 2024-09-13T08:47:15.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73506.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73506.log.gz 2024-09-13T08:47:15.620 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80581.log 2024-09-13T08:47:15.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55550.log 2024-09-13T08:47:15.621 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42281.log.gz 2024-09-13T08:47:15.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80581.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80581.log.gz 2024-09-13T08:47:15.622 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73901.log 2024-09-13T08:47:15.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76801.log 2024-09-13T08:47:15.623 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55550.log.gz 2024-09-13T08:47:15.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73901.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73901.log.gz 2024-09-13T08:47:15.624 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71864.log 2024-09-13T08:47:15.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56967.log 2024-09-13T08:47:15.626 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76801.log.gz 2024-09-13T08:47:15.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71864.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.71864.log.gz 2024-09-13T08:47:15.627 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75012.log 2024-09-13T08:47:15.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56967.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56967.log.gz 2024-09-13T08:47:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74722.log 2024-09-13T08:47:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48497.log 2024-09-13T08:47:15.629 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75012.log.gz 2024-09-13T08:47:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74722.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.74722.log.gz 2024-09-13T08:47:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43793.log 2024-09-13T08:47:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55322.log 2024-09-13T08:47:15.631 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48497.log.gz 2024-09-13T08:47:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43793.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43793.log.gz 2024-09-13T08:47:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80894.log 2024-09-13T08:47:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66480.log 2024-09-13T08:47:15.634 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55322.log.gz 2024-09-13T08:47:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80894.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80894.log.gz 2024-09-13T08:47:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43709.log 2024-09-13T08:47:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66480.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66480.log.gz 2024-09-13T08:47:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57041.log 2024-09-13T08:47:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41945.log 2024-09-13T08:47:15.641 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43709.log.gz 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57041.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57041.log.gz 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72997.log 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41945.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41945.log.gz 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77690.log 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76717.log 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72997.log.gz 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77690.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.77690.log.gzgzip 2024-09-13T08:47:15.642 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75766.log 2024-09-13T08:47:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45098.log 2024-09-13T08:47:15.643 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76717.log.gz 2024-09-13T08:47:15.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75766.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.75766.log.gz 2024-09-13T08:47:15.644 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75107.log 2024-09-13T08:47:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66690.log 2024-09-13T08:47:15.645 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45098.log.gz 2024-09-13T08:47:15.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75107.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75107.log.gz 2024-09-13T08:47:15.646 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42491.log 2024-09-13T08:47:15.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66690.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66690.log.gz 2024-09-13T08:47:15.647 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71618.log 2024-09-13T08:47:15.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66564.log 2024-09-13T08:47:15.649 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42491.log.gz 2024-09-13T08:47:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71618.log: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.71618.log.gz 2024-09-13T08:47:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52855.log 2024-09-13T08:47:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68701.log 2024-09-13T08:47:15.651 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66564.log.gz 2024-09-13T08:47:15.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70124.log 87.2% 2024-09-13T08:47:15.652 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52855.log.gz 2024-09-13T08:47:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68701.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68701.log.gz 2024-09-13T08:47:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57223.log 2024-09-13T08:47:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78107.log 2024-09-13T08:47:15.655 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70124.log.gz 2024-09-13T08:47:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57223.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57223.log.gz 2024-09-13T08:47:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70334.log 2024-09-13T08:47:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78107.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78107.log.gz 2024-09-13T08:47:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82109.log 2024-09-13T08:47:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49546.log 2024-09-13T08:47:15.658 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70334.log.gz 2024-09-13T08:47:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82109.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82109.log.gz 2024-09-13T08:47:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46989.log 2024-09-13T08:47:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50131.log 2024-09-13T08:47:15.660 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49546.log.gz 2024-09-13T08:47:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46989.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46989.log.gz 2024-09-13T08:47:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55280.log 2024-09-13T08:47:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78149.log 2024-09-13T08:47:15.662 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50131.log.gz 2024-09-13T08:47:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80791.log 87.0% 2024-09-13T08:47:15.663 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55280.log.gz 2024-09-13T08:47:15.664 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78149.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78149.log.gz 2024-09-13T08:47:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75724.log 2024-09-13T08:47:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69781.log 2024-09-13T08:47:15.666 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80791.log.gz 2024-09-13T08:47:15.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75724.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75724.log.gz 2024-09-13T08:47:15.667 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36184.log 2024-09-13T08:47:15.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44996.log 2024-09-13T08:47:15.668 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69781.log.gz 2024-09-13T08:47:15.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36184.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36184.log.gz 2024-09-13T08:47:15.669 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81505.log 2024-09-13T08:47:15.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42748.log 2024-09-13T08:47:15.671 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44996.log.gz 2024-09-13T08:47:15.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54219.log 2024-09-13T08:47:15.672 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.81505.log.gz 2024-09-13T08:47:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42748.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42748.log.gz 2024-09-13T08:47:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47592.log 2024-09-13T08:47:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57327.log 2024-09-13T08:47:15.674 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54219.log.gz 2024-09-13T08:47:15.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47592.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47592.log.gz 2024-09-13T08:47:15.675 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71782.log 2024-09-13T08:47:15.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57327.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57327.log.gz 2024-09-13T08:47:15.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2024-09-13T08:47:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72380.log 2024-09-13T08:47:15.677 INFO:teuthology.orchestra.run.smithi031.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.71782.log.gz 2024-09-13T08:47:15.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48185.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2024-09-13T08:47:15.678 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58881.log 2024-09-13T08:47:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72380.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72380.log.gz 2024-09-13T08:47:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66438.log 2024-09-13T08:47:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72081.log 2024-09-13T08:47:15.681 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58881.log.gz 2024-09-13T08:47:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66438.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.66438.log.gz 2024-09-13T08:47:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76675.log 2024-09-13T08:47:15.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72081.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.72081.log.gz 2024-09-13T08:47:15.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81295.log 2024-09-13T08:47:15.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63777.log 2024-09-13T08:47:15.684 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76675.log.gz 2024-09-13T08:47:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65563.log 2024-09-13T08:47:15.685 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81295.log.gz 2024-09-13T08:47:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63777.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63777.log.gz 2024-09-13T08:47:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73081.log 2024-09-13T08:47:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54799.log 2024-09-13T08:47:15.687 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.65563.log.gz 2024-09-13T08:47:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63102.log 2024-09-13T08:47:15.689 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73081.log.gz 2024-09-13T08:47:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70471.log 2024-09-13T08:47:15.690 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54799.log.gz 2024-09-13T08:47:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63102.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63102.log.gz 2024-09-13T08:47:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45461.log 2024-09-13T08:47:15.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70513.log 2024-09-13T08:47:15.692 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70471.log.gz 2024-09-13T08:47:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45461.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45461.log.gz 2024-09-13T08:47:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51540.log 2024-09-13T08:47:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70513.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70513.log.gz 2024-09-13T08:47:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T08:47:15.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45844.log 2024-09-13T08:47:15.696 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51540.log.gz 2024-09-13T08:47:15.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51905.log 2024-09-13T08:47:15.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42747.log 2024-09-13T08:47:15.698 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45844.log.gz 2024-09-13T08:47:15.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60199.log 2024-09-13T08:47:15.699 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51905.log.gz 2024-09-13T08:47:15.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58784.log 2024-09-13T08:47:15.700 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42747.log.gz 2024-09-13T08:47:15.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73464.log 2024-09-13T08:47:15.701 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60199.log.gz 2024-09-13T08:47:15.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69907.log 2024-09-13T08:47:15.703 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58784.log.gz 2024-09-13T08:47:15.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70206.log 2024-09-13T08:47:15.704 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73464.log.gz 2024-09-13T08:47:15.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54364.log 2024-09-13T08:47:15.705 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69907.log.gz 2024-09-13T08:47:15.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72122.log 2024-09-13T08:47:15.706 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70206.log.gz 2024-09-13T08:47:15.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71536.log 2024-09-13T08:47:15.708 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54364.log.gz 2024-09-13T08:47:15.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42449.log 2024-09-13T08:47:15.709 INFO:teuthology.orchestra.run.smithi031.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.72122.log.gz 2024-09-13T08:47:15.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71495.log 2024-09-13T08:47:15.710 INFO:teuthology.orchestra.run.smithi031.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.71536.log.gz 2024-09-13T08:47:15.711 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71270.log 2024-09-13T08:47:15.711 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42449.log.gz 2024-09-13T08:47:15.712 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71495.log: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.71495.log.gz 2024-09-13T08:47:15.712 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62447.log 2024-09-13T08:47:15.714 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53814.log 2024-09-13T08:47:15.714 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.71270.log.gz 2024-09-13T08:47:15.715 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45540.log 2024-09-13T08:47:15.715 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62447.log.gz 2024-09-13T08:47:15.716 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.53814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53814.log.gz 2024-09-13T08:47:15.716 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56196.log 2024-09-13T08:47:15.717 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-09-13T08:47:15.717 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45540.log.gz 2024-09-13T08:47:15.718 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73636.log 2024-09-13T08:47:15.718 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56196.log.gz 2024-09-13T08:47:15.720 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48971.log 2024-09-13T08:47:15.720 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-09-13T08:47:15.721 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63228.log 2024-09-13T08:47:15.721 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73636.log.gz 2024-09-13T08:47:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2024-09-13T08:47:15.723 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48971.log.gz 2024-09-13T08:47:15.724 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46218.log 2024-09-13T08:47:15.724 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63228.log.gz 2024-09-13T08:47:15.724 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58040.log 2024-09-13T08:47:15.725 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2024-09-13T08:47:15.726 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76233.log 2024-09-13T08:47:15.726 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46218.log.gz 2024-09-13T08:47:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59259.log 2024-09-13T08:47:15.727 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58040.log.gz 2024-09-13T08:47:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48227.log 2024-09-13T08:47:15.728 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76233.log.gz 2024-09-13T08:47:15.730 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46680.log 2024-09-13T08:47:15.730 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.59259.log.gz 2024-09-13T08:47:15.731 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2024-09-13T08:47:15.731 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48227.log.gz 2024-09-13T08:47:15.732 INFO:teuthology.orchestra.run.smithi045.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T08:47:15.732 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72040.log 2024-09-13T08:47:15.732 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46680.log.gz 2024-09-13T08:47:15.733 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79323.log 2024-09-13T08:47:15.733 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2024-09-13T08:47:15.734 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77490.log 2024-09-13T08:47:15.735 INFO:teuthology.orchestra.run.smithi031.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.72040.log.gz 2024-09-13T08:47:15.736 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43429.log 2024-09-13T08:47:15.736 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79323.log.gz 2024-09-13T08:47:15.737 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50617.log 2024-09-13T08:47:15.737 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77490.log.gz 2024-09-13T08:47:15.738 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74546.log 2024-09-13T08:47:15.738 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43429.log.gz 2024-09-13T08:47:15.739 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60962.log 2024-09-13T08:47:15.739 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50617.log.gz 2024-09-13T08:47:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69122.log 2024-09-13T08:47:15.741 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74546.log.gz 2024-09-13T08:47:15.742 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81859.log 2024-09-13T08:47:15.742 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60962.log.gz 2024-09-13T08:47:15.743 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69122.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T08:47:15.743 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69122.log.gz 2024-09-13T08:47:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74825.log 2024-09-13T08:47:15.744 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81859.log.gz 2024-09-13T08:47:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67116.log 2024-09-13T08:47:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74886.log 2024-09-13T08:47:15.747 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74825.log.gz 2024-09-13T08:47:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64025.log 2024-09-13T08:47:15.748 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67116.log.gz 2024-09-13T08:47:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74886.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74886.log.gz 2024-09-13T08:47:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77544.log 2024-09-13T08:47:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64025.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64025.log.gz 2024-09-13T08:47:15.751 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59325.log 2024-09-13T08:47:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77544.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77544.log.gz 2024-09-13T08:47:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2024-09-13T08:47:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48582.log 2024-09-13T08:47:15.754 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59325.log.gz 2024-09-13T08:47:15.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54925.log 2024-09-13T08:47:15.755 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2024-09-13T08:47:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77097.log 2024-09-13T08:47:15.756 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.48582.log.gz 2024-09-13T08:47:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48049.log 2024-09-13T08:47:15.758 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54925.log.gz 2024-09-13T08:47:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.77097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.77097.log.gz 2024-09-13T08:47:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61042.log 2024-09-13T08:47:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62351.log 2024-09-13T08:47:15.760 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48049.log.gz 2024-09-13T08:47:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61042.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61042.log.gz 2024-09-13T08:47:15.761 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.55112.log 2024-09-13T08:47:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62351.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62351.log.gz 2024-09-13T08:47:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74383.log 2024-09-13T08:47:15.764 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51007.log 2024-09-13T08:47:15.765 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55112.log.gz 2024-09-13T08:47:15.765 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.74383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74383.log.gz 2024-09-13T08:47:15.766 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51007.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51007.log.gz 2024-09-13T08:47:15.766 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76109.log 2024-09-13T08:47:15.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2024-09-13T08:47:15.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-09-13T08:47:15.770 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76109.log.gz 2024-09-13T08:47:15.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51273.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz -5 2024-09-13T08:47:15.771 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68617.log 2024-09-13T08:47:15.772 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36362.log: gzip 86.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz --verbose 2024-09-13T08:47:15.772 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.58295.log 2024-09-13T08:47:15.773 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58082.log 2024-09-13T08:47:15.773 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68617.log.gz 2024-09-13T08:47:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58295.log: gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.0.62012.log -- replaced with /var/log/ceph/ceph-client.admin.58295.log.gz 2024-09-13T08:47:15.775 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58082.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55847.log 2024-09-13T08:47:15.776 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58082.log.gz 2024-09-13T08:47:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.62012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62012.log.gz 2024-09-13T08:47:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53392.log 2024-09-13T08:47:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71436.log 2024-09-13T08:47:15.778 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55847.log.gz 2024-09-13T08:47:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53392.log: gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53392.log.gz 2024-09-13T08:47:15.780 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.51460.log 2024-09-13T08:47:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71436.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.71436.log.gz 2024-09-13T08:47:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77218.log 2024-09-13T08:47:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2024-09-13T08:47:15.782 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51460.log.gz 2024-09-13T08:47:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48353.log 2024-09-13T08:47:15.784 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77218.log.gz 2024-09-13T08:47:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2024-09-13T08:47:15.785 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T08:47:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73302.log 2024-09-13T08:47:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60783.log 2024-09-13T08:47:15.786 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48353.log.gz 2024-09-13T08:47:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79909.log 2024-09-13T08:47:15.787 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73302.log.gz 2024-09-13T08:47:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56382.log 2024-09-13T08:47:15.789 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60783.log.gz 2024-09-13T08:47:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79909.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79909.log.gz 2024-09-13T08:47:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.56382.log: --verbose -- /var/log/ceph/ceph-client.admin.66732.log 2024-09-13T08:47:15.791 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56382.log.gz 2024-09-13T08:47:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58696.log 2024-09-13T08:47:15.794 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69200.log 2024-09-13T08:47:15.794 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66732.log.gz 2024-09-13T08:47:15.795 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58696.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58696.log.gz 2024-09-13T08:47:15.795 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74087.log 2024-09-13T08:47:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69200.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69200.log.gz 2024-09-13T08:47:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42575.log 2024-09-13T08:47:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68180.log 2024-09-13T08:47:15.798 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74087.log.gz 2024-09-13T08:47:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2024-09-13T08:47:15.799 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-09-13T08:47:15.800 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50173.log 2024-09-13T08:47:15.800 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68180.log.gz 2024-09-13T08:47:15.802 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43089.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2024-09-13T08:47:15.802 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-09-13T08:47:15.803 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50173.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53098.log 2024-09-13T08:47:15.803 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50173.log.gz 2024-09-13T08:47:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79993.log 2024-09-13T08:47:15.805 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-09-13T08:47:15.806 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50555.log 2024-09-13T08:47:15.806 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53098.log.gz 2024-09-13T08:47:15.807 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51602.log 2024-09-13T08:47:15.807 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79993.log.gz 2024-09-13T08:47:15.809 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50555.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50555.log.gz 2024-09-13T08:47:15.809 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-09-13T08:47:15.809 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51602.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51602.log.gz 2024-09-13T08:47:15.810 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42755.log 2024-09-13T08:47:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80162.log/var/log/ceph/ceph-client.admin.42755.log: 2024-09-13T08:47:15.812 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42755.log.gz 2024-09-13T08:47:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68949.log 2024-09-13T08:47:15.815 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53052.log 2024-09-13T08:47:15.815 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80162.log.gz 2024-09-13T08:47:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68949.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49458.log 2024-09-13T08:47:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53052.log: -- replaced with /var/log/ceph/ceph-client.admin.68949.log.gz 2024-09-13T08:47:15.817 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53052.log.gz 2024-09-13T08:47:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73207.log 2024-09-13T08:47:15.820 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49458.log.gz 2024-09-13T08:47:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61123.log 2024-09-13T08:47:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73207.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73207.log.gz 2024-09-13T08:47:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.61123.log: /var/log/ceph/ceph-client.admin.47550.log 2024-09-13T08:47:15.823 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61123.log.gz 2024-09-13T08:47:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60741.log 2024-09-13T08:47:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47550.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47550.log.gz 2024-09-13T08:47:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80851.log 2024-09-13T08:47:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60741.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60741.log.gz 2024-09-13T08:47:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76447.log 2024-09-13T08:47:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80851.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80851.log.gz 2024-09-13T08:47:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55803.log 2024-09-13T08:47:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76447.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76447.log.gz 2024-09-13T08:47:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67763.log 2024-09-13T08:47:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55803.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55803.log.gz 2024-09-13T08:47:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.67763.log: -- /var/log/ceph/ceph-client.admin.43751.log 2024-09-13T08:47:15.834 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67763.log.gz 2024-09-13T08:47:15.835 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48311.log 2024-09-13T08:47:15.836 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43751.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43751.log.gz 2024-09-13T08:47:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75411.log 2024-09-13T08:47:15.838 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48311.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48311.log.gz 2024-09-13T08:47:15.839 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2024-09-13T08:47:15.840 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75411.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75411.log.gz 2024-09-13T08:47:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36090.log 2024-09-13T08:47:15.842 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43439.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2024-09-13T08:47:15.842 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T08:47:15.843 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36090.log: 77.9% -- replaced with /var/log/ceph/ceph-client.admin.36090.log.gz 2024-09-13T08:47:15.844 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70248.log 2024-09-13T08:47:15.849 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.70248.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70248.log.gz 2024-09-13T08:47:15.849 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42757.log 2024-09-13T08:47:15.849 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T08:47:15.850 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2024-09-13T08:47:15.850 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2024-09-13T08:47:15.851 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81631.log 2024-09-13T08:47:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67423.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2024-09-13T08:47:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51335.log 2024-09-13T08:47:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81631.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81631.log.gz 2024-09-13T08:47:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-09-13T08:47:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51335.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51335.log.gz 2024-09-13T08:47:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42155.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-09-13T08:47:15.858 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80623.log 2024-09-13T08:47:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62489.log 2024-09-13T08:47:15.860 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80623.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80623.log.gz 2024-09-13T08:47:15.861 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80749.log 2024-09-13T08:47:15.862 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62489.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62489.log.gz 2024-09-13T08:47:15.862 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49798.log 2024-09-13T08:47:15.863 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80749.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80749.log.gz 2024-09-13T08:47:15.865 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49798.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49798.log.gz 2024-09-13T08:47:15.865 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78443.log 2024-09-13T08:47:15.867 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46775.log 2024-09-13T08:47:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78443.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78443.log.gz 2024-09-13T08:47:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49864.log 2024-09-13T08:47:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46775.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46775.log.gz 2024-09-13T08:47:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69468.log 2024-09-13T08:47:15.872 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49864.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49864.log.gz 2024-09-13T08:47:15.872 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75640.log 2024-09-13T08:47:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69468.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69468.log.gz 2024-09-13T08:47:15.874 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76633.log 2024-09-13T08:47:15.875 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75640.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75640.log.gz 2024-09-13T08:47:15.876 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2024-09-13T08:47:15.876 INFO:teuthology.orchestra.run.smithi031.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-13T08:47:15.877 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76633.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76633.log.gz 2024-09-13T08:47:15.877 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52386.log 2024-09-13T08:47:15.878 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45802.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.74003.log 2024-09-13T08:47:15.879 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2024-09-13T08:47:15.880 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59133.log 2024-09-13T08:47:15.880 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.52386.log.gz 2024-09-13T08:47:15.881 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-09-13T08:47:15.881 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74003.log.gz 2024-09-13T08:47:15.883 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59133.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59133.log.gz 2024-09-13T08:47:15.883 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.46138.log 2024-09-13T08:47:15.884 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43625.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-09-13T08:47:15.884 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45266.log 2024-09-13T08:47:15.886 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63396.log 2024-09-13T08:47:15.886 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46138.log.gz 2024-09-13T08:47:15.887 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-09-13T08:47:15.887 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.45266.log.gz 2024-09-13T08:47:15.888 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72458.log 2024-09-13T08:47:15.888 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63396.log.gz 2024-09-13T08:47:15.889 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53350.log 2024-09-13T08:47:15.889 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gz 2024-09-13T08:47:15.890 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73961.log 2024-09-13T08:47:15.890 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72458.log.gz 2024-09-13T08:47:15.892 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81589.log 2024-09-13T08:47:15.892 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53350.log.gz 2024-09-13T08:47:15.893 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46905.log 2024-09-13T08:47:15.893 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73961.log.gz 2024-09-13T08:47:15.894 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55982.log 2024-09-13T08:47:15.894 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81589.log.gz 2024-09-13T08:47:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46905.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46905.log.gz 2024-09-13T08:47:15.896 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66294.log 2024-09-13T08:47:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55982.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55982.log.gz 2024-09-13T08:47:15.897 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78191.log 2024-09-13T08:47:15.898 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52769.log 2024-09-13T08:47:15.898 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66294.log.gz 2024-09-13T08:47:15.900 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72516.log 2024-09-13T08:47:15.900 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78191.log.gz 2024-09-13T08:47:15.901 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52769.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52769.log.gz 2024-09-13T08:47:15.901 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53935.log 2024-09-13T08:47:15.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72516.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72516.log.gz 2024-09-13T08:47:15.902 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69865.log 2024-09-13T08:47:15.903 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53935.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53935.log.gz 2024-09-13T08:47:15.904 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81253.log 2024-09-13T08:47:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73687.log 2024-09-13T08:47:15.905 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69865.log.gz 2024-09-13T08:47:15.907 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81253.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81253.log.gz 2024-09-13T08:47:15.907 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.80539.log 2024-09-13T08:47:15.908 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73687.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73687.log.gz 2024-09-13T08:47:15.908 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72163.log 2024-09-13T08:47:15.909 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77881.log 2024-09-13T08:47:15.909 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80539.log.gz 2024-09-13T08:47:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63522.log 2024-09-13T08:47:15.911 INFO:teuthology.orchestra.run.smithi031.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.72163.log.gz 2024-09-13T08:47:15.912 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.77881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78527.log 2024-09-13T08:47:15.912 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77881.log.gz 2024-09-13T08:47:15.913 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63522.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63522.log.gz 2024-09-13T08:47:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78527.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78527.log.gz 2024-09-13T08:47:15.915 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66032.log 2024-09-13T08:47:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52470.log 2024-09-13T08:47:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45638.log 2024-09-13T08:47:15.917 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66032.log.gz 2024-09-13T08:47:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52470.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52470.log.gz 2024-09-13T08:47:15.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-09-13T08:47:15.920 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45638.log.gz 2024-09-13T08:47:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70685.log 2024-09-13T08:47:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69510.log 2024-09-13T08:47:15.922 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-09-13T08:47:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75285.log 2024-09-13T08:47:15.923 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.70685.log.gz 2024-09-13T08:47:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76319.log 2024-09-13T08:47:15.925 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69510.log.gz 2024-09-13T08:47:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75285.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75285.log.gz 2024-09-13T08:47:15.926 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51211.log 2024-09-13T08:47:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76319.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76319.log.gz 2024-09-13T08:47:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50883.log 2024-09-13T08:47:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63963.log 2024-09-13T08:47:15.929 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51211.log.gz 2024-09-13T08:47:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50883.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50883.log.gz 2024-09-13T08:47:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51947.log 2024-09-13T08:47:15.931 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63963.log.gz 2024-09-13T08:47:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75453.log 2024-09-13T08:47:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63819.log 2024-09-13T08:47:15.933 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51947.log.gz 2024-09-13T08:47:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75453.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75453.log.gz 2024-09-13T08:47:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46096.log 2024-09-13T08:47:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63819.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63819.log.gz 2024-09-13T08:47:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54883.log 2024-09-13T08:47:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36489.log 2024-09-13T08:47:15.937 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46096.log.gz 2024-09-13T08:47:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54883.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54883.log.gz 2024-09-13T08:47:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70044.log 2024-09-13T08:47:15.940 INFO:teuthology.orchestra.run.smithi031.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36489.log.gz 2024-09-13T08:47:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55154.log 2024-09-13T08:47:15.942 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79534.log 2024-09-13T08:47:15.942 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70044.log.gz 2024-09-13T08:47:15.943 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55154.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55154.log.gz 2024-09-13T08:47:15.943 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80223.log 2024-09-13T08:47:15.945 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79040.log 2024-09-13T08:47:15.945 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79534.log.gz 2024-09-13T08:47:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80223.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80223.log.gz 2024-09-13T08:47:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62175.log 2024-09-13T08:47:15.948 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59846.log 2024-09-13T08:47:15.948 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79040.log.gz 2024-09-13T08:47:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54406.log 2024-09-13T08:47:15.949 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62175.log.gz 2024-09-13T08:47:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59846.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.59846.log.gz 2024-09-13T08:47:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63048.log 2024-09-13T08:47:15.952 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54406.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54406.log.gz 2024-09-13T08:47:15.952 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62921.log 2024-09-13T08:47:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72558.log 2024-09-13T08:47:15.954 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63048.log.gz 2024-09-13T08:47:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70769.log 2024-09-13T08:47:15.955 INFO:teuthology.orchestra.run.smithi031.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.62921.log.gz 2024-09-13T08:47:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61632.log 2024-09-13T08:47:15.955 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72558.log.gz 2024-09-13T08:47:15.957 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2024-09-13T08:47:15.957 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.70769.log.gz 2024-09-13T08:47:15.958 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61632.log: gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61632.log.gz 2024-09-13T08:47:15.958 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.72600.log 2024-09-13T08:47:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45308.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2024-09-13T08:47:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52428.log 2024-09-13T08:47:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56841.log 2024-09-13T08:47:15.961 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72600.log.gz 2024-09-13T08:47:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52428.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52428.log.gz 2024-09-13T08:47:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65478.log 2024-09-13T08:47:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56841.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56841.log.gz 2024-09-13T08:47:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36455.log 2024-09-13T08:47:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79788.log 2024-09-13T08:47:15.965 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.65478.log.gz 2024-09-13T08:47:15.966 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48708.log 2024-09-13T08:47:15.966 INFO:teuthology.orchestra.run.smithi031.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36455.log.gz 2024-09-13T08:47:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.79788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79788.log.gz 2024-09-13T08:47:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73859.log 2024-09-13T08:47:15.969 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48708.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48708.log.gz 2024-09-13T08:47:15.969 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.69384.log 2024-09-13T08:47:15.970 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73859.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73859.log.gz 2024-09-13T08:47:15.970 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52968.log 2024-09-13T08:47:15.972 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47833.log 2024-09-13T08:47:15.972 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69384.log.gz 2024-09-13T08:47:15.973 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52645.log 2024-09-13T08:47:15.973 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52968.log.gz 2024-09-13T08:47:15.975 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47833.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47833.log.gz 2024-09-13T08:47:15.975 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.52645.log: --verbose -- /var/log/ceph/ceph-client.admin.78485.log 2024-09-13T08:47:15.975 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52645.log.gz 2024-09-13T08:47:15.976 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36006.log 2024-09-13T08:47:15.978 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51664.log 2024-09-13T08:47:15.978 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78485.log.gz 2024-09-13T08:47:15.979 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36006.log: 74.7% -- replaced with /var/log/ceph/ceph-client.admin.36006.log.gz 2024-09-13T08:47:15.979 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67469.log 2024-09-13T08:47:15.981 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52260.log 2024-09-13T08:47:15.981 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51664.log.gz 2024-09-13T08:47:15.982 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72286.log 2024-09-13T08:47:15.982 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67469.log.gz 2024-09-13T08:47:15.983 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52260.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52260.log.gz 2024-09-13T08:47:15.984 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72955.log 2024-09-13T08:47:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72286.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.72286.log.gz 2024-09-13T08:47:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2024-09-13T08:47:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67200.log 2024-09-13T08:47:15.986 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72955.log.gz 2024-09-13T08:47:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45182.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2024-09-13T08:47:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-09-13T08:47:15.989 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67200.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60657.log 2024-09-13T08:47:15.989 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67200.log.gz 2024-09-13T08:47:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41987.log 2024-09-13T08:47:15.990 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-09-13T08:47:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60657.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60657.log.gz 2024-09-13T08:47:15.992 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81547.log 2024-09-13T08:47:15.992 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41987.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41987.log.gz 2024-09-13T08:47:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70378.log 2024-09-13T08:47:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65877.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81547.log.gz 2024-09-13T08:47:15.994 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70378.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70378.log.gz 2024-09-13T08:47:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48834.log 2024-09-13T08:47:15.997 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76909.log 2024-09-13T08:47:15.997 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65877.log.gz 2024-09-13T08:47:15.998 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67244.log 2024-09-13T08:47:15.998 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48834.log.gz 2024-09-13T08:47:15.999 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58570.log 2024-09-13T08:47:15.999 INFO:teuthology.orchestra.run.smithi031.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76909.log.gz 2024-09-13T08:47:16.001 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59926.log 2024-09-13T08:47:16.001 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.67244.log.gz 2024-09-13T08:47:16.002 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-09-13T08:47:16.002 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58570.log.gz 2024-09-13T08:47:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59926.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59926.log.gz 2024-09-13T08:47:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49175.log 2024-09-13T08:47:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58124.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-09-13T08:47:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48269.log 2024-09-13T08:47:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45886.log 2024-09-13T08:47:16.006 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49175.log.gz 2024-09-13T08:47:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48269.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48269.log.gz 2024-09-13T08:47:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.45886.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.78317.log 2024-09-13T08:47:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45886.log.gz 2024-09-13T08:47:16.009 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63480.log 2024-09-13T08:47:16.011 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79867.log 2024-09-13T08:47:16.011 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78317.log.gz 2024-09-13T08:47:16.012 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63480.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63480.log.gz 2024-09-13T08:47:16.013 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2024-09-13T08:47:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79867.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79867.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45928.log 2024-09-13T08:47:16.014 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:16.015 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62133.log 2024-09-13T08:47:16.015 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-09-13T08:47:16.017 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52565.log 2024-09-13T08:47:16.017 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45928.log.gz 2024-09-13T08:47:16.018 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62133.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62133.log.gz -5 2024-09-13T08:47:16.018 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57265.log 2024-09-13T08:47:16.019 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45056.log 2024-09-13T08:47:16.019 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52565.log.gz 2024-09-13T08:47:16.021 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57265.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57265.log.gz 2024-09-13T08:47:16.021 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.41535.log 2024-09-13T08:47:16.021 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45056.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45056.log.gz 2024-09-13T08:47:16.022 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43079.log 2024-09-13T08:47:16.023 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50331.log 2024-09-13T08:47:16.023 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41535.log.gz 2024-09-13T08:47:16.024 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53308.log 2024-09-13T08:47:16.025 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43079.log.gz 2024-09-13T08:47:16.026 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46817.log 2024-09-13T08:47:16.026 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50331.log.gz 2024-09-13T08:47:16.027 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53308.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53308.log.gz 2024-09-13T08:47:16.027 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54985.log 2024-09-13T08:47:16.028 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46817.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46817.log.gz 2024-09-13T08:47:16.029 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60699.log 2024-09-13T08:47:16.030 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67288.log 2024-09-13T08:47:16.030 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54985.log.gz 2024-09-13T08:47:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63648.log 2024-09-13T08:47:16.031 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60699.log.gz 2024-09-13T08:47:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67288.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67288.log.gz 2024-09-13T08:47:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.63648.log: --verbose -- /var/log/ceph/ceph-client.admin.55898.log 2024-09-13T08:47:16.034 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63648.log.gz 2024-09-13T08:47:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2024-09-13T08:47:16.036 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75369.log 2024-09-13T08:47:16.036 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55898.log.gz 2024-09-13T08:47:16.037 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55759.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2024-09-13T08:47:16.037 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42323.log 2024-09-13T08:47:16.039 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36432.log 2024-09-13T08:47:16.039 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75369.log.gz 2024-09-13T08:47:16.040 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53977.log 2024-09-13T08:47:16.040 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42323.log.gz 2024-09-13T08:47:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36432.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36432.log.gz 2024-09-13T08:47:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-09-13T08:47:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53977.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53977.log.gz 2024-09-13T08:47:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75598.log 2024-09-13T08:47:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54177.log 2024-09-13T08:47:16.044 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-09-13T08:47:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75598.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75598.log.gz 2024-09-13T08:47:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54177.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54177.log.gz 2024-09-13T08:47:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54757.log 2024-09-13T08:47:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48750.log 2024-09-13T08:47:16.049 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72871.log 2024-09-13T08:47:16.049 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54757.log.gz 2024-09-13T08:47:16.050 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48750.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48750.log.gz 2024-09-13T08:47:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72871.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72871.log.gz 2024-09-13T08:47:16.052 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64143.log 2024-09-13T08:47:16.053 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81379.log 2024-09-13T08:47:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63186.log 2024-09-13T08:47:16.055 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64143.log.gz 2024-09-13T08:47:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81379.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81379.log.gz 2024-09-13T08:47:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66648.log 2024-09-13T08:47:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79239.log 2024-09-13T08:47:16.058 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63186.log.gz 2024-09-13T08:47:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47894.log 2024-09-13T08:47:16.059 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66648.log.gz 2024-09-13T08:47:16.060 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79239.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79239.log.gz 2024-09-13T08:47:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60283.log 2024-09-13T08:47:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47894.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47894.log.gz 2024-09-13T08:47:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52091.log 2024-09-13T08:47:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56424.log 2024-09-13T08:47:16.063 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60283.log.gz 2024-09-13T08:47:16.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52091.log: gzip -5 87.5% --verbose -- /var/log/ceph/ceph-client.admin.46859.log 2024-09-13T08:47:16.065 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52091.log.gz 2024-09-13T08:47:16.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56424.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56424.log.gz 2024-09-13T08:47:16.066 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2024-09-13T08:47:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46859.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46859.log.gz 2024-09-13T08:47:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73733.log 2024-09-13T08:47:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42113.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2024-09-13T08:47:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-09-13T08:47:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63879.log 2024-09-13T08:47:16.071 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73733.log.gz 2024-09-13T08:47:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79450.log 2024-09-13T08:47:16.072 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-09-13T08:47:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49133.log 2024-09-13T08:47:16.073 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63879.log.gz 2024-09-13T08:47:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81757.log 2024-09-13T08:47:16.075 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79450.log.gz 2024-09-13T08:47:16.076 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70429.log 2024-09-13T08:47:16.076 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49133.log.gz 2024-09-13T08:47:16.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-09-13T08:47:16.077 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81757.log.gz 2024-09-13T08:47:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56112.log 2024-09-13T08:47:16.079 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70429.log.gz 2024-09-13T08:47:16.080 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42365.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-09-13T08:47:16.080 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63438.log 2024-09-13T08:47:16.081 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56112.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56112.log.gz 2024-09-13T08:47:16.081 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81805.log 2024-09-13T08:47:16.082 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36302.log 2024-09-13T08:47:16.083 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.63438.log.gz 2024-09-13T08:47:16.084 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75555.log 2024-09-13T08:47:16.084 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81805.log.gz 2024-09-13T08:47:16.085 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70919.log 2024-09-13T08:47:16.085 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36302.log.gz 2024-09-13T08:47:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75555.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75555.log.gz 2024-09-13T08:47:16.087 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47634.log 2024-09-13T08:47:16.087 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70919.log.gz 2024-09-13T08:47:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43086.log 2024-09-13T08:47:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47634.log.gz 2024-09-13T08:47:16.090 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43086.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43086.log.gz 2024-09-13T08:47:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72245.log 2024-09-13T08:47:16.093 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63270.log 2024-09-13T08:47:16.093 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz 2024-09-13T08:47:16.094 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49588.log 2024-09-13T08:47:16.094 INFO:teuthology.orchestra.run.smithi031.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.72245.log.gz 2024-09-13T08:47:16.095 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57674.log 2024-09-13T08:47:16.095 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63270.log.gz 2024-09-13T08:47:16.096 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49588.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49588.log.gz 2024-09-13T08:47:16.097 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59718.log 2024-09-13T08:47:16.098 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57674.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.59800.log 2024-09-13T08:47:16.098 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57674.log.gz 2024-09-13T08:47:16.099 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55028.log 2024-09-13T08:47:16.099 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59718.log.gz 2024-09-13T08:47:16.101 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59800.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.59800.log.gz 2024-09-13T08:47:16.101 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53140.log 2024-09-13T08:47:16.101 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55028.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55028.log.gz 2024-09-13T08:47:16.102 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56762.log 2024-09-13T08:47:16.104 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70559.log 2024-09-13T08:47:16.104 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53140.log.gz 2024-09-13T08:47:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.56762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56762.log.gz 2024-09-13T08:47:16.106 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46340.log 2024-09-13T08:47:16.106 INFO:teuthology.orchestra.run.smithi045.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T08:47:16.106 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70559.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70559.log.gz 2024-09-13T08:47:16.106 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log 2024-09-13T08:47:16.108 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46340.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T08:47:16.108 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46340.log.gz 2024-09-13T08:47:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36037.log: 77.2% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-09-13T08:47:16.111 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46594.log 2024-09-13T08:47:16.113 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61716.log 2024-09-13T08:47:16.113 INFO:teuthology.orchestra.run.smithi045.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-13T08:47:16.114 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46594.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46594.log.gz 2024-09-13T08:47:16.115 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46724.log 2024-09-13T08:47:16.115 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61716.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61716.log.gz 2024-09-13T08:47:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78275.log 2024-09-13T08:47:16.117 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46724.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46724.log.gz 2024-09-13T08:47:16.118 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78868.log 2024-09-13T08:47:16.119 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78275.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78275.log.gz 2024-09-13T08:47:16.120 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76191.log 2024-09-13T08:47:16.122 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78868.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78868.log.gz 2024-09-13T08:47:16.122 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78359.log 2024-09-13T08:47:16.123 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.76191.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76191.log.gz 2024-09-13T08:47:16.124 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78359.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78359.log.gz 2024-09-13T08:47:16.125 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79408.log 2024-09-13T08:47:16.126 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67721.log 2024-09-13T08:47:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79408.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79408.log.gz 2024-09-13T08:47:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45760.log 2024-09-13T08:47:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67721.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67721.log.gz 2024-09-13T08:47:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-13T08:47:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45760.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45760.log.gz 2024-09-13T08:47:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57716.log 2024-09-13T08:47:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.57716.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57716.log.gz 2024-09-13T08:47:16.135 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42752.log 2024-09-13T08:47:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78065.log 2024-09-13T08:47:16.138 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42752.log.gz 2024-09-13T08:47:16.139 INFO:teuthology.orchestra.run.smithi045.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-13T08:47:16.140 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41903.log 2024-09-13T08:47:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.78065.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78065.log.gz 2024-09-13T08:47:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47791.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41903.log.gz 2024-09-13T08:47:16.143 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59886.log 2024-09-13T08:47:16.146 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.47791.log.gz 2024-09-13T08:47:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55365.log 2024-09-13T08:47:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59886.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.59886.log.gz 2024-09-13T08:47:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69739.log 2024-09-13T08:47:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55365.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55365.log.gz 2024-09-13T08:47:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66168.log 2024-09-13T08:47:16.154 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.69739.log.gz 2024-09-13T08:47:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.66168.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66168.log.gz 2024-09-13T08:47:16.158 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59634.log 2024-09-13T08:47:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75495.log 2024-09-13T08:47:16.161 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59634.log.gz 2024-09-13T08:47:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66859.log 2024-09-13T08:47:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75495.log: 86.8%/var/log/ceph/ceph-client.admin.66859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70853.log 2024-09-13T08:47:16.168 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75495.log.gz 2024-09-13T08:47:16.169 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66859.log.gz 2024-09-13T08:47:16.170 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70643.log 2024-09-13T08:47:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70853.log: /var/log/ceph/ceph-client.admin.70643.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56925.log 2024-09-13T08:47:16.176 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70853.log.gz 2024-09-13T08:47:16.176 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70643.log.gz 2024-09-13T08:47:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-09-13T08:47:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43667.log: /var/log/ceph/ceph-client.admin.56925.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.56925.log.gz 2024-09-13T08:47:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60573.log 2024-09-13T08:47:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80978.log 2024-09-13T08:47:16.186 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0%/var/log/ceph/ceph-client.admin.60573.log: -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-09-13T08:47:16.187 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60573.log.gz 2024-09-13T08:47:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53224.log 2024-09-13T08:47:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80978.log: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T08:47:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53224.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53224.log.gz 2024-09-13T08:47:16.193 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80978.log.gz 2024-09-13T08:47:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2024-09-13T08:47:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66336.log 2024-09-13T08:47:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48876.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2024-09-13T08:47:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68781.log 2024-09-13T08:47:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66336.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66336.log.gz 2024-09-13T08:47:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.68781.log: /var/log/ceph/ceph-client.admin.74129.log 2024-09-13T08:47:16.199 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68781.log.gz 2024-09-13T08:47:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35908.log 2024-09-13T08:47:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74129.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.74129.log.gz 2024-09-13T08:47:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35908.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51069.log 2024-09-13T08:47:16.203 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35908.log.gz 2024-09-13T08:47:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71958.log 2024-09-13T08:47:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51069.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51069.log.gz 2024-09-13T08:47:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79197.log 2024-09-13T08:47:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71958.log: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.71958.log.gz 2024-09-13T08:47:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73382.log 2024-09-13T08:47:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.79197.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79197.log.gz 2024-09-13T08:47:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73382.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73382.log.gz 2024-09-13T08:47:16.235 INFO:teuthology.orchestra.run.smithi031.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T08:47:16.326 INFO:teuthology.orchestra.run.smithi045.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-13T08:47:16.357 INFO:teuthology.orchestra.run.smithi045.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T08:47:16.360 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-13T08:47:16.361 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m1.096s 2024-09-13T08:47:16.361 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m5.249s 2024-09-13T08:47:16.361 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.507s 2024-09-13T08:47:16.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-13T08:47:16.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-13T08:47:16.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-13T08:47:16.551 INFO:teuthology.orchestra.run.smithi031.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-13T08:47:16.569 INFO:teuthology.orchestra.run.smithi031.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T08:47:16.676 INFO:teuthology.orchestra.run.smithi031.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T08:47:16.787 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-13T08:47:16.851 INFO:teuthology.orchestra.run.smithi031.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T08:47:16.854 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-13T08:47:16.854 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m1.593s 2024-09-13T08:47:16.854 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m5.731s 2024-09-13T08:47:16.854 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m2.220s 2024-09-13T08:47:16.855 INFO:tasks.ceph:Archiving logs... 2024-09-13T08:47:16.855 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi031/log 2024-09-13T08:47:16.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T08:47:18.153 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi045/log 2024-09-13T08:47:18.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T08:47:18.835 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T08:47:18.844 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T08:47:18.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T08:47:18.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T08:47:18.963 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T08:47:18.972 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T08:47:18.972 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T08:47:18.977 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T08:47:19.008 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-13T08:47:19.010 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +539us[ +524us] +/- 60ms 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -585us[ -585us] +/- 76ms 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -48us[ -62us] +/- 62ms 2024-09-13T08:47:19.016 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +531us[ +529us] +/- 60ms 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 -599us[ -601us] +/- 76ms 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -45us[ -45us] +/- 62ms 2024-09-13T08:47:19.017 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T08:47:19.018 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T08:47:19.027 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T08:47:19.027 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T08:47:19.036 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T08:47:19.204 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T08:47:19.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T08:47:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:19.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T08:47:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:19.455 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T08:47:19.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T08:47:19.474 INFO:teuthology.task.internal:Duration was 1090.336691 seconds 2024-09-13T08:47:19.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T08:47:19.483 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T08:47:19.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T08:47:19.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T08:47:19.573 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T08:47:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T08:47:19.875 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T08:47:19.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-09-13T08:47:19.875 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T08:47:19.925 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-09-13T08:47:19.926 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T08:47:19.975 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T08:47:19.976 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T08:47:19.981 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T08:47:20.075 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T08:47:20.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T08:47:20.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T08:47:20.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T08:47:20.264 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T08:47:20.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T08:47:20.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T08:47:20.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T08:47:20.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T08:47:20.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T08:47:20.459 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-09-13T08:47:20.462 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-09-13T08:47:20.492 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T08:47:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:20.526 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T08:47:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T08:47:20.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T08:47:20.569 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T08:47:20.570 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi031 2024-09-13T08:47:20.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T08:47:20.637 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2024-09-13_08:07:46-fs:volumes-wip-athakkar-testing-2024-09-12-1849-distro-default-smithi/7903322/remote/smithi045 2024-09-13T08:47:20.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T08:47:20.701 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T08:47:20.702 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T08:47:20.706 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T08:47:20.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T08:47:20.754 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T08:47:20.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T08:47:20.764 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T08:47:20.764 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T08:47:20.782 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T08:47:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: 393253 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 08:47 /home/ubuntu/cephtest 2024-09-13T08:47:20.816 INFO:teuthology.orchestra.run.smithi045.stdout: 393851 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 08:47 /home/ubuntu/cephtest 2024-09-13T08:47:20.818 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T08:47:20.828 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T08:47:20.851 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a client.1-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} duration: 1090.3366906642914 flavor: default owner: scheduled_khiremat@teuthology success: true 2024-09-13T08:47:20.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T08:47:20.927 INFO:teuthology.run:pass